Peter007
Hi Gurus,
I've been having some problem in using FIND or FINDSTR command to find a particular string in a log file and return its output with the matching string along with 1 line above and 1 line below the matching line.
So far, this is the only command that I've tried. I've tried to search online, but couldn't find anything useful.
findstr /n "NETWORK ISSUE DETECTED" c:\Log.txt
Sample log data (Log.txt):
1371524155 Tue Jun 18 10:55:55 2013
1371524160 Tue Jun 18 10:56:00 2013
1371524165 Tue Jun 18 10:56:05 2013
NETWORK ISSUE DETECTED
1371523243 Tue Jun 18 10:40:43 2013
1371523248 Tue Jun 18 10:40:48 2013
1371523253 Tue Jun 18 10:40:53 2013
Desire output (Example):
1371524160 Mon Jun 10 08:50:01 2013
NETWORK ISSUE DETECTED
1371523241 Mon Jun 10 08:20:41 2013
1371524165 Tue Jun 18 10:56:05 2013
NETWORK ISSUE DETECTED
1371523243 Tue Jun 18 10:40:43 2013
There maybe more than 1 occurance of the string "NETWORK ISSUE DETECTED" in the log file as it can contains a lot of data. So the script should be able to handle it and output the multiple result of the searched string as shown in the above example.
Appreciate for your kind help.
Thank you.
----------------------------
#2 20 Jun 2013 02:54
Aacini
Code: Select all
@echo off
setlocal EnableDelayedExpansion
findstr /N /C:"NETWORK ISSUE DETECTED" c:\Log.txt > matchingLines.txt
set lastLine=0
< c:\Log.txt (for /F "delims=:" %%a in (matchingLines.txt) do (
set /A skip=%%a-lastLine-1, lastLine=%%a+2
for /L %%i in (1,1,!skip!) do set /P line=
for /L %%i in (1,1,3) do set /P "line=!line!" & echo/
echo/
))
del matchingLines.txt----------------------------
#3 20 Jun 2013 03:01
RG
Here's another way that does not need a temporary file:
Code: Select all
@echo off
set Exec=:FindIssue
for /f "tokens=*" %%a IN (c:\Log.txt) do CALL %%Exec%% "%%a"
pause
:Findissue
echo.%~1 | findstr "NETWORK ISSUE DETECTED" >nul
if not errorlevel 1 set SecondLine=%~1 & set Exec=:EchoLines
if "%SecondLine%"=="" set FirstLine=%~1
goto :eof
:EchoLines
echo.%Firstline%
echo.%SecondLine%
echo.%~1
echo.
set Exec=:FindIssue
goto :eof
Windows Shell Scripting and InstallShield
----------------------------
#4 20 Jun 2013 05:50
Aacini
@RG:
Your solution have a couple unusual details.
- I think your code missed one or two SET SECONDLINE= commands. In the next point below I inserted such commands in the right place.
- Your method to select a certain code segment from two of them is somewhat peculiar; I haven't seen something like this since many years ago ("self-modifying code"). The same result may be achieved in the traditional way via an IF command:
Code: Select all
@echo off
SET SECONDLINE=
set Exec=:FindIssue
for /f "tokens=*" %%a IN (c:\Log.txt) do CALL :Select "%%a"
pause
:Select
if %Exec% == :FindIssue (
echo.%~1 | findstr "NETWORK ISSUE DETECTED" >nul
if not errorlevel 1 set SecondLine=%~1 & set Exec=:EchoLines
if "%SecondLine%"=="" set FirstLine=%~1
) else (
rem :EchoLines
echo.%Firstline%
echo.%SecondLine%
echo.%~1
echo.
SET SECONDLINE=
set Exec=:FindIssue
)
goto :eof
- Your solution execute a CALL command, a pipe (that requires to execute two copies of cmd.exe) and FINDSTR.EXE command with every line of the input file, so this solution is much slower than another one that does not execute such amount of commands/programs.
On the other hand, I modified my solution so the new version does not need a temporary file. Here it is:
Code: Select all
@echo off
setlocal EnableDelayedExpansion
set lastLine=0
< c:\Log.txt (for /F "delims=:" %%a in (
'findstr /N /C:"NETWORK ISSUE DETECTED" c:\Log.txt') do (
set /A skip=%%a-lastLine-1, lastLine=%%a+2
for /L %%i in (1,1,!skip!) do set /P line=
for /L %%i in (1,1,3) do set /P "line=!line!" & echo/
echo/
))
#5 20 Jun 2013 07:55
Peter007
Hi Gurus,
Thank you for your kind reply and help. Appreciate it.
Most of the given scripts are working as expected which is what I've been looking for. Great !
@RG, the script works OK for a smaller log data, however for a bigger log data with about 800.000 lines, it took about 2 hours.
@Aacini, your fist script works OK and able to handle big log data, but if I terminate the script half way, it'll go into indefinite looping.
These are just my feedback for your review and improvement.
Thanks.
----------------------------
#6 20 Jun 2013 09:41
bluesxman
Here's an alternative method, with known limitations, but the method perhaps could be refined to work around this.
The limitation being that too many matches will breach the maximum line length that "findstr" can accept (later on it'll breach the maximum line length of a "set" command -- but the "findstr" limit comes first).
NB -- I've REMmed out an alternate method using an input file for findstr that sadly didn't get around that.
Code: Select all
@echo off
set "file=log.txt"
REM set "tempfile=%TEMP%\%~n0.tmp"
set "string=NETWORK ISSUE DETECTED"
set "match.lines="
setlocal enabledelayedexpansion
type nul > "%tempfile%"
REM find the lines numbers and calculate -1 and +2 values, store all values to use later
for /f "usebackq tokens=1 delims=:" %%a in (`type "%file%" ^| findstr /b /i /l /n "%string%"`) do (
set /a "line=%%a - 1"
if !line! GTR 1 (
set "match.lines=!match.lines! !line!:"
REM echo:!line!:>> "%tempfile%"
)
set "match.lines=!match.lines! %%a:"
REM echo:%%a:>> "%tempfile%"
set /a "line=%%a + 1"
set "match.lines=!match.lines! !line!:"
REM echo:!line!:>> "%tempfile%"
)
REM reprocess the input file using the line numbers generated above to grab the relevant lines
REM this also removes the line numbers -- could be simplified, by removing the "for" wrapper, if you don't care about seeing the numbers
for /f "usebackq tokens=1* delims=:" %%a in (`type "%file%" ^| findstr /n "." ^| findstr /b "%match.lines%"`) do set /p "out=%%b" <nul & echo:
REM for /f "usebackq tokens=1* delims=:" %%a in (`type "%file%" ^| findstr /n "." ^| findstr /b /g:"%tempfile%"`) do (set /p "out=%%b" <nul & echo:)
cmd | *sh | ruby | chef
----------------------------
#7 20 Jun 2013 09:48
bluesxman
PS -- If you can deploy a third party tool, you could use a Windows port of "grep" with the "-C" switch to get context very quickly and easily. Natively, a VBScript or PowerShell might be a better option. If you're forced to work within default the CMD toolset, you're going to suffer the inherent limitations.
Last edited by bluesxman (20 Jun 2013 09:48)
cmd | *sh | ruby | chef
----------------------------
#8 20 Jun 2013 11:33
Peter007
Hi bluesxman,
Thanks for your reply and help.
The script works as expected and the processing time is also quite fast.
The only thing that I'd like to ask is, how to add a space between the output?
For example:
1371524160 Mon Jun 10 08:50:01 2013
NETWORK ISSUE DETECTED
1371523241 Mon Jun 10 08:20:41 2013
1371524165 Tue Jun 18 10:56:05 2013
NETWORK ISSUE DETECTED
1371523243 Tue Jun 18 10:40:43 2013
Thanks.
----------------------------
#9 21 Jun 2013 12:07
bluesxman
We could alter the final "for" as follows, this should hopefully do the trick:
Code: Select all
set "FLAG="
for /f "usebackq tokens=1* delims=:" %%a in (`type "%file%" ^| findstr /n "." ^| findstr /b "%match.lines%"`) do (
if not defined FLAG (
if "%%b" EQU "%string%" (set FLAG=1)
set /p "out=%%b" <nul & echo:
) else (
set /p "out=%%b" <nul & echo:
echo:
set FLAG=
)
)
----------------------------
#10 21 Jun 2013 14:00
dbenham
Following up on bluesxman's suggestion to use VBScript or PowerShell for an efficient native solution - I have already written a hybrid batch/JScript utility called REPL.BAT that provides robust and efficient regex search and replace capability to batch. The utility uses only native batch and JScript commands.
Assuming REPL.BAT is either in your PATH, or in your current directory, then your desired output is easily achieved using:
Code: Select all
type log.txt | repl "\r?\n(NETWORK ISSUE DETECTED)\r?\n" "$1" m | findstr /c:"NETWORK ISSUE DETECTED" | repl "(NETWORK ISSUE DETECTED)(.*)" "\r\n$1\r\n$2\r\n" xThe second step uses FINDSTR to preserve only lines containing NETWORK ISSUE DETECTED
The final step uses REPL.BAT to reinsert the new lines, including an extra one at the end of the string.
You might find it more convenient to have the prior and subsequent line on the same line of output. And since you already know the output concerns NETWORK ISSUEs, you don't really need to preserve that text.
Assuming your log file does not already contain the - character, output like the following:
Code: Select all
1371524160 Mon Jun 10 08:50:01 2013 - 1371523241 Mon Jun 10 08:20:41 2013
1371524165 Tue Jun 18 10:56:05 2013 - 1371523243 Tue Jun 18 10:40:43 2013Code: Select all
type log.txt | repl "([^\r\n]*)\r?\nNETWORK ISSUE DETECTED\r?\n([^\r\n]*)" "$1 - $2" m | findstr /c:"-"Dave Benham
Last edited by dbenham (21 Jun 2013 14:01)
----------------------------
#11 22 Jun 2013 14:21
Peter007
Hi bluesxman,
Thanks for your reply and help.
I've tried the modified for statement, but it did not work.
The output is still not showing any spaces.
Any other trick to produce the desired output?
Thanks.
----------------------------
#12 22 Jun 2013 14:24
Peter007
Hi dbenham,
Thanks for your reply. Appreciated.
I've tried your method, but it didn't produce the correct output.
Reason being could be the searched string "NETWORK ISSUE DETECTED" has some additional information behind it (suffix).
Thanks.
----------------------------
#13 22 Jun 2013 15:24
foxidrive
Post a sample log file where we can download it, and test the solution.
----------------------------
#14 22 Jun 2013 16:37
Peter007
Hi foxidrive,
Thanks for your reply.
Sample of the log entries had been given on my initial post.
Here is a sample of the error string:
NETWORK ISSUE DETECTED: possible intruder has been detected
I've tried using REPL.BAT method and it's able to produce the correct output with space.
However, I'd prefer not to use any additional script in my script, if possible.
I think using findstr method could be the best way, like script from bluesxman which needs a liitle bit of tuning.
Thank you.
----------------------------
#15 25 Jun 2013 02:39
Peter007
Hi Guys,
Anybody can help out?
Thanks.
----------------------------
#16 25 Jun 2013 08:15
Aacini
Aacini wrote:
... I modified my solution so the new version does not need a temporary file. Here it is:
Code: Select all
@echo off setlocal EnableDelayedExpansion set lastLine=0 < c:\Log.txt (for /F "delims=:" %%a in ( 'findstr /N /C:"NETWORK ISSUE DETECTED" c:\Log.txt') do ( set /A skip=%%a-lastLine-1, lastLine=%%a+2 for /L %%i in (1,1,!skip!) do set /P line= for /L %%i in (1,1,3) do set /P "line=!line!" & echo/ echo/ ))
Excuse me, but I don't understand what "if I terminate the script half way" means. What is the reason because you would want to "terminate the script half way"? Anyway, you may always terminate a Batch file even if you need to close the command-line window, so "it'll go into indefinite looping" phrase has no sense. If you want I help you, then you must help me first. Did my program produce correct results? If not, what exactly the problem is? I can't try to solve a problem if I don't know what the problem is!Peter007 wrote:
@Aacini, your fist script works OK and able to handle big log data, but if I terminate the script half way, it'll go into indefinite looping.
Antonio
----------------------------
#17 29 Jun 2013 01:22
Peter007
Hi Antonio,
Thanks for your update and help. Appreciate it.
The revised script works much better now.
Have a nice day ahead.
- Peter