#1 03 Dec 2013 16:20

npocmaka
Member
From: Bulgaria
Registered: 03 Dec 2009
Posts: 430

FOR /F options priority

While the other options are more or less obvious skip and eol were not for me.skip is with higher prio and is executed first:

"2" is not skipped , but only -1 (which is also omitted because of the eol=- ):

C:\>for /f "skip=1 eol=-" %n in ('echo -1 ^& echo 2 ^& echo 3') do @echo %n
2
3

Last edited by npocmaka (04 Dec 2013 08:31)

Offline

#2 04 Dec 2013 01:19

Aacini
Member
Registered: 05 Dec 2012
Posts: 142

Re: FOR /F options priority

It seems rather logical to me: first skip the number of lines, then check the eol in the rest. This way, the skip option indicate to skip a number of lines from the beginning of the file, not after the first line that pass the eol test.

Offline

#3 04 Dec 2013 10:52

npocmaka
Member
From: Bulgaria
Registered: 03 Dec 2009
Posts: 430

Re: FOR /F options priority

It's seems logical because 'eol' check looks more complicated (and slow) and it's better to be left after 'skip' in sake of the performance .
But as it is not explicitly mentioned in  'FOR /?' it could be misleading.

Offline

#4 04 Dec 2013 12:54

foxidrive
Member
Registered: 04 Apr 2013
Posts: 339

Re: FOR /F options priority

If EOL was in force at the beginning then skip would be unpredictable in the actual number of lines that were skipped.

It makes sense for skip to function on an exact number of lines.

Offline

#5 04 Dec 2013 13:34

dbenham
Member
From: U.S. east coast
Registered: 15 Apr 2012
Posts: 111

Re: FOR /F options priority

The thing that surprised me a bit is that SKIP includes empty lines, but after the SKIP is complete, FOR /F ignores (does not iterate) empty lines.

So if the first line of FILE.TXT is empty, then the following commands will give identical results:

for /f "delims=" %A in (file.txt) do @echo %A
for /f "skip=1 delims=" %A in (file.txt) do @echo %A

Another thing that surprised me a bit is that a line that only contains delimiters is not ignored. The row is iterated, but all FOR variables are undefined (empty). I like the fact that it is not ignored, but it does not seem logical to me that a line of delimiters is not ignored, but an empty line is. (I guess it really comes down to the fact that I hate that empty lines are ignored)


Dave Benham

Offline

#6 04 Dec 2013 13:47

foxidrive
Member
Registered: 04 Apr 2013
Posts: 339

Re: FOR /F options priority

dbenham wrote:

(I guess it really comes down to the fact that I hate that empty lines are ignored)

I agree, that was a stupid design decision.

You could have just tested for "" in the code and you'd know it was an empty line.

Having a default EOL character (which is really a Start Of Line character) was equally stupid IMO.

Last edited by foxidrive (04 Dec 2013 13:47)

Offline

#7 21 Jan 2014 10:57

npocmaka
Member
From: Bulgaria
Registered: 03 Dec 2009
Posts: 430

Re: FOR /F options priority

another thing that came to me to test."tokens=*" is faster thant "delims="  :


@echo off
echo "delims="
for /f %%t in ('powershell "(get-date).tofiletime()"') do set mst=%%t
for /l %%a in (;;;1=1=500000;;) do (
		for /f "delims=" %%z in ("text with delimiters. word word word word word word word word word word word word word word word word word word word word word word word word") do break
)
powershell ((get-date).tofiletime() - %mst%)

echo(
echo( 
echo "tokens=*"
for /f %%t in ('powershell "(get-date).tofiletime()"') do set mst=%%t
@echo off
for /l %%a in (;;;1=1=500000;;) do (
		for /f "tokens=*" %%z in ("text with delimiters. word word word word word word word word word word word word word word word word word word word word word word word word") do break
)
powershell ((get-date).tofiletime() - %mst%)

echo(
echo(
echo delims^^^=
for /f %%t in ('powershell "(get-date).tofiletime()"') do set mst=%%t
for /l %%a in (;;;1=1=500000;;) do (
		for /f delims^= %%z in ("text with delimiters. word word word word word word word word word word word word word word word word word word word word word word word word") do break
)
powershell ((get-date).tofiletime() - %mst%)

echo(
echo(
echo delims^^^=^ tokens^^^=*
for /f %%t in ('powershell "(get-date).tofiletime()"') do set mst=%%t
for /l %%a in (;;;1=1=500000;;) do (
		for /f delims^=^ tokens^=* %%z in ("text with delimiters. word word word word word word word word word word word word word word word word word word word word word word word word") do break
)
powershell ((get-date).tofiletime() - %mst%)

echo(
echo(
echo tokens^^^=*
for /f %%t in ('powershell "(get-date).tofiletime()"') do set mst=%%t
for /l %%a in (;;;1=1=500000;;) do (
		for /f tokens^=* %%z in ("text with delimiters. word word word word word word word word word word word word word word word word word word word word word word word word") do break
)
powershell ((get-date).tofiletime() - %mst%)

echo(
echo( 
echo "tokens=* delims="
for /f %%t in ('powershell "(get-date).tofiletime()"') do set mst=%%t
@echo off
for /l %%a in (;;;1=1=500000;;) do (
		for /f "tokens=* delims=" %%z in ("text with delimiters. word word word word word word word word word word word word word word word word word word word word word word word word") do break
)
powershell ((get-date).tofiletime() - %mst%)

(are they always replaceable with each other? )

EDIT:
At least on my machine tokens^=*   gives the fastest results.It was consistent after a few plays.While the "tokens=* delims=" is the slowest.

Last edited by npocmaka (21 Jan 2014 13:32)

Offline

#8 21 Jan 2014 14:21

foxidrive
Member
Registered: 04 Apr 2013
Posts: 339

Re: FOR /F options priority

tokens=* removes all leading whitespace, while delims= retains any leading spaces in filenames.

If your filenames could have leading spaces then using tokens=* will mean they will not be processed properly when gathering a list of filenames.

Last edited by foxidrive (21 Jan 2014 14:23)

Offline

#9 21 Jan 2014 23:37

npocmaka
Member
From: Bulgaria
Registered: 03 Dec 2009
Posts: 430

Re: FOR /F options priority

foxidrive wrote:

tokens=* removes all leading whitespace, while delims= retains any leading spaces in filenames.

If your filenames could have leading spaces then using tokens=* will mean they will not be processed properly when gathering a list of filenames.

Yep.And now it's sounds reasonable after I saw it written :-) .Looks like tokens is doing more work in this case , but anyway it's faster.

Offline

#10 22 Jan 2014 07:03

foxidrive
Member
Registered: 04 Apr 2013
Posts: 339

Re: FOR /F options priority

This is the results I got from your code.  Is the figure returned 7.93 seconds the first one?

"delims="
79369416


"tokens=*"
78598961


delims^=
80249059


delims^= tokens^=*
80929589


tokens^=*
78299241


"tokens=* delims="
79875805

Offline

#11 22 Jan 2014 09:17

npocmaka
Member
From: Bulgaria
Registered: 03 Dec 2009
Posts: 430

Re: FOR /F options priority

foxidrive wrote:

This is the results I got from your code.  Is the figure returned 7.93 seconds the first one?

Yes.Just used filetime format for easier calculations.

Offline

#12 06 Feb 2014 14:38

MC ND
Member
Registered: 06 Feb 2014
Posts: 3

Re: FOR /F options priority

Maybe this is known and this post is not needed, and maybe it is a little off topic with the "priority" title, but i've seen (sorry dbenham, you were the first)

dbenham wrote:

I hate that empty lines are ignored

And, as i see it, for does not skip empty lines. for skips lines in where none of the replaceable parameters (declared or generated by the token list) get data assigned.

With this file

a,b,c
d,e,f

g,h,
i,j,k

(there is no space after the h,) and this code

for /f "delims=, tokens=3" %f in (file.txt) do echo %f

all that i see is c,f and k

Offline

#13 06 Feb 2014 23:12

MC ND
Member
Registered: 06 Feb 2014
Posts: 3

Re: FOR /F options priority

Just to try to make it clear. Running

for /f "tokens=2,3" %a in ('@echo a,b,') do ( echo %a & echo %b)

we get `b` and `echo is on` in screen, that is, a line with data, but with an empty variable.

But running

for /f "tokens=3" %f in ('@echo a,b,') do ( echo %a & echo %b)

we get no output. No echo status line. Line is ignored/skipped as no variable gets data.

So, for command is ignoring not the emtpy lines but the lines that get no data asigned into the replaceable parameters.

Offline

#14 07 Feb 2014 03:53

foxidrive
Member
Registered: 04 Apr 2013
Posts: 339

Re: FOR /F options priority

Another way to look at this is that lines with data can be echoed and lines without data cannot be echoed or included in the procedure.

As there is no way that empty lines can be included in the procedure then it is fair to say that it is ignoring them.

Offline

#15 07 Feb 2014 09:09

MC ND
Member
Registered: 06 Feb 2014
Posts: 3

Re: FOR /F options priority

foxidrive wrote:

Another way to look at this is that lines with data can be echoed and lines without data cannot be echoed or included in the procedure.

As there is no way that empty lines can be included in the procedure then it is fair to say that it is ignoring them.

Of course, FOR command is ignoring them, but the question is why they are ignored, because lines with data are also ignored when this data is not assigned into any of the FOR replaceable parameters.

And this "why" is the reason for the npocmaka first post.

When FOR command start to process data, SKIP is evaluated, and lines are skipped with or without data, with or without EOL. Just skip n lines.

Then, lines start to be parsed. And here TOKENS, DELIMS and EOL determine if the line is ignored or not. If after parse, any of the replaceable parameters get data, the code into the FOR is executed. If no data is assigned, line is ignored.

With only "tokens=*" leading spaces are removed because they are a delimiter, and if they are at the beginning of the line, no data has still be retrieved to be asigned, so, FOR continues trying to find the first token in the line to see if the line should be processed or not. Once data is found, no further processing is done in the line (no need to search for end of token), as we are asking to retrieve all the remaining data.

"delims=" does not remove the leading spaces obviously because we are excluding them as delimiters. But the rest of the line is parsed to determine start and end of the tokens that will be mapped into replaceable parameters.

Offline

#16 07 Feb 2014 10:48

foxidrive
Member
Registered: 04 Apr 2013
Posts: 339

Re: FOR /F options priority

MC ND wrote:

the question is why they are ignored

The coder managed to do it that way.  smile

Like you, there are several blokes who delight in dissecting cmd and creating wonderful workarounds to perform batch-only solutions, just for the pleasure of coding.

I had that bug too for many many years, in times when there were no other ways to solve the problem,
but these days there is sed and repl and findrepl and WSH with Jscript and VBS, and Wmic,
and command line compilers in windows, so my enjoyment now is solving a question in an easy/simple/robust way.

You might like to create an account at dostips.com too, if you haven't already, where a lot of discussion of techniques goes on.
A recent thread devised a pure batch method to create binary files with all 256 characters.

Offline

#17 17 Sep 2014 16:31

npocmaka
Member
From: Bulgaria
Registered: 03 Dec 2009
Posts: 430

Re: FOR /F options priority

One more thing that caught my attention.   If there are few option of the same type only the last one is taken into consideration:

>for /f "eol=1 eol=2" %a in ('echo 1^&echo 2') do @echo %a
1

Offline

Board footer

Powered by FluxBB