#1 06 May 2021 09:11

dvx
New Member
Registered: 06 May 2021
Posts: 3

FOR /F command bug: unexpected tokenization of "equal" character

Something in the FOR /F processing of a 'command string' is eliding the = character.

The command is ECHO, with the string A=5
C:>ECHO A=5
C:>A=5

But when used as a FOR /F command string, the '=' character is converted to a space:

C:>FOR /F %R IN ('ECHO A=5') DO ECHO %R
C:>ECHO A
C:>A
(in this example, the first token is  "A")

C:>FOR /F (TOKENS=*) %R IN ('ECHO A=5') DO ECHO %R
C:>ECHO A 5
C:>A 5
(in this example, subsequent lines would be subsequent tokens.)

This is not a problem with ECHO. Any program has the same effect. As shown, at the command prompt ECHO does not have this problem. Using a generic CMDLINE program to echo the command line demonstrates the same effect in FOR /F
C:>CMDLINE MYVAL=6
C:>MYVAL=6
C:>FOR /F %R IN ('CMDLINE MYVAL=6') DO ECHO %R
C:>ECHO MYVAL 6
C:>MYVAL 6
C:>FOR /F %R IN ('CMDLINE MYVAL=6') DO CMDLINE %R
C:>CMDLINE MYVAL 6
C:>MYVAL 6

The problem is not in the %R processing: the problem exists before the parameters are given to the command:
C:>PARAMCOUNT MYVAL=6
C:>1
C:>FOR /F %R IN ('PARAMCOUNT MYVAL=6') DO ECHO %R
C:>ECHO 2
C:>2

The problem exists in the way 'program.exe [parameter list]' is evaluated.

The same behavior is exhibited in a BATCH file (%%R used instead of %R)

Is there any other part of DOS that evaluates strings in the same way?
Is there a work-around for this problem?
Is this behavior documented?

(Win7 and WinXP. Not attempted on other versions).

Offline

#2 06 May 2021 15:37

T3RRY
Member
Registered: 15 Oct 2020
Posts: 16

Re: FOR /F command bug: unexpected tokenization of "equal" character

The issue is expected, and si not because its being treated as a command token, rather it is a standard delimiter. all standard delimiters are removed. An example:

@for /f "Delims=" %G in ('Echo( A ; B = C , D')Do @Echo(%G

Will output: A B C D

Remove the Delims option, and only the first token, "A" will be output

Last edited by T3RRY (06 May 2021 15:38)

Offline

#3 07 May 2021 02:08

dvx
New Member
Registered: 06 May 2021
Posts: 3

Re: FOR /F command bug: unexpected tokenization of "equal" character

T3RRY wrote:

The issue is expected, ... all standard delimiters are removed.

Why is it expected?

Is there any other part of DOS that evaluates strings in the same way?
Is there a work-around for this problem?
Is this behavior documented?

Offline

#4 07 May 2021 03:55

T3RRY
Member
Registered: 15 Oct 2020
Posts: 16

Re: FOR /F command bug: unexpected tokenization of "equal" character

dvx wrote:
T3RRY wrote:

The issue is expected, ... all standard delimiters are removed.

Why is it expected?

Is there any other part of DOS that evaluates strings in the same way?
Is there a work-around for this problem?
Is this behavior documented?

It is expected as it is how standard delim characters are parsed. standard delim characters are detailed here and cmds parsing rules are detailed here. Use CTRL+F  and the term delim to skip through sections of the parsing rules related to how delims are handled.

The 'workaround' for delims is to ensure the string is doublequoted. to iterate over a string using For /f:

@for /f "Delims=" %%G in ("A;B=C,D")Do @Echo(%%~G

And for a For command set:

@for /f "usebackQ Delims=" %%G in (`"Echo(A;B=C,D"`)Do @Echo(%%~G

Last edited by T3RRY (07 May 2021 03:57)

Offline

#5 07 May 2021 05:08

Rekrul
Member
Registered: 17 Apr 2016
Posts: 98

Re: FOR /F command bug: unexpected tokenization of "equal" character

T3RRY wrote:

The issue is expected, and si not because its being treated as a command token, rather it is a standard delimiter. all standard delimiters are removed. An example:

@for /f "Delims=" %G in ('Echo( A ; B = C , D')Do @Echo(%G

Will output: A B C D

Remove the Delims option, and only the first token, "A" will be output

The page on For /F;

https://ss64.com/nt/for_f.html

Claims that if you don't specify any delims, they default to space and tab for text strings. It says nothing about the equals sign. Also, that same page claims that you can eliminate all delims by using "Delims=", which will then place all of the text into the token. Yet, your example shows that this clearly isn't the case.

Apparently the page above is wrong?

Offline

#6 07 May 2021 06:43

T3RRY
Member
Registered: 15 Oct 2020
Posts: 16

Re: FOR /F command bug: unexpected tokenization of "equal" character

Rekrul wrote:
T3RRY wrote:

The issue is expected, and si not because its being treated as a command token, rather it is a standard delimiter. all standard delimiters are removed. An example:

@for /f "Delims=" %G in ('Echo( A ; B = C , D')Do @Echo(%G

Will output: A B C D

Remove the Delims option, and only the first token, "A" will be output

The page on For /F;

https://ss64.com/nt/for_f.html

Claims that if you don't specify any delims, they default to space and tab for text strings. It says nothing about the equals sign. Also, that same page claims that you can eliminate all delims by using "Delims=", which will then place all of the text into the token. Yet, your example shows that this clearly isn't the case.

Apparently the page above is wrong?

It is more correct to say the page is unclear or incomplete, hence why it was not among the pages I linked for you - it does not contain the information relevant to explaining for your understanding how batch syntax works regarding delims.

from the linked ss64 syntax page, regarding doublequoting:

If you are passing a parameter to a batch file that contains any of these delimiter characters, it will split the parameter into two parameters unless you surround the whole thing with double quotes: "this is;one=param,"

Many characters such as \ = ( ) do not need to be escaped when they are used within a "quoted string" typically these are characters you might find in a filename/path. The percent character is one exception to this rule, even though under NTFS % is a valid filename character.

hence the earlier examples showing examples of doublequoting strings to retain delim characters. The alternative is to escape them, however this is only practical if the string is hardcoded:

@for /f "Delims=" %%G in ('Echo(A^;B^=C^,D')Do @Echo(%%~G

Offline

#7 07 May 2021 09:13

dvx
New Member
Registered: 06 May 2021
Posts: 3

Re: FOR /F command bug: unexpected tokenization of "equal" character

T3RRY wrote:

It is more correct to say the page is unclear or incomplete, hence why it was not among the pages I linked for you - it does not contain the information relevant to explaining for your understanding how batch syntax works regarding delims.

Well, having looked at all the pages before I posted, I'd say "that" page is wrong, and the other linked pages "don't" contain the relevant information.

in particular (perhaps I missed this), the 'other linked pages' don't seem to acknowledge that

ECHO A=5

is parsed the same as

(ECHO A=5)

but is different than

for /f %%G in ('Echo A=5') 

I agree that there is an avoidable error around the term 'quoting', made more pointed by the fact that other example uses the term "double quoting", and that another possible area for improvement is where it is written that:

If you are passing a parameter to a batch file

T3RRY wrote:

hence the earlier examples showing examples of doublequoting strings to retain delim characters. The alternative is to escape them, however this is only practical if the string is hardcoded:

@for /f "Delims=" %%G in ('Echo(A^;B^=C^,D')Do @Echo(%%~G

Well here we have it. My examples (from my question) are hardcoded, and here is an example of escaping the symbols which are delimiters even though they are in a quoted string and are not specified. [I'VE DONE THAT NOW AND IT WORKS]

The double quoted examples above are certainly interesting. The odd (~ annotations seem to be irrelevant, since they have no effect in the examples. The first I take to be just a demonstration of principle, since all it does is pass a single token to the secondary command. Since I want to run a command, not evaluate a string, and since the point of a FOR loop is to loop, it wasn't immediately obvious that it had any relevance. It's relevance is shown in the second example where (contrary to expectations), a quoted quoted command is executable: ' "DIR *.*" '.  "DIR" is not a valid command, and it is by no means obvious that this is "passing a parameter to a batch file", but I can't argue with reality.

It looks like, so far, 2 out of 3:
1) There is a workaround
2) "Passing a parameter to a batch file" does the same thing.
3) It's not documented anywhere.

Offline

#8 07 May 2021 17:33

Simon Sheppard
Super Administrator
Registered: 27 Aug 2005
Posts: 1,118
Website

Re: FOR /F command bug: unexpected tokenization of "equal" character

OK the key thing to understand here is that 'Echo(A;B=C,D' is not a quoted string but a command to process.
Even though ' is a quotation mark, in the FOR command that denotes a command not a string and so the delimiters for commands are the standard CMD delimiters not the FOR command ones.

So a quoted string has to be surrounded in DOUBLE quotes "Echo(A;B=C,D" or more simply "(A;B=C,D" because we don't need the ECHO.

Offline

Board footer

Powered by FluxBB