SS64 Forum

Rekrul

Member

Registered: 17 Apr 2016

Posts: 98

Is there a guide to all the loopholes regarding variables?

Since learning about DelayedExpansion, it's been drilled into my head that once DelayedExpansion is enabled, you have to surround a variable with "!" to make it work. Yet there are apparently a bunch of places where you're supposed to break that rule. For example;

echo !name[!c!]!

Doesn't work, but this does;

echo !name[%c%]!

Similarly, this doesn't work;

set name=!test:!a!=!b!!

But this does;

set name=!test:%a%=%b%!

Yet both appeared to breaking the rules and gave me MUCH frustration before I stumbled across the fact that the "enclose your variables in !!" rule didn't apply to some variables.

I wish there was a comprehensive guide that explained all the times when you're not supposed to follow the rules. Frankly it's the haphazard approach to variable handling that frustrates me the most. It really seems like there are no coherent rules and they just created a different kludge for each situation.

bluesxman

Member

From: UK

Registered: 29 Dec 2006

Posts: 1,129

Re: Is there a guide to all the loopholes regarding variables?

"DelayedExpansion" does not mean you have to use "!" everywhere (and I don't think I've seen that sentiment expressed per se), it merely enables you to use "!" to elicit different a behaviour from your variables.

It's not that there are "rules" per se.  There are behaviours and ways to combine and exploit them to your advantage.  Having a firm understanding of the behaviours is key here, not so much an arbitrary set of rules.

I'll try to compose my thoughts about these behaviours and come back on this later.

Last edited by bluesxman (05 Dec 2019 16:24)

---

cmd | *sh | ruby | chef

bluesxman

Member

From: UK

Registered: 29 Dec 2006

Posts: 1,129

Re: Is there a guide to all the loopholes regarding variables?

The behaviours I'm referring to go roughly like this (excuse my misuse of quote blocks to group stuff below):

%var% will be expanded before !var! (hence "Delayed Expansion").

Each pair % or ! (from left to right) in a line of code will be regarded as a variable, regardless (I think?) of the characters between.

Special case: %% will be translated as a literal % (this is useful when using the "call" work around to avoid enabling delayed expansion).  Presumably !! too, but I'm not in front of a Windows machine to confirm.

Hence !var:%x%=%y%! works -- the % are evaluated earlier, thus become their text equivalents, allowing the substitution to work.

I don’t think any other combination of nesting the two types will work in a helpful manner

In your failing examples:

echo !name[!c!]!

CMD will see is this as a variable called "name[", the literal letter "c", followed by another variable named "]"
Net result: you probably just see "c" echoed to screen.

set name=!test:!a!=!b!!

Similarly I think that would be seen as follows:

Variable "test:" (may even report a syntax error here), a literal "a", variable "=", a literal "b", a literal "!" (not 100% sure about this last one?)

Replacing the inner "!" with "%" means that CMD processes everything twice, making the "%" vars into literal text before the "!" gets a look in.

The key issue here in CMD versus other languages, which better support nesting, is that we indicate the start and end of a variable with the same character, instead of (say) bash which uses ${ } (for syntax like ${var[${i}]}), thus is can readily identify the nesting.  CMD has no insight into our intent so processes strictly left to right detecting opening and closing of variables in sequence.

There's no black art here.

Addendum: There are other issues you will need to keep in mind around how % variables behave inside blocks of code wrapped with ( ) -- however the variable looked at the point of the first "(" it will remain entirely unchanged when referenced via % until the very last ")"

Last edited by bluesxman (06 Dec 2019 14:52)

---

cmd | *sh | ruby | chef

Rekrul

Member

Registered: 17 Apr 2016

Posts: 98

Re: Is there a guide to all the loopholes regarding variables?

"DelayedExpansion" does not mean you have to use "!" everywhere (and I don't think I've seen that sentiment expressed per se), it merely enables you to use "!" to elicit different a behaviour from your variables.

A different and highly confusing behavior.

Maybe it's not explicitly stated that you always have to use "!", but that's been my experience trying to use variables in a loop.

To be clear, my prior programming experience was in BASIC (and a little ML) on the C64. I was never a great programmer, but I learned that when you define a variable, it has that content until you change it. Any time you check the contents of the variable, it will return exactly what you put into it. There is never any situation where you could change the contents of a variable and it would return the previous contents. To me that is completely logical. Try as hard as I can, I can not see any logical reason why someone would want to design a programming language where the default behavior in some instances is to not return the latest value placed into a variable and where you would have to go out of your way to make variables act in a logical, easy to understand manner.

That's like opening a bank account and after a year of depositing your paycheck into it, your balance is zero. "Oh, your money is in there, you just can't see it. If you want to see the actual balance as soon as you make a deposit, you'll need to enable Delayed Deposit."

Even the name "Delayed Expansion" is illogical. To me, the word "delayed" would seem to indicate that the variables will be expanded later but enabling it allows them to be expanded immediately. It seems more like delaying the expansion is the default behavior since reading a variable inside a loop gives you the initial value rather than the current one.

To me, if I set the value of X to "10", it should always return "10" under all conditions. There should never be a situation where you read the value of X and it returns any other value. That's logical to me. To me, it's illogical to have to think "Wait a minute, I want to read the value of X, but based on where I'm using it, is it going to return the current value or the initial value? Do I need to use Delayed Expansion and change how the variable is referenced in order to get the current value from it?" That's a needless complication that gives a minor benefit, but causes major headaches. So you can read both the initial value and the current value from a variable, is that really worth all the confusion when the same thing could accomplished just as easily by copying the initial value to a second variable before changing it? I really can't see any situation where this behavior couldn't be easily duplicated if Delayed Expansion didn't exist and variables always returned the current value.

In my pictures script, almost the entire thing executes inside of a loop. So to make variables behave in the way I expect them to, I have to use Delayed Expansion, otherwise I change the contents of a variable, but it still shows the old value. But I also have to set the filename variable without Delayed Expansion or it will strip any "!" from the name. Strangely once a normal variable is defined with it disabled, it can be referenced with it enabled with no problems. It's only the act of setting the normal variable to the contents of the FOR variable that causes problems. However this then causes problems in that any variables defined with Delayed Expansion enabled lose their contents when you issue the ENDLOCAL command. Which is logical I suppose, but how do you pass a value from the portion inside the SETLOCAL ENABLEDELAYEDEXPANSION section to the script outside of it? I couldn't find a way to do it and ended up writing the values I wanted to preserve to a file and then reading them back in on the next loop. All of which could have been avoided if Delayed Expansion was the default behavior and you didn't need to use special commands to enable it.

It's not that there are "rules" per se.  There are behaviours and ways to combine and exploit them to your advantage.  Having a firm understanding of the behaviours is key here, not so much an arbitrary set of rules.

Well, things like that the variables in a FOR loop always take the form of %%X, which can mostly be used like a normal variable, but there are limitations with how you can use it. You can't set such a variable manually, and although I can't think of any examples at the moment, I know I've run into situations where using such a variable in a command didn't work properly, but putting the contents into a normal variable did.

Or the fact that arguments on the command line are referenced as %1, %2, etc. If I had designed it, I would have included a function in the SET command to put any arguments into user defined variables, such as SET /C ARG1 ARG2 ARG3. Here again, I've run into situations where using %1 in a command didn't work, but putting the contents into a normal variable did.

Hence !var:%x%=%y%! works -- the % are evaluated earlier, thus become their text equivalents, allowing the substitution to work.

But yet if you include;

SET x=x+1
ECHO %x%

You get the old value (inside a loop). Which lead me to believe that only using !x! would return the current value rather than the initial value, hence why I find it confusing that when used in a replace function or an array, you use %x%, which doesn't work properly anywhere else inside a loop, as far as I know.

The key issue here in CMD versus other languages, which better support nesting, is that we indicate the start and end of a variable with the same character, instead of (say) bash which uses ${ } (for syntax like ${var[${i}]}), thus is can readily identify the nesting.  CMD has no insight into our intent so processes strictly left to right detecting opening and closing of variables in sequence.

Do any other languages have an equivalent of Delayed Expansion where you have to take special steps to read the current contents of a variable in a loop?

Addendum: There are other issues you will need to keep in mind around how % variables behave inside blocks of code wrapped with ( ) -- however the variable looked at the point of the first "(" it will remain entirely unchanged when referenced via % until the very last ")"

Unless it's part of a string replace operation or referencing an array variable.

For example;

echo off
set c=1
setlocal enabledelayedexpansion

for /f %%A in (test.txt) do (
set line[!c!]=%%A
set /A c=c+1)

set max=!c!
set c=1

:print
echo !line[%c%]!
set /a c=c+1
if "!c!"=="!max!" goto end
goto print

:end

Once Delayed Expansion has been enabled, you HAVE to use "!" enclose the variables or it won't work properly. For example, the following doesn't work;

echo off
set c=1
setlocal enabledelayedexpansion

for /f %%A in (test.txt) do (
set line[!c!]=%%A
set /A c=c+1)

set max=%c%
set c=1

:print
echo %line[%c%]%
set /a c=c+1
if %c%==%max% goto end
goto print

:end

Hence my thinking that with Delayed Expansion, you always have to enclose variables in "!" to make them work properly. And hence my confusion at the fact that every variable needs to be enclosed in "!", but the array counter needs "%". Even though those variables are outside the loop, Delayed Expansion still demands that you use !c! instead of %c%. Well, except in !line[%c%]!.

BTW, you're probably wondering why I would make a script like this. As usual, I'm just testing the basic functions. My ultimate goal is to process a file of unknown length, but only have it do ten lines at time. I couldn't figure out any way to do that other than by putting each line into an array and then manually looping through ten variables at a time.

bluesxman

Member

From: UK

Registered: 29 Dec 2006

Posts: 1,129

Re: Is there a guide to all the loopholes regarding variables?

I understand your frustration. It is confusing. It does not line up properly with how other languages work.

I’ve done my share of back flips with DE to get exclamation marks to stay in file names. In the end I avoided DE like the plague if I was working with filenames, employing it only when absolutely necessary.

That said, the work arounds required (such as using call echo %%var%% instead of echo !var! or wholesale calling out to a sub routine from a FOR loop in order to work with its tokens as variables) do incur performance penalties which mount up over high numbers of iterations in loops.

To touch on a couple of your points, you should not conflate a %variable% with a FOR token %%f or a command line parameter %1. They are not the same thing at all (even though they share %). Though FOR tokens and command line parameters are equivalent in regard to their expansion options.

Also, arrays can be tricky to work with, but I feel I need to draw attention to the fact that it’s a hack. You see CMD doesn’t actually do arrays, it’s an exploitation of certain behaviours to ape functionality that wasn’t actually designed in. Given that it’s a hack (and it very much is) you shouldn’t expect it to work flawlessly or without limitations.

To reiterate what I said, the key here is understanding the limitations and their work arounds. It does what it does; you’re not going to get it to behave exactly like other languages, but you can employ strategies to get pretty close. I’ve largely stepped away from CMD now, but part of the fun for me, in the past, was working within its confines and making it do things the designers didn’t intend or expect.

Frankly, if you keep finding yourself coming up against problems due to the inherent limitations, I would strongly consider using a different language. “Bash” for Windows is a thing. It’s not perfect by any means, but variable handling is a lot more transparent.

Last edited by bluesxman (07 Dec 2019 13:30)

---

cmd | *sh | ruby | chef

Rekrul

Member

Registered: 17 Apr 2016

Posts: 98

Re: Is there a guide to all the loopholes regarding variables?

I understand your frustration. It is confusing. It does not line up properly with how other languages work.

I'll just never understand why anyone would create a programming/scripting language with such a confusing behavior.

That said, the work arounds required (such as using call echo %%var%% instead of echo !var! or wholesale calling out to a sub routine from a FOR loop in order to work with its tokens as variables) do incur performance penalties which mount up over high numbers of iterations in loops.

It seems like that would be even more confusing than using DE and enclosing (nearly) every variable in "!".

To touch on a couple of your points, you should not conflate a %variable% with a FOR token %%f or a command line parameter %1. They are not the same thing at all (even though they share %). Though FOR tokens and command line parameters are equivalent in regard to their expansion options.

The thing is that they seem like variables to me. I mean, they hold a temporary value and are used in place of an absolute value. Here again, I would have done things much differently. I would have allowed the user to place any valid variable name in the FOR statement. I'm aware that you can do some special things with the tokens, like getting the filename minus the extension, etc, but that functionality could have also been designed to work with normal variables as well. Personally I would have also made it a requirement that variables always be referenced the same way no matter what the case. As in SET %name%=test removing any confusion over what's a variable versus what's a string.

Also, arrays can be tricky to work with, but I feel I need to draw attention to the fact that it’s a hack. You see CMD doesn’t actually do arrays, it’s an exploitation of certain behaviours to ape functionality that wasn’t actually designed in. Given that it’s a hack (and it very much is) you shouldn’t expect it to work flawlessly or without limitations.

OK, is there some other way to accomplish what I want to do?

I have a text file with an unknown number of lines. All the lines are the same format and have no spaces. I want to fetch ten lines, process them, then fetch the next ten, process them, etc, to the end of the file.

I know that I can make the FOR command fetch an arbitrary number of items from a file, but I'm unaware of any way to make it fetch an arbitrary number of items on each repetition of the loop. In other words, fetch items 1-10 on the first loop, items 11-20 on the second, etc. All the examples I've seen either go through the entire file or fetch a pre-determined number of items. I don't suppose you can use variables to tell it what items to fetch? That would be too easy...

Frankly, if you keep finding yourself coming up against problems due to the inherent limitations, I would strongly consider using a different language. “Bash” for Windows is a thing. It’s not perfect by any means, but variable handling is a lot more transparent.

Part of the reason I keep working with CMD is that anything I come up with can be shared with others without requiring them to install something extra. For example, there's a command line program to compress/uncompress ISO files to CSO. It can also test compressed files. I knew my friend could make use of this for ISO images for use in emulators, so I made a script that presents a menu allowing all files in the directory to be compressed, uncompressed or tested. It requires the command line program MaxCSO.exe, but he already had that. More recently, I created a fairly complex script that serves as an installer for the DOS game Dark Forces to be run with DOSBox and various enhancements. I posted the first version in a couple places, but I've made quite a few enhancements to it since then. Nobody needs anything extra to use it (other than a copy of the game). All the required special programs are included, none of which need to be installed. I had to include the CHOICE command to ensure that it worked on every system from XP to Win10 though, since you can't count on XP systems having it. Luckily the only FOR command it needed was the one to read the current parameters in from a text file so that they would persist between sessions, so I didn't need to bother with DE at all.

It's not that I want to share everything I create, but I like the idea that if I do want to share it, the scripts have no special requirements other than Windows and whatever specialized command line programs they may use.

Rekrul

Member

Registered: 17 Apr 2016

Posts: 98

Re: Is there a guide to all the loopholes regarding variables?

After posting my last message, it occurs to me that I can use a counter in the FOR loop and call a subroutine when it reaches ten. Sometimes I overlook the most obvious answers...