SETLOCAL ENABLEDELAYEDEXPANSION

Microsoft Windows
Post Reply
User avatar
MigrationUser
Posts: 336
Joined: 2021-Jul-12, 1:37 pm
Contact:

SETLOCAL ENABLEDELAYEDEXPANSION

Post by MigrationUser »

24 Apr 2010 00:46
RG

Whoever is interested,

In another post on this forum there were some questions about enabling delayed expansion. I decided to start another post and expound on it to give the topic the attention it deserves. There are probably many ways to explain delayed expansion. Here is my explanation. I encourage you to play with this a little and then we will all have benefited from this exercise.

For those of you that are 'programmer type' people, you could compare this to compile-time and run-time behavior.
In this environment it is referred to as load-time and run-time behavior.
Variables are normally expanded when loaded.
Enclosing a variable in ! instead of % allows us to 'delay expansion' of the variable until run-time.
Keep in mind that shell scripts (bat files) are loaded , then executed one line at a time.
We need to have a clear understanding of what a 'line' is. Typically it is 1 line.
However in the case of FOR loops and IF statements the entire FOR and IF constructs are treated as a 'line'... even if they span multiple lines.
So in my example below, everything from the 'for' though the ')' is loaded, variables expanded and then executed. Then we move on to the next line.
So when you use %InnerVar% you are getting the 'load-time' value... when you use !InnerVar! you are getting the 'run-time' value.
Therefore it is correct to use %OuterVar% within the loop because it was set outside the loop... in a previous line, so it was already set at the time that we loaded the FOR construct.
In the same manner, it is correct to use %InnerVar% once we are outside the FOR construct because that line is loaded after the FOR construct has executed.
Take a look at the simple example below and the results generated by the echo statements.
Note that the use of %InnerVar% within the FOR loop gives us the load-time value (Wrong InnerVar).
The use of %InnerVar% outside the FOR loop gives us the expected value. It is still the load-time value, but the value was already set at the time the line was loaded.

I used a FOR construct for this example. The same applies to an IF construct.

Code: Select all

@echo off
setlocal enabledelayedexpansion
set /a OuterVar=1
set /a InnerVar=1
REM Notice that variables set inside the loop must be enclosed in !
REM Variables set outside the loop are enclosed in %
REM The replaceable parameter %%i is an exception to this
for /L %%i IN (1,1,5) do (
   set /a InnerVar+=%OuterVar%
   echo.OuterVar=%OuterVar% InnerVar=!InnerVar! Wrong InnerVar=%InnerVar%
)
echo.OuterVar= %OuterVar% InnerVar=%InnerVar%
endlocal
pause
Result:
OuterVar=1 InnerVar=2 Wrong InnerVar=1
OuterVar=1 InnerVar=3 Wrong InnerVar=1
OuterVar=1 InnerVar=4 Wrong InnerVar=1
OuterVar=1 InnerVar=5 Wrong InnerVar=1
OuterVar=1 InnerVar=6 Wrong InnerVar=1
OuterVar= 1 InnerVar=6

Forgive my long-windedness, but I use this extensively and I think this is a very important concept.
Take this for a spin and ask questions if you have them... there are plenty of folks on here that will answer your questions.

Windows Shell Scripting and InstallShield

----------------------------

#2 24 Apr 2010 18:42
retro-starr

Would it hurt anything to have all variables with '!'? https://ss64.com/nt/setlocal.html makes me believe that you must have all variables with '!'. I would imagation that you would have a slightly slower script if all variables were expanded right at run-time.

----------------------------

#3 24 Apr 2010 20:42
RG
retro-starr wrote:

Would it hurt anything to have all variables with '!'? https://ss64.com/nt/setlocal.html makes me believe that you must have all variables with '!'. I would imagation that you would have a slightly slower script if all variables were expanded right at run-time.
I think you are saying if you have DELAYEDEXPANSIONENABLED, why not always use !.
I don't know if it would hurt anything but I don't think it is good practice to do so.
For one thing, if you moved the SET or ENDLOCAL so that variables enclosed in ! are now outside the area where delayed expansion is enabled the following statement:
echo.Var=!Var!
Woud display
Var=!Var!
Not what you wanted.

Windows Shell Scripting and InstallShield

----------------------------

#4 25 Apr 2010 02:08
retro-starr

If this is what your example was showing then of course outside the localization wouldn't do anything.

Code: Select all

setlocal enabledelayedexpansion
set "var=[whatever]"
echo.!var!
endlocal

:: notice it's out of the localization
echo.var=!var!
Output:

[whatever]
!var!

Where might a good place to be setting delayed variables? For me it was just guess and check that it worked!

----------------------------

#5 25 Apr 2010 03:05
RG

Generally you put the SETLOCAL near the beginning of the bat file or subroutine(s) if you localize them.
I should have mentioned that there are times when you don't want to enable delayed expansion, such as when there are ! present in the data because they will be consumed.
Go ahead and add delayed expansion to the example below and see what happens (exclamation point will not be displayed).

You indirectly brought up a subject that I intentionally did not get into in the post above. There are times when you need to 'return' a value(s) from a routine whose variables are localized. This is easily done with a technique called 'tunneling'. See the endlocal line in my example below. Keeping in mind that everything on the line is expanded at load time... %var% is expanded at load time. The 'set var' is done at run time. so we made a new variable with the same name that exists outside the localization, using the localized value. I could have used another name instead of var... your choice. It is indeed another variable though.

Code: Select all

@echo off
setlocal
set "var=[whatever!]"
echo.%var%
endlocal & set var=%var%

:: notice it's out of the localization
echo.var=%var%
Windows Shell Scripting and InstallShield

----------------------------

#6 25 Apr 2010 05:31
retro-starr

I intentionally didn't tunnel as it was how I thought you worded your example, but hey! If the tunneling technique expands the whole concept of 'setlocal' why not put it! My qeustion was more of where to put !var! instead of 'setlocal'. Like in the other thread your helping me on, nobody pointed out that !upx! was going to work or even !cmd_string! (for those who don't know, upx is a tool set as a variable and cmd_string was a using upx's variable to run inside of a subroutine that ran it). Was it just luck that I remembered that I had to do !upx! (which was also luck when I figured it out long time ago)?

----------------------------

#7 25 Apr 2010 14:28
RG

Use !var! when delayed expansion is enabled and the variable is referenced within an IF or FOR construct. In those cases you want the expansion of the variable to occur at run-time instead of at load-time. Outside the IF or FOR construct the same variable is referenced as %var%.

Windows Shell Scripting and InstallShield

----------------------------

#8 25 Apr 2010 19:08
retro-starr

Since 'setlocal enabledelayedexpansion' provides localization would it be better to just use that instead of 'setlocal'? I think I can remember to use it for 'for' and 'if' functions. Thanks!

----------------------------

#9 25 Apr 2010 19:42
RG
retro-starr wrote:

Since 'setlocal enabledelayedexpansion' provides localization would it be better to just use that instead of 'setlocal'? I think I can remember to use it for 'for' and 'if' functions. Thanks!
I don't think so. It is good practice to always use one or the other. If you don't use either, you run the risk of 'overloading' variables that you did not intend to.
I think I posted an example in one of these posts that shows that you can lose the ! in text if delayed expansion is enabled, so I only use it when needed... but as you sort of implied, that can be most of the time.

I have one very long script that I did years ago. It gathers data about the machine and runs many scripts to gather tons of information out of a SQL database. I used SET LOCAL ENABLEDELAYEDEXPANSION almost exclusively through out it. Then I got into trouble because one of the registry entries I was picking up occasionally had an ! in it. Got to be messy to go back and sort out where I should have delayed expansion enabled and where not.

Windows Shell Scripting and InstallShield

----------------------------

#10 19 Nov 2010 22:41
jeb

https://ss64.com/nt/setlocal.html
It describes it not very correct (like this: With delayed expansion the caret ^ escapes each special character all the time, not just for one command)

I suppose the main problem with variables, expansion and special characters is to understand the moments of there "activity".

As long as you don't understand how and when they work, the results are unpredictable,
therefore I make some (thousands) experiments.

The short form is:
A line is a line (Like RG posted), but can spawn multiple lines in parenthesis blocks or with multiline characters (^)

Each line is parsed and expanded by the BatchLineParser in multiple steps
1. Replace all %vars%
2. remove all <CarrigeReturn> (most of the time of no importance)
3. special char parser for carets ^ quotes " redirection <> pipes |, ampersands & and parenthesis, stops on the first <LineFeed>
REM, IF, FOR recognizer - they have their own special handling way
Building the primary token list
5. echo the result if echo is on
6. Expanding of %%v in for-loops (Not really step6 - but it makes no difference)
7. DELAYED-EXPANSION (if it is enabled) recognized only ^ and ! (therefor you have to write echo ^^!),
this can expand also the primary tokens, but without creating or removing tokens from the list!
the expanded result will not analyzed anymore!
8. call starts the parser again, but with a new step - the ZERO step
0. doubling all carets ^ -the most of them are removed in step 3
9. execute the cmd - the command decide of using the primary token list or take something else, like set "var=bla bla"

And a more complete description are here https://web.archive.org/web/20101109165 ... f=12&t=615

In my opinion, delayed expansion is nearly always better than percent expansion.
You never have to take care of the variable content, like quotes, ampersands, carets or spaces.

hope it helps
jeb

----------------------------

#11 20 Nov 2010 11:23
Simon Sheppard
jeb wrote:

https://ss64.com/nt/setlocal.html
It describes it not very correct (like this: With delayed expansion the caret ^ escapes each special character all the time, not just for one command)
Good point, I've reworded it now to (hopefully) make this a bit clearer:

https://ss64.com/nt/setlocal.html#enabl ... dexpansion

----------------------------

#12 20 Nov 2010 12:57
jeb

Sorry, now you are completely wrong smile

From your link
Simon wrote:

Setting EnableDelayedExpansion will reverse this behaviour.
No. It only adds a new way of expanding variables, nothing else.
Simon wrote:

Escaping control characters:

@echo off
setlocal
Set _html=Hello^>World
Echo %_html%

In the above, the Echo command will create a text file called 'world' - not quite what we wanted! This is because the '^' caret works once for the SET command, but then vanishes.
If we now try the same thing with EnableDelayedExpansion, the caret works all the way through the script:

SETLOCAL EnableDelayedExpansion
Set _html=^<title^>Hello world ^</title^>
Echo !_html!
<title>Hello world </title>

With delayed expansion the caret ^ escapes each special character all the time, not just for one command.
This makes it possible to work with HTML and XML formatted strings in a variable.
The content in both variants is the same
Set _html=Hello^>World
always set the content to Hello >World
The difference is, that expanding with percents are just before the special characters are handeld.
So
echo %_html% expands to echo Hello > World --- Step 1 of the BatchLineParser
Detecting the ">" --- Step 3 of the BatchLineParser
So it fails (you could get it working with set _html=Hello ^^^> World)

But with delayed expansion it is:
echo !_html! nothing happens in step 1 to 6
In step7 of the BatchLineParser it expands to
echo Hello > World --- And then it is executed, parsing ends after this step!

That's the cause why this work this way

Code: Select all

@echo off
setlocal DisableDelayedExpansion
set "var1=One !var2!"
set "var2=Two %%var3%%"
set "var3=Three !var4!"
set "var4=Four"
set var
echo Sample1 %var1%
setlocal EnableDelayedExpansion
echo Sample2 %var1%
call echo Sample3 %var1%
call call call call echo Sample4 "^"

Output:
var1=One !var2!
var2=Two %var3%
var3=Three !var4!
var4=Four
Sample1 One !var2!
Sample2 One Two %var3%
Sample3 One Two Three !var4!
Sample4 "^^^^^^^^^^^^^^^^"
hope it helps
jeb

----------------------------

#13 20 Nov 2010 14:58
Simon Sheppard

OK I think the page is right now
https://ss64.com/nt/setlocal.html#enabl ... dexpansion

The best explanation of Delayed Expansion I've found is this page on Raymond Chens blog:
https://devblogs.microsoft.com/oldnewth ... 0/?p=29993

----------------------------

#14 28 Feb 2017 04:36
Rekrul

Every single time I try to write a script that uses variables, they NEVER work properly. Then I have to spend an hour or so Googling to figure out why. Invariably the reason turns out to be the idiotic delayedexpansion. I don't mean it takes me an hour to figure that part out, what takes the most time is trying to figure out how to work around this incredibly stupid "feature" that the Microsoft programmers decided to inflict upon everyone.

I find examples, I adapt them to my script and naturally they fail. I try more things, more failure. More Googling, more examples, more failure. Why failure? Because it seems that there is absolutely no consistency in variable usage. Everything you want to do with them has its own unique format that they have to be in. Sometimes variables are enclosed in %%, but sometimes you need to use !!. You put them on each end of the variable name, except for the occasions where you only put them in front of it. You can use variables as arguments in the search and replace function, but sometimes you have to use %%%.

If you copy an example script 100% it works great, but make the tiniest change and it fails miserably even though common sense says it should work. Why does it fail? Because of some obscure condition that the programmers put in that you have to be intimately familiar with to make it work.

Of course the one variable you can't use inside a loop is the errorlevel. Some sites claim that you can use it inside a loop, but frankly I think they're lying because no matter what I do, it absolutely refuses to work.

What I want to know is why the hell the MS programmers thought it would be a good idea to inflict this mess on the world. Since this is the default behavior, someone considered it important enough that people would be using it all the time, but I'm having to rack my brain for examples of why you would ever want such convoluted behavior from variables. And any examples that I do come up would be better handled by making those the exception rather than the rule. There's something very wrong when you have to go out of your way to use variables in the most basic functions. It would be like making a home phone where you always need to press "9" before dialing a number. Or a TV where you have to press all the buttons on the TV itself before the power button on the remote will work.

So far the two most likely explanations for why variable handling is so unintuitive and cumbersome is that either the MS programmers were completely incompetent, or that they got really baked and thought it would be a hilarious way to screw with people.

I mean how often do people actually need variables to act in such a strange manner?

----------------------------

#15 28 Feb 2017 05:43
Shadow Thief

Delayed expansion didn't exist in the original version of batch. The %variable% behavior being default is kept for backwards compatibility for people who are using scripts that are older than you are.

----------------------------

#16 28 Feb 2017 07:26
Rekrul
Shadow Thief wrote:

Delayed expansion didn't exist in the original version of batch. The %variable% behavior being default is kept for backwards compatibility for people who are using scripts that are older than you are.
What I'm getting at is why would anyone in their right mind design a scripting language where variables behave completely contrary to common sense?

If you're designing a scripting language from scratch, what makes you stop and think "You know what would be a really good idea? If the default behavior for variables was for them to NOT to contain the value that people expect, and people had to jump through hoops just to get them to behave the way you'd expect them to."

Wherever this behavior originated, what possible reason could they have had for making this the default? Surely it's not common enough that a majority of people would want it to be the rule and normal variable behavior to be the exception?

There must have been a reason the original authors made this the default behavior, but for the life of me I can't see a logical reason why they would do so. As I stated in my last message, for the tiny percentage of cases where you might possibly need it, it would be better if those were the exceptions where you needed to do something special to get abnormal behavior from the variables. The default behavior should have been to make variables behave according to common sense, and anyone with half a brain should have known that.

----------------------------

#17 28 Feb 2017 07:47
Aacini

These are my thoughts about this point:

- Batch .BAT files is a feature presented with MS-DOS version 1.0 about 1981. As years went on Batch files evolved, but most features of previous versions were preserved in order to keep backwards compatibility. There are just three types of entities that can vary their values in a Batch file: environment variables, FOR command replaceable parameters, and Batch file parameters. Although these three features use the same special character (the percent sign) their use is governed by perfectly clear rules.

- Batch file parameters are accessed in a Batch file via a single percent sign and a digit. Period. There is no other entity that is accessed this way. In other words: if you want to access a Batch file parameter, use a single percent sign and a digit (and vice versa: a single percent sign followed by a digit always represent a Batch file parameter).

- FOR command replaceable parameters are accessed in a Batch file via a double percent sign and one letter. There is no other entity that is accessed this way.

- The value of environment variables are accessed in a Batch file via a name enclosed between a single percent sign at each side of the name. There is no other entity that is accessed this way.

- When a Batch file is executed, the command processor parse complete lines and complete compound commands. In this step the entities that use one percent sign (Batch file parameters and environment variables) are replaced by their values just once, BEFORE the commands of the line are executed. For example:

Code: Select all

set "var=Previous"
set "var=New" & echo Value: %var%
Of course, this line show "Value: Previous" for the reasons explained in previous paragraph.

==> Why this happen? Because this is the way that Batch files have worked since the very beginning and for more than 35 years! At that time the expansion of a %variable% value just one time was enough. Note that the execution of .BAT files is done via an interpreter that read each .BAT file line each time it is executed, so the %variable% expansion is a simple way to implement this feature.

A compound command is one that is comprised of two or more commands, and there are just two commands of this type: IF and FOR. In this case the compound command is formed by all the lines that comprise it. As mentioned before, all %variables% inside the compound command are replaced by their values just once, before the compound command is executed. For example:

Code: Select all

set "file=Previous"
for %%a in (*.txt) do (
   set "file=%%a"
   echo This file: %file%
)
Of course, this code show "This file: Previous" as many times as files exists.

In order to facilitate this type of management, MS Batch file developers later introduced Delayed !variable! Expansion. Note that this feature was introduced at end of 1999, that is, about 20 years after the usual %variable% expansion and there were millions of Batch files that use it. Delayed expansion is easy of understand: if a variable is enclosed between exclamation marks (and Delayed Expansion was previously enabled) its expansion will happen several times, that is, each time that the command be executed. In other words: a %variable% expansion happen before the command is executed, but a !variable! expansion is delayed until the command be executed. An example:

Code: Select all

set "var=Previous"
set "var=New" & echo Normal: %var%, Delayed: !var!
This feature is also easy to use: if you need to access the new value of a variable that was modified inside a compound command, use delayed !variable! expansion. Period. For example:

Code: Select all

for %%a in (*.txt) do (
   set "file=%%a"
   echo This file: !file!
)
Note that both %normal% and !delayed! expansions are not good nor bad by themselves; they are just two different features, so it is up to us to appropriately use them. For example, the next line exchange the value of two variables in the simplest possible way:

Code: Select all

set "var1=%var2%" & set "var2=%var1%"
Antonio

Last edited by Aacini (28 Feb 2017 17:17)

----------------------------

#18 28 Feb 2017 23:36
RG

Aacini, Very nice explanation.

Rekrul, using the knowledge that Aacini provided and keeping in mind that the entire IF and FOR constructs as well as lines within () are loaded as 1 line... you may find it helpful to ask yourself if you want the 'load-time' value or the 'run-time' value when using variables.

Windows Shell Scripting and InstallShield

----------------------------

#19 09 Mar 2017 00:32
Simon Sheppard

Merged two topics on this.

Also I have updated https://ss64.com/nt/delayedexpansion.html with a couple of Aacini's points

----------------------------

#20 09 Mar 2017 09:41
jeb

1)
Simon wrote:

In the same way, setting DelayedExpansion also affects the point at which escape characters (^) and redirection characters are evaluated:
Nitpicking, delayed expansion doesn't affects the point, it simply evaluates after the phase where the special characters are evaluated.

2)
There is one point missing.
When delayed expansion is enabled AND at least one exclamation mark in a line is present then carets will be working as escape charater a second time.
And in spite of the first caret escaping these caret escapes works independet of quotes!

Examples

Code: Select all

setlocal DisableDelayedExpansion
echo #1 Test 1 Caret^ 2 Carets^^ 3 Carets^^^ "1 Caret^ 2 Carets^^ 3 Carets^^^".
echo #2 Test 1 Caret^ 2 Carets^^ 3 Carets^^^ "1 Caret^ 2 Carets^^ 3 Carets^^^". !

setlocal EnableDelayedExpansion
echo #3 Test 1 Caret^ 2 Carets^^ 3 Carets^^^ "1 Caret^ 2 Carets^^ 3 Carets^^^".
echo #4 Test 1 Caret^ 2 Carets^^ 3 Carets^^^ "1 Caret^ 2 Carets^^ 3 Carets^^^". !
See the difference of #3 and #4
Output wrote:

#1 Test 1 Caret 2 Carets^ 3 Carets^ "1 Caret^ 2 Carets^^ 3 Carets^^^".
#2 Test 1 Caret 2 Carets^ 3 Carets^ "1 Caret^ 2 Carets^^ 3 Carets^^^". !
#3 Test 1 Caret 2 Carets^ 3 Carets^ "1 Caret^ 2 Carets^^ 3 Carets^^^".
#4 Test 1 Caret 2 Carets 3 Carets "1 Caret 2 Carets^ 3 Carets^".
3)
Simon wrote:

Bugs when using delayed variable expansion
If DelayedExpansion is processing a file with a ! in the filename, that will be interpreted as a variable, this is not a common character used in filenames, but it can cause scripts to fail.
Very unclear description, delayed expansion has nothing to do with filenames, but perhaps you mean For loops over filenames.
In that case the parameter expansion evaluates the exclamation marks (and carets too, when there was at east one exclamation mark).
This behaviour is expected as the parameter expansion (%%P) is just before the delayed expansion phase.

Code: Select all

@echo off
setlocal DisableDelayedExpansion
echo dummy > "Test^test!.tmp"
For %%A in (Test*.tmp) do (
   echo #1 "%%A"
   setlocal EnableDelayedExpansion
   echo #2 "%%A"
)
Output wrote:

#1 "Test^test!.tmp"
#2 "Testtest.tmp"
4)
Simon wrote:
Bugs when using delayed variable expansion
When DelayedExpansion is used inside a code block (one or several commands grouped between parentheses) whose output is Piped, the variable expansion will be skipped. This is an incompatibility when using these three features together.
Not quite right, it's not a bug, it's expected behaviour.
The cause is obvious wink when you use a pipe, both parts of the pipe will be executed in a new cmd.exe instance and these instances are started by default with disabled delayed expansion.
See also Why does delayed expansion fail when inside a piped block of code?

----------------------------
#2309 Mar 2017 23:11
Simon Sheppard
Excellent points as always Jeb, I'm always learning something new.
The cause is obvious ;) when you use a pipe, both parts of the pipe will be executed in a new cmd.exe instance
Thats a real - slap forehead - why didnt I realise that?

I have reworded various bits of the page again now

Thanks all

----------------------------

#2210 Apr 2017 22:30
psyl0w


Would either of you have a reasonable explanation why the delayed expansion fails in the case of a command line (not batch) of for loop ?

Code: Select all

>FOR %A IN (1 2 3) DO SET VAR=!VAR!%A&&echo !VAR!
Result in
!VAR!1
!VAR!12
!VAR!123
Putting aside the batch implentation, any work around is welcome.

Offline

#2311 Apr 2017 04:23
Shadow Thief


Because the command prompt doesn't turn delayed expansion on by default.

Run
cmd /V:ON
first.

----------------------------

#2411 Apr 2017 07:12
psyl0w


Nope, if the delayed expansion wasn't enabled, the result would be
!VAR!
I haven't mentioned but of course the deayed expansion is turned ON

The issue is the delayed expansion fails if VAR is not defined. But I haven't figured out how to initialize with a kind of nul value.

----------------------------

#2511 Apr 2017 07:45
psyl0w


Just to mention the issue is certainly a bit out of the thread topic since it's the same in the case of undelayed expansion:

Code: Select all

>FOR %A IN (1 2 3) DO (SET "VAR=%VAR%%A")
Gives
VAR=%VAR%3
EDIT: Jeb has actually documented in details the differences of processing between the CommandLine parser and the batch parser:
how-does-the-windows-command-interpreter-cmd-exe-parse-scripts

Last edited by psyl0w (11 Apr 2017 18:54)
Post Reply