You are not logged in.

#1 15 Jul 2019 03:25

Rekrul
Member
Registered: 17 Apr 2016
Posts: 98

Delayed expansion and variables in a loop: Why does this work?

Normally I post here when I can't get something to work. This time, I have a working script, but I'm not clear on why it works. Well, logically there's no reason it shouldn't work, but considering how delayed expansion messes with virtually every variable, I'm surprised that it works. Here's my script;

echo off
setlocal disabledelayedexpansion

for /R %%F in ("*.jpg", "*.jpeg", "*.png", "*.gif", "*.webp", "*.bmp") do (
   for /F %%m in ('identify -format "%%m" "%%F[0]"') do (

set name1=%%F
set name2=%%~nF
set extension=%%~xF

setlocal enabledelayedexpansion

set type=%%m

if "!type!"=="JPEG" set type=jpg
if "!type!"=="PNG" set type=png
if "!type!"=="GIF" set type=gif
if "!type!"=="WEBP" set type=webp
if "!type!"=="BMP3" set type=bmp

if /I not "!type!"=="!extension!" (
 set extension=!type!

ren "!name1!" "!name2!.!extension!"
)
endlocal
))

The purpose of this script is to check the first frame of all pictures (JPG, PNG, GIF, WEBP and BMP) in all subdirectories using Image Magick's Identify command, compare the format string to the filename's extension and correct it if they differ. In other words, if an image has a PNG extension, but it's really a JPG, the script will rename it.

It's 99% my own, but I was having problems dealing with filenames that contained exclamation points. After much Googling, hair pulling, swearing and banging my head on my desk, I stumbled across a post that said to turn off delayed expansion when putting the filename into a variable, then turn it on for the rest of the loop. So I copied the given example and it worked.

The part that confuses me is that I'd been lead to believe that you couldn't set variables inside a loop without using delayed expansion. If you remove the line that enables it, and change all the "!" to "%", it doesn't work. So if you can't properly set variables inside a loop without using delayed expansion, why are you able to put the filename into a variable inside a loop without using it? In other words, why does this part work;

setlocal disabledelayedexpansion

for /R %%F in ("*.jpg", "*.jpeg", "*.png", "*.gif", "*.webp", "*.bmp") do (
   for /F %%m in ('identify -format "%%m" "%%F[0]"') do (

set name1=%%F
set name2=%%~nF
set extension=%%~xF

But this part needs delayed expansion;

set type=%%m

if "!type!"=="JPEG" set type=jpg
if "!type!"=="PNG" set type=png
if "!type!"=="GIF" set type=gif
if "!type!"=="WEBP" set type=webp
if "!type!"=="BMP3" set type=bmp

if /I not "!type!"=="!extension!" (
 set extension=!type!

Also, why does delayed expansion strip "!" from strings in the first place? Yes, I know that "!" is used to denote variables, but the exclamation point is inside the string. If a string contains the word "echo" the script doesn't interpret it as the batch ECHO command and execute it, it's simply treated as part of the string. So why isn't "!" treated as a valid part of the string?

Offline

#2 15 Jul 2019 18:34

Simon Sheppard
Admin
Registered: 27 Aug 2005
Posts: 1,130
Website

Re: Delayed expansion and variables in a loop: Why does this work?

A classic batch script without Delayed Expansion will expand all variables at the beginning of each line.
A FOR...DO command is considered a single line of the script even though you can write it across multiple lines for better readability.

So within a FOR loop you can set all the variables you like, but you wont be able to READ any of those values until the next line when the variables are expanded again.
Thats why you need delayed expansion.

Now in your script you have something unusual - you have switched on SETLOCAL Delayed Expansion in the middle of the FOR command. Most people just set it to either on or off at the top of the script and are done with it.

When using SETLOCAL like this you need to think about where the corresponding ENDLOCAL will come in, for most scripts there is one SETLOCAL at the start of the whole script and an implied ENDLOCAL at the end. Because you are doing this in a loop you are effectively instantiating SETLOCAL (and setting a new set of all the variables) for every execution of the loop.

The reason this may cause you a problem is that 32 is the deepest level of setlocal when used in batch file, (although as explored in that thread you can extend it with CALL)
So what I think you need is either to add an ENDLOCAL at the end of the DO (...) clause, or just run the whole script under Delayed Expansion.

Offline

#3 16 Jul 2019 04:59

Rekrul
Member
Registered: 17 Apr 2016
Posts: 98

Re: Delayed expansion and variables in a loop: Why does this work?

Simon Sheppard wrote:

A classic batch script without Delayed Expansion will expand all variables at the beginning of each line.
A FOR...DO command is considered a single line of the script even though you can write it across multiple lines for better readability.

So within a FOR loop you can set all the variables you like, but you wont be able to READ any of those values until the next line when the variables are expanded again.

A very ass-backwards way of doing things. It over-complicates things that should be simple and runs counter to common sense.

Every time I try to write a script that uses variables, I end up spending literally hours trying to figure out why things don't work, even though common sense says they should.

Simon Sheppard wrote:

Thats why you need delayed expansion.

Seems to me that waiting until the next line for the variable to be expanded would be "delayed" expansion, but I suppose you can't expect logic to apply...

Simon Sheppard wrote:

Now in your script you have something unusual - you have switched on SETLOCAL Delayed Expansion in the middle of the FOR command. Most people just set it to either on or off at the top of the script and are done with it.

I copied what was in the example. I suppose I probably don't need the SETLOCAL DISABLEDELAYEDEXPANSION line, but I didn't want to take any chances. I had originally enabled it at the start of the script, but then it would fail on any filename that contained an exclamation point. Setting the name variable with delayed expansion off was the only way to ensure that the exact filename would be preserved. Because for some reason, trying to use a string containing "!", even if it's contained within another variable and not written explicitly in the script, causes it to be filtered out. Yet that doesn't happen with strings containing "%" when delayed expansion is disabled. It seems that a string can include any valid character if delayed expansion is disabled, but once you enable it, exclamation points are off limits, unless the variable was defined while it was still disabled.

Simon Sheppard wrote:

When using SETLOCAL like this you need to think about where the corresponding ENDLOCAL will come in, for most scripts there is one SETLOCAL at the start of the whole script and an implied ENDLOCAL at the end. Because you are doing this in a loop you are effectively instantiating SETLOCAL (and setting a new set of all the variables) for every execution of the loop.

The reason this may cause you a problem is that 32 is the deepest level of setlocal when used in batch file, (although as explored in that thread you can extend it with CALL)
So what I think you need is either to add an ENDLOCAL at the end of the DO (...) clause, or just run the whole script under Delayed Expansion.

If I understand the logic correctly, the ENDLOCAL is at the end of the DO clause. I made a small revision, to set the %%m variable right at the end of the second DO as I previously had that FOR/DO left open for no valid reason. And looking at it now, I'm not even sure I need the () around the SET command. In any case, there's now only one DO clause active at once. delayed expansion is enabled after the filename variables are defined, the script evaluates each file to decide if the extension is correct and renames it if it's not. Then it does ENDLOCAL before the next iteration of the loop.

Here is my revised script;

echo off
setlocal disabledelayedexpansion
echo Fixing incorrect extensions...
echo.

for /R %%F in ("*.jpg", "*.jpeg", "*.png", "*.gif", "*.webp", "*.bmp") do (
   for /F %%m in ('identify -quiet -format "%%m" "%%F[0]"') do (set type=%%m)

set name1=%%F
set name2=%%~nF
set extension=%%~xF

setlocal enabledelayedexpansion

if "!type!"=="JPEG" set type=jpg
if "!type!"=="PNG" set type=png
if "!type!"=="GIF" set type=gif
if "!type!"=="WEBP" set type=webp
if "!type!"=="BMP3" set type=bmp

if /I not "!type!"=="!extension!" (
 set extension=!type!

ren "!name1!" "!name2!.!extension!"
)
endlocal
)

The ")" above ENDLOCAL closes the previous IF clause, then ENDLOCAL, then the last ")" causes the loop to repeat for the next file.

I've run this on a directory containing several sub-directories with hundreds of files in them, and if worked fine. It's actually one part of a longer script. I'm a pack-rat when it comes to files. I save pictures off the net all the time and when I accumulate enough, I burn them to disc. I wanted a script that would automate the process of making sure the files were all named properly. So my full script renames incorrect extensions, fixes incorrect width labels in the filenames of pictures saved from Tumblr, and then checks to see if any of the files have a name over 102 characters and if so, writes the path and name to a Results.txt file.

It makes three passes through the files and thinking about it now, I'm wondering if I couldn't streamline it to perform all the various operations in one pass...

Offline

#4 16 Jul 2019 18:23

Simon Sheppard
Admin
Registered: 27 Aug 2005
Posts: 1,130
Website

Re: Delayed expansion and variables in a loop: Why does this work?

Sorry I missed the ENDLOCAL you have in there - so it does work and hopefully you understand why now.

It over-complicates things that should be simple and runs counter to common sense.

Thats the batch language for you, no one promised it would be easy!

Offline

#5 17 Jul 2019 01:29

Rekrul
Member
Registered: 17 Apr 2016
Posts: 98

Re: Delayed expansion and variables in a loop: Why does this work?

Simon Sheppard wrote:

Sorry I missed the ENDLOCAL you have in there - so it does work and hopefully you understand why now.

Except for why "!" is stripped out of variables if you define them with delayed expansion enabled.

Is there any way to get around that, other than defining the variable with it disabled and then enabling it afterwards?

Note that I'm not even talking about defining a literal string (although that would be helpful too), but rather a string that is already contained within the FOR variable, like the filename. If I enable delayed expansion at the start, then try to put those filenames into a variable, "Look at this!.jpg" becomes "Look at this.jpg" and causes a file not found error.

Is there any way to preserve the string exactly as it should be while delayed expansion is enabled?

Simon Sheppard wrote:

Thats the batch language for you, no one promised it would be easy!

Definitely not. I'll never understand why it was designed like this. No matter what the origins are, how could anyone look at this behavior and think "Yeah, that makes complete sense?"

Not to mention that there's no consistency in the variable handling. Use just the name to SET a variable, but %name% to reference it elsewhere. Oh, unless you're doing a substring operation, then you just put % at the start and end, and use the name by itself. Oh, unless you're using variables for the search/replace strings, then you have to enclose those in %. Oh, and sometimes you need to add an extra % at the end just because. Variables in a FOR loop have to have two %% at the start, while parameters passed from the command line only have one % and have to be a number. When using delayed expansion, you have to replace the % with !. Oh, unless you're using FOR variables or command line arguments, then you just use the normal %. Oh and you can't use !name! variables as the search and replace portions of a substring operation, you have to put those values into FOR variables and then use those. String variables can contain any valid printable characters. Oh, unless you're using delayed expansion, then they can't contain "!". Oh, unless the variable was defined while delayed expansion was disabled...

I always get this image of some programmer designing this system according to his own internal ideas while ignoring all logic and common sense. If the same guy had designed a light switch, flipping it up and down wouldn't do anything, you'd have to push in on the switch and turn it 180 degrees while moving it up to turn on the light, but turn it 90 degrees while tilting it sideways to turn the light off.

Offline

#6 17 Jul 2019 19:58

Simon Sheppard
Admin
Registered: 27 Aug 2005
Posts: 1,130
Website

Re: Delayed expansion and variables in a loop: Why does this work?

Rekrul wrote:
Simon Sheppard wrote:

Sorry I missed the ENDLOCAL you have in there - so it does work and hopefully you understand why now.

Except for why "!" is stripped out of variables if you define them with delayed expansion enabled.

Because thats the character batch uses to identify a variable when delayed expansion is in use.

You have to remember that the batch language largely dates back to the earliest days of MS-DOS when memory and disk space was tiny: 16 kB – 256 kB and no hard drive.
A lot of attention was placed not in ease of use but in shaving every bit and byte to make the programs smaller. That small memory space is for everything, the OS, drivers, programs, data, everything.

That has the nice side effect that the programs are FAST, even today with fast hardware it can make a difference.

Some of the syntax is ugly but that was never the first design goal, if you want consistent syntax use PowerShell.

Offline

#7 18 Jul 2019 05:00

Rekrul
Member
Registered: 17 Apr 2016
Posts: 98

Re: Delayed expansion and variables in a loop: Why does this work?

Simon Sheppard wrote:
Rekrul wrote:

Except for why "!" is stripped out of variables if you define them with delayed expansion enabled.

Because thats the character batch uses to identify a variable when delayed expansion is in use.

Yes, but why isn't it protected when it's already inside a variable from the FOR command?

When delayed expansion is off, it uses % to identify variables, but if a filename contains %, such as "Get 15% more free.txt" it will happily accept the name as is. Having % in the string doesn't confuse it.

I could understand if you tried to do something like;

SET name=This is a test!

But when a variable already contains a string, why is the script looking inside that variable and trying to interpret parts of the string as part of the script? If you include the word "echo" in a string, why doesn't it try to print the rest of the string since ECHO is a valid command?

Simon Sheppard wrote:

You have to remember that the batch language largely dates back to the earliest days of MS-DOS when memory and disk space was tiny: 16 kB – 256 kB and no hard drive.
A lot of attention was placed not in ease of use but in shaving every bit and byte to make the programs smaller. That small memory space is for everything, the OS, drivers, programs, data, everything.

The Commodore VIC-20 only had 5K of RAM, only 3.5K of which was available for BASIC programs, but they still managed to make variables work in a logical manner. wink

Simon Sheppard wrote:

That has the nice side effect that the programs are FAST, even today with fast hardware it can make a difference.

My script is quite slow, even when it's not doing a lot of renaming. Even on a directory only containing handful of files, it still takes several seconds to execute.

Simon Sheppard wrote:

Some of the syntax is ugly but that was never the first design goal, if you want consistent syntax use PowerShell.

The reason I use the Windows batch is for portability. Sometimes I give my scripts to others and I don't want to have to tell them to install something extra to use them.

Offline

Board footer

Powered by