#1 30 Jan 2020 07:39

Rekrul
Member
Registered: 17 Apr 2016
Posts: 69

Title change: Pass variable definitions OUT of SETLOCAL "bubble"?

I have a script that uses a FOR loop to read in one line at a time from a text file. Because I also need to change and read the values of variables in the loop, I enable delayedexpansion. I do this because trying to deal with variables in a loop any other way is a colossal pain in the ass that has me pulling out my hair in frustration.

The problem is that if any of the lines contain an exclamation mark "!", it will be stripped out.

Is there some trick that will allow me to read the contents of a file into a token while delayed expansion is enabled and NOT lose any "!" in the lines?

Here's sample code;

echo off
set name=test.txt

setlocal enabledelayedexpansion

for /F "delims=" %%F in (!Name!) do (
set Line=%%F
echo !Line!
pause)

Since I know it's inevitable that someone is going to tell me I don't need delayed expansion for this snippet, allow me to again explain that this is only a test segment from a much larger script that does other stuff that DOES require delayed expansion to be enabled to preserve my sanity.

So, is there any way to alter the above example to preserve any "!" that might be in the lines of the text file (other than removing the SETLOCAL command)?

Last edited by Rekrul (31 Jan 2020 01:32)

Offline

#2 30 Jan 2020 15:31

RG
Member
From: Minnesota
Registered: 18 Feb 2010
Posts: 354

Re: Title change: Pass variable definitions OUT of SETLOCAL "bubble"?

It is a PIA... but one way is to make a pre-process that scans your text file and creates a temp txt file substituting each "!" with a string that will never be found in the text like "#Rekrul#". Then run your code on the temp txt file. When done run a post-process that scans the temp txt file and substitutes each "#Rekrul#" with "!".


Windows Shell Scripting and InstallShield

Offline

#3 31 Jan 2020 01:31

Rekrul
Member
Registered: 17 Apr 2016
Posts: 69

Re: Title change: Pass variable definitions OUT of SETLOCAL "bubble"?

RG wrote:

It is a PIA... but one way is to make a pre-process that scans your text file and creates a temp txt file substituting each "!" with a string that will never be found in the text like "#Rekrul#". Then run your code on the temp txt file. When done run a post-process that scans the temp txt file and substitutes each "#Rekrul#" with "!".

That won't work because the script needs to be able to process each line as-is and while it's unlikely that a line will contain an exclamation mark, it's not impossible. I don't want a script that works sometimes, if the right conditions are always met, I want one that can deal with any valid line.

The reason it has to process each one as-is is because the lines will be URLs, which will be fed to another program within the loop. Each line will be checked to first verify that it is an URL, then it's sent to another program. While this is going on, the proper functioning of the script also relies on incrementing counters to keep track of how many URLs are being processed concurrently and how many URLs there are in total. If an URL has been changed to swap "!" with something else, it's no longer a valid URL.

Let me try a different tack;

Is there any way to preserve a variable past an ENDLOCAL command?

For example;

set test=HELLO

setlocal enabledelayedexpansion

echo !test!

endlocal

This prints HELLO, because the variable test carried its value into the "bubble" of SETLOCAL. However this doesn't work;


setlocal enabledelayedexpansion

set test=HELLO

endlocal

echo %test%

Is there any way to pass the definition of a variable defined within the bubble to the rest of the script?

Offline

#4 31 Jan 2020 09:17

Rekrul
Member
Registered: 17 Apr 2016
Posts: 69

Re: Title change: Pass variable definitions OUT of SETLOCAL "bubble"?

RG wrote:

It is a PIA... but one way is to make a pre-process that scans your text file and creates a temp txt file substituting each "!" with a string that will never be found in the text like "#Rekrul#". Then run your code on the temp txt file. When done run a post-process that scans the temp txt file and substitutes each "#Rekrul#" with "!".

That won't work because the script needs to be able to process each line as-is and while it's unlikely that a line will contain an exclamation mark, it's not impossible. I don't want a script that works sometimes, if the right conditions are always met, I want one that can deal with any valid line.

The reason it has to process each one as-is is because the lines will be URLs, which will be fed to another program within the loop. Each line will be checked to first verify that it is an URL, then it's sent to another program. While this is going on, the proper functioning of the script also relies on incrementing counters to keep track of how many URLs are being processed concurrently and how many URLs there are in total. If an URL has been changed to swap "!" with something else, it's no longer a valid URL.

Let me try a different tack;

Is there any way to preserve a variable past an ENDLOCAL command?

For example;

set test=HELLO

setlocal enabledelayedexpansion

echo !test!

endlocal

This prints HELLO, because the variable test carried its value into the "bubble" of SETLOCAL. However this doesn't work;


setlocal enabledelayedexpansion

set test=HELLO

endlocal

echo %test%

Is there any way to pass the definition of a variable defined within the bubble to the rest of the script?

EDIT: Never mind, I just discovered the ENDLOCAL & SET trick.

EDIT 2: Naturally what at first seems logical doesn't actually work in practice. At first glance using ENDLOCAL & SET seems to pass the value of variables on to the rest of the script, but it fails in practice.

What's wrong with this?

echo off
set URLCount=0
set Count=0

setlocal enabledelayedexpansion

for /F "delims=" %%F in (urls.txt) do (

set /a URLCount=URLCount+1
)

endlocal & set URLCount=%URLCount%

for /F "delims=" %%F in (urls.txt) do (

setlocal enabledelayedexpansion

set /a Count=Count+1
echo !Count! of !URLCount!

endlocal & set Count=%Count%

)
echo Total: %Count% of %URLCount%

The first variable "URLCount" is preserved and survive the rest of the script intact, however the second variable "Count" gets reset each time the loop repeats. WHY? I preserve it after the ENDLOCAL statement, so why is it getting set back to zero every time???

Why isn't the value of "Count" preserved when the loop starts again and how do I fix this so that it is?

NOTE: For this example, I have stripped out other parts of the script that have no bearing on the problem at hand, to make it as simple as possible.

Last edited by Rekrul (31 Jan 2020 13:04)

Offline

#5 31 Jan 2020 16:03

RG
Member
From: Minnesota
Registered: 18 Feb 2010
Posts: 354

Re: Title change: Pass variable definitions OUT of SETLOCAL "bubble"?

@Rekrul,
It is helpful to consider both the 'load time' and 'run time' behavior' of bat files. The 'trick' you refer to is a technique called 'tunneling'. It works because everything in an IF statement, FOR loop, or everything between parentheses is loaded as one line. So variables are expanded (at load time) before they are set (at run time). The problem you are having with the value of "Count" is because it is within the FOR loop.

Anyway, I recommend that you do at least some of this without delayed expansion. I know you said that can't be done for other reasons... but I can't see the rest of your code to verify that. In the code below we pick up each line in a FOR loop without enabling delayed expansion to preserve the "!". Then we call a subroutine to determine if this is a valid URL (insert your other code there... some of which may need delayed expansion).

   @echo off
   set /a Count=0
   set /a URLCount=0
   
   for /F "delims=" %%F in (urls.txt) do call :IsItURL "%%F"
   echo Total: %URLCount% of %Count%
   pause
   
   :IsItURL
   REM 1=Text string
   set /a Count+=1
   rem <Your other code goes here... if "%~1" is URL> set /a URLCount+=1
   goto :eof

Windows Shell Scripting and InstallShield

Offline

#6 31 Jan 2020 17:27

OJBakker
Member
Registered: 05 Aug 2011
Posts: 4

Re: Title change: Pass variable definitions OUT of SETLOCAL "bubble"?

Rekrul wrote:

The first variable "URLCount" is preserved and survive the rest of the script intact, however the second variable "Count" gets reset each time the loop repeats. WHY? I preserve it after the ENDLOCAL statement, so why is it getting set back to zero every time???

Why isn't the value of "Count" preserved when the loop starts again and how do I fix this so that it is?

NOTE: For this example, I have stripped out other parts of the script that have no bearing on the problem at hand, to make it as simple as possible.

The for-loop counting URLCount is finished before you use ENDLOCAL, so the value of URLCount is uptodate in the 'Set URLCount=%URLCount%'.

The second for-loop counting Count has the SETLOCAL ... ENDLOCAL within te commandblock of the for-loop.
So the value of %Count% will remain the initial value of Count when the for-loop started, that is zero.
The uptodate value of Count is available in !Count!, you already use this in the echo line.
Change the ENDLOCAL line to 'ENDLOCAL & set COUNT-!COUNT!' and you will have both counts uptodate after the second for-loop is completed.

Offline

#7 31 Jan 2020 23:06

Rekrul
Member
Registered: 17 Apr 2016
Posts: 69

Re: Title change: Pass variable definitions OUT of SETLOCAL "bubble"?

RG wrote:

It is helpful to consider both the 'load time' and 'run time' behavior' of bat files. The 'trick' you refer to is a technique called 'tunneling'. It works because everything in an IF statement, FOR loop, or everything between parentheses is loaded as one line. So variables are expanded (at load time) before they are set (at run time). The problem you are having with the value of "Count" is because it is within the FOR loop.

As much as I try, I still can't keep the two straight. Logic keeps telling me "It's a variable, it holds a value, when you read it, you get that value." Anything else seems illogical to me.

RG wrote:

Anyway, I recommend that you do at least some of this without delayed expansion. I know you said that can't be done for other reasons... but I can't see the rest of your code to verify that.

When I get everything working, I plan to release it so that... it can be largely ignored. smile

I don't want to release it before it's finished, so I'll describe what it does. Actually it works well enough that I've been using it for a while, but it still has potential bugs, like stripping any potential "!" from URLs, and the other day I realized that I wasn't checking for numeric input at the prompts that expect numeric input.

The main script is a menu-driven front end for the command line video downloader YouTube-DL.exe. It allows you to download video URLs in both Single and Batch modes. Single mode allows you to enter one URL at a time, which are each launched in a new process to avoid tying up the main script. It does this by echoing the commands to create a numbered download script and transfers control to it. It does this rather than launching the command directly so that if an URL fails (which happens some times) it will retry it up to the user-defined number of times and if they all fail, it writes the URL to a text file. YouTube-DL does have its own retry mechanism, but it only handles minor errors, not aborted downloads, false error messages from the site that will be bypassed if you keep trying, etc. When it finishes, the download script erases itself using a trick I found in another forum.

Batch mode allows you to process a text file full of URLs, which is the part I've been posting snippets of here. When you enter the name of a valid file, the main script then echos all the commands needed to create the Batch script and launches it as a separate process. I did this rather than just making a separate script so that everything (with the exception of the EXE files) is contained within a single script.

The Batch script first reads the text file passed to it to count the URLs, checking each line for "http" to determine whether it's an URL or not. That value is stored in URLCount. Then the main loop reads the file one line at a time and jumps to a subroutine. There, the current line is checked to verify that it's an URL and if so, a bunch of echo commands are executed that create another script file, this time using the variable "Count" added to the filename so that there are no conflicts. This new script file is basically identical to the ones launched by the main script in Single mode, except with a different filename so there's no conflicts between the two and it prints a message indicating what number file is being downloaded. It will retry each download up to the user-defined limit, write failed URLs to a file and erase itself when done.

From there the Batch script then checks to see if Count and URLCount are the same and if so, skips the next portion of the script. Otherwise it then jumps to a routine that counts how many Batch download scripts exist and compares that number to a user-defined limit. If the numbers match, it means the max number of concurrent downloads has been reached. The routine waits 2 seconds (I felt it was better to add a delay rather than let it run as fast as possible) before jumping back to the start of the routine to check again. When the number of existing download scripts is less than the limit due to some of them having finished and erased themselves, it exits back to the main loop to launch another URL, and the whole process starts over again.

Once there are no more URLs to process it goes to the ending routine which waits for all the downloads to finish, by checking how many download scripts still exist and when that number hits zero, the batch script erases itself, leaving just the downloaded files (and possibly a list of failed URLs). Actually, the Batch script could end and erase itself after the last URL has been launched, but since it prints status messages of which number URL is being processed, I felt it was more appropriate to leave it open until all the downloads finish. Even though the messages aren't always accurate. If URL #3 is a huge file and the rest are small, the last update will say it's downloading URL #25, but #3 is the only download that hasn't finished yet. Now that I think about it, I suppose I could have a dynamic display that bases the status messages on the existing download scripts so that it's always accurate. And this is how my hap-hazard design process normally goes, I keep thinking of things to tweak smile

Originally I only had it create one download script and that script would create numbered FLAGxxx.txt files, which the Batch script would check for. Then I realized that if it's creating additional files anyway, just create numbered download scripts that can act as the flags. While they're larger than the flag files were, they're still microscopically tiny by today's standards. Hell, the entire script with several pages of help text is only a hair over 14K.

My latest addition was to add an option in the main script to gracefully stop the Batch script. It does this by writing a file called STOP.txt to the directory being used, and the Batch script checks for this file before launching each download. If it's detected, it immediately goes to the ending routine and will exit once all current downloads finish. You can of course stop the whole process by closing the Batch window manually, but that leaves the Batch script behind. Harmless, but messy. Although the Batch script would get overwritten and erase itself the next time you used Batch mode in that directory.

In addition to setting the max number of concurrent downloads for Batch mode and the number of retries, the main script also lets you select to launch the Batch window open or minimized, view help information (now out of date) and should eventually also let you enter a destination path for downloads other than the current directory. I added the code to enter and display it, but currently it doesn't use it for anything. Oh, you can also save the current settings, load them (done automatically if the settings file exists at launch) and delete them. I should probably add a reset option as well...

Of course I realize that by the time I finish this, someone will have written a graphical front end for YouTube-DL.exe (if they haven't already), but I'm enjoying creating something useful that not only will I use, but maybe others will as well. Plus, I can customize it to include the options that I want. I know that it would have been easier to just make separate scripts for the functions that are launched separately, but I like the idea of everything being contained in the single script and that it create secondary (and even third-ary) scripts as needed. And to be honest, I kind of like impressing those with less computer knowledge than myself with my ability to create something like this, even though I know that my programming knowledge falls far short of that of others

RG wrote:

In the code below we pick up each line in a FOR loop without enabling delayed expansion to preserve the "!". Then we call a subroutine to determine if this is a valid URL (insert your other code there... some of which may need delayed expansion).

So when you CALL a subroutine, that subroutine behaves as if DelayedExpansion is enabled? I added an echo command to the subroutine and it properly showed the value of Count and URLCount as if it was.

OJBakker wrote:

The second for-loop counting Count has the SETLOCAL ... ENDLOCAL within te commandblock of the for-loop.
So the value of %Count% will remain the initial value of Count when the for-loop started, that is zero.
The uptodate value of Count is available in !Count!, you already use this in the echo line.
Change the ENDLOCAL line to 'ENDLOCAL & set COUNT-!COUNT!' and you will have both counts uptodate after the second for-loop is completed.

That didn't work for me.

One trick I know I can use is to write the values to a temporary file at the end of the loop and read them in again at the start of the next iteration. I didn't mention this sooner because I was hoping there was a more elegant solution and I didn't want people to just think I'd found my perfect solution.

Last edited by Rekrul (01 Feb 2020 04:01)

Offline

#8 20 Feb 2020 08:57

jeb
Member
From: Germany
Registered: 19 Nov 2010
Posts: 109

Re: Title change: Pass variable definitions OUT of SETLOCAL "bubble"?

1. Reading from a file (or command output)

with a FOR loop without destroying exclamation marks and carets (yes carets have problems too) is to use the delayed toggling technique.

setlocal DisableDelayedExpansion
FOR /F "delims=" %%L in ('findstr /n "^" test.txt') DO (
    set "line=%%L"
    setlocal EnableDelayedExpansion
    set "line=!line:*:=!"
    echo(!line!
    endlocal
)

Transfer the value from %%L to a variable has to be done without delayed expansion, else you lose exclamation marks and carets (carets only when at least one exclam is present, too).
But to handle the line variable, delayed expansion should be enabled.
The set "line=!line:*:=!" simply removes the prefixed line number. The line number is needed to handle empty lines and lines beginning with a semi colon (the EOL-char).

That's said.

2. How to return a value out of the endlocal scope?

It can be done by using the percent expansion in the same command block (a command block can be build by & or parenthesis) with a endlocal.
That works, because the percent expansion expands when a block is read, even before any part of the block is executed.

That's the cause why your first URLCount sample works.
Your second sample can't work, as the command bock is the complete FOR-loop

set Count=OUT OF LOOP
for /F "delims=" %%F in (urls.txt) do (
    setlocal EnableDelayedExpansion

    set /a Count+=1
    echo !Count! of !URLCount!

    endlocal & set Count=%Count%
)

The count variable is expanded in the moment the block is parsed, therefor it's expanded to the value "OUT OF LOOP".

You can fix it with another FOR-loop

for /F "delims=" %%F in (urls.txt) do (
    setlocal EnableDelayedExpansion

    set /a Count+=1
    echo !Count! of !URLCount!

    FOR /F "delims=" %%# in ("!count!") DO (
        endlocal 
        set Count=%%#
    )
    echo Count is still set after the endlocal, proof:
    set "count"
)

This technique can be used also for more than one variable to transfer out of scope

for /F "delims=" %%F in (urls.txt) do (
    setlocal EnableDelayedExpansion

    set /a Count+=1
    echo !Count! of !URLCount!
    set "inner_url_temp=%%F"

    FOR /F "delims=" %%U in ("!inner_url_temp!") DO (
	FOR /F "delims=" %%# in ("!count!") DO (
		endlocal 
		set "Count=%%#"
		set "URL=%%U"
	)
	)
	echo Count is still set after the endlocal, proof:
	set "count"
	echo And URL contains data, too:
	set "URL"
)

This works good for most of contents.
But there are limits  smile
It fails for:
- Line feeds in the variable content
- The outer context is also a delayed expansion context and your variable contains ! and/or carets

Both can be solved, but with more advanced methods.
But it's not necessary to understand the stuff, simply use it !

for /F "delims=" %%F in (urls.txt) do (
    setlocal EnableDelayedExpansion
	set "tmp=<something nasty, complex>"
	
	%ENDLOCAL% tmp
	
	echo tmp is working
	set "tmp"
)

ENDLOCAL is a variable, containing a batch macro, see preserving exclamation marks in variable between setlocals batch

Last edited by jeb (20 Feb 2020 08:58)

Offline

Board footer

Powered by FluxBB