#1 12 Jan 2020 14:27

Rekrul
Member
Registered: 17 Apr 2016
Posts: 54

Someone please help me understand why this doesn't work

I have a command line video downloader program, YouTube-DL. I'm trying to make a script that will read a text file of URLs, one per line, and launch each one in its own separate window so that they will run concurrently. However, as I don't know ahead of time how many lines will be in the file, I don't want it launching dozens of windows at once. I want to limit it to five active downloads at once.

To do this, I made a separate download script that will retry each URL up to ten times in case of errors. The main script sets a counter and the secondary script creates a file named "flag-_" with the underscore being replaced with the current counter value. This ensures that each of these "flag" files has a different name. When the secondary script finishes, the "flag" file is deleted and the secondary script ends. All of that is working fine.

The part I'm having trouble with is that I want the main script to count the number of "flag" files and loop if the number is less than five. Meaning that less than five copies of the secondary script are currently running. If the number of "flag" files is less than 5, I want it to loop and start another download. The idea being that there will always be five downloads active, until it finishes the list. When one finishes the main script will detect it and start another. Of course it doesn't work.

echo off
set count=1

cls
echo.
echo.
set /P name=Enter list name:

setlocal enabledelayedexpansion

for /F "delims=" %%F in (!name!) do (
set url=%%F
start Download
set /a count=count+1
call :Check
)
goto end


:Check
set active=0
for %%X in (flag-*) do set /a active=active+1
if !active! lss 5 exit /b
sleep 2
goto Check

:end
exit

The Check subroutine should exit back to the main portion only if the count of "flag" files is less than 5. If it's 5, it should wait 2 seconds and then run the check again. Of course what actually happens is that it opens a random number of download windows. It might be six, it might be nine. I have no idea what's happening.

I've tested the code of the subroutine by making a script that counts the number of files in a directory and exits when they fall below a certain number, so the code seems to work, it just doesn't work here, even though it's exactly the same, other than the files it's checking for. For example;

echo off

setlocal enabledelayedexpansion

Echo checking
call :check
Echo files = !active!
goto end

:Check
set active=0
for %%X in (*.jpg) do set /a active=active+1
if !active! lss 5 exit /b
sleep 2
goto Check

:end
exit

Put some jpeg files in a directory and run the script. If the number of files is greater than 4, the script keeps checking. As soon as the number of files falls below 5, the script prints the number of files and exits. It works here, so why does it completely fail in my real script???

Here's the secondary script "Download.bat";

echo off
set rt=1
echo test >flag-%count%
:start
youtube-dl "%url%"
if errorlevel 1 goto Retry

:End
del flag-%count%
exit

:Retry
set /a rt=rt+1
if %rt% lss 11 goto start
echo %url% >>Failed.txt
goto End

To test it, you'll need a copy of YouTube-DL and a list of video URLS in a text file.

What am I doing wrong?

Offline

#2 13 Jan 2020 17:00

OJBakker
Member
Registered: 05 Aug 2011
Posts: 3

Re: Someone please help me understand why this doesn't work

I did not download and run your scripts but I see a possible problem in the code.

You start the download.bat in a separate process.
This creates a flag-file indicating 'this file is being downloaded' and deletes this flag-file when download is done.

After the 'start download' the main process increments counter and calls subroutine Check.
This is were I see a potential problem.

Subroutine check is probably running before download makes the flag-file (remember download starts in a separate process!)
If there is no flag-file or less than 5 flag-files then immediately exit check and start the next download.

Solution:
Add an extra "sleep 5" before the 'calll :Check' and then run your main batchfile again to see if this solves the problem.

Offline

#3 13 Jan 2020 23:32

Rekrul
Member
Registered: 17 Apr 2016
Posts: 54

Re: Someone please help me understand why this doesn't work

OJBakker wrote:

Subroutine check is probably running before download makes the flag-file (remember download starts in a separate process!)
If there is no flag-file or less than 5 flag-files then immediately exit check and start the next download.

Solution:
Add an extra "sleep 5" before the 'calll :Check' and then run your main batchfile again to see if this solves the problem.

Thank you! That was exactly the problem. I solved it without introducing any additional delays by moving the creation of the flag files from the secondary script to the main one, just after the second script is called. This way a flag file is always created before the Check subroutine runs.

Now that it works I want to add options to customize the number of concurrent downloads and the number of retries. I'm kind of an option junkie. smile

Offline

Board footer

Powered by FluxBB