ftp script - MoveIT mput only uploads one file

Microsoft Windows
Post Reply
User avatar
MigrationUser
Posts: 336
Joined: 2021-Jul-12, 1:37 pm
Contact:

ftp script - MoveIT mput only uploads one file

Post by MigrationUser »

08 Mar 2010 20:14
GrandPixel

I have a script that runs daily to upload files with ftps.exe (MOVEit Freely by Ipswitch - similar syntax as ftp.exe but supports passive and secure transfers). The script uses passive and binary modes to transfer with the mput command. Often there is only one file to transfer, but when there are two or three, only the first file gets uploaded. These files are 100MB so perhaps the connection is timing out before the first file is complete. But then I would think it should not time out during a transfer...

Any ideas? Let me know if you need to see the scripts.

----------------------------

#2 08 Mar 2010 20:50
bluesxman


Visibility of your scripts would be nigh on essential :-)

cmd | *sh | ruby | chef

----------------------------

#3 08 Mar 2010 22:34
GrandPixel

Code: Select all

@ECHO OFF

REM *****************************************************************************
REM * This script creates an incremental backup of the [***] repositories,
REM * compresses the backup into a RAR archive, and uploads the archive to the
REM * [***] FTP server.
REM *****************************************************************************


REM CREATE VARIABLES
REM *****************************************************************************
SET today=%date:~10,4%_%date:~4,2%_%date:~7,2%
SET source=[X:\***source***]
SET targetpath=[X:\***targetpath***\]
SET targetfile=Backup.[***].%today%.Incremental


REM CREATE BACKUP
REM *****************************************************************************
REM * /d "Set Created %today%"
REM * Specifies a label for each backup set.
REM *
REM * /v:yes
REM * Verifies the data after the backup is complete.
REM *
REM * /r:no
REM * Restricts access to this tape to the owner or members of the Administrators
REM * group.
REM * 
REM * /rs:no
REM * Backs up the migrated data files located in Remote Storage. The /RS
REM * command-line option is not required to back up the local Removable Storage
REM * database (that contains the Remote Storage placeholder files). When you
REM * backup the %systemroot% folder, Backup automatically backs up the Removable
REM * Storage database as well.
REM * 
REM * /hc:off
REM * Uses hardware compression, if available, on the tape drive.
REM * 
REM * /m incremental
REM * Specifies the backup type. It must be one of the following: normal, copy,
REM * differential, incremental, or daily.
REM * 
REM * /j "Incremental Backup"
REM * Specifies the job name to be used in the backup report. The job name
REM * usually describes the files and folders you are backing up in the current
REM * backup job.
REM * 
REM * /l:s
REM * Specifies the type of log file: f=full, s=summary, n=none (no log file is
REM * created).
REM * 
REM * /f %targetpath%%targetfile%.bkf
REM * Logical disk path and file name. You must not use the following switches
REM * with this switch: /P /G /T.
REM *****************************************************************************
ntbackup backup "%source%" /d "Set Created %today%" /v:yes /r:no /rs:no /hc:off /m incremental /j "Incremental Backup" /l:s /f %targetpath%%targetfile%.bkf


REM COMPRESS BACKUP INTO ARCHIVE
REM *****************************************************************************
REM *  m       move files into archive (delete originals)
REM * -ep      exclude paths from names
REM * -m5      use max compression
REM * -os      save NTFS streams (file descriptions, etc.)
REM * -ow      save owner, permissions, etc.
REM * -rr      add 1% recovery record in case file becomes damaged
REM * -t       test files after archiving (will not delete source files unless
REM *          test passes)
REM * -ts      save file times (modification, creation, access)
REM * -v100m   split into 100MB volumes
REM *****************************************************************************
rar m -ep -m5 -os -ow -rr -t -ts -v100m %targetpath%%targetfile%.rar %targetpath%%targetfile%.bkf


REM CREATE FTP RESPONSE FILE TO UPLOAD ARCHIVE
REM *****************************************************************************
ECHO open ftp.[***].net>> %targetpath%%targetfile%.ftp
ECHO [***username***]>> %targetpath%%targetfile%.ftp
ECHO [***password***]>> %targetpath%%targetfile%.ftp
ECHO binary>> %targetpath%%targetfile%.ftp
ECHO prompt>> %targetpath%%targetfile%.ftp
ECHO mput %targetpath%%targetfile%*rar>> %targetpath%%targetfile%.ftp
ECHO bye>> %targetpath%%targetfile%.ftp


REM UPLOAD ARCHIVE TO [***] FTP SERVER
REM *****************************************************************************
REM * Many FTP servers support only passive transfers, including our host for
REM * [***]. The command-line FTP client included with Windows (ftp.exe)
REM * does not support passive transfers. MOVEit Freely by Ipswitch (ftps.exe) is
REM * an alternative command-line FTP cilent with very similar syntax and a more
REM * complete feature set, including passive transfers.
REM * 
REM * -a
REM * Starts the program in passive mode.
REM * 
REM * -d
REM * Displays commands (except passwords) sent to the server.
REM * 
REM * -s:%targetpath%%targetfile%.ftp
REM * Reads commands from a script file instead of the console.
REM *****************************************************************************
FTPS -a -d -s:%targetpath%%targetfile%.ftp > %targetpath%%targetfile%.ftp.log


REM MOVE FTP RESPONSE FILE AND FTP LOG OUT OF BACKUP DIRECTORY
REM *****************************************************************************
move %targetpath%%targetfile%.ftp "%targetpath%FTP Scripts & Transfer Logs"
move %targetpath%%targetfile%.ftp.log "%targetpath%FTP Scripts & Transfer Logs"
Last edited by GrandPixel (08 Mar 2010 23:35)

----------------------------

#4 09 Mar 2010 09:23
bluesxman


Hmmm, nothing obvious wrong there.

I'd be a bit surprised if it was due to a timeout, but that could be tested by pushing the files up individually with "PUT". Does this modification shed any light?

Code: Select all

REM CREATE FTP RESPONSE FILE TO UPLOAD ARCHIVE
REM *****************************************************************************
ECHO open ftp.[***].net>> %targetpath%%targetfile%.ftp
ECHO [***username***]>> %targetpath%%targetfile%.ftp
ECHO [***password***]>> %targetpath%%targetfile%.ftp
ECHO binary>> %targetpath%%targetfile%.ftp
ECHO prompt>> %targetpath%%targetfile%.ftp
for %%F in ("%targetpath%%targetfile%*rar") do (
    ECHO put "%%~F"
) >> %targetpath%%targetfile%.ftp
ECHO bye>> %targetpath%%targetfile%.ftp
cmd | *sh | ruby | chef

----------------------------

#5 09 Mar 2010 19:22
GrandPixel


I am getting an error:

%%F was unexpected at this time.

----------------------------

#6 09 Mar 2010 20:20
bluesxman


I'll hazard an educated guess that you are running the "for" command from the command prompt and not as part of a script, in which case that error is to be expected.

cmd | *sh | ruby | chef

----------------------------

#7 09 Mar 2010 21:47
GrandPixel


you are right, I need to use %F perhaps - and it works, i am uploading now using the .ftp input file - we will see if it makes it past the first file

----------------------------

#8 09 Mar 2010 23:10
GrandPixel


Here is the .ftp input file created by the modified script:

Code: Select all

open ftp.[***].net
[***username***]
[***password***]
binary
prompt
put "[X:\***\Backup.[***].part1.rar"
put "[X:\***\Backup.[***].part2.rar"
put "[X:\***\Backup.[***].part3.rar"
bye
Looks good... Here is the log of the FTP session wijth the 3 individual PUT commands
ftp> 220-Welcome to the [***] Web Hosting FTP server.
220-
220-No anonymous logins accepted.
Connected to ftp.[***].net.
User: ---> USER [***username***]
331-Enter your password
331
Password: ---> PASS (hidden)
230
ftp> ---> TYPE I
200 Type set to I.
ftp> Interactive mode Off .
ftp> ---> PASV
227 Entering Passive Mode (##,###,###,###,###,##)
---> STOR Backup.[***].part1.rar
150 Opening BINARY mode data connection for /***/Backup.[***].part1.rar .
ftp: 104857600 bytes sent in 4136.66Seconds 24.75Kbytes/sec.
ftp> ---> PASV
ftp: 0 bytes sent in 0.01Seconds 0.00Kbytes/sec.
ftp> Not connected.
ftp> ---> QUIT
Still being cut off at the end of the first file. Any more possibilities? Maybe there is an issue with my FTP host.

----------------------------

#9 10 Mar 2010 12:19
bluesxman


Not overly familiar with the intricacies of passive mode, but 70 minutes to upload a single file does sound like a prime candidate for being timed out.

You'll likely have to make a new connection for each file. Like so, perhaps?

Code: Select all

REM CREATE FTP RESPONSE FILE TO UPLOAD ARCHIVE
REM *****************************************************************************
for %%F in ("%targetpath%%targetfile%*rar") do (
    ECHO open ftp.[***].net>> %targetpath%%targetfile%.ftp
    ECHO [***username***]>> %targetpath%%targetfile%.ftp
    ECHO [***password***]>> %targetpath%%targetfile%.ftp
    ECHO binary>> %targetpath%%targetfile%.ftp
    ECHO prompt>> %targetpath%%targetfile%.ftp
    ECHO put "%%~F"
    ECHO close>> %targetpath%%targetfile%.ftp
) >> %targetpath%%targetfile%.ftp
ECHO bye>> %targetpath%%targetfile%.ftp
cmd | *sh | ruby | chef

----------------------------

#10 10 Mar 2010 17:49
GrandPixel


I am trying to create a separate ftp response file for each file to be uploaded. But for whatever reason, it is putting everything into "Backup.[***].part0.ftp

Code: Select all

SET /a counter = 0


for %%F in ("%targetpath%%targetfile%*rar") do (
    SET /a counter += 1
    ECHO open ftp.[***].net>> %targetpath%%targetfile%.part%counter%.ftp
    ECHO [***username***]>> %targetpath%%targetfile%.part%counter%.ftp
    ECHO [***password***]>> %targetpath%%targetfile%.part%counter%.ftp
    ECHO binary>> %targetpath%%targetfile%.part%counter%.ftp
    ECHO prompt>> %targetpath%%targetfile%.part%counter%.ftp
    ECHO put "%%~F">> %targetpath%%targetfile%.part%counter%.ftp
    ECHO bye>> %targetpath%%targetfile%.part%counter%.ftp
)
----------------------------

#11 10 Mar 2010 18:26
GrandPixel


This is the output at the command prompt:

Code: Select all

X:\[***]>SET /a counter = 0

X:\[***]>for %F in ("[X:\[***]*rar") do (
SET /a counter += 1
 ECHO open ftp.[***].net 1>>X:\[***].part0.ftp
 ECHO [***username***] 1>>X:\[***].part0.ftp
 ECHO [***password***] 1>>X:\[***].part0.ftp
 ECHO binary 1>>X:\[***].part0.ftp
 ECHO prompt 1>>X:\[***].part0.ftp
 ECHO put "%~F" 1>>X:\[***].part0.ftp
 ECHO bye 1>>X:\[***].part0.ftp
)

X:\[***]>(
SET /a counter += 1
 ECHO open ftp.[***].net 1>>X:\[***].part0.ftp
 ECHO [***username***] 1>>X:\[***].part0.ftp
 ECHO [***password***] 1>>X:\[***].part0.ftp
 ECHO binary 1>>X:\[***].part0.ftp
 ECHO prompt 1>>X:\[***].part0.ftp
 ECHO put "X:\[***].part1.rar" 1>>X:\[***].part0.ftp
 ECHO bye 1>>X:\[***].part0.ftp
)

X:\[***]>(
SET /a counter += 1
 ECHO open ftp.[***].net 1>>X:\[***].part0.ftp
 ECHO [***username***] 1>>X:\[***].part0.ftp
 ECHO [***password***] 1>>X:\[***].part0.ftp
 ECHO binary 1>>X:\[***].part0.ftp
 ECHO prompt 1>>X:\[***].part0.ftp
 ECHO put "X:\[***].part2.rar" 1>>X:\[***].part0.ftp
 ECHO bye 1>>X:\[***].part0.ftp
)

X:\[***]>(
SET /a counter += 1
 ECHO open ftp.[***].net 1>>X:\[***].part0.ftp
 ECHO [***username***] 1>>X:\[***].part0.ftp
 ECHO [***password***] 1>>X:\[***].part0.ftp
 ECHO binary 1>>X:\[***].part0.ftp
 ECHO prompt 1>>X:\[***].part0.ftp
 ECHO put "X:\[***].part3.rar" 1>>X:\[***].part0.ftp
 ECHO bye 1>>X:\[***].part0.ftp
)

X:\[***]>
and this is the resulting .ftp response file:

Code: Select all

open ftp.[***].net
[***username***]
[***password***]
binary
prompt
put "X:\[***].part1.rar"
bye
open ftp.[***].net
[***username***]
[***password***]
binary
prompt
put "X:\[***].part2.rar"
bye
open ftp.[***].net
[***username***]
[***password***]
binary
prompt
put "X:\[***].part3.rar"
bye
----------------------------

#12 10 Mar 2010 18:41
bluesxman


You'd need "setlocal enabledelayedexpansion" for that. But your method is somewhat clunky.

I'd do it this way, myself:

Code: Select all

for %%F in ("%targetpath%%targetfile%*rar") do (
    (
        ECHO open ftp.[***].net
        ECHO [***username***]
        ECHO [***password***]
        ECHO binary
        ECHO prompt
        ECHO put "%%~F"
        ECHO bye
    ) >> "%%~dpnF.ftp"
)
PS -- I've realised my previous suggestion was a little bit wrong. But since you've moved on from that, I shan't bother to correct it smile

Last edited by bluesxman (10 Mar 2010 18:44)

cmd | *sh | ruby | chef

----------------------------

#13 10 Mar 2010 18:56
GrandPixel


I thought I was using your suggestion smile but I thought to make it work I would have to create a separate .ftp response file for each file upload. I will input your latest suggestion and see what happens

----------------------------

#14 10 Mar 2010 18:58

GrandPixel


How do you do that!

----------------------------

#15 11 Mar 2010 10:00
bluesxman
GrandPixel wrote:

I thought to make it work I would have to create a separate .ftp response file for each file upload
I don't see why you would -- the script I proposed would create one long response file, recreating the connection before each upload.

Here is my earlier method for creating a single FTP response file but corrected for the mistake I made:

Code: Select all

REM CREATE FTP RESPONSE FILE TO UPLOAD ARCHIVE
REM *****************************************************************************
for %%F in ("%targetpath%%targetfile%*rar") do (
    ECHO open ftp.[***].net
    ECHO [***username***]
    ECHO [***password***]
    ECHO binary
    ECHO prompt
    ECHO put "%%~F"
    ECHO close
) >> %targetpath%%targetfile%.ftp
ECHO bye>> %targetpath%%targetfile%.ftp
Last edited by bluesxman (11 Mar 2010 10:01)

cmd | *sh | ruby | chef

----------------------------

#16 11 Mar 2010 18:02
GrandPixel


I see what you were doing now. I may change it to use a single response file, but either way, I think I have a working solution. Thanks very much for your help.

----------------------------

#17 11 Mar 2010 20:23
bluesxman


You're welcome.
Post Reply