GrandPixel
I have a script that runs daily to upload files with ftps.exe (MOVEit Freely by Ipswitch - similar syntax as ftp.exe but supports passive and secure transfers). The script uses passive and binary modes to transfer with the mput command. Often there is only one file to transfer, but when there are two or three, only the first file gets uploaded. These files are 100MB so perhaps the connection is timing out before the first file is complete. But then I would think it should not time out during a transfer...
Any ideas? Let me know if you need to see the scripts.
----------------------------
#2 08 Mar 2010 20:50
bluesxman
Visibility of your scripts would be nigh on essential
cmd | *sh | ruby | chef
----------------------------
#3 08 Mar 2010 22:34
GrandPixel
Code: Select all
@ECHO OFF
REM *****************************************************************************
REM * This script creates an incremental backup of the [***] repositories,
REM * compresses the backup into a RAR archive, and uploads the archive to the
REM * [***] FTP server.
REM *****************************************************************************
REM CREATE VARIABLES
REM *****************************************************************************
SET today=%date:~10,4%_%date:~4,2%_%date:~7,2%
SET source=[X:\***source***]
SET targetpath=[X:\***targetpath***\]
SET targetfile=Backup.[***].%today%.Incremental
REM CREATE BACKUP
REM *****************************************************************************
REM * /d "Set Created %today%"
REM * Specifies a label for each backup set.
REM *
REM * /v:yes
REM * Verifies the data after the backup is complete.
REM *
REM * /r:no
REM * Restricts access to this tape to the owner or members of the Administrators
REM * group.
REM *
REM * /rs:no
REM * Backs up the migrated data files located in Remote Storage. The /RS
REM * command-line option is not required to back up the local Removable Storage
REM * database (that contains the Remote Storage placeholder files). When you
REM * backup the %systemroot% folder, Backup automatically backs up the Removable
REM * Storage database as well.
REM *
REM * /hc:off
REM * Uses hardware compression, if available, on the tape drive.
REM *
REM * /m incremental
REM * Specifies the backup type. It must be one of the following: normal, copy,
REM * differential, incremental, or daily.
REM *
REM * /j "Incremental Backup"
REM * Specifies the job name to be used in the backup report. The job name
REM * usually describes the files and folders you are backing up in the current
REM * backup job.
REM *
REM * /l:s
REM * Specifies the type of log file: f=full, s=summary, n=none (no log file is
REM * created).
REM *
REM * /f %targetpath%%targetfile%.bkf
REM * Logical disk path and file name. You must not use the following switches
REM * with this switch: /P /G /T.
REM *****************************************************************************
ntbackup backup "%source%" /d "Set Created %today%" /v:yes /r:no /rs:no /hc:off /m incremental /j "Incremental Backup" /l:s /f %targetpath%%targetfile%.bkf
REM COMPRESS BACKUP INTO ARCHIVE
REM *****************************************************************************
REM * m move files into archive (delete originals)
REM * -ep exclude paths from names
REM * -m5 use max compression
REM * -os save NTFS streams (file descriptions, etc.)
REM * -ow save owner, permissions, etc.
REM * -rr add 1% recovery record in case file becomes damaged
REM * -t test files after archiving (will not delete source files unless
REM * test passes)
REM * -ts save file times (modification, creation, access)
REM * -v100m split into 100MB volumes
REM *****************************************************************************
rar m -ep -m5 -os -ow -rr -t -ts -v100m %targetpath%%targetfile%.rar %targetpath%%targetfile%.bkf
REM CREATE FTP RESPONSE FILE TO UPLOAD ARCHIVE
REM *****************************************************************************
ECHO open ftp.[***].net>> %targetpath%%targetfile%.ftp
ECHO [***username***]>> %targetpath%%targetfile%.ftp
ECHO [***password***]>> %targetpath%%targetfile%.ftp
ECHO binary>> %targetpath%%targetfile%.ftp
ECHO prompt>> %targetpath%%targetfile%.ftp
ECHO mput %targetpath%%targetfile%*rar>> %targetpath%%targetfile%.ftp
ECHO bye>> %targetpath%%targetfile%.ftp
REM UPLOAD ARCHIVE TO [***] FTP SERVER
REM *****************************************************************************
REM * Many FTP servers support only passive transfers, including our host for
REM * [***]. The command-line FTP client included with Windows (ftp.exe)
REM * does not support passive transfers. MOVEit Freely by Ipswitch (ftps.exe) is
REM * an alternative command-line FTP cilent with very similar syntax and a more
REM * complete feature set, including passive transfers.
REM *
REM * -a
REM * Starts the program in passive mode.
REM *
REM * -d
REM * Displays commands (except passwords) sent to the server.
REM *
REM * -s:%targetpath%%targetfile%.ftp
REM * Reads commands from a script file instead of the console.
REM *****************************************************************************
FTPS -a -d -s:%targetpath%%targetfile%.ftp > %targetpath%%targetfile%.ftp.log
REM MOVE FTP RESPONSE FILE AND FTP LOG OUT OF BACKUP DIRECTORY
REM *****************************************************************************
move %targetpath%%targetfile%.ftp "%targetpath%FTP Scripts & Transfer Logs"
move %targetpath%%targetfile%.ftp.log "%targetpath%FTP Scripts & Transfer Logs"
----------------------------
#4 09 Mar 2010 09:23
bluesxman
Hmmm, nothing obvious wrong there.
I'd be a bit surprised if it was due to a timeout, but that could be tested by pushing the files up individually with "PUT". Does this modification shed any light?
Code: Select all
REM CREATE FTP RESPONSE FILE TO UPLOAD ARCHIVE
REM *****************************************************************************
ECHO open ftp.[***].net>> %targetpath%%targetfile%.ftp
ECHO [***username***]>> %targetpath%%targetfile%.ftp
ECHO [***password***]>> %targetpath%%targetfile%.ftp
ECHO binary>> %targetpath%%targetfile%.ftp
ECHO prompt>> %targetpath%%targetfile%.ftp
for %%F in ("%targetpath%%targetfile%*rar") do (
ECHO put "%%~F"
) >> %targetpath%%targetfile%.ftp
ECHO bye>> %targetpath%%targetfile%.ftp
----------------------------
#5 09 Mar 2010 19:22
GrandPixel
I am getting an error:
%%F was unexpected at this time.
----------------------------
#6 09 Mar 2010 20:20
bluesxman
I'll hazard an educated guess that you are running the "for" command from the command prompt and not as part of a script, in which case that error is to be expected.
cmd | *sh | ruby | chef
----------------------------
#7 09 Mar 2010 21:47
GrandPixel
you are right, I need to use %F perhaps - and it works, i am uploading now using the .ftp input file - we will see if it makes it past the first file
----------------------------
#8 09 Mar 2010 23:10
GrandPixel
Here is the .ftp input file created by the modified script:
Code: Select all
open ftp.[***].net
[***username***]
[***password***]
binary
prompt
put "[X:\***\Backup.[***].part1.rar"
put "[X:\***\Backup.[***].part2.rar"
put "[X:\***\Backup.[***].part3.rar"
bye
Still being cut off at the end of the first file. Any more possibilities? Maybe there is an issue with my FTP host.ftp> 220-Welcome to the [***] Web Hosting FTP server.
220-
220-No anonymous logins accepted.
Connected to ftp.[***].net.
User: ---> USER [***username***]
331-Enter your password
331
Password: ---> PASS (hidden)
230
ftp> ---> TYPE I
200 Type set to I.
ftp> Interactive mode Off .
ftp> ---> PASV
227 Entering Passive Mode (##,###,###,###,###,##)
---> STOR Backup.[***].part1.rar
150 Opening BINARY mode data connection for /***/Backup.[***].part1.rar .
ftp: 104857600 bytes sent in 4136.66Seconds 24.75Kbytes/sec.
ftp> ---> PASV
ftp: 0 bytes sent in 0.01Seconds 0.00Kbytes/sec.
ftp> Not connected.
ftp> ---> QUIT
----------------------------
#9 10 Mar 2010 12:19
bluesxman
Not overly familiar with the intricacies of passive mode, but 70 minutes to upload a single file does sound like a prime candidate for being timed out.
You'll likely have to make a new connection for each file. Like so, perhaps?
Code: Select all
REM CREATE FTP RESPONSE FILE TO UPLOAD ARCHIVE
REM *****************************************************************************
for %%F in ("%targetpath%%targetfile%*rar") do (
ECHO open ftp.[***].net>> %targetpath%%targetfile%.ftp
ECHO [***username***]>> %targetpath%%targetfile%.ftp
ECHO [***password***]>> %targetpath%%targetfile%.ftp
ECHO binary>> %targetpath%%targetfile%.ftp
ECHO prompt>> %targetpath%%targetfile%.ftp
ECHO put "%%~F"
ECHO close>> %targetpath%%targetfile%.ftp
) >> %targetpath%%targetfile%.ftp
ECHO bye>> %targetpath%%targetfile%.ftp
----------------------------
#10 10 Mar 2010 17:49
GrandPixel
I am trying to create a separate ftp response file for each file to be uploaded. But for whatever reason, it is putting everything into "Backup.[***].part0.ftp
Code: Select all
SET /a counter = 0
for %%F in ("%targetpath%%targetfile%*rar") do (
SET /a counter += 1
ECHO open ftp.[***].net>> %targetpath%%targetfile%.part%counter%.ftp
ECHO [***username***]>> %targetpath%%targetfile%.part%counter%.ftp
ECHO [***password***]>> %targetpath%%targetfile%.part%counter%.ftp
ECHO binary>> %targetpath%%targetfile%.part%counter%.ftp
ECHO prompt>> %targetpath%%targetfile%.part%counter%.ftp
ECHO put "%%~F">> %targetpath%%targetfile%.part%counter%.ftp
ECHO bye>> %targetpath%%targetfile%.part%counter%.ftp
)
#11 10 Mar 2010 18:26
GrandPixel
This is the output at the command prompt:
Code: Select all
X:\[***]>SET /a counter = 0
X:\[***]>for %F in ("[X:\[***]*rar") do (
SET /a counter += 1
ECHO open ftp.[***].net 1>>X:\[***].part0.ftp
ECHO [***username***] 1>>X:\[***].part0.ftp
ECHO [***password***] 1>>X:\[***].part0.ftp
ECHO binary 1>>X:\[***].part0.ftp
ECHO prompt 1>>X:\[***].part0.ftp
ECHO put "%~F" 1>>X:\[***].part0.ftp
ECHO bye 1>>X:\[***].part0.ftp
)
X:\[***]>(
SET /a counter += 1
ECHO open ftp.[***].net 1>>X:\[***].part0.ftp
ECHO [***username***] 1>>X:\[***].part0.ftp
ECHO [***password***] 1>>X:\[***].part0.ftp
ECHO binary 1>>X:\[***].part0.ftp
ECHO prompt 1>>X:\[***].part0.ftp
ECHO put "X:\[***].part1.rar" 1>>X:\[***].part0.ftp
ECHO bye 1>>X:\[***].part0.ftp
)
X:\[***]>(
SET /a counter += 1
ECHO open ftp.[***].net 1>>X:\[***].part0.ftp
ECHO [***username***] 1>>X:\[***].part0.ftp
ECHO [***password***] 1>>X:\[***].part0.ftp
ECHO binary 1>>X:\[***].part0.ftp
ECHO prompt 1>>X:\[***].part0.ftp
ECHO put "X:\[***].part2.rar" 1>>X:\[***].part0.ftp
ECHO bye 1>>X:\[***].part0.ftp
)
X:\[***]>(
SET /a counter += 1
ECHO open ftp.[***].net 1>>X:\[***].part0.ftp
ECHO [***username***] 1>>X:\[***].part0.ftp
ECHO [***password***] 1>>X:\[***].part0.ftp
ECHO binary 1>>X:\[***].part0.ftp
ECHO prompt 1>>X:\[***].part0.ftp
ECHO put "X:\[***].part3.rar" 1>>X:\[***].part0.ftp
ECHO bye 1>>X:\[***].part0.ftp
)
X:\[***]>
Code: Select all
open ftp.[***].net
[***username***]
[***password***]
binary
prompt
put "X:\[***].part1.rar"
bye
open ftp.[***].net
[***username***]
[***password***]
binary
prompt
put "X:\[***].part2.rar"
bye
open ftp.[***].net
[***username***]
[***password***]
binary
prompt
put "X:\[***].part3.rar"
bye
#12 10 Mar 2010 18:41
bluesxman
You'd need "setlocal enabledelayedexpansion" for that. But your method is somewhat clunky.
I'd do it this way, myself:
Code: Select all
for %%F in ("%targetpath%%targetfile%*rar") do (
(
ECHO open ftp.[***].net
ECHO [***username***]
ECHO [***password***]
ECHO binary
ECHO prompt
ECHO put "%%~F"
ECHO bye
) >> "%%~dpnF.ftp"
)
Last edited by bluesxman (10 Mar 2010 18:44)
cmd | *sh | ruby | chef
----------------------------
#13 10 Mar 2010 18:56
GrandPixel
I thought I was using your suggestion smile but I thought to make it work I would have to create a separate .ftp response file for each file upload. I will input your latest suggestion and see what happens
----------------------------
#14 10 Mar 2010 18:58
GrandPixel
How do you do that!
----------------------------
#15 11 Mar 2010 10:00
bluesxman
I don't see why you would -- the script I proposed would create one long response file, recreating the connection before each upload.GrandPixel wrote:
I thought to make it work I would have to create a separate .ftp response file for each file upload
Here is my earlier method for creating a single FTP response file but corrected for the mistake I made:
Code: Select all
REM CREATE FTP RESPONSE FILE TO UPLOAD ARCHIVE
REM *****************************************************************************
for %%F in ("%targetpath%%targetfile%*rar") do (
ECHO open ftp.[***].net
ECHO [***username***]
ECHO [***password***]
ECHO binary
ECHO prompt
ECHO put "%%~F"
ECHO close
) >> %targetpath%%targetfile%.ftp
ECHO bye>> %targetpath%%targetfile%.ftp
cmd | *sh | ruby | chef
----------------------------
#16 11 Mar 2010 18:02
GrandPixel
I see what you were doing now. I may change it to use a single response file, but either way, I think I have a working solution. Thanks very much for your help.
----------------------------
#17 11 Mar 2010 20:23
bluesxman
You're welcome.