MSSFTP
Section: User Commands (1C)
Updated: 16 May 2008
NAME
mssftp - File transfer program for NCSA's mass storage system
SYNOPSIS
mssftp [options] [cmds]
DESCRIPTION
mssftp is a GridFTP-enabled client that supports both interactive use
and FTP commands on the command line to transfer files between NCSA's
mass storage system and the local system. It is based upon NCSA's
UberFTP client. mssftp supports GSI authentication, parallel data
channels and striping. For more information about GridFTP, see the
GridFTP URL in the "SEE ALSO" section below.
mssftp connects only to NCSA's mass storage system (mss.ncsa.uiuc.edu).
User authentication is done without requiring a password, so you can
access the mass storage system from a batch job without putting the
password into a .netrc file.
Upon execution, mssftp immediately attempts to establish a connection
to the FTP server on mss.ncsa.uiuc.edu. If no commands are given on
the command, the client will enter its command interpreter and await
instructions from the user. If commands are given on the commandline,
the client will execute these commands in order and exit upon the first
error encountered or upon successful completion of all given commands.
This list must be enclosed in quotes. Multiple commands are semicolon
or comma delimited.
OPTIONS
-active
Use ACTIVE mode for data transfers.
-ascii Use ASCII mode for data transfers.
-binary
Use BINARY mode for data transfers.
-blksize n
Set the internal buffer size to n.
-cksum [on|off]
Enable/Disable CRC checks after file transfers.
-d Enable debugging. Same as '-debug 3'. Deprecated.
-debug n
Set the debug level to n.
-family name
Set the remote storage family to name.
-g Disable filename globbing. Same as '-glob off'. Deprecated.
-mode [E|S]
Switch the transfer mode to extended block (E) or streams mode
(S).
-parallel n
Use n parallel data channels during extended block transfers.
-passive
Use PASSIVE mode for data transfers.
-pbsz n
Set the data protection buffer size to n n bytes.
-prot [C|S|E|P]
Set the data protection lelvel to clear (C), safe (S), confiden-
tial (E) or private (P).
-retry n
Retry commands that fail with transient errors n times.
-resume path
Retry the recursive transfer starting at path.
-tcpbuf n
Set the TCP read/write buffers to n bytes.
-wait This will cause the client to wait for remote files to stage
before attempting to transfer them.
-v Enable verbose mode. Same as '-debug 3'. Deprecated.
-version
Print mssftp version information and exit.
-versions
Print version information about all used globus modules and
exit.
DEFAULT TRANSFER MODE
By default, without any special environment variables, command line
options or commands, mssftp will transfer files in ACTIVE STREAMS mode.
ACTIVE means that the server will initiate the data connection which
increases performance to NCSA's mass storage system. STREAMS mode
implies that GRIDFTP features including striping and parallel data con-
nections are not used. In order to take advantage of these features
with GridFTP capable servers, you must either change the mode directly
using -m command line switch or the mode interactive command, or you
can change the mode indirectly by specifying more than one parallel
data connection using the -c command line switch or by using the paral-
lel interactive command.
ascii Change the data transfer type to ASCII which causes the server
to do some simple transformations to the file being transferred.
This is mostly useful for changing EOL (end of line) in text
files when moving between platforms. This option is almost
never necessary today. The default is BINARY mode also known as
IMAGE mode.
binary Change the data transfer type to BINARY (aka IMAGE) which causes
the server to not perform transformations to the file being
transferred. This is the default and is faster than an ASCII
transfer.
blksize size
Change the size of the memory buffer used to read and write data
to disks to size bytes. The default block size is 1024*1024
(1048576) bytes but it can be changed at compile time. The block
size can be increased to improve file transfer performance. This
is not related to the extended block mode block size used to
determine the ratio of data to header for data transferred on
the data channel.
bugs Prints information regarding bug reporting and feature requests.
bye Close all control and data connections and exit.
cat file1 [file2 ... filen]
Print the contents of the remote file(s) to stdout.
cd [dir]
Change the remote working directory to dir. If dir is not given,
the client will make every attempt to change to the user's home
directory.
cdup Change the remote working directory up one level.
chgrp [-r] group object [object2 ... objectn]
Change group ownership on the remote object(s).
-r Recursively chgrp everything in the given directory.
chmod [-r] perms object [object2 ... objectn]
Change permissions on the remote object(s).
-r Recursively chmod everything in the given directory.
cksum [on|off]
Enable file cksum comparison after each file transfer. This only
works with NCSA's mass storage system.
on Enable checksum comparison
off Disable checksum comparison
close Close the control connection to the remote host.
3 Print all information
delete [-r] object1 [object1...objectn]
Alias for rm. This command has been deprecated.
-r Recursively remove the given directory.
dir [-r] [target]
List the contents of the remote target directory. If target is
not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.
disconnect
Alias for close. This command has been deprecated.
family name
Sets the tape family to name on the FTP service if the service
supports it. If name is omitted, the current family is printed.
get [-r] source [destination]
Retreive file(s) from the remote service. If source implies mul-
tiple transfers, either through regular expressions or by using
the recursive feature, then destination must be a directory. If
destination is not specified, source is used.
-r Recursively transfer the given directory.
glob [on|off]
Enable or disable filename globbing. If no option is given, this
command will toggle the current setting.
on Enable filename globbing
off Disable filename globbing
hash Print hash marks during data transfers. This does not work dur-
ing third party transfers.
help [command]
If command is given, print a helpful blurb about command. Oth-
erwise, list all commands.
keepalive [seconds]
Attempts to keep the control channel from being blocked by fire-
walls during long data channel operations. mssftp sends a NOOP
command to the service at intervals equal to the specified num-
ber of seconds. Setting it to zero will disable keepalive. If
seconds are not given, the current timeout is displayed. This
feature is disabled by default.
seconds number of seconds between NOOPs. Disabled if zero.
lcat file1 [file2 ... filen]
Print the contents of the local file(s) to stdout.
lcd [dir]
ldir [-r] [target]
List the contents of the local target directory. If target is
not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.
lls [-r] [target]
List the contents of the local target directory. If target is
not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.
lmkdir dir1 [dir2 ... dirn]
Create the local directory(ies).
lpwd Prints the current local working directory.
lquote cmd
Pass cmd to the local FTP service. This allows the user to use
server-specific commands that are not available through the mss-
ftp interface.
lrename src dst
Rename the local object src to dst.
lrm [-r] object1 [object1...objectn]
Removes the local file system object(s).
-r Recursively remove the given directory.
lrmdir dir1 [dir2...dirn]
Removes the given directories from the local service.
ls [-r] [target]
List the contents of the remote target directory. If [target] is
not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.
lsize file1 [file2...filen]
Prints the size of the given object(s).
lstage [-r] seconds object1 [object2...objectn]
Attempt to stage all matching files within the given number of
seconds on the local service.
seconds number of seconds to attempt staging
-r Recursively stage all files in the given subdirectory.
mput [-r] object1 [object2...objectn]
Retrieve file(s) from the remote service. This is similiar to
making multiple calls to get without specifying a destination.
-r Recursively transfer the given directory.
mput [-r] object1 [object2...objectn]
Store file(s) to the remote service. This is similiar to making
multiple calls to put without specifying a destination.
-r Recursively transfer the given directory.
open Opens a control channel to mss.ncsa.uiuc.edu and that host
becomes the 'remote' machine. GSI authentication is used.
order [type]
Changes the order of lists returned from ls and lls to the given
scheme. If type is not given, the current order is displayed.
type Ordering scheme to use. Value options are:
none Do not order listings
name Order listings by name
size Order listings by size
type Order listings by type
parallel [number]
Set the number of parallel data connections to number. This is
only useful for extended block mode transfers. The default num-
ber of data connections is one. If no number is given, the cur-
rent setting for the number of parallel connects is printed.
passive
Change to PASSIVE mode which causes the client to initiate the
data connection. The default is ACTIVE mode. If you are behind
a firewall you must use PASSIVE mode.
pbsz [size]
Change the length of the protection buffer. The protection
buffer is used to encrypt data on the data channel. The length
of the protection buffer represents the largest encoded message
that is allowed on the data channel. By default, the protection
buffer is grown to match the internal buffer used. For efficient
transfers, pbsz should be sufficiently larger than blksize so
that the wrapped buffer fits within the protection buffer. Oth-
erwise, the blksize buffer is broken into multiple pieces so
that each write is less than pbsz when wrapped. If pbsz is not
given, the current size is displayed.
size length of protection buffer. 0 will set it to its
default.
pget offset size srcfile [destfile]
Retrieve only the specified portion of the file(s). If srcfile
is a regular expression and expands to multiple files, and des-
tination is given, destination must refer to a directory.
offset Offset within the file
size Amount of data to retrieve
srcfile Name of remote file
destfile Name of local file. srcfile is used if destfile
is not specified
will be integrity protected meaning that altered data will be
detected. Confidential means that the data will be unreadable to
third parties. Private mode means the data will be confidential
and safe.
C Set protection level to clear.
S Set protection level to safe.
E Set protection level to confidential.
P Set protection level to private.
get [-r] source [destination]
Store file(s) to the remote service. If source implies multiple
transfers, either through regular expressions or by using the
recursive feature, then destination must be a directory. If des-
tination is not specified, source is used.
-r Recursively transfer the given directory.
pwd Prints the current working directory.
quit Close all control and data connections and exit.
quote cmd
Pass cmd to the remote FTP service. This allows the user to use
server-specific commands that are not available through the mss-
ftp interface.
rename src dst
Rename the remote object src to dst.
retry [cnt]
Configures retry on failed commands that have transient errors.
cnt represents the number of times a failed command is retried.
A value of zero effectively disables retry. Zero is the default.
If no value is given the current setting is displayed.
cnt Number of times a failed command is retried.
resume [-d] path
Sets a restart point for recursive transfers. If a long recur-
sive transfer fails, you can set resume to the path that failed
and mssftp will skip all file and directory creations up to the
given path.
path Path to resume transfer at. If path is not given, print
the current
resume target.
-d Remove the current resume path.
rm [-r] object1 [object1...objectn]
Removes the remote file system object(s).
-r Recursively remove the given directory.
rmdir dir1 [dir2...dirn]
Removes the given directories from the remote service.
Toggles the client to store files using unique names during get
operations.
tcpbuf [size]
Set the data channel TCP buffer size to size bytes. If size is
not given, the current TCP buffer size will be printed.
versions
Prints the versions of all Globus modules being used.
wait Toggles whether the client should wait for files to stage before
attempting to retrieve them.
IMPROVING FILE TRANSFER PERFORMANCE
When tranferring files over long distances, use a large value (for
example, 16777216) for tcpbuf. When there is high network traffic, you
may be able to improve performance using the parallel command to
increase the number of parallel data connections to 2-4.
THIRD-PARTY TRANSFERS
All remote service commands have "l*" counterparts that allow you to
specifiy that the command is to be performed on the local host.
EXIT VALUES
mssftp will exit with a value of 0 if no errors occurred during the
session, otherwise it will exit with a value of 1. In non interactive,
commandline mode, it will exit after the first error occurs.
EXAMPLES
Connect to NCSA's mass storage system interactively from NCSA's Tera-
Grid cluster:
% mssftp
...
220 UNIX Archive FTP server ready.
230 User consult logged in.
ftp>
Use the command-line interface to copy a file from NCSA's TeraGrid
cluster to the mass storage system:
% mssftp "cd work; get file.tar"
SEE ALSO
uberftp(1), msscmd(1), ftp(1)
GridFTP:
http://www.globus.org/datagrid/gridftp.html
TCP Window Size:
http://dast.nlanr.net/Articles/GettingStarted/TCP_window_size.html
Active vs. Passive FTP:
http://slacksite.com/other/ftp.html
Note: The links above are not under NCSA's control so they may become obsolete.
|
MSSCMD
Section: User Commands (1)
Updated: August 1999
NAME
msscmd - command line interface to mssftp
SYNOPSIS
msscmd [-h] [-v] [-c] [-s] ftp-cmd,ftp-cmd,...
or
msscmd [-h] [-v] [-c] [-s] -f cmd-file
DESCRIPTION
The msscmd command provides an interface to mssftp(1) allowing FTP
directives to be specified as command line arguments or read from a
file.
msscmd also adds additional functionality such as retry, asynchronous
file transfer after batch job completion, and mapping directives to
UniTree extended features.
FTP DIRECTIVES
If the FTP directives are specified on the command line, by default,
they must be separated by commas. For example:
msscmd "mkdir J, cd J, put y"
Another non-alphanumeric character may be used if it is placed before
the first directive as well as between directives. For example:
msscmd "+ mkdir J + cd J + put y,v"
The FTP directives can be stored in a file, one directive per line, and
the file specified with the -f option:
msscmd -f file
UNITREE EXTENSIONS:
To ease use of the UniTree extended features, directives beginning with
chgrp, chmod, chown, gtrsh, ln, mstage, nmdup, setfam, setpurgei,
stage, strsh, sum, swap, umask, or wait are automatically converted to
the appropriate form involving quote or quote site.
MSSCMDRC
If the file $HOME/.msscmdrc exists, the directives contained in that
file are supplied to mssftp preceding the directives on the command
line. Typically, these directives will be ones such as umask that
alter the behavior of subsequent directives rather than having a direct
effect of their own. To specify the use of another file in place of
$HOME/.msscmdrc, place its name in the environment variable MSSCMDRC.
RETRY
If mssftp fails to execute all of the FTP directives, msscmd will wait
five minutes and retry the directives. It will continue to retry until
-v Runs mssftp in verbose mode.
-c Computes checksums for the local and remote files.
-f Read the FTP directives from a file instead of on the command
line. If the file is a minus sign (-), the commands are read from
stdin. For example:
msscmd -f - << EOF
cd backupdir
get y
EOF
-s Suppresses the reading of the MSSCMDRC file.
USING TAR
msscmd has a built-in tar command. The syntax is:
tar {c|x|t}[vf] <remote-tarfile> <local-files>
The 'c' option runs tar on the local-files and puts the result into the
remote-file.
The 'x' option runs tar on the remote-tarfile and extracts files onto
the local filesystem.
The 't' option is like 'x' except that instead of extracting the files,
it simply lists the contents of the remote-tarfile.
If the name of the remote-tarfile ends with ".gz", then the tar output
will be piped through gzip ('c' option) or gunzip ('x' and 't'
options).
If the name of the remote-tarfile end with ".Z", then the tar output
will be piped through compress ('c' option) or uncompress ('x' and 't'
options).
Examples using built-in tar command
The following command creates a tar file of the current directory and
stores it in UniTree:
msscmd "tar cvf dir.tar ."
SHELL COMMANDS WITH GET AND PUT
The general form of the mssftp put and get is:
put local-file remote-file
get remote-file local-file
It is possible to use a shell command in place of "local-file":
put "|command" remote-file
get remote-file "|command"
The double quotes (") are required by mssftp. However, in order to get
these directives to mssftp using msscmd, the double quotes (and the
pipe symbol) need to be enclosed in single quotes('):
msscmd put '"|command"' remote-file
msscmd get remote-file '"|command"'
Examples
For example, to create a tarfile of the current directory (the hard way
- see USING TAR)
msscmd put '"|tar cf - . "' dir.tar
To extract the contents of "dir.tar" into the current directory:
msscmd get dir.tar '"|tar xf -"'
This syntax will also work if a list of files, using shell wildcards,
is specified:
msscmd put '"|tar cf - *.dat"' dir.tar
The shell wildcards are protected inside single or double qoutes. How-
ever, shell variables are not interpreted within single quotes. The
following will not work:
cd $SCR
set run=test5
msscmd put '"|tar cf - $run.*"' $run.tar
The problem is that $run is enclosed within the single quotes and will
not be expanded. One solution is to place the $run outside the single
quotes:
msscmd put '"|tar cf - '$run'.*"' $run.tar
Another is to use -f:
put |tar cf - test5.* test5.tar
Msscmd will notice that the pipe(|) is not preceeded by the double
quote("). It will add the double quotes before it sends the directive
to mssftp:
put "|tar cf - test5.*" test5.tar
But the simplest solution is to use the the tar directive:
msscmd "tar cf $run.tar $run.*"
WARNINGS
Shell Quotes
As is apparent from the previous section, it may be necessary to use
quotes in order to use special characters in directives.
In particular, it is a good idea to quote "wildcards" so they will be
interpreted by mssftp rather than the shell. If this is not done, the
wildcard expansion may result in a directive that is too long for mss-
ftp to handle, or expanded relative to the wrong filesystem or direc-
tory.
The quotes are necessary for mget, since the files to be matched exist
in unitree, not in the local file system.
msscmd mget *.z
would not get the correct files unless all the *.z files on the local
filesystem were the same as the *.z files in unitree. Quotes must be
used, as follows:
msscmd "mget *.z"
A subtle problem exists in the use of the command separator and wild-
cards. The following example will not expand properly:
msscmd mput *.z, put y
The problem is the placement of the comma. The expansion of *.z, which
includes the comma, may not match any files and msscmd would actually
see:
msscmd mput put y
The fix is to either add a space before the comma:
msscmd mput *.z , put y
set x = dir
msscmd get ${x}/file1
msscmd get "${x}/file2"
msscmd get {$x}/file3
msscmd get "{$x}/file4"
These four msscmd's will be
msscmd get dir/file1
msscmd get dir/file2
msscmd get dir/file3
msscmd get {dir}/file4
With the last msscmd, the placement of the { before the $ within double
quotes causes the csh to keep the {}. This problem does not occur if
the {$x} is replaced by ${x}.
In general it is recommended to use the form ${var} instead of {$var}.
Verbose Mode
Execution of mssftp begins in its nonverbose mode. Some UniTree direc-
tives provide information that is displayed only in the verbose mode,
so these directives may appear to produce no response unless the ver-
bose mode is explicitly requested. The verbose mode can only be
requested with the -v option. The "verbose" FTP directive is ignored.
Retry
If a communications error occurs in the middle of executing the
sequence of directives, or if there are get or put failures, msscmd
will retry the entire sequence, so care should be taken in the use of
directives that are not safely repeatable (for example, deleting a file
and then renaming a different file to have that name).
SEE ALSO
mssftp(1)
NCSA August 1999 MSSCMD(1)
|
UBERFTP
Section: User Commands (1C)
Updated: 16 May 2008
NAME
uberftp - GridFTP-enabled client to transfer files
SYNOPSIS
uberftp [options] [host options] [host]
uberftp [options] [host options] host "cmd"
uberftp [options] srcurl dsturl
uberftp [options] -f urlfile
uberftp [options] -cmd url
DESCRIPTION
uberftp is a GridFTP-enabled client that supports both interactive use
and FTP commands on the uberftp command line to transfer files between
two computers. It is intended for use with computers that have a
GridFTP server installed. Uberftp supports GSI authentication, parallel
data channels and striping. For more information about GridFTP, see the
GridFTP URL in the "SEE ALSO" section below.
Only the first usage shown above will create an interactive session. If
host is specified, uberftp immediately attempts to establish a connec-
tion to the GridFTP server on host, enters its command interpreter and
awaits commands from the user. If host is not specified, uberftp imme-
diately drops into the command interpreter without connecting to any
GridFTP server.
The second usage option denotes a non interactive session in which
"cmd" is a series of one or more commands as described in the COMMANDS
section below to run once the control connection is established. These
commands are run exactly as if they had been entered from the interac-
tive prompt. This list must be enclosed in quotes. Multiple commands
are semicolon or comma delimited. uberftp will execute these commands
and then exit. Uberftp will exit upon the first error encountered.
The third and forth usage statements use the URL style format for spec-
ifying the source and destination for the files to transfer. The third
usage statement places these URLs on the command line. The forth usage
allows the user to specify multiple URL pairs in a separate file for
Uberftp to transfer one at a time. The supported URL syntaxes are
gsiftp://[user@]host[:port]/file, ftp://[user[:pass]@]host[:port]/file
and file://path.
The fifth usage statement allows for commands that take pathnames to
accept URLs instead. The allowable commands are listed in the -cmds
section below.
HOST OPTIONS
-P port
Connect to port instead of the default. The default for GSI
authentication is 2811. The default for password authentication
Use ACTIVE mode for data transfers.
-ascii Use ASCII mode for data transfers.
-binary
Use BINARY mode for data transfers.
-blksize n
Set the internal buffer size to n.
-cksum [on|off]
Enable/Disable CRC checks after file transfers.
-debug n
Set the debug level to n.
-family name
Set the remote storage family to name.
-glob [on|off]
Enable/Disable filename expansion.
-hash Enable printing of hash marks during transfers.
-keepalive n
Send control channel keepalive messages every n seconds during
data transfers.
-mlsx [on|off]
Enable/Disable MLSD usage. This feature is used for client-side
globbing and error checking. This is enabled by default.
-mode [E|S]
Switch the transfer mode to extended block (E) or streams mode
(S).
-parallel n
Use n parallel data channels during extended block transfers.
-passive
Use PASSIVE mode for data transfers.
-pbsz n
Set the data protection buffer size to n n bytes.
-prot [C|S|E|P]
Set the data protection lelvel to clear (C), safe (S), confiden-
tial (E) or private (P).
-retry n
Retry commands that fail with transient errors n times.
-versions
Print version information about all used globus modules and
exit.
Supported -cmds
-cat url
Print to stdout the contents of the remote file.
-chgrp [-r] group url
Set the group ownership of the remote object(s).
-chmod [-r] perms url
Set the permissions of the remote object(s).
-dir [-r] url
List the contents of the remote object.
-ls [-r] url
List the contents of the remote object.
-mkdir url
Create the remote directory.
-rename url path
Rename the remote object to the given path.
-rm [-r] url
Remove the remote object(s).
-rmdir url
Remove the remote directory.
-size url
Return the size of the remote object.
-stage [-r] seconds url
Attempt to stage the remote object(s) over the time period given
in seconds.
DEFAULT TRANSFER MODE
By default, without any special environment variables, command line
options or commands, uberftp will transfer files in PASSIVE STREAMS
mode. PASSIVE means that the client will initiate the data connection
which is useful for users behind firewalls. STREAMS mode implies that
GRIDFTP features including striping and parallel data connections are
not used. In order to take advantage of these features with GridFTP
capable servers, you must either change the mode directly using -m com-
mand line switch or the mode interactive command, or you can change the
mode indirectly by specifying more than one parallel data connection
using the -c command line switch or by using the parallel interactive
! [command]
Run the command using a shell on the local machine. If no com-
mand is given, invoke an interactive shell.
? [command]
If command is given, print a (hopefully) helpful blurb about it.
Otherwise, list all commands.
active Change to ACTIVE mode which causes the server to initiate the
data connection. The default is PASSIVE mode unless the variable
UBERFTP_ACTIVE_MODE is set in the environment. If you are behind
a firewall you must use PASSIVE mode.
ascii Change the data transfer type to ASCII which causes the server
to do some simple transformations to the file being transferred.
This is mostly useful for changing EOL (end of line) in text
files when moving between platforms. This option is almost
never necessary today. The default is BINARY mode also known as
IMAGE mode.
binary Change the data transfer type to BINARY (aka IMAGE) which causes
the server to not perform transformations to the file being
transferred. This is the default and is faster than an ASCII
transfer.
blksize size
Change the size of the memory buffer used to read and write data
to disks to size bytes. The default block size is 1024*1024
(1048576) bytes but it can be changed at compile time. The block
size can be increased to improve file transfer performance. This
is not related to the extended block mode block size used to
determine the ratio of data to header for data transferred on
the data channel.
bugs Prints information regarding bug reporting and feature requests.
bye Close all control and data connections and exit.
cat file1 [file2 ... filen]
Print the contents of the remote file(s) to stdout.
cdup Change the remote working directory up one level.
cd [dir]
Change the remote working directory to dir. If dir is not given,
the client will make every attempt to change to the user's home
directory.
chgrp [-r] group object [object2 ... objectn]
Change group ownership on the remote object(s).
-r Recursively chgrp everything in the given directory.
dcau [N|A|S subject]
Change the data channel authentication settings. If the service
does not support DCAU, these settings are ignored.
N Disabled dcau.
A Expect the remote identity to be mine. (Default)
S subject Expect the remote identity to be subject.
debug [0-3]
Turn debug statements on/off. If no value is given, this command
will toggle between debug(2) and non debug(1) mode. Otherwise
the debug level is set to the given level.
0 Only errors are printed
1 Default. Errors and some helpful messages are printed
2 Print useful control channel information
3 Print all information
family name
Sets the tape family to name on the FTP service if the service
supports it. If name is omitted, the current family is printed.
glob [on|off]
Enable or disable filename globbing. If no option is given, this
command will toggle the current setting.
on Enable filename globbing
off Disable filename globbing
dir [-r] [target]
List the contents of the remote target directory. If target is
not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.
get [-r] source [destination]
Retreive file(s) from the remote service. If source implies mul-
tiple transfers, either through regular expressions or by using
the recursive feature, then destination must be a directory. If
destination is not specified, source is used.
-r Recursively transfer the given directory.
hash Print hash marks during data transfers. This does not work dur-
ing third party transfers.
help [command]
If command is given, print a helpful blurb about command. Oth-
erwise, list all commands.
keepalive [seconds]
Attempts to keep the control channel from being blocked by fire-
walls during long data channel operations. UberFTP sends a NOOP
command to the service at intervals equal to the specified num-
ber of seconds. Setting it to zero will disable keepalive. If
seconds are not given, the current timeout is displayed. This
Change group ownership on the local object(s).
-r Recursively chgrp everything in the given directory.
lchmod [-r] perms object [object2 ... objectn]
Change permissions on the local object(s).
-r Recursively chmod everything in the given directory.
lclose Close the control connection to the local host.
ldir [-r] [target]
List the contents of the local target directory. If target is
not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.
lls [-r] [target]
List the contents of the local target directory. If target is
not given, then the current working directory is used.
-r Recursively list target.
target Directory or file to list. '.' is used by default.
lmkdir dir1 [dir2 ... dirn]
Create the local directory(ies).
lopen [-P port] [-u user] [-p pass | X] host
Opens a control channel to host and that host becomes the
'local' machine. After using lopen, all local (l*) commands
perform their respective operations on host rather than the
local machine. This is how third party transfers are accom-
plished. GSI authentication is used unless the -p option is
used.
-P port Connect to port (Default 2811 for GSI, 21 for pass-
word).
-u user Connect as alternate user.
-p pass | X
Use password pass when authenticating with host.
If pass equals X, read the password from STDIN with
character echoing turned off.
host Connect to host.
lpwd Prints the current local working directory.
lrename src dst
Rename the local object src to dst.
lrm [-r] object1 [object1...objectn]
Removes the local file system object(s).
-r Recursively remove the given directory.
lrmdir dir1 [dir2...dirn]
Removes the given directories from the local service.
lstage [-r] seconds object1 [object2...objectn]
Attempt to stage all matching files within the given number of
seconds on the local service.
seconds number of seconds to attempt staging
-r Recursively stage all files in the given subdirectory.
mput [-r] object1 [object2...objectn]
Retrieve file(s) from the remote service. This is similiar to
making multiple calls to get without specifying a destination.
-r Recursively transfer the given directory.
mkdir dir
Create the remote directory.
mlsx [on|off]
Enable/Disable MLSD usage. This feature is used for client-side
globbing and error checking. This is enabled by default.
mode [E|S]
Toggle the data transfer mode between Streams mode and Extended
Block mode. The default is Streams mode. If no option is given,
it will display the current mode.
E Extended block mode
S Streams mode
mput [-r] object1 [object2...objectn]
Store file(s) to the remote service. This is similiar to making
multiple calls to put without specifying a destination.
-r Recursively transfer the given directory.
open [-P port] [-u user] [-p pass | X] host
Opens a control channel to host and that host becomes the
'remote' machine. GSI authentication is used unless the -p
option is used.
-P port Connect to port (Default 2811 for GSI, 21 for pass-
word).
-u user Connect as user.
-p pass | X
Use password pass when authenticating with host.
If pass equals X, read the password from STDIN with
character echoing turned off.
host Connect to host.
order [type]
Changes the order of lists returned from ls and lls to the given
scheme. If type is not given, the current order is displayed.
type Ordering scheme to use. Value options are:
none Do not order listings
name Order listings by name
size Order listings by size
type Order listings by type
Change the length of the protection buffer. The protection
buffer is used to encrypt data on the data channel. The length
of the protection buffer represents the largest encoded message
that is allowed on the data channel. By default, the protection
buffer is grown to match the internal buffer used. For efficient
transfers, pbsz should be sufficiently larger than blksize so
that the wrapped buffer fits within the protection buffer. Oth-
erwise, the blksize buffer is broken into multiple pieces so
that each write is less than pbsz when wrapped. If pbsz is not
given, the current size is displayed.
size length of protection buffer. 0 will set it to its
default.
pget offset size srcfile [destfile]
Retrieve only the specified portion of the file(s). If srcfile
is a regular expression and expands to multiple files, and
destination is given, destination must refer to a directory.
offset Offset within the file
size Amount of data to retrieve
srcfile Name of remote file
destfile Name of local file. srcfile is used if destfile
is not specified
pput offset size srcfile [destfile]
Store only the specified portion of the file(s). If srcfile is a
regular expression and expands to multiple files, and destina-
tion is given, destination must refer to a directory.
offset Offset within the file
size Amount of data to retrieve
srcfile Name of local file
destfile Name of remote file. srcfile is used if destfile
is not specified
prot [C|S|E|P]
This command configures the level of security on the data chan-
nel after data channel authentication has completed. Clear means
that the data will not be protected. Safe means that the data
will be integrity protected meaning that altered data will be
detected. Confidential means that the data will be unreadable to
third parties. Private mode means the data will be confidential
and safe.
C Set protection level to clear.
S Set protection level to safe.
E Set protection level to confidential.
P Set protection level to private.
get [-r] source [destination]
Store file(s) to the remote service. If source implies multiple
transfers, either through regular expressions or by using the
recursive feature, then destination must be a directory. If des-
tination is not specified, source is used.
-r Recursively transfer the given directory.
Configures retry on failed commands that have transient errors.
cnt represents the number of times a failed command is retried.
A value of zero effectively disables retry. Zero is the default.
If no value is given the current setting is displayed.
cnt Number of times a failed command is retried.
resume [-d] path
Sets a restart point for recursive transfers. If a long recur-
sive transfer fails, you can set resume to the path that failed
and UberFTP will skip all file and directory creations up to the
given path.
path Path to resume transfer at. If path is not given, print
the current
resume target.
-d Remove the current resume path.
rm [-r] object1 [object1...objectn]
Removes the remote file system object(s).
-r Recursively remove the given directory.
rmdir dir1 [dir2...dirn]
Removes the given directories from the remote service.
runique
Toggles the client to store files using unique names during put
operations.
size file1 [file2...filen]
Prints the size of the given object(s).
stage [-r] seconds object1 [object2...objectn]
Attempt to stage all matching files within the given number of
seconds on the remote service.
seconds number of seconds to attempt staging
-r Recursively stage all files in the given subdirectory.
sunique
Toggles the client to store files using unique names during get
operations.
tcpbuf [size]
Set the data channel TCP buffer size to size bytes. If size is
not given, the current TCP buffer size will be printed.
versions
Prints the versions of all Globus modules being used.
wait Toggles whether the client should wait for files to stage before
attempting to retrieve them.
the former local machine. In UberFTP terminology, it is referred to as
"opening a new local service" since, from the perspective of the user,
the new local service will appear as though the user initiated the FTP
session from that machine.
All remote service commands have "l*" counterparts that allow you to
specifiy that the command is to be performed on the local service,
whether that service is the local machine or a new local service. So
to open a new local service, use the "l*" version of the open command:
UberFTP> lopen mss.ncsa.teragrid.org
UberFTP> lclose
Once you have connected to both services, files can be transferred as
before with the change that files you "get" go to the new local service
and files you "put" are sent from the new local service.
EXIT VALUES
UberFTP will exit with a value of 0 if no errors occurred during the
session, otherwise it will exit with a value of 1. In non interactive,
commandline mode, it will exit after the first error occurs.
EXAMPLES
Set the environment variable to set active mode FTP (improves file
transfer performance to the mass storage system). Connect to NCSA's
UniTree mass storage system interactively from NCSA's TeraGrid cluster:
setenv UBERFTP_ACTIVE_MODE on
% uberftp mss.ncsa.teragrid.org
...
220 UNIX Archive FTP server ready.
230 User consult logged in.
UberFTP>
Use the command-line interface to copy a file from NCSA's TeraGrid
cluster to the UniTree mass storage system. (There is no need to set
tcpbuf since it is over a LAN but active mode is turned on to improve
file transfer performance to the mass storage system.):
uberftp mss.ncsa.teragrid.org \
"active; cd work; get file.tar"
Copy a file from SDSC's TeraGrid cluster to NCSA's TeraGrid cluster.
(Note that tcpbuf is set to 16777216 since there is a long network
latency between NCSA and SDSC):
uberftp tg-gridftp.sdsc.teragrid.org \
"tcpbuf 16777216; cd scr; put file.tar"
SEE ALSO
mssftp(1), msscmd(1), ftp(1), ls(1), ftpd(8)
Uberftp User Guide:
"http://teragrid.ncsa.uiuc.edu/Doc/Data/uberftp.html
GridFTP:
http://www.globus.org/datagrid/gridftp.html
TCP Window Size:
http://dast.nlanr.net/Articles/GettingStarted/TCP_window_size.html
Active vs. Passive FTP:
http://slacksite.com/other/ftp.html
Note: Some of the links above are not under NCSA's control so they may become obsolete.
|
tgcp
Section: User Commands (1)
Updated: 03 December 2004
Index
NAME
tgcp - teragrid copy (remote file copy program for the teragrid)
SYNOPSIS
tgcp
[options] <sourceURL> <destURL>
<sourceURL> and <destURL> can be of the form:
<file> or
<host>:/<file> or
file:///<file> or
<protocol>://<host>:<port>/<file>
<protocol> can be 'gsiftp', 'http' or 'https'
<sourceURL> may contain wildcard characters ? and [ ] character
ranges in the filename only.
Any url specifying a directory must end with a forward slash '/'.
If <sourceURL> is a directory, all files within that directory will be copied.
<destURL> must be a directory if multiple files are being copied.
DESCRIPTION
tgcp
copies files/directories between hosts on the TeraGrid. It uses
GridFTP for data transfer. It uses globus-url-copy (GUC) or Reliable File
Transfer (RFT) service (both are GridFTP clients) to perform the copy.
The default is globus-url-copy but RFT can be forced with -rft option.
RFT supports only third-party transfers and not client/server transfers.
The users must have a valid proxy certificate to run tgcp. A proxy
certificate can be generated using grid-proxy-init.
OPTIONS
Informational options
- -help -usage
-
Print help
- -version
-
Print the version of this program
- -guc-options
-
Print globus-url-copy options
- -guc-version
-
Print the version of globus-url-copy
- -guc-versions
-
Print the versions of all modules that globus-url-copy uses
- -rft-options
-
Print RFT options
Performance options
- -big
-
Recommended when the file being transferred is large. Use
GridFTP striped server (if available) for the transfer
- -bs <block size> -block-size <block size>
-
Specify the size (in bytes) of the buffer to be used by the
underlying transfer methods
- -tcp-bs <size> -tcp-buffer-size <size>
-
Specify the size (in bytes) of the buffer to be used by the
underlying ftp data channels
- -p <parallelism> -parallel <parallelism>
-
Specify the number of parallel data connections should be used.
Security related options
- -s <subject> -subject <subject>
-
If you run the GridFTP servers by hand, use this subject to match
with both the source and dest servers
- -ss <subject> -source-subject <subject>
-
If you run the source GridFTP server by hand, use this subject to
match with the source server
- -ds <subject> -dest-subject <subject>
-
If you run the destination GridFTP server by hand, use this subject
to match with the destination server
- -nodcau -no-data-channel-authentication
-
Turn off data channel authentication for ftp transfers
Reliability option
- -rft
-
Use RFT to do the transfer
URL translation options
- -nt -no-translate
-
Do not translate sourceURL and destURL. By default, source and
dest URL's are translated (if need be) to use the appropriate
GridFTP servers for the transfer
- -nts -no-translate-source
-
Do not translate sourceURL
- -ntd -no-translate-destination
-
Do not translate destURL
File type options
- -a -ascii
-
Convert the file to/from ASCII format to/from local file format
- -b -binary
-
Do not apply any conversion to the files. *default*
Workaround option
- -notpt -no-third-party-transfers
-
Turn third-party transfers off (on by default). This is useful
if a third party transfer between two gridftp servers can not be
done (for example, because of some firewall problem). Data goes
from server->client->server rather than server->server.
globus-url-copy (GUC) specific options
GUC informational options
- -q -quiet
-
Suppress all output for successful operation
- -v -verbose
-
Display urls being transferred
- -vb -verbose-perf
-
During the transfer, display the number of bytes transferred
and the transfer rate per second. Show urls being transferred
- -dbg -debugftp
-
Debug ftp connections. Prints control channel communication to stderr.
GUC performance options
- -stripe
-
Enable striped transfers on supported servers
- -fast
-
Recommended when using GridFTP servers. Use MODE E for all data transfers,
including reusing data channels between list and transfer operations.
GUC ease of use options
- -f <filename>
-
Read a list of url pairs from filename. Each line should contain <sourceURL> <destURL>.
Enclose URLs with spaces in double qoutes (").
Blank lines and lines beginning with # will be ignored. See EXAMPLES below.
- -rp -relative-paths
-
The path portion of ftp urls will be interpereted as relative to the user's starting
directory on the server. By default, all paths are root-relative. When this flag
is set, the path portion of the ftp url must start with %2F if it designates a
root-relative path.
- -r -recurse
-
Copy files in subdirectories
GUC partial file transfer options
- -off -partial-offset
-
Offset for partial ftp file transfers, defaults to 0
- -len -partial-length
-
Length for partial ftp file transfers, used only for the source
url, default is the full file.
GUC security related options
- -dcsafe -data-channel-safe
-
Set data channel protection mode to SAFE
- -dcpriv -data-channel-private
-
Set data channel protection mode to PRIVATE
GUC reliability options
- -c -continue-on-error
-
Do not die after any errors. By default, program will exit after most errors.
- -rst -restart
-
Restart failed ftp operations.
- -rst-interval <seconds>
-
The interval in seconds to wait after a failure before retrying the transfer.
Use 0 for an exponential backoff. Default is 0.
- -rst-timeout <seconds>
-
Maximum time after a failure to keep retrying. Use 0 for no timeout. Default is 0.
GUC experimental option
- -ipv6
-
use ipv6 when available (EXPERIMENTAL)
RFT specific options
RFT service information options
- -host <host>
-
Specify the hostname or ip address of the machine where a grid
service container that contains a RFT factory service is running
- -port <port>
-
Specify the port at which the grid service container is listening
RFT resource option
- -t <life time> -termination-time <life time>
-
Specify the lifetime of the resource created for this transfer in minutes.
If the transfer takes longer than this, it will be terminated. Default is 60 minutes
RFT performance option
- -con <concurrency> -concurrency <concurrency>
-
Specify the number of concurrent transfers to be used by RFT
RFT ease of use option
- -f <filename>
-
Read the transfer options and the urls from filename. The file
should contain the options/urls in the following order.
- transfer type (true=binary, false=ascii)
- block size in bytes
- TCP buffer size in bytes
- no thirdPartyTransfer (true/false)
- number of parallel streams
- data channel authentication (true/false)
- concurrency of the request
- subject to match the source gridftp server
- subject to match the destination gridftp server
- transfer all or none (true/false)
- maximum number of retries
- one or more source and destination url pair(s)
Each of these options/urls should be on a separate line. Blank
lines and lines beginning with # will be ignored. See EXAMPLES below.
RFT security option
- -z <authorization mechanism>
-
Specify the authorization mechanism used to authorize with the RFT service.
It can either 'host' or 'self'. Default is 'host'.
RFT miscellaneous option
- -aon -all-or-none
-
Either the entire file (or directory) gets transferred or none
EXAMPLES
A sample RFT transfer file
#true=binary false=ascii
true
#Block size in bytes
100000
#TCP Buffer size in bytes
256000
#Notpt (No thirdPartyTransfer)
false
#Number of parallel streams
8
#Data Channel Authentication (DCAU)
true
# Concurrency of the request
1
#If you run the source GridFTP server by hand, you will need to specify
#the subject to expect here. Otherwise specify "null" here.
null
#If you run the destination GridFTP server by hand, you will need to
#specify the subject to expect here. Otherwise specify "null" here.
null
#Transfer all or none of the transfers
false
#Maximum number of retries
1
#Source/Dest URL Pairs
gsiftp://tg-gridftprr.uc.teragrid.org:2811/disks/scratchgpfs1/kettimut/log1
gsiftp://tg-gridftp.sdsc.teragrid.org:2811/users/kettimut/
gsiftp://tg-gridftprr.uc.teragrid.org:2811/disks/scratchgpfs1/kettimut/log2
gsiftp://tg-gridftp.sdsc.teragrid.org:2811/users/kettimut/
A sample GUC URLs file
gsiftp://tg-gridftprr.uc.teragrid.org:2811/disks/scratchgpfs1/kettimut/log1 gsiftp://tg-gridftp.sdsc.teragrid.org:2811/users/kettimut/
gsiftp://tg-gridftprr.uc.teragrid.org:2811/disks/scratchgpfs1/kettimut/log2 gsiftp://tg-gridftp.sdsc.teragrid.org:2811/users/kettimut/
gsiftp://tg-gridftprr.uc.teragrid.org:2811/disks/scratchgpfs1/kettimut/log3 gsiftp://tg-gridftp.sdsc.teragrid.org:2811/users/kettimut/
~
Index
- NAME
-
- SYNOPSIS
-
- DESCRIPTION
-
- OPTIONS
-
- Informational options
-
- Performance options
-
- Security related options
-
- Reliability option
-
- URL translation options
-
- File type options
-
- Workaround option
-
- globus-url-copy (GUC) specific options
-
- GUC informational options
-
- GUC performance options
-
- GUC ease of use options
-
- GUC partial file transfer options
-
- GUC security related options
-
- GUC reliability options
-
- GUC experimental option
-
- RFT specific options
-
- RFT service information options
-
- RFT resource option
-
- RFT performance option
-
- RFT ease of use option
-
- RFT security option
-
- RFT miscellaneous option
-
- EXAMPLES
-
- A sample RFT transfer file
-
- A sample GUC URLs file
-
This document was created by
man2html,
using the manual pages.
Time: 20:47:59 GMT, June 17, 2005
|
|
