Selected Category Information

GridFTP Transfers

Transfers that use a GridFTP server installation at one or both endpoints.

Example Listing

MSS (AR) -> Co (guc)

Attempt to transfer an archived file via globus-url-copy

globus-url-copy returns an error if the file is archived.
Archive retrieval is initiated by the (failed) request for the file.
A workaround for this would be to loop until no error is returned, or use uberftp with 'quote wait'.
The "-rp" or relative path option allows the paths to be specified relative to $HOME on either system. %2F in the path forces root directory.

> globus-url-copy -nodcau -tcp-bs 65536 -rp gsiftp://mss.ncsa.teragrid.org/1GB -2 gsiftp://gridftp-co.ncsa.teragrid.org/%2F/scratch/users/jdoe/

error: globus_ftp_client: the server responded with an error
550 1GB-2: is being retrieved from the archive...

MSS (AR) -> Hg

Retrival of an archived file from MSS using uberftp with the "quote wait" command.

Note the use of the "quote wait" command. This instructs the client to wait for the file(s) to be staged from the tape archive and then carry out the transfer.

> uberftp mss.ncsa.teragrid.org "lopen gridftp-hg.ncsa.teragrid.org; lcd /gpfs _scratch1/dadams; active; mode stream; quote wait; get 1GB-2"

UNIX Archive FTP server (DiskXtender Version 2 .9) active. Checking DiskXtender.conf

220 ;UNIX Archive FTP server ready.
230 User&nbs p;dadams logged in.
220 tg-s037.ncsa.teragrid.org ;GridFTP Server 2.1 (gcc64dbg, 1122653280-63) ready .
230 User dadams logged in.
250 CWD& nbsp;command successful.
Active mode
Stream mod e
258 WAIT on
Transfer of 1073741824 bytes completed in 128.07 seconds. 8384.30 KB/ sec

guc large file

Third party transfer of a large file from NCSA to SDSC.

The stripe option is used here to utilize all of the servers at each end.
Large files benefit the most from striping.
A 10 MB tcp window size is used (tcp-bs) to keep the pipe full between SDSC and NCSA (~60ms RTT).

> globus-url-copy -vb -tcp-bs 10000000 -stripe gsiftp://gridftp-hg.ncsa.teragr id.org/gpfs_scratch1/nopurge/dadams/data/10GB/10GB-1 gsiftp://tg-gridftp.sds c.teragrid.org/gpfs/dadams/

Source: gsiftp://gridftp-hg.ncsa.teragrid.org/gpfs_scratch1/nopurge/dad ams/data/10GB/
Dest:   gsiftp://tg-gridftp.sdsc.teragr id.org/gpfs/dadams/
  10GB-1
  10737418240 bytes       567.29 MB/sec ;avg       634.24 MB/sec inst

tgcp large file

Transfer of a large file from NCSA to SDSC using the TeraGrid copy (tgcp) tool.

Passing the -big option to tgcp tells it to employ striping.
The URL sytax is more relaxed here. Basically, you get the third party benifits without all the typing.
Notice that the full globus-url-copy comand is printed. This is a good way to develop globus-url-copy commands.

> tgcp -vb -big ./10GB-1 tg-gridftp.sdsc.teragrid.org:/gpfs/dadams/

/usr/local/globus-4.0.1-r3//bin/globus-url-copy -vb  -stripe& nbsp;-p 4 -tcp-bs 4000000 gsiftp://gridftp-hg.ncsa.terag rid.org/gpfs_scratch1/nopurge/dadams/data/10GB/10GB-1 gsiftp://tg-gridf tp.sdsc.teragrid.org/gpfs/dadams/

Source: gsiftp://gridf tp-hg.ncsa.teragrid.org/gpfs_scratch1/nopurge/dadams/data/10GB/
Dest:    gsiftp://tg-gridftp.sdsc.teragrid.org/gpfs/dadams/
  10GB-1
  10737418240 bytes  &nbs p;    582.65 MB/sec avg     ;   918.36 MB/sec inst

tgcp RFT

Transfer a single file using the RFT transfee management system.

RFT functionality is implemented with the -rft option.
tgcp automatically generates an input file for the rft command then executed the rft command.
Using the -v option, the file name of the temporary input file for rft is displayed, and it's contents can be inspected.

> tgcp -v -rft 1GB tg-gridftp.sdsc.teragrid.org:/gpfs/dadams/

/usr/local/globus-4.0.1-r3/bin/rft -h rft-hg.ncsa.teragrid.org&nbs p;-r 8443 -l 60 -z host -f /tmp/filelTzYM v

Number of transfers in this reques t: 1
Subscribed for overall status
Termina tion time to set: 60 minutes

O verall status of transfer:
Finished/Active/Failed/Retr ying/Pending
0/1/0/0/0

Overall status of transfer:
Finished/Active/Failed/Retrying/Pending
1/0/ 0/0/0
All Transfers are completed

Client -> Server

Pushing data from client process to waiting GridFTP server.

The "-vb" option is included to show performance data.

> globus-url-copy -vb file:///gpfs_scratch1/nopurge/dadams/1GB gsiftp://gridft p-co.ncsa.teragrid.org/scratch/users/dadams/1GB

Source: file:///gpfs_scratch1/nopurge/dadams/
Dest: & nbsp;gsiftp://gridftp-co.ncsa.teragrid.org/scratch/users/dadams/
&nbs p; 1GB
1045430272 bytes 28.58 MB/sec avg &n bsp; 30.00 MB/sec inst

guc: Client -> Server

Pushing data from client process to waiting GridFTP server

The parallel streams parameter is adjusted here to claim more "fair shares" of the network.

> globus-url-copy -vb -p 8 file:///gpfs_scratch1/nopurge/dadams/1GB gsiftp://g ridftp-co.ncsa.teragrid.org/scratch/users/dadams/1GB

Source: file:///gpfs_scratch1/nopurge/dadams/
Dest: & nbsp;gsiftp://gridftp-co.ncsa.teragrid.org/scratch/users/dadams/
&nbs p; 1GB
1073741824 bytes 72.62 MB/sec avg &n bsp; 70.49 MB/sec inst

guc: whole directory

Copy an entire directory hierarchy to another location.

The "fast" option tells the GridFTP servers to always use Mode E to complete transfers. This, among other things enables the reuse of the control channel connection reducing the overhead of each individual file transfer.
Adding the "-r>" and "-cd" options ensures that directories are recursively copied and created when needed.
The trailing slashes on the source and destination URLs indicate a directory is being copied rather than a file.

> globus-url-copy -vb -tcp-bs 8388608 -fast -r -cd gsiftp://gridftp-hg.ncsa.te ragrid.org/gpfs_scratch1/nopurge/dadams/data/dfiles/ gsiftp://tg-gridftp.sds c.teragrid.org/gpfs/dadams/dfiles/

Source: gsiftp://gridftp-hg.ncsa.teragrid.org/gpfs_scratch1/nopurge/dad ams/data/dfiles/
Dest:   gsiftp://tg-gridftp.sdsc.tera grid.org/gpfs/dadams/dfiles/
  d1/

&nb sp;d2/

  d3/

Source: gsiftp://g ridftp-hg.ncsa.teragrid.org/gpfs_scratch1/nopurge/dadams/data/dfiles/d1/
Dest:   gsiftp://tg-gridftp.sdsc.teragrid.org/gpfs/dadams /dfiles/d1/
  m_000001
      ;10485760 bytes        16.67&nb sp;MB/sec avg        16.67 ;MB/sec inst
Source: gsiftp://gridftp-hg.ncsa.teragrid.org/ gpfs_scratch1/nopurge/dadams/data/dfiles/d2/
Dest:   g siftp://tg-gridftp.sdsc.teragrid.org/gpfs/dadams/dfiles/d2/
&nb sp;m_000001

Source: gsiftp://gridftp-hg.ncsa.teragrid.or g/gpfs_scratch1/nopurge/dadams/data/dfiles/d3/
Dest:    ;gsiftp://tg-gridftp.sdsc.teragrid.org/gpfs/dadams/dfiles/d3/
& nbsp;m_000001

Data Transfer Example Database