Selected Category Information

`globus-url-copy` Transfers

Transfers using the globus-url-copy transfer client. globus-url-copy is a command line client provided by the Globus project that will transfer data using different protocols when providesd with a URL description of the source and destination files.

Use the "-vb" option to see performance statistics and source/destination recap.
Providing a appropriate value to the "-tcp-bs option will ensure efficient buffer usage and can often have a dramatic effect on performance.

Example Listing

MSS (AR) -> Co (guc)

Attempt to transfer an archived file via globus-url-copy

globus-url-copy returns an error if the file is archived.
Archive retrieval is initiated by the (failed) request for the file.
A workaround for this would be to loop until no error is returned, or use uberftp with 'quote wait'.
The "-rp" or relative path option allows the paths to be specified relative to $HOME on either system. %2F in the path forces root directory.

> globus-url-copy -nodcau -tcp-bs 65536 -rp gsiftp://mss.ncsa.teragrid.org/1GB -2 gsiftp://gridftp-co.ncsa.teragrid.org/%2F/scratch/users/jdoe/

error: globus_ftp_client: the server responded with an error
550 1GB-2: is being retrieved from the archive...

MSS -> Co (guc)

third party transfer from MSS to a remote TG site using globus-url-copy

> time globus-url-copy -nodcau -tcp-bs 65536 -rp gsiftp://mss.ncsa.teragrid.or g/2GB gsiftp://gridftp-co.ncsa.teragrid.org/%2F/scratch/users/dadams/

real    1m13.367s
user    0m0 .197s
sys     0m0.016s

guc large file

Third party transfer of a large file from NCSA to SDSC.

The stripe option is used here to utilize all of the servers at each end.
Large files benefit the most from striping.
A 10 MB tcp window size is used (tcp-bs) to keep the pipe full between SDSC and NCSA (~60ms RTT).

> globus-url-copy -vb -tcp-bs 10000000 -stripe gsiftp://gridftp-hg.ncsa.teragr id.org/gpfs_scratch1/nopurge/dadams/data/10GB/10GB-1 gsiftp://tg-gridftp.sds c.teragrid.org/gpfs/dadams/

Source: gsiftp://gridftp-hg.ncsa.teragrid.org/gpfs_scratch1/nopurge/dad ams/data/10GB/
Dest:   gsiftp://tg-gridftp.sdsc.teragr id.org/gpfs/dadams/
  10GB-1
  10737418240 bytes       567.29 MB/sec ;avg       634.24 MB/sec inst

Client -> Server

Pushing data from client process to waiting GridFTP server.

The "-vb" option is included to show performance data.

> globus-url-copy -vb file:///gpfs_scratch1/nopurge/dadams/1GB gsiftp://gridft p-co.ncsa.teragrid.org/scratch/users/dadams/1GB

Source: file:///gpfs_scratch1/nopurge/dadams/
Dest: & nbsp;gsiftp://gridftp-co.ncsa.teragrid.org/scratch/users/dadams/
&nbs p; 1GB
1045430272 bytes 28.58 MB/sec avg &n bsp; 30.00 MB/sec inst

guc: Client -> Server

Pushing data from client process to waiting GridFTP server

The parallel streams parameter is adjusted here to claim more "fair shares" of the network.

> globus-url-copy -vb -p 8 file:///gpfs_scratch1/nopurge/dadams/1GB gsiftp://g ridftp-co.ncsa.teragrid.org/scratch/users/dadams/1GB

Source: file:///gpfs_scratch1/nopurge/dadams/
Dest: & nbsp;gsiftp://gridftp-co.ncsa.teragrid.org/scratch/users/dadams/
&nbs p; 1GB
1073741824 bytes 72.62 MB/sec avg &n bsp; 70.49 MB/sec inst

guc: whole directory

Copy an entire directory hierarchy to another location.

The "fast" option tells the GridFTP servers to always use Mode E to complete transfers. This, among other things enables the reuse of the control channel connection reducing the overhead of each individual file transfer.
Adding the "-r>" and "-cd" options ensures that directories are recursively copied and created when needed.
The trailing slashes on the source and destination URLs indicate a directory is being copied rather than a file.

> globus-url-copy -vb -tcp-bs 8388608 -fast -r -cd gsiftp://gridftp-hg.ncsa.te ragrid.org/gpfs_scratch1/nopurge/dadams/data/dfiles/ gsiftp://tg-gridftp.sds c.teragrid.org/gpfs/dadams/dfiles/

Source: gsiftp://gridftp-hg.ncsa.teragrid.org/gpfs_scratch1/nopurge/dad ams/data/dfiles/
Dest:   gsiftp://tg-gridftp.sdsc.tera grid.org/gpfs/dadams/dfiles/
  d1/

&nb sp;d2/

  d3/

Source: gsiftp://g ridftp-hg.ncsa.teragrid.org/gpfs_scratch1/nopurge/dadams/data/dfiles/d1/
Dest:   gsiftp://tg-gridftp.sdsc.teragrid.org/gpfs/dadams /dfiles/d1/
  m_000001
      ;10485760 bytes        16.67&nb sp;MB/sec avg        16.67 ;MB/sec inst
Source: gsiftp://gridftp-hg.ncsa.teragrid.org/ gpfs_scratch1/nopurge/dadams/data/dfiles/d2/
Dest:   g siftp://tg-gridftp.sdsc.teragrid.org/gpfs/dadams/dfiles/d2/
&nb sp;m_000001

Source: gsiftp://gridftp-hg.ncsa.teragrid.or g/gpfs_scratch1/nopurge/dadams/data/dfiles/d3/
Dest:    ;gsiftp://tg-gridftp.sdsc.teragrid.org/gpfs/dadams/dfiles/d3/
& nbsp;m_000001

Data Transfer Example Database