Selected Category Information

MSS Transfers

Transfers to or from the NCSA Mass Storage System (MSS). MSS is a tape archival permanent data storage facility. A disk cache is used to stage data from tapes which is then available for download. For more details about MSS visit the MSS homepage. To browse example MSS transfers, stay here.

Connectivity of each cluster into Mass Storage System (MSS) varies. In general, multiple transfer streams
will achieve the best aggregate transfer rates. To maximize the efficiency of each
individual transfer use the following guidelines:

Always specify the "active" option when *writing* to MSS.
Gets are more forgiving of the FTP "MODE" used. So when retriving data from MSS more options are availiable that won't severly impact performace.
Be mindful of the archive retrieval process. A "get" will request the staging of a file, if it resides on the tape archive, then return without retrieving the file. A subsequent "get" command must be issued after tape retrieval is completed.
Avoid globus-url-copy; especially when writing.
Use msscmd or mssftp or uberftp whenever possible.

Example Listing

MSS -> Hg (3rd Party)

Transfer to MSS from NCSA IA-64 (Mercury) using uberftp and third party.

> uberftp mss.ncsa.teragrid.org "lopen gridftp-hg.ncsa.teragrid.org; lcd /gpfs _scratch1/dadams; active; put 2GB"

UNIX Archive FTP server (DiskXtender Version 2 .9) active. Checking DiskXtender.conf

220 ;UNIX Archive FTP server ready.
230 User&nbs p;dadams logged in.
220 tg-s039.ncsa.teragrid.org ;GridFTP Server 2.1 (gcc64dbg, 1122653280-63) ready .
230 User dadams logged in.
250 CWD& nbsp;command successful.
Active mode
Transfer o f 2147287040 bytes completed in 39.49 seconds. 54374.96 KB/sec

tg-login.uc -> MSS

Third party transfer to MSS from remote TG site.

Note the increased tcp buffer size setting.

> uberftp mss.ncsa.teragrid.org "lopen tg-gridftp.uc.teragrid.org; active; mod e stream; tcpbuf 4000000; lcd /disks/scratchgpfs1/adams; put 2GB"

UNIX Archive FTP server (DiskXtender Version 2 .9) active. Checking DiskXtender.conf

220 ;UNIX Archive FTP server ready.
230 User&nbs p;dadams logged in.
220 tg-s003.uc.teragrid.org G ridFTP Server 2.1 (gcc32dbg, 1122653280-63) ready.< br /> 230 User adams logged in.
Active mode
Stream mode
TCP buffer set to 400000 0 bytes
250 CWD command successful.
Transf er of 2147287040 bytes completed in 294.12&nbs p;seconds. 7300.82 KB/sec

MSS -> Hg

Transfer to MSS from NCSA IA-64 (Mercury) using uberftp.

> uberftp mss.ncsa.teragrid.org "active; put 2GB"

UNIX Archive FTP server (DiskXtender Version 2 .9) active. Checking DiskXtender.conf

220 ;UNIX Archive FTP server ready.
230 User&nbs p;dadams logged in.
Active mode
Transfer o f 2147287040 bytes completed in 48.30 seconds. 44457.07 KB/sec

MSS -> Co (guc)

third party transfer from MSS to a remote TG site using globus-url-copy

> time globus-url-copy -nodcau -tcp-bs 65536 -rp gsiftp://mss.ncsa.teragrid.or g/2GB gsiftp://gridftp-co.ncsa.teragrid.org/%2F/scratch/users/dadams/

real    1m13.367s
user    0m0 .197s
sys     0m0.016s

MSS (AR) -> Co (guc)

Attempt to transfer an archived file via globus-url-copy

globus-url-copy returns an error if the file is archived.
Archive retrieval is initiated by the (failed) request for the file.
A workaround for this would be to loop until no error is returned, or use uberftp with 'quote wait'.
The "-rp" or relative path option allows the paths to be specified relative to $HOME on either system. %2F in the path forces root directory.

> globus-url-copy -nodcau -tcp-bs 65536 -rp gsiftp://mss.ncsa.teragrid.org/1GB -2 gsiftp://gridftp-co.ncsa.teragrid.org/%2F/scratch/users/jdoe/

error: globus_ftp_client: the server responded with an error
550 1GB-2: is being retrieved from the archive...

MSS tarfile

Make tarfile of current directory and send to MSS.

Tar file is only created on MSS.

> msscmd "mkdir test-tar, cd test-tar, tar cvf test.tar ."

Using binary mode to transfer files.
Using&n bsp;65536 byte TCP window size.
./
./d1/ ./d1/m_000001
./d2/
./d2/m_000001
./d3/
./ d3/m_000001

msscmd: list tar

List contents of tar file on MSS using msscmd.

Tar archive exists only on MSS.

> msscmd "cd test-tar, tar tvf test.tar"

Using binary mode to transfer files.
Using&n bsp;65536 byte TCP window size.
drwxr-x--- d adams/aaa        0 2006-09-28&n bsp;11:37:38 ./
drwxr-x--- dadams/aaa   &nbs p;    0 2006-09-28 11:47:37 ./d1/
-rw-r----- dadams/aaa 10485760 2006-09-28 11:37:28 ./d1/m_000001
drwxr-x--- dadams/aaa      ;   0 2006-09-28 11:47:45 ./d2/
-rw-r-- --- dadams/aaa 10485760 2006-09-28 11:37:32 ./d2/m_ 000001
drwxr-x--- dadams/aaa         0 2006-09-28 11:47:49 ./d3/
-rw-r-----&nbs p;dadams/aaa 10485760 2006-09-28 11:37:40 ./d3/m_000001

MSS with uberftp

Transfer a file from MSS using the uberftp client.

Note the use of the "quote wait" command. This tells uberftp to wait for the tape retrieval process to complete before attempting to access the file.

> uberftp mss.ncsa.teragrid.org "lopen gridftp-w.ncsa.teragrid.org; lcd /cfs/s cratch/users/arnoldg; cd junk; quot wait; get new.tar"

220 UNIX Archive FTP server ready.
230 User arnoldg logged in.
220 tunf001.ncsa.uiuc.edu GridFTP Server 2.1 (gcc32dbg, 1122653280-63) ready.
230 User arnoldg logged in.
WAIT&nb sp;is now turned on.
new.tar: 24948008960&n bsp;bytes in 1433.74 seconds. 17400.62 KB/sec

Data Transfer Example Database