NCSA File Systems

Directories
Permanent File Storage
File Transfer Options

1. Directories

1.1 Home Directories

Your home directory is the default directory you are placed in when you log on. You should use this space for storing files you want to keep long term such as source code, scripts, input data sets, etc.

The command to see your disk usage and limits is quota. Home directories are backed up daily.

Computing System Quota (GB)

Intel 64 Tesla Cluster (lincoln) ^(*) 50

Intel 64 Cluster (abe)^(*)

SGI Altix (cobalt)

IBM IA-64 Linux Cluster (mercury) 10

^(*)Home directories are shared on abe and lincoln

1.2 Scratch Directories

Scratch file systems are intended for short term use and should be considered volatile. The size of scratch file systems varies with the system.

Please note that backups are not performed on the scratch directories. In the event of a disk crash or file purge, files on the scratch directories cannot be recovered. Therefore, you should make sure to back up your files to permanent storage as significant changes are made (at least daily).

SGI Altix (cobalt)

The common scratch filesystem /scratch/users is available for all interactive work. There is a soft link scratch-global in your home directory that points to your scratch directory. For batch jobs, see the section Disk Space for Batch Jobs.
- Starting Monday Aug 22, 2005, files in the common scratch filesystem (/scratch/users) are removed on the basis of size and time since the last access:
```
File Size    Removed after
>= 10 GB            4 days
<  10 GB           14 days
```
  Please do not attempt to circumvent this removal scheme (e.g., with touch). Such attempts may result in the loss of access to the scratch file systems.
- Files in the batch scratch filesystem (/scratch/batch) on the SGIAltix may be purged as soon as the batch job that it is associated with completes. Users should use the saveafterjob utility for automated, guaranteed saving of files from batch jobs.
Intel 64 Cluster (abe)/ Intel 64 Tesla Cluster (lincoln)

The common scratch filesystem /scratch/users is available for all interactive work. There is a soft link scratch-global in your home directory that points to your scratch directory. For batch jobs, see the section Disk Space for Batch Jobs.

No scratch disk quotas are currently in place. No guarantees are made as to how much scratch space will be available at any time. Files may be removed at any time should disk space become critically low.

The purge policy is as follows:
- Files in the common scratch filesystem (/scratch/users) are removed on the basis of size and time since the last access:
```
File Size    Removed after
>= 10 GB            2 days
<  10 GB           14 days
```
  Please do not attempt to circumvent this removal scheme (e.g., with touch). Such attempts may result in the loss of access to the scratch file systems.
- Files in the batch scratch filesystem (/scratch/batch) on Abe may be purged as soon as the batch job that it is associated with completes. Users should use the saveafterjob utility for automated, guaranteed saving of files from batch jobs.

TeraGrid IA64 Cluster

With the exception of the Node Local scratch file system, all scratch file systems have a subdirectory for each user.

GPFS (General Parallel File System)
The NCSA cluster has a GPFS scratch file system that uses Network Shared Disk Server (NSD) configuration model accessible from all the cluster nodes.

There are 55 TB of scratch space available in the GPFS NSD file system, /gpfs_scratch1. It is accessible from all nodes in the cluster (login, gridftp, and compute). It is the recommended scratch file system.

Starting May 19, 2005, files in the GPFS NSD filesystem that have not been modified in 5 days will be purged. Please do not attempt to circumvent this removal scheme (e.g., with touch). Such attempts may result in the loss of access to the scratch file systems.

Starting November 6 2007, there will be a 5 TB quota per user in GPFS scratch.
NFS Scratch Directories
(September 2007) The NFS scratch directories have been removed from service
Node-Local Scratch Directories
The scratch directory local to each machine is /scr. Each scratch directory has about 50GB of space available.

Files in the local scratch space are not available to any other nodes and hence are not directly accessible to your processes running on other nodes as part of your job. Only processes running on the two CPUs which make up a node have direct access to files in the local scratch space.

All files are automatically deleted after your batch job completes and the nodes are deallocated. All files you want to save must be copied from local scratch as part of your job. You will not be able to access files in local scratch after your job has completed.

TeraGrid Scratch Directory (GPFS-WAN)
TeraGrid GPFS-WAN (Global Parallel File System-Wide Area Network) is available. See the GPFS-WAN section of the TeraGrid documentation for details. GPFS-WAN is not mounted by default on the mercury compute nodes. It is available from the login nodes.

(January 2008) GPFS-WAN is no longer available on Mercury.

go to Top

1.3 /tmp, /usr/tmp, and /var/tmp Directories

The /tmp, /usr/tmp, and /var/tmp directories are intended for temporary files that are used during the execution of a process or job. Please do not use these directories for storage of user files. Files placed in /tmp, /usr/tmp, or /var/tmp may be purged at any time.

1.4 Reserved Project Space

NCSA has reserved project space available on the HPC systems. They are intended for users who require larger amounts of scratch space than exist in the normal scratch file systems for short durations. These directories are intended for short term use and are not backed up. Contact consult@ncsa.uiuc.edu for requesting space.

Please include your name, project (PSN), preferred start date, duration of need, approximate disk space needed, and a short description of the project. Include reasons why regular scratch space is insufficient for your needs.

1.5 TeraGrid Environment Variables

Environment variables have been set up for the TeraGrid to help make scripts work no matter which cluster they are run on. They provide a common user environment. You may choose to usethese variables instead of hardcoding paths. Availability on NCSA systems is denoted by the path name.

File System	Variable	Mercury	Cobalt	Abe
Home Directory	$TG_CLUSTER_HOME	/home/{group}/$USER	/u/{group}/$USER	/u/{group}/$USER
Default Parallel File System	$TG_CLUSTER_PFS	/gpfs_scratch1/$USER	/scratch/users/$USER	/scratch/users/$USER
Scratch Directory	$TG_CLUSTER_SCRATCH	/gpfs_scratch1/$USER	/scratch/users/$USER	/scratch/users/$USER
Node-Local Scratch Directory	$TG_NODE_SCRATCH	/scr

Note: these variables include the user ($USER) subdirectory for those file systems that have user subdirectories.

2. Permanent File Storage

Permanent storage of your files is available using the UniTree archival storage system. While there are no limits on the number or size of files that may be saved to UniTree, please send email to consult@ncsa.uiuc.edu if you want to store more than 5 Tbytes of data within a three day period. UniTree files are read accessible for 4 months after your account has been deleted.

The NCSA UniTree Page has links to detailed information on how to use UniTree.

3. File Transfer Options

A variety of methods are available for transferring files between computer systems. These are outlined, with examples, here: NCSA Data Transfers