NCSA Home
Contact Us | Intranet | Search

File Systems

  1. Directories
    1. Home Directories
    2. Scratch Directories
    3. /tmp, /usr/tmp, and /var/tmp Directories
    4. Reserved Project Space
    5. TeraGrid Environment Variables
  2. Permanent File Storage
  3. File Transfer Options


1. Directories

1.1 Home Directories

Your home directory is the default directory you are placed in when you log on. You should use this space for storing files you want to keep long term such as source code, scripts, input data sets, etc.

The command to see your disk usage and limits is quota. Home directories are backed up daily.

Computing System Quota (GB)
Intel 64 Tesla Cluster (lincoln) (*) 50
Intel 64 Cluster (abe)(*)
SGI Altix (cobalt)
IBM IA-64 Linux Cluster (mercury) 10
(*)Home directories are shared on abe and lincoln

1.2 Scratch Directories

Scratch file systems are intended for short term use and should be considered volatile. The size of scratch file systems varies with the system.

Please note that backups are not performed on the scratch directories. In the event of a disk crash or file purge, files on the scratch directories cannot be recovered. Therefore, you should make sure to back up your files to permanent storage as significant changes are made (at least daily).

  • SGI Altix (cobalt)

    The common scratch filesystem /scratch/users is available for all interactive work. There is a soft link scratch-global in your home directory that points to your scratch directory. For batch jobs, see the section Disk Space for Batch Jobs.

    • Starting Monday Aug 22, 2005, files in the common scratch filesystem (/scratch/users) are removed on the basis of size and time since the last access:
      File Size    Removed after
      >= 10 GB            4 days
      <  10 GB           14 days
      Please do not attempt to circumvent this removal scheme (e.g., with touch). Such attempts may result in the loss of access to the scratch file systems.
    • Files in the batch scratch filesystem (/scratch/batch) on the SGIAltix may be purged as soon as the batch job that it is associated with completes. Users should use the saveafterjob utility for automated, guaranteed saving of files from batch jobs.
  • Intel 64 Cluster (abe)/ Intel 64 Tesla Cluster (lincoln)

    The common scratch filesystem /scratch/users is available for all interactive work. There is a soft link scratch-global in your home directory that points to your scratch directory. For batch jobs, see the section Disk Space for Batch Jobs.

    No scratch disk quotas are currently in place. No guarantees are made as to how much scratch space will be available at any time. Files may be removed at any time should disk space become critically low.

    The purge policy is as follows:

    • Files in the common scratch filesystem (/scratch/users) are removed on the basis of size and time since the last access:
      File Size    Removed after
      >= 10 GB            2 days
      <  10 GB           14 days
      Please do not attempt to circumvent this removal scheme (e.g., with touch). Such attempts may result in the loss of access to the scratch file systems.
    • Files in the batch scratch filesystem (/scratch/batch) on Abe may be purged as soon as the batch job that it is associated with completes. Users should use the saveafterjob utility for automated, guaranteed saving of files from batch jobs.
  • TeraGrid IA64 Cluster

    With the exception of the Node Local scratch file system, all scratch file systems have a subdirectory for each user.

    • GPFS (General Parallel File System)

      The NCSA cluster has a GPFS scratch file system that uses Network Shared Disk Server (NSD) configuration model accessible from all the cluster nodes.

      There are 55 TB of scratch space available in the GPFS NSD file system, /gpfs_scratch1. It is accessible from all nodes in the cluster (login, gridftp, and compute). It is the recommended scratch file system.

      Starting May 19, 2005, files in the GPFS NSD filesystem that have not been modified in 5 days will be purged. Please do not attempt to circumvent this removal scheme (e.g., with touch). Such attempts may result in the loss of access to the scratch file systems.

      Starting November 6 2007, there will be a 5 TB quota per user in GPFS scratch.

    • NFS Scratch Directories

      (September 2007) The NFS scratch directories have been removed from service

    • Node-Local Scratch Directories

      The scratch directory local to each machine is /scr. Each scratch directory has about 50GB of space available.

      Files in the local scratch space are not available to any other nodes and hence are not directly accessible to your processes running on other nodes as part of your job. Only processes running on the two CPUs which make up a node have direct access to files in the local scratch space.

      All files are automatically deleted after your batch job completes and the nodes are deallocated. All files you want to save must be copied from local scratch as part of your job. You will not be able to access files in local scratch after your job has completed.

    • TeraGrid Scratch Directory (GPFS-WAN)

      TeraGrid GPFS-WAN (Global Parallel File System-Wide Area Network) is available. See the GPFS-WAN section of the TeraGrid documentation for details. GPFS-WAN is not mounted by default on the mercury compute nodes. It is available from the login nodes.

      (January 2008) GPFS-WAN is no longer available on Mercury.

  • go to Top

    1.3 /tmp, /usr/tmp, and /var/tmp Directories

    The /tmp, /usr/tmp, and /var/tmp directories are intended for temporary files that are used during the execution of a process or job. Please do not use these directories for storage of user files. Files placed in /tmp, /usr/tmp, or /var/tmp may be purged at any time.

    1.4 Reserved Project Space

    NCSA has reserved project space available on the HPC systems. They are intended for users who require larger amounts of scratch space than exist in the normal scratch file systems for short durations. These directories are intended for short term use and are not backed up. Contact consult@ncsa.uiuc.edu for requesting space.

    Please include your name, project (PSN), preferred start date, duration of need, approximate disk space needed, and a short description of the project. Include reasons why regular scratch space is insufficient for your needs.

    1.5 TeraGrid Environment Variables

    Environment variables have been set up for the TeraGrid to help make scripts work no matter which cluster they are run on. They provide a common user environment. You may choose to usethese variables instead of hardcoding paths. Availability on NCSA systems is denoted by the path name.

    File System Variable Mercury Cobalt Abe
    Home Directory $TG_CLUSTER_HOME /home/{group}/$USER /u/{group}/$USER /u/{group}/$USER
    Default Parallel File System $TG_CLUSTER_PFS /gpfs_scratch1/$USER /scratch/users/$USER /scratch/users/$USER
    Scratch Directory $TG_CLUSTER_SCRATCH /gpfs_scratch1/$USER /scratch/users/$USER /scratch/users/$USER
    Node-Local Scratch Directory $TG_NODE_SCRATCH /scr

    Note: these variables include the user ($USER) subdirectory for those file systems that have user subdirectories.

    2. Permanent File Storage

    Permanent storage of your files is available using the UniTree archival storage system. While there are no limits on the number or size of files that may be saved to UniTree, please send email to consult@ncsa.uiuc.edu if you want to store more than 5 Tbytes of data within a three day period. UniTree files are read accessible for 4 months after your account has been deleted.

    The NCSA UniTree Page has links to detailed information on how to use UniTree.

    3. File Transfer Options

    A variety of methods are available for transferring files between computer systems. These are outlined, with examples, here: NCSA Data Transfers

    go to Top