Request Guidelines
The request form below pertains specifically to scratch high-performance parallel file system space. Users are expected to make a reasonable effort to comply with the scratch file system purging policies (below) for each system if at all possible. The purpose of this form is to make available extended non-purged scratch file system space for a specific period of time, if it can be shown to better facilitate a project's computational goals.
Please provide as much information as possible about the nature of the data requirements. Understanding how the file systems are being used will help us determine which new system capabilities such as new scheduler enhancements, transfer tools, database systems are most useful
Request Implementation
- If the request is approved, a subdirectory on the scratch file system will be created for you in the same way as in normal scratch.
- Granted space may be removed and given to another user after your requested duration has been exceeded.
- Though not subject to purging, no backups of project file system allocation are maintained. Maintaining copies of important data on the Mass Storage System is recommended. Automated backups can be requested for a limited portion of requested project space.
- All requests will be reviewed on the basis of size*time and project needs. Requests to occupy a significant percentage of file system space may be granted contingent on subsequent user requests.
Example File System Usage
- Frequent use of same data for an extended set of runs. Rather than repeatedly staging the same data, keep the dataset persistent on disk.
- Large restart files needed for subsequent runs. Moving throw-away data to and from archival storage, just to facilitate a subsequent run, can be avoided by requesting persistent project space.
- To facilitate high data-availability for multi-site runs or other data transfer requirements.
General Scratch File System Policies
File Purging
Refer to system specific documentation for file size and duration limits. These limits are dictated by the performance characteristics and capacity of the locally accessible disk and therefore will vary for each system. Limits are enforced using the following guidelines:
- No files belonging to a particular user shall be deleted while that user has a batch job in the local queue.
- Large files are deleted first.
This policy is intended to accommodate the pre-staging of input data for batch runs. Files can be placed in scratch prior to job submission and no purging will take place until the job finishes. This eliminates the need for lengthy cycle-wasting file transfers within batch jobs.
Please keep in mind that parallel file systems are a shared resource. Codes with multi-terabyte data requirements can be accommodated, but must be managed in such a way that free space remains for other jobs. A significant percentage of file system space must be treated in similar fashion as a dedicated computational block. Clearing large sections of file system in a timely fashion will help accommodate data from new jobs just as relinquishing computational resources allows more jobs to run.
Backups
Scratch space on parallel file systems is not automatically backed up. Consult system specific backup policies for details about which file systems are backed up. High-performance parallel file systems are large and complex, and file system failures do still happen occasionally; though rarely is any data lost. Even though these file systems have become more stable recently, they should still be considered volatile storage and any non-recoverable data should either be moved to a backed up location (such as home directory) or to the Mass Storage System. As stated above: Automated backups can be requested for a limited portion of requested project space.
To Request Form
