Previous: Scratch_Local
Next: New_Software_Stack
View by Date; View by Name
Lustre Striping

Information on New Striping Capability on Tungsten LUSTRE Filesystem

The Lustre filesystem on Tungsten has recently been enhanced to allow striping
of files across multiple Object Storage Targets (OSTs) to provide increased
overall bandwidth to the filesystem.  Please refer to the following usage
guidelines to decide if your application will benefit from this new capability.

NOTE:  NOT ALL APPLICATIONS WILL BENEFIT FROM STRIPING

If I/O performance is not an issue with your application please use the
non-striped Lustre filesystem on $HOME/scratch-global

Striping will have minimal or no impact (and may in fact degrade performance)
in the following cases:

- A single node performs all of the I/O for an application.
- Multiple nodes perform I/O, but the accesses are performed by different nodes
  at different times.
- Multiple nodes perform I/O simultaneously to different files that are small
  (each < 100 MB).

Striping can improve I/O performance in the following case:

- Multiple nodes simultaneous accessing one or more large files (> 100 MB each).
  Note that such access patterns are common if you use MPI-IO or parallel HDF5.

I/O can be directed to Tungsten's pre-configured striped filesystem where
striped I/O is automatically enabled.

- Files created on the /cfs/projects/widescratch filesystem will automatically
  inherit a stripe size of 4 MBytes, stripe over 8 OSTs, and begin writing on
  a random OST.
- This configuration offers good performance for many applications meeting the
  above I/O access criteria.

Alternatively, striping parameters can be configured by users on a per-directory
or per-file basis for any directory or file created under the /cfs hierarchy.
For applications with exceptional I/O requirements, this option allows users to
fine-tune the filesystem behavior to achieve optimal performance for their
application.

- For directories, first create the directory for which you wish to set striping
  parameters.   If you set parameters for a directory, all files created below
  that directory inherit the striping parameters unless explicitly overridden
  with new striping options.

  For example:
  > mkdir /cfs/projects/widescratch/users/$LOGNAME/FS_8_64

- The command "lfs setstripe" is used to set the striping parameters.  The
  command arguments specify:
  + A new filename or an existing directory.
  + The stripe size in bytes, with "0" used to specify the default stripe
    size associated with the filesystem overall.
  + The offset of the OST where the initial stripe is written, with "0" being
    the first OST, "1" being the second OST, and so on. A value of "-1"
    specifies the default OST offset associated with the filesystem overall,
    which cycles through the pool of available OSTs so that initial stripes
    of files do not all begin on the same OST. In most cases, "-1" should be
    used.
  + The stripe count, that is, the number of OSTs the stripes of the file will
    be distributed over, assuming the file is large enough to "wrap around"
    stripe-count OSTs.   A value of "0" uses the default stripe count for the
    filesystem and "-1" stripes over all available OSTs.

  For example, to set stripe size to 64k, start with a random OST, and stripe
  over 8 OSTs for the directory created earlier:
  > lfs setstripe /cfs/projects/widescratch/users/$LOGNAME/FS_8_64 65536 -1 8

- The command "lfs getstripe" is used to list the striping parameters for a
  file or directory.  All available OSTs will be listed.  When querying a
  directory the default striping parameters (as specified in the setstripe
  command) are listed.  When querying a file, the actual OSTs where the file
  resides are listed.
  > lfs getstripe /cfs/projects/widescratch/users/$LOGNAME/FS_8_64

- See "man lfs" for additional information.
Dell Xeon Cluster:usr/news/Lustre_Striping
Last Modified: September 19, 2005
ncsa

Lustre Striping
