Previous: SGI_MathLibraries
Next: Viz_Software
View by Date; View by Name
gsn

                            UPDATED 7/26/02

balder (256-processor Origin2000) has been split into two 128-processor
systems, so the 512_gsn queue is unavailable until further notice.
Please contact consult@ncsa.uiuc.edu if you have an urgent need to use
more than 128 processors in a single batch job on the Origin2000 systems.


----------------------------------------------------------------------
                   Running Jobs in the 512_gsn queue
                           April 24, 2001

Three of NCSA's Origin 2000 systems are connected by a very high
speed network called Gigabit System Network (GSN).  The network is 
full duplex and runs at 800MB per second. (For more information about 
GSN, see http://www.sgi.com/peripherals/networking/gsn_overview.html)

GSN is used to connect balder (256p), aegir (128p) and forseti1 (128p)
into a GSN cluster of 512 processors.  Since MPI currently does not 
support clusters with machines having more than 128 processors, balder 
is treated as two hosts of size 128 processors each.  

Users may run MPI jobs on the GSN cluster by submitting them to the
512_gsn queue.  Access to the queue is limited.  To request access, 
please send e-mail to consult@ncsa.uiuc.edu.  The default time limit
for the 512_gsn queue is 48 hours.  As with other dedicated queues, 
you are charged for all the processors whether you use them or not.

Set the following environment variables to use GSN: 

    setenv MPI_GSN_ON
    setenv MPI_GSN_DEVS 01

If you want a verbose mode, the following variable may be set:

    setenv MPI_GSN_VERBOSE

The meaning of other environment variables in the example below can be
found in the MPI man page (man mpi).   You may also add other variables
based on your needs.   

Note: SGI found that MPI_DSM_MUSTRUN environment variable causes
performance problems when using the split partitions on balder so 
its use is not recommended.

The following mpirun command uses 4x128 processors for running linpack:

   mpirun -v -a gsn -d $HOME/linpack/src \
      -prefix "%g %h %l %@: " \
      aegir, forseti1, balder, balder \
      -np 128 linpack <input >& output.ncsa.4x128 

The "-np 128" in the mpirun command represents the number of processors
on each partition.  Since there are 4 partitions (aegir, forsetil, balder, 
and balder) this translates into 4x128 = 512 processors.  The meanings of 
the options for the mpirun command (e.g. -v, -a, etc) can be found in 
the mpirun man page (man mpirun).     

The sample job below runs linpack on 4x128 processors.

---cut here---cut here---cut here---cut here---cut here---cut here---

#!/bin/csh
#
#  Sample Batch Script for an Origin2000 MPI GSN job
#
#  Submit this script using the "bsub" command: bsub < batch.sample
#  [NOTE THE "<" SIGN in the above command]
#
#  Use the "bjobs" command to check the status of a job.
#
# The following are embedded BSUB options. The syntax is #BSUB (the # does
# _not_  denote that the lines are commented out so do not remove).
#
#BSUB -q 512_gsn        # Submit job to 512_gsn
#BSUB -W 1:00           # Specify job run time limit of 1 hour
#BSUB -P abc            # Charge job to project abc (recommended for users
                        # with multiple projects)
#BSUB -o testjob.o      # Store the standard output and standard error of the
                        # job in file testjob.o (optional)
#BSUB -N                # Send mail when job terminates (optional)
#BSUB -J testjob        # Specify job name (optional)
# End of embedded BSUB options
#
#set echo               # echo commands before execution; use for debugging

# Change to machine-local scratch directory (/scratch-balder)
cd $SCR

# After the job is done, tar test.output* files and put the resulting file
# named test.output.tar in UniTree.  It is important to issue this command
# after changing to the $SCR directory, and before the executable begins.
msscmd -b "tar cf test.output.tar test.output*"

# Get the files a.out and test.input from directory test in UniTree
msscmd cd test,get a.out,get test.input

# Since UniTree gives you the same permissions as what your umask is
# set to (generally only read and write), you need to give yourself
# executable permission to run a.out
chmod u+x a.out

# These environment variables enable the GSN interconnect.  If they
# are not specified, the job will run across an alternate network
setenv MPI_GSN_ON
setenv MPI_GSN_DEVS 01
setenv MPI_GSN_VERBOSE

# These environment variables are application specific.  The idea
# is to provide as much memory as necessary for the MPI job message
# and data buffers.  If the application sends a large amount of
# mpi messages, the user should increase these as necessary.  See the
# mpi man pages (man mpi) for more information on these variables.
setenv MPI_MSGS_PER_HOST 1024
setenv MPI_MSGS_PER_PROC 1024
setenv MPI_BUFS_PER_HOST 256
setenv MPI_BUFS_PER_PROC 256
setenv MPI_MSG_RETRIES 10

# Note: SGI found that MPI_DSM_MUSTRUN environment variable causes
# performance problems when using the split partitions on balder.
##setenv MPI_DSM_MUSTRUN

# Run the MPI program on 128 processors on each of 4 partitions
# (balder is listed twice because it is treated as two hosts of
# 128 processors each)
mpirun -v -a gsn -prefix "%g %h %l %@: " \
  aegir, forseti1, balder, balder -np 128 ./a.out

#busage $LSB_JOBID           # appends job usage report to the end of the job
                             # standard output file.

---cut here---cut here---cut here---cut here---cut here---cut here---
Silicon Graphics Origin2000:usr/news/gsn
Last Modified: July 26, 2002