NCSA SGI Altix UV System: Programming Environment

Compilers

Introduction
Compiler Commands
Vendor Documentation
Useful Compiler/Linker Options

Libraries and Application Software

Intel's Math Kernel Library (MKL)
Locally Installed Software
Hierarchical Data Format (HDF) Library

1. Compilers

1.1 Introduction

NCSA supports the Intel compilers on the SGI Altix UV system. (The GNU compilers are also available, but they generally produce slower code.)

1.2 Compiler Commands

To compile and link a serial program in Fortran, C, and C++ enter:

  ifort myprog.f
  icc myprog.c
  icpc myprog.cc

To compile and link an OpenMP program in Fortran, C, and C++ use the -openmp option:

  ifort -openmp myprog.f
  icc -openmp myprog.c
  icpc -openmp myprog.cc

Before you run an OpenMP program, set the environment variable OMP_NUM_THREADS to the number of thtreads you want. For example, to run program a.out interactively with two threads:

  setenv OMP_NUM_THREADS 2
  ./a.out

Before you run an MPI OpenMP hybrid program, set the environment variable MPI_OPENMP_INTEROP to 1 or "true" so that OpenMP threads are allowed to use their own processors along with each MPI rank.

  setenv OMP_NUM_THREADS 2
  setenv MPI_OPENMP_INTEROP true
  mpirun -np 4 ./a.out_openmp

The following environment variables may also be useful in running your OpenMP programs:

OMP_SCHEDULE	Sets the schedule type and (optionally) the chunk size for DO and PARALLEL DO loops declared with a schedule of RUNTIME. The default is STATIC.
KMP_LIBRARY	sets the run-time execution mode. The default is `throughput`, but it can be set to `turnaround` so worker threads do not yield while waiting for work.
OMP_STACKSIZE	Sets the number of bytes to allocate for the stack of each parallel thread. You can use a suffix k, m, or g to specify kilobytes, megabytes or gigabytes. The default is 4m.

To compile and link an MPI program in Fortran, C, and C++ add -lmpi at the end of the compile command:

  ifort myprog.f -lmpi
  icc myprog.c -lmpi
  icpc myprog.cc -lmpi -lmpi++

When you run an MPI program, you need to specify the number of processors with the -np option. For example, to run program a.out on two processors:

  mpirun -np 2 a.out

Notes:

The MPICH compile commands (mpicc, mpif90, etc.) are not supported on the SGI Altix UV system.
For information about running OpenMP and MPI programs in a batch job, see Running Jobs.
(From the mpi man page) The -lmpi option may be replaced with -lmpi_mt for programs that require the MPI_THREAD_MULTIPLE multithreading level. See MPI_Init_thread(3) for more information.

1.3 Vendor Documentation

1.4 Useful Compiler/Linker Options

Compatibility Options

-fpp[n]: runs the Fortran preprocessor on source files prior to compilation.
-assume byterecl: Many compilers assume that in the OPEN statement, the record length (recl=) of direct-access unformatted files is specified in bytes. The default is in four-byte words. If you use the option above, it assumes the record length is specified in bytes.
-Dname[=value]: specifies name as a definition to use with conditional compilation directives or the Fortran preprocessor (-fpp).
-w90, -w95: suppress messages about use of non-standard Fortran
-i8: set default KIND of integer variables is 8
-integer_size size: specifies the default size of integer and logical variables (size: 16, 32, 64)
-r8: set default size of REAL to 8 bytes
-real_size size: specify the size of REAL and COMPLEX declarations, constants, functions, and intrinsics (size: 32, 64, 128)
-save: save all variables (static allocation)

Little endian to big endian conversion

This is intended for Fortran unformatted input/output operations. This enables the development and processing of files with big-endian data organization on the Intel processors, which use little endian. It is implemented as an environment variable F_UFMTENDIAN. The syntax is:

  csh/tcsh: setenv  F_UFMTENDIAN u[,u]...
  sh/bash:  export  F_UFMTENDIAN=u[,u]...

where u[,u]... are unit numbers of those files that are to be treated as big endian.

Detecting Programming Errors

-g: produce symbolic debug information in object file (implies -O0)
-CB: array bounds checking
-fpen: specifies behavior on floating point exceptions (n = 0, 1 or 3)
-zero: implicitly initialize all data to zero
-u: equivalent to having "IMPLICIT NONE" throughout the code
-traceback: allow the display of source file traceback information at runtime when a severe error occurs. For mpi programs, also add "setenv MPI_COREDUMP INHIBIT" to the runtime environment to get effective use of the -traceback flag.

Optimization

-O2: enable optimizations (DEFAULT)
-O1: optimize for maximum speed, but disable some optimizations which increase code size for a small speed benefit
-O3: enable -O2 plus more aggressive optimizations that may not improve performance for all programs
-O0: disable optimizations
-O: same as -O2
-ipo: enable multi-file IP optimizations (between files)
-mp: maintain floating point precision (disables some optimizations)
-mp1: improve floating-point precision (speed impact is less than -mp)
-xHost: generate instructions for the highest instruction set and processor available on the compilation host.

Parallel Programming Options

-openmp: tells the parallelizer to generate multithreaded code based on OpenMP directives.
-parallel: tells the auto-parallelizer to generate multithreaded code for loops that can be safely executed in parallel. To use this option, you must also specify -O2 or -O3.
-lmpi: link with MPI libraries. This is required for MPI programs. It must be specified at the end of the compile command.

Other Options

-v

display verbose information about each compiler phase.

-V

Displays the compiler version information

shared-intel

Causes Intel-provided libraries to be linked in dynamically.

-mcmodel[=value]

Tells the compiler to use a specific memory model to generate code and store data.
value =

small	Tells the compiler to restrict code and data to the first 2GB of address space. (DEFAULT)
medium	Tells the compiler to restrict code to the first 2GB; it places no memory restriction on data.
large	Places no memory restriction on code or data. All accesses of code and data must be done with absolute addressing.

NOTE: When you specify -mcmodel=medium or -mcmodel=large, you must also specify compiler option -shared-intel to ensure that the correct dynamic versions of the Intel run-time libraries are used.

-sox

Tells the compiler to save the compiler options and version number in the executable. The default is -no-sox. To retrieve the information:

        strings --all a.out | grep comment:

2. Libraries and Application Software

2.1 Intel Math Kernel Library

The Math Kernel Library (MKL) contains the complete set of functions from the basic linear algebra subprograms (BLAS), the extended BLAS (sparse), and the complete set of LAPACK routines. In addition, there is a set of fast Fourier transforms (FFT) in single- and double-precision, real and complex data types with both Fortran and C interfaces. The library also includes the cblas interfaces, which allow the C programmer to access all the functionality of the BLAS without considering C-Fortran issues. ScaLAPACK, BLACS and the PARDISO solver are also provided by Intel MKL.

See here for linking information.

See the Intel Math Kernel Library Reference Manual for detailed information.

2.2 Locally Installed Software

NCSA has a variety of applications software available in different areas of concentration, including chemistry, computational fluid dynamics, mathematics, solid mechanics, and visualization. An online list is maintained in the Software Repository. The list is organized by system and by area of science. If you have any questions about available software, contact the appropriate software coordinator or the staff in Consulting Services. The /usr/apps directory contains third party applications with most located in subdirectories according to the applications areas.

2.3 Hierarchical Data Format (HDF) Library

HDF is a library and platform independent data format for the storage and exchange of scientific data. It includes Fortran and C calling interfaces, and utilities for analyzing and converting HDF data files. There are two HDF formats, HDF4 and HDF5/PHDF5. These formats are completely different and not compatible. Although HDF4 is still funded, new users that are not constrained to using HDF4 should use HDF5. HDF4 is located under /usr/apps/hdf/hdf4 and HDF5/PHDF5 is located under /usr/apps/hdf/hdf5 and /usr/apps/hdf/phdf5. Both static and shared library versions of HDF5 are available. The Szip compression library is also provided for use with HDF in /usr/apps/hdf/szip.

Ember has HDF5/PHDF5 version 1.8 installed. HDF5 Release 1.8 represents a major update to the HDF5 library, utilities, and file format. It contains many new features, file size optimizations, and performance enhancements along with more than 300 new APIs. Many of the function signatures in 1.8 have been upgraded since 1.6, so separate versions of each are maintained (the 1.8 bindings generally having additional property list arguments). Please refer to the What's New in HDF5 1.8 document for more information on the changes in the HDF5 1.8 release.

Users can recompile existing HDF5 1.6 applications using the compilation flag:

         -D_H5_USE_16_API

without changing existing source code. For more on the transition to the HDF5 1.8 libraries see API Compatibility Macros in HDF5.

[Note that the default version of HDF5 on Cobalt is 1.6.7.]

Programming Environment on NCSA SGI Altix UV System