NCSA SGI Altix System: Programming Environment

Compilers

Introduction
Compiler Commands
Vendor Documentation
Useful Compiler/Linker Options

Libraries and Application Software

Intel's Math Kernel Library (MKL)
Locally Installed Software
Hierarchical Data Format (HDF) Library

1. Compilers

1.1 Introduction

NCSA supports the Intel compilers on the SGI Altix system. (The GNU compilers are also available, but they generally produce slower code.)

1.2 Compiler Commands

To compile and link a serial program in Fortran, C, and C++ enter:

  ifort myprog.f
  icc myprog.c
  icpc myprog.cc

To compile and link an OpenMP program in Fortran and C, use the -openmp option:

  ifort -openmp myprog.f
  icc -openmp myprog.c

Before you run an OpenMP program, set the environment variable OMP_NUM_THREADS to the number of thtreads you want. For example, to run program a.out interactively with two threads:

  setenv OMP_NUM_THREADS 2
  ./a.out

Before you run an MPI OpenMP hybrid program, set the environment variable MPI_OPENMP_INTEROP to 1 or "true" so that OpenMP threads are allowed to use their own processors along with each MPI rank.

  setenv OMP_NUM_THREADS 2
  setenv MPI_OPENMP_INTEROP true
  mpirun -np 4 ./a.out_openmp

The following environment variables may also be useful in running your OpenMP programs:

OMP_SCHEDULE	Sets the schedule type and (optionally) the chunk size for DO and PARALLEL DO loops declared with a schedule of RUNTIME. The default is STATIC.
KMP_LIBRARY	sets the run-time execution mode. The default is `throughput`, but it can be set to `turnaround` so worker threads do not yield while waiting for work.
KMP_STACKSIZE	Sets the number of bytes to allocate for the stack of each parallel thread. You can use a suffix k, m, or g to specify kilobytes, megabytes or gigabytes. The default is 4m.

To compile and link an MPI program in Fortran, C, and C++ add -lmpi at the end of the compile command:

  ifort myprog.f -lmpi
  icc myprog.c -lmpi
  icpc myprog.cc -lmpi -lmpi++

When you run an MPI program, you need to specify the number of processors with the -np option. For example, to run program a.out on two processors:

  mpirun -np 2 a.out

Notes:

The MPICH compile commands (mpicc, mpif90, etc.) are not supported on the SGI Altix system.
The default behavior of icc for the C preprocessor (CPP) macros is to appear as if it is the GNU gcc 3.0 compiler. To list what macros are being passed to the preprocessor, use the -v option. Use the -no-gcc option to turn off these GNU CPP macros.
For information about running OpenMP and MPI programs in a batch job, see Running Jobs.

1.3 Vendor Documentation

1.4 Useful Compiler/Linker Options

Compatibility Options

-fpp[n]: runs the Fortran preprocessor on source files prior to compilation.
-assume byterecl: Many compilers assume that in the OPEN statement, the record length (recl=) of direct-access unformatted files is specified in bytes. With the Intel 8.X/9.X compilers the default is in four-byte words. If you use the option above, it assumes the record length is specified in bytes.
-Dname[=value]: specifies name as a definition to use with conditional compilation directives or the Fortran preprocessor (-fpp).
-w90, -w95: suppress messages about use of non-standard Fortran
-i8: set default KIND of integer variables is 8
-integer_size size: (8.X/9.X compilers) specifies the default size of integer and logical variables (size: 16, 32, 64)
-r8: set default size of REAL to 8 bytes
-real_size size: (8.X/9.X compilers) specify the size of REAL and COMPLEX declarations, constants, functions, and intrinsics (size: 32, 64, 128)
-save: save all variables (static allocation)

Little endian to big endian conversion

This is intended for Fortran unformatted input/output operations. This enables the development and processing of files with big-endian data organization on the Intel processors, which use little endian. It is implemented as an environment variable F_UFMTENDIAN. The syntax is:

  csh/tcsh: setenv  F_UFMTENDIAN u[,u]...
  sh/bash:  export  F_UFMTENDIAN=u[,u]...

where u[,u]... are unit numbers of those files that are to be treated as big endian.

Detecting Programming Errors

-g: produce symbolic debug information in object file (implies -O0)
-CB: array bounds checking
-fpen: (8.X/9.X compilers) specifies behavior on floating point exceptions (n = 0, 1 or 3)
-zero: implicitly initialize all data to zero
-u: equivalent to having "IMPLICIT NONE" throughout the code
-traceback: (Fortran 8.X/9.X compiler) allow the display of source file traceback information at runtime when a severe error occurs. For mpi programs, also add "setenv MPI_COREDUMP INHIBIT" to the runtime environment to get effective use of the -traceback flag.

Optimization

-O2: enable optimizations (DEFAULT)
-O1: optimize for maximum speed, but disable some optimizations which increase code size for a small speed benefit
-O3: enable -O2 plus more aggressive optimizations that may not improve performance for all programs
-O0: disable optimizations
-O: same as -O2
-ipo: enable multi-file IP optimizations (between files)
-ftz: flush denormal values to zero when the application is in the gradual underflow mode. By default, the compiler lets results to gradually underflow. Flushing the denormal values to zero may improve performance of your application. Use this option if the denormal values are not critical to application behavior.
-mp: maintain floating point precision (disables some optimizations)
-mp1: improve floating-point precision (speed impact is less than -mp)
-tpp2: Optimize for Intel Itanium 2 processors. (default)

Parallel Programming Options

-openmp: tells the parallelizer to generate multithreaded code based on OpenMP directives.
-parallel: tells the auto-parallelizer to generate multithreaded code for loops that can be safely executed in parallel. To use this option, you must also specify -O2 or -O3.
-lmpi: link with MPI libraries. This is required for MPI programs. It must be specified at the end of the compile command.

Other Options

-v

display verbose information about each compiler phase.

-V

Displays the compiler version information

-sox

Tells the compiler to save the compiler options and version number in the executable. The default is -no-sox. To retrieve the information:

        strings --all a.out | grep comment:

2. Libraries and Application Software

2.1 Intel Math Kernel Library

The Math Kernel Library (MKL) contains the complete set of functions from the basic linear algebra subprograms (BLAS), the extended BLAS (sparse), and the complete set of LAPACK routines. In addition, there is a set of fast Fourier transforms (FFT) in single- and double-precision, real and complex data types with both Fortran and C interfaces. The library also includes the cblas interfaces, which allow the C programmer to access all the functionality of the BLAS without considering C-Fortran issues. ScaLAPACK, BLACS and the PARDISO solver are also provided by Intel MKL.

See Intel Math Kernel Library for detailed information, including release notes and reference manual.

The libraries are installed under ${MKL_HOME}/lib/64. To link to the libraries statically using the layered, pure form as recommended by Intel please do the following:

(The linking schemes below are recommended by Intel and resolves a known numerical issue with linking to the dynamic libraries. We are working with Intel to resolve the issue with the dynamic libraries.)

Also note that the linker options -Wl,-Bstatic -Wl,--start-group and -Wl,--end-group -Wl,-Bdynamic are needed to properly link to the static libraries and to resolve cyclic dependencies in the static libraries.

Note: Please do not use the Intel compiler option -fast as it adds -static and there are not static libraries for the SGI MPT (MPI) libraries. If you would like to link to the Intel compiler libraries statically please add -static-intel -static-libgcc.

sequential versions of:

BLAS and LAPACK: -Wl,-Bstatic -Wl,--start-group -L ${MKL_HOME}/lib/64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -Wl,-Bdynamic

ScaLAPACK and BLACS:-L ${MKL_HOME}/lib/64 -lmkl_scalapack_lp64 -lmkl_blacs_sgimpt_lp64 -Wl,-Bstatic -Wl,--start-group -L ${MKL_HOME}/lib/64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -Wl,-Bdynamic -lmpi

threaded versions of:

BLAS and LAPACK: -Wl,-Bstatic -Wl,--start-group -L ${MKL_HOME}/lib/64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -Wl,-Bdynamic -lguide -lpthread

For FFT and PARDISO documentation, please consult the userguide and the software database entry for more information on using Intel MKL.

Note: Some MKL routines can run in parallel if the environment variable $MKL_NUM_THREADS is set to the number of threads you wish to use and when linking to the libmkl_intel_thread as documented above. To ensure only one thread is being used in MKL routines set the value of $MKL_NUM_THREADS to 1 (this is the default).

2.2 Scientific Computing Software Library

Under SGI ProPack5, the SCS and SDSM libraries are no longer supported by SGI. We recommned using Intel MKL to provide the functionality supplied by SCS and SDSM. Please see the Migrating to ProPack5 Math libraries for more information.

2.3 Locally Installed Software

NCSA has a variety of applications software available in different areas of concentration, including chemistry, computational fluid dynamics, mathematics, solid mechanics, and visualization. An online list is maintained in the Software Repository. The list is organized by system and by area of science. If you have any questions about available software, contact the appropriate software coordinator or the staff in Consulting Services. The /usr/apps directory contains third party applications with most located in subdirectories according to the applications areas.

2.3 Hierarchical Data Format (HDF) Library

HDF is a library and platform independent data format for the storage and exchange of scientific data. It includes Fortran and C calling interfaces, and utilities for analyzing and converting HDF data files. There are two HDF formats, HDF (4.x and previous releases) and HDF5. These formats are completely different and NOT compatible. Use SoftEnv for information on accessing the software.

Information on support is available at the HDF Support Issues page. The HDF Home Page has detailed information on HDF and HDF5, including documentation, tutorials, examples, and FAQs.

Programming Environment on NCSA SGI Altix System