NCSA TeraGrid: Programming Environment

Compilers

Introduction
Compiler Commands
Intel Compiler Documentation
Useful Intel Compiler/Linker Options
Notes

Message Passing Libraries

Introduction
MPICH-GM
Virtual Machine Interface (VMI)
MPICH Softenv Keys
MPICH Compiler Commands

Intel Math Kernel Library
Hierarchical Data Format (HDF) Library

1. Compilers

1.1 Introduction

NCSA supports the Intel compilers on its TeraGrid cluster. The GNU compilers are also available, but they generally produce slower code.

1.2 Compiler Commands

	Language	Help	Version	Example
Intel	C	`man icc`	`icc -V`	`icc mycode.c -o mycode`
	C++	`man icpc`	`icpc -V`	`icpc mycode.cpp -o mycode`
	F77/F90/F95	`man ifort`	`ifort -V`	`ifort mycode.f -o mycode`
GNU	C	`man gcc`	`gcc -v`	`gcc mycode.c -o mycode`
	C++	`man g++`	`g++ -v`	`g++ mycode.cpp -o mycode`
	F77	`man g77`	`g77 -v`	`g77 mycode.f -o mycode`

1.3 Intel Compiler Documentation

1.4 Useful Intel Compiler/Linker Options

Compatibility Options

-fpp[n]: runs the Fortran preprocessor on source files prior to compilation.
-Dname[=value]: specifies name as a definition to use with conditional compilation directives or the Fortran preprocessor (-fpp).
-w90, -w95: suppress messages about use of non-standard Fortran
-i8: set default KIND of integer variables is 8
-integer_size size: specifies the default size of integer and logical variables (size: 16, 32, 64)
-r8: set default size of REAL to 8 bytes
-real_size size: specify the size of REAL and COMPLEX declarations, constants, functions, and intrinsics (size: 32, 64, 128)
-save: save all variables (static allocation)

Little endian to big endian conversion

This is intended for Fortran unformatted input/output operations. This enables the development and processing of files with big-endian data organization on the Intel processors, which use little endian. It is implemented as an environment variable F_UFMTENDIAN. The syntax is:

  csh/tcsh: setenv  F_UFMTENDIAN u[,u]...
  sh/bash:  export  F_UFMTENDIAN=u[,u]...

where u[,u]... are unit numbers of those files that are to be treated as big endian.

Detecting Programming Errors

-g: produce symbolic debug information in object file (implies -O0)
-CB: array bounds checking
-fpen: specifies behavior on floating point exceptions (n = 0, 1 or 3)
-zero: implicitly initialize all data to zero
-u: equivalent to having "IMPLICIT NONE" throughout the code
-traceback: (Fortran 8.X compiler) allow the display of source file traceback information at runtime when a severe error occurs.

Optimization

-O2: enable optimizations (DEFAULT)
-O1: optimize for maximum speed, but disable some optimizations which increase code size for a small speed benefit
-O3: enable -O2 plus more aggressive optimizations that may not improve performance for all programs
-O0: disable optimizations
-O: same as -O2
-ipo: enable multi-file interprocedural optimizations (between files)
-ftz: flush denormal values to zero when the application is in the gradual underflow mode. By default, the compiler lets results gradually underflow. Flushing the denormal values to zero may improve performance of your application. Use this option if the denormal values are not critical to application behavior.
-mp: maintain floating point precision (disables some optimizations)
-mp1: improve floating-point precision (speed impact is less than -mp)
-tpp2: Optimize for Intel Itanium 2 processors. (default)

Other Options

-v: display verbose information about each compiler phase.
-V: Displays the compiler version information

1.4 Notes

[03/04/05] If you have problems compiling with the current version of the Intel compilers (version 8.0), the latest version of version 8.1 is available. To use this, replace the "@teragrid" line in your ${HOME}/.soft file with:
```
@teragrid-basic
@intel-8.1.028-stack
```
Note that C++ codes need to be compiled with icpc rather than icc with the 8.1 compiler.
To go back to the default version, remove the .soft file and log off. A new .soft with the default settings will be created for you the next time you log in.

If you continue to have problems with the Intel 8.1 compilers, please report the problem to help@teragrid.org. In addition to reporting compiler bugs to Intel, they will provide assistance in finding a workaround.
The default behavior of icc for the C preprocessor (CPP) macros is to appear as if it is the GNU gcc 3.0 compiler. To list what macros are being passed to the preprocessor, use the -v option. Use the -no-gcc option to turn off these GNU CPP macros.

2. Message Passing Libraries

2.1 Introduction

The NCSA TeraGrid cluster has a couple of implementations of MPI message passing libraries installed: MPICH-GM and Virtual Machine Interface (VMI).

2.2 MPICH-GM

MPICH is a portable implementation of MPI, the Message Passing Interface standard. MPICH was developed by Argonne National Laboratory, is designed to be highly portable, and is currently used by a large number of providers of MPI implementations. MPICH is often run over TCP/IP over Ethernet. In order to exploit the lower latency and higher data rates of Myrinet networks, Myricom developed MPICH-GM as a port of MPICH on top of GM (ch_gm).

MPICH-GM is in the default environment on the NCSA TeraGrid system.

2.3 Virtual Machine Interface (VMI)

Virtual Machine Interface (VMI) is a high-performance messaging API that supports multiple underlying communication devices in a cluster environment. At run time, it chooses from the interconnects available to communicate between processors to provide the best performance for a job transparent to the application. The layering of MPI on top of VMI is accomplished by means of a ch_vmi MPI device running on the Argonne National Laboratory MPICH distribution.

2.4 MPICH Softenv Keys

The default message passing libraries use MPICH-GM with the Intel compilers. To use a different implementation or version of the message passing libraries, use the appropriate SoftEnv key and commands to modify your environment. The following are the SoftEnv keys that point to the current version of each type. If you need to use older versions, you'll need to use the SoftEnv key for that specific version which can be found by looking at the output of the softenv command.

MPICH-GM	Intel	+mpich-gm-intel (default)
MPICH-GM	GNU	+mpich-gm-gcc
VMI	Intel	+mpich-vmi-intel
VMI	GNU	+mpich-vmi-gcc

See the SoftEnv section for instructions on how to set up your environment to use it.

Example ~/.soft file to use the Intel version of MPICH-VMI:

# don't include the default MPICH version
@remove +mpich-gm-intel     

# add implementation/version we want
+mpich-vmi-intel

# add in the rest of the default software environment
@teragrid-basic

# TeraGrid wide Globus 4 and Grid software suite
@globus-4.0

# Platform recommended development software suite
@teragrid-dev

2.5 MPICH Compiler Commands

All the implementations of MPICH on the NCSA TeraGrid cluster provide compiler wrapper scripts (mpicc/mpiCC/mpif77/mpif90) that automatically include all the necessary include and library directories and MPICH libraries.

Language	Command	Example
C	mpicc	mpicc mympi.c -o mympi
C++	mpiCC	mpiCC mympi.cpp -o mympi
F77	mpif77	mpif77 mympi.f -o mympi
F90 (Intel only)	mpif90	mpif90 mympi.f90 -o mympi

If a makefile needs the MPICH include and library paths, the following command will print helpful information:

soft-dbq -a `/bin/whatami` <MPICH softenv key>

3. Intel Math Kernel Library

The Math Kernel Library (MKL) from Intel contains:

Basic linear algebra subprograms (BLAS) with additional sparse routines.
Fast Fourier Transforms (FFT) in 1 and 2 dimensions, complex and real.
The linear algebra package, LAPACK
A C interface to BLAS
Vector Math Library (VML)
Vector Statistical Library (VSL)
Multi-dimensional Discrete Fourier Transforms (DFTs)

To use MKL, add the following to the end of your .soft file:

+intel-mkl

Documentation is available in HTML and PDF formats in ${MKL_HOME}/doc.

The libraries for Itanium processors are installed under ${MKL_HOME}/lib/64.

To link the MKL libraries dynamically:

For blas only: -L${MKL_HOME}/lib/64 -lmkl -lguide -lpthread
For lapack and blas: -L${MKL_HOME}/lib/64 -lmkl_lapack64 -lmkl -lguide -lpthread

To link the MKL libraries statically:

For blas only: -L${MKL_HOME}/lib/64 -lmkl_ipf -lguide -lpthread
For lapack and blas: -L${MKL_HOME}/lib/64 -lmkl_lapack -lmkl_ipf -lguide -lpthread

Notes:

Some MKL routines can run in parallel (using OpenMP) if the environment variable OMP_NUM_THREADS is set to the number of threads. OMP_NUM_THREADS is not set by default.
To force Intel MKL to serial mode, environment variable MKL_SERIAL should be set to YES. It works regardless of OMP_NUM_THREADS value. MKL_SERIAL is not set by default.
If you get unresolved symbols when calling MKL routines from C/C++ programs with the non-default Intel 7.1 compilers you will also need to link with:
-lPEPCF90 -lCEPCF90 -lF90 -lintrins.

4. Hierarchical Data Format (HDF) Library

HDF is a library and platform independent data format for the storage and exchange of scientific data. It includes Fortran and C calling interfaces, and utilities for analyzing and converting HDF data files. There are two HDF formats, HDF (4.x and previous releases) and HDF5. These formats are completely different and NOT compatible. Use SoftEnv for information on accessing the software.

Information on support is available at the HDF Support Issues page. The HDF Home Page has detailed information on HDF and HDF5, including documentation, tutorials, examples, and FAQs.