User Information Home
Compute Resources
Software
Data
Security
Allocations
Consulting
Training
TeraGrid Advanced Application Support

mpiP at NCSA

The mpiP library for MPI profiling is available on Mercury (TeraGrid Itanium 2) cluster and the SGI Altix (Cobalt) cluster. Please note that the stack walking part (identification of callsite to file name and line number) of mpiP currently does not work on Cobalt yet.

mpiP is an MPI profiling library from Lawrence Livermore National Laboratory that provides an easy-to-use way of measuring overall message-passing statistics for your MPI application. mpiP is not a tracing or logging library that records information about each individual message sent between processors: instead, it collects statistics about communication volume and wall clock time spent within MPI for your application. Additionally, mpiP records the call site information for MPI calls - this allows you to identify the exact spot within your application where measured MPI time occurred and can be very helpful to isolate the areas of your application that are dominated by communication time.

The output of mpiP is a single text file with the suffix ".mpiP" written from MPI task 0 at the time your application calls MPI_Finalize(). Please refer to the mpiP documentation listed below or to the mpiP project web site for examples of the information that the mpiP output file contains.

---

Basic Use at NCSA

mpiP is installed on NCSA's Mercury cluster in the directory:

/usr/projects/perftools/mpip

mpiP is installed on NCSA's Cobalt cluster in the directory:

/usr/apps/tools/mpip

To use mpiP, you'll only need to relink your application with the mpiP profiling library. You should select the correct library based on the MPI implementation you are using by including one of the subdirectories lib/mpich or lib/vmi (for Mercury depending on which MPI implementation you are using), or lib/mpt (for Cobalt) in the library search path used at link time (-L option). You'll also want to include the GNU libbfd and libiberty libraries that mpiP uses. You do not need to include the library -lintl at NCSA. On Cobalt, you should also include the math library libm.

Note: call site identification on Mercury and Cobalt requires the Hewlett-Packard version of the library "libunwind".

To link your program with mpiP on Mercury, you should include the following library in your link line and you must use the full path name:

/usr/projects/perftools/libunwind/lib/libunwind.a

To link your program with mpiP on Cobalt, you should include the following library in your link line and you must use the full path name:

/usr/apps/tools/libunwind/lib/libunwind.a

If you'd like mpiP to translate program addresses to source code locations within your application, you'll want to include the option -g when you compile your program. If you don't include this option, mpiP will still work, but will only display raw program counter values where MPI calls occurred.

Here are example link lines that can be used to link the C program myprog for mpiP profiling:

Mercury MPICH (with H-P's libunwind)

mpicc -o myprog myprog.o \
	-L/usr/projects/perftools/mpip/lib/mpich -lmpiP -lbfd -liberty \
	/usr/projects/perftools/libunwind/lib/libunwind.a

Mercury VMI (with H-P's libunwind)

mpicc -o myprog myprog.o \
	-L/usr/projects/perftools/mpip/lib/vmi -lmpiP -lbfd -liberty \
	/usr/projects/perftools/libunwind/lib/libunwind.a

MPT (Cobalt with H-P's libunwind)

icc -o myprog myprog.o \
	-L/usr/apps/tools/mpip/lib/mpt -lmpiP -lbfd -liberty -lm \
	/usr/apps/tools/libunwind/lib/libunwind.a -lmpi

Once you have linked your program with mpiP, you can run your application as you normally would, using mpirun.

---

To Learn More

The mpiP project web site provides a great deal of information about the software as well as examples of use. You can also find documentation in the subdirectory doc within the mpiP installation directory.