User Information Home
Data
Security
Allocations
Consulting
Training

mpiP at NCSA

The mpiP library for MPI profiling is available on Forge (version 3.3).

mpiP is an MPI profiling library originally from Lawrence Livermore National Laboratory that provides an easy-to-use way of measuring overall message-passing statistics for your MPI application. mpiP is not a tracing or logging library that records information about each individual message sent between processors: instead, it collects statistics about communication volume and wall clock time spent within MPI for your application. Additionally, mpiP records the call site information for MPI calls - this allows you to identify the exact spot within your application where measured MPI time occurred and can be very helpful to isolate the areas of your application that are dominated by communication time.

The output of mpiP is a single text file with the suffix ".mpiP" written from MPI task 0 at the time your application calls MPI_Finalize(). Please refer to the mpiP documentation listed below or to the mpiP project web site for examples of the information that the mpiP output file contains.

---

Basic Use at NCSA

mpiP is installed on NCSA's Forge cluster in the directory:

/usr/apps/tools/mpip/3.3-openmpi-forge (for OpenMPI)
/usr/apps/tools/mpip/3.3-mvapich2-1.7rc1-intel-forge (for MVAPICH2)

To use mpiP, you'll need to relink your application with the mpiP profiling library. You'll also need to include the GNU libbfd and libunwind libraries that mpiP uses.

To link your program with mpiP on Forge, you should include the following in your link line and add /usr/apps/tools/unsupported/libunwind/1.0.1-forge/lib in your LD_LIBRARY_PATH environment variable:

-L/usr/apps/tools/unsupported/libunwind/1.0.1-forge/lib -lunwind

If you'd like mpiP to translate program addresses to source code locations within your application, you'll want to include the option "-g" when you compile your program. If you don't include this option, mpiP will still work, but will only display raw program counter values where MPI calls occurred.

Here are example link lines that can be used to link the C program myprog for mpiP profiling:

Forge OpenMPI

mpicc -g -c myprog.c
mpicc -o myprog myprog.o \
        -L/usr/apps/tools/mpip/3.3-openmpi-forge/lib -lmpiP -lbfd \
        -L /usr/apps/tools/unsupported/libunwind/1.0.1-forge/lib -lunwind

Forge MVAPICH2

mpicc -g -c myprog.c
mpicc -o myprog myprog.o \
        -L/usr/apps/tools/mpip/3.3-mvapich2-1.7rc1-intel-forge/lib -lmpiP -lbfd \
        -L /usr/apps/tools/unsupported/libunwind/1.0.1-forge/lib -lunwind

Once you have linked your program with mpiP, you can run your application as you normally would, using mpirun.

Current status: on Forge, the stack unwinding and source code mapping to file name and line number work well with OpenMPI for both gcc and icc (the modules "openmpi_gcc-1.4.2" and "openmpi-1.4.3-intel-12.0.4"); however, for MVAPICH2, it could not successfully perform the source code mapping, so the other info are there, but the file name and line number are not available.

---

To Learn More

The mpiP project web site provides a great deal of information about the software as well as examples of use. You can also find documentation in the subdirectory doc within the mpiP installation directory.