Debugging on Cobalt

gdb

[classic serial debuggers]

The GNU gdb debugger is available and may be used with serial programs or core files from serial or parallel programs. For information on  using gdb see the online man page and the  gdb user manual. 

serial or openmp/threaded

best debugger for gnu compilers

can attach to running processes

gdb sometimes has difficulty with Intel compiled code

The ddd graphical interface for gdb is available on the login nodes. ddd can be used with c, c++, fortran, and perl source code. See the ddd user guide for more information and examples.

The Intel debugger idb is installed along with the Intel compilers. For more information on the Intel debugger, see the idb man page. idb is similar to gdb in operation and it will recognize most gdb commands if started with the -gdb flag. idb works well with C, C++, and Fortran codes.

idb provides a graphical interface when started with the -gui flag [example idb -gui screenshot].

The Totalview debugger [with graphical user interface] works with the supported MPI environment. It is our recommended debugger for MPI code.

Information on starting Totalview on Cobalt (and other NCSA platforms) can be found here.

parallel, serial, or openmp/threaded

The Electric Fence malloc debugger is installed on the login hosts.  It can debug malloc() and pointer related bugs in c/c++ code. Electric Fence does not work simply with the default MPI environment, but it can be used with any of the mpich-tcp MPI environments available via softenv.  Electric fence slows execution and should only be used when debugging.

C/C++ programs using malloc(), calloc(), or realloc() can set the  MALLOC_CHEC_ env. variable.  From "man malloc": If MALLOC_CHECK_ is set to 0, any detected heap corruption is silently ignored; if set to 1, a diagnostic is printed on stderr; if set to 2, abort() is called immediately.

These libraries can be linked with your program to provide runtime checks for common MPI programming problems and MPI deadlock detection. This debugging aid can scale with your MPI application to the maximum number of processes you can employ. It's a good option when bugs appear when running at scalle

The MPICH2 MPI environment is installed and supports a text mode gdb interface. This debugging setup has been tested at large scale [> 200 processes] and is the only interactive debugging option available for large scale runs.

The techniques for trapping floating point exceptions [which are not always bugs] vary by compiler and operating system.  Since each case is a little different, see the link at left for the examples that match your situation.  Fortran compilers tend to have a more straightforward approach to floating point exceptions than c compilers.

Splint [for C code] and ftnchek [Fortran] are available. While not true debuggers, source code analysis tools can be very helpful when trying to track down a program bug. They can also help you write clean maintainable code by providing useful feedback about coding style, unused variables, non-portable practices, etc. Don't be alarmed by the number of warnings generated by these tools; they are designed to detect a great variety of potential problems.

Strace produces a system call trace for any program you can run.

Each line in the trace contains the system call name as used by your program, followed by its arguments in parentheses and its return value.

If you know what sort of system call may be failing, strace can be quite powerful.

For more information, see:

man strace