IBM pSeries 690 Message Passing FAQ

• Where are the commands to compile my MPI program (mpicc, mpif90, etc.)?
• Where is the mpirun command?
• Why can't I use the -procs option on poe to run on different numbers of processors in a LoadLeveler job?
• Why do I get "ERROR: 0031-309 Connect failed during message passing initialization...There is not enough memory available now" when I run my MPI program?

Where are the commands to compile my MPI program (mpicc, mpif90, etc.)?

The usual commands for compiling MPI programs (mpicc, mpif77, mpif90, etc.) are not available on the p690. To find out how to compile your MPI program, see NCSA's p690 Programming Environment web page.

Where is the mpirun command?

The mpirun command is not available on the p690. To run an MPI job, you must use poe. See the poe man page and the Running Jobs web page for details.

Why can't I use the -procs option on poe to run on different numbers of processors in a LoadLeveler job?

When poe is used in a LoadLeveler job, the following POE environment variables, and associated command line options, are validated, but not used, for batch jobs submitted using LoadLeveler.
• MP_PROCS
• MP_RMPOOL
• MP_EUIDEVICE
• MP_EUILIB
• MP_MSG_API (except for programs that use LAPI and also use the LoadLeveler requirements keyword to specify Adapter="hps_user")
• MP_HOSTFILE
• MP_SAVEHOSTFILE
• MP_RESD
• MP_RETRY
• MP_RETRYCOUNT
• MP_ADAPTER_USE
• MP_CPU_USE
• MP_NODES
• MP_TASKS_PER_NODE
poe uses the LoadLeveler directive "tasks_per_node" to determine how many MPI processes to run. To run the same program on different numbers of processors, you need to submit multiple jobs or use job steps.

Why do I get "ERROR: 0031-309 Connect failed during message passing initialization...There is not enough memory available now" when I run my MPI program?

The environment variable $MP_BUFFER_MEM specifies the size of the buffer used for early arrivals. The default setting is 64m (64 Mbytes). If the ConsumableMemory is set too small on a batch job such that it does not include extra memory for MPI buffers, the job will fail with this error message. Change the ConsumableMemory to be at least 200 Mbytes more than you expect your program to use.

Or if you know how big of a MPI buffer your program will need, you can try setting the environment variable MP_BUFFER_MEM to 32m, 16m, or 8m.