IBM LoadLeveler for AIX 5L: Using and Administering

Step 14: Enable checkpointing

This section tells you how to set up checkpointing for jobs. Checkpointing is a method of periodically saving the state of a job step so that if the step does not complete it can be restarted from the saved state. When checkpointing is enabled, checkpoints can be initiated from within the application at major milestones, or by the user, administrator or LoadLeveler external to the application. Both serial and parallel job steps can be checkpointed.

Once a job step has been successfully checkpointed, if that step terminates before completion, the checkpoint file can be used to resume the job step from its saved state rather than from the beginning. When a job step terminates and is removed from the LoadLeveler job queue, it can be restarted from the checkpoint file by submitting a new job and setting the restart_from_ckpt = yes job command file keyword. When a job is terminated and remains on the LoadLeveler job queue, such as when a job step is vacated, the job step will automatically be restarted from the latest valid checkpoint file. A job can be vacated as a result of flushing a node, issuing checkpoint and hold, stopping or recycling LoadLeveler or as the result of a node crash.

Checkpoint keywords

The following is a summary of keywords associated with the checkpoint and restart function.

Administration file keyword summary

Table 21. Administration file keyword summary

Configuration file keyword summary

Table 22. Configuration file keyword summary

Job command file keyword summary

Table 23. Job command file keyword summary

Naming checkpoint files and directories

At checkpoint time, a checkpoint file and potentially an error file will be created. For jobs which are enabled for checkpoint, a control file may be generated at the time of job submission. The directory which will contain these files must pre-exist and have sufficient space and permissions for these files to be written. The name and location of these files will be controlled through keywords in the job command file or the LoadLeveler configuration. The file name specified is used as a base name from which the actual checkpoint file name is constructed. To prevent another job step from writing over your checkpoint file, make certain that your checkpoint file name is unique. For serial jobs and the master task (POE) of parallel jobs, the checkpoint file name will be <basename>.Tag. For a parallel job, a checkpoint file is created for each task. The checkpoint file name will be <basename>.Taskid.Tag.

The tag is used to differentiate between a current and previous checkpoint file. A control file may be created in the checkpoint directory. This control file contains information LoadLeveler uses for restarting certain jobs. An error file may also be created in the checkpoint directory. The data in this file is in a machine readable format. The information contained in the error file is available in mail, LoadLeveler logs or is output of the checkpoint command. Both of these files are named with the same base name as the checkpoint file with the extensions .cntl and .err, respectively.

See How to checkpoint a job for more information.

Naming checkpoint files for serial and batch parallel jobs

The following describes the order in which keywords are checked to construct the full path name for a serial or batch checkpoint file:

• Base name for the checkpoint file name
  1. The ckpt_file keyword in the job command file
  2. The default file name [< jobname.>]<job_step_id>.ckpt

     Where:

     jobname

     The job_name specified in the Job Command File. If job_name is not specified, it is omitted from the default file name

     job_step_id

     Identifies the job step that is being checkpointed

• Checkpoint Directory Name
  1. The ckpt_file keyword in the job command file, if it contains a "/" as the first character
  2. The ckpt_dir keyword in the job command file
  3. The ckpt_dir keyword specified in the class stanza of the LoadLeveler admin file
  4. The default directory is the initial working directory

Note that two or more job steps running at the same time cannot both write to the same checkpoint file, since the file will be corrupted.

Naming checkpointing files for interactive parallel jobs

The following describes the order in which keywords and variables are checked to construct the full path name for the checkpoint file for an interactive parallel job.

• Checkpoint File Name
  1. The value of the MP_CKPTFILE environment variable within the POE process
  2. The default file name, poe.ckpt.<pid>
• Checkpoint Directory Name
  1. The value of the MP_CKPTFILE environment variable within the POE process, if it contains a full path name.
  2. The value of the MP_CKPTDIR environment variable within the POE process.
  3. The initial working directory.