1) modification of job script to save files to UniTree before the wall clock limit is reached:

To save files to UniTree before the wall clock limit is reached, you will need to set a soft wall clock limit as well as a hard limit. When the soft wall clock limit is reached, the batch system will send a XCPU signal to your script. Your script can catch the signal and run "msscmd" to save the files before the hard limit is reached. See the following example scripts in /usr/local/doc/ll/advanced:

mpi_wallclock.ll      openmp_wallclock.ll   serial_wallclock.ll

2) using job steps with LoadLeveler

Job steps let you combine multiple execution runs into a single script and introduce some additional flexibility for running your jobs.  Think of using job steps when you're running sets of jobs while varying input files, or number of processors.  The example files in /usr/local/doc/ll/advanced show some possible uses of scripts with multiple job steps.  The files containing "_min" in their name show our most abbreviated samples for that job type.

mpi_steps.ll          openmp_steps.ll       serial_steps.ll
mpi_steps_min.ll      openmp_steps_min.ll

See section 4.1 of the Workload Management with LoadLeveler book for more information about job steps and additional sample usage.