IBM LoadLeveler for AIX 5L: Using and Administering

LoadLeveler 2.2 and 3.1 coexistence

LoadLeveler 3.1 can coexist with LoadLeveler 2.2. It is possible to operate a "mixed" LoadLeveler cluster consisting of some machines running the new 3.1 software while the remaining machines continue to run LoadLeveler 2.2. The operation of this mixed cluster is subject to a number of restrictions. Most of these restrictions exist because LoadLeveler 3.1 has new features that are not supported by LoadLeveler 2.2. Other restrictions are due to the incompatibilities of the Parallel Environment software levels that support LoadLeveler 3.1 and LoadLeveler 2.2.

The requirements, restrictions, and operating characteristics of a mixed LoadLeveler cluster include:

• The Central Manager must run on a LoadLeveler 3.1 machine.
• Machines running version 2.2 software must have LoadLeveler service PTF 2.2.0.10 (U478841) or later installed.
• The job command, configuration, and administration files on all machines in the cluster should not use any keywords that are new and unique to version 3.1.
• Starting with LoadLeveler 3.1, 64-bit integer values can be assigned to a number of LoadLeveler variables and keywords that previously accepted only 32-bit integers. To avoid compatibility problems in a mixed cluster, values outside the range of a 32-bit integer should not be associated with these variables and keywords.
• Checkpointing can not be enabled in a mixed cluster. LoadLeveler 3.1 and 2.2 use different methodologies to provide support for checkpointing applications and they are not compatible.
• The LoadLeveler interface to AIX Workload Manager (WLM) should not be enabled in a mixed cluster.
• In a mixed cluster, the configuration keyword SCHEDULER_TYPE should not be set to GANG. GANG scheduling is supported only when all machines in a cluster run LoadLeveler 3.1 software.
• In a mixed cluster, the llacctmrg and llsummary commands should be run only on a 3.1 machine. The llacctmrg command is normally used by a LoadLeveler administrator to merge the job accounting information in the history files of all the schedd machines into a single global history file. This file is then used by the llsummary command to generate a cluster-wide accounting report. History files of 3.1 machines contain 64-bit integers. Since the llsummary command of LoadLeveler 2.2 does not support 64-bit integer data type, the llacctmrg and llsummary commands should not be executed on machines running LoadLeveler 2.2.
• No special action is required by LoadLeveler administrators to provide support for serial jobs. Jobs submitted from a 2.2 machine can be executed on a 3.1 machine and jobs submitted from a 3.1 machine can run on a 2.2 machine.
• If support for parallel POE jobs is required, administrators must be aware that LoadLeveler uses Parallel Environment for parallel job submission, and that the PE software requires the same level of PE to be used throughout the parallel job. Different levels of PE cannot be mixed. PE 3.2 supports only LoadLeveler 3.1, and PE 3.1 supports only LoadLeveler 2.2. Therefore, a POE parallel job can not run some of its tasks on LoadLeveler 2.2 machines and the remaining tasks on LoadLeveler 3.1 machines.

  The requirements keyword of the job command file can be used to ensure that all the tasks of a POE job run the same levels of PE and LoadLeveler software in a mixed cluster. Here are three examples showing different ways this can be done:

  1. If the statement "# @ requirements = (LL_Version >= "3.1") && (OpSys == "AIX51")" is included in the job command file, LoadLeveler's Central Manager will select only 3.1.0.0 or higher machines with the appropriate OpSys level for this job step. The requirements expression should contain the OpSys specification because the llsubmit command automatically adds the OpSys of the submitting machine to the other job requirements unless an OpSys requirement has already been explicitly specified.
  2. The tasks of a POE job will see a consistent environment if a statement such as "# @ requirements = (Machine == { "hostname1" "hostname2" }) && (OpSys == "AIX51")" is specified and "hostname1" and "hostname2" run the same levels of PE and LoadLeveler software.
  3. If the mixed cluster has been partitioned into 2.2 and 3.1 LoadLeveler pools then requirement specifications such as "# @ requirements = (Pool == 31) && (OpSys == "AIX51")" or "# @ requirements = (Pool == 22) && (OpSys == "AIX43")" can be used to select machines running the same levels of software. Here, it is assumed that all the 2.2 machines in this mixed cluster are assigned to pool 22 and all 3.1 machines are assigned to pool 31. A LoadLeveler administrator can use the "pool_list" keyword of the machine stanza of the LoadLeveler administration file to assign machines to pools.

  If a statement such as "# @ executable = /bin/poe" is specified in a job command file, and if the job is intended to be run on 3.1 machines, then it is important that the job be submitted from a 3.1 machine. When the "executable" keyword is used, LoadLeveler will copy the associated binary on the submitting machine and send it to a running machine for execution. In this example, the POE program will fail if the submitting and the running machines are at different software levels. In a mixed cluster, this problem can be circumvented by not using the "executable" keyword in the job command file. By omitting this keyword, the job command file itself is the shell script that will be executed. If this script invokes a local version of the POE binary then there is no compatibility problem at run time.