IBM Books

IBM LoadLeveler for AIX 5L: Using and Administering

Step 12: Record and control log files

The LoadLeveler daemons and processes keep log files according to the specifications in the configuration file. A number of keywords are used to describe where LoadLeveler maintains the logs and how much information is recorded in each log. These keywords, shown in Table 20, are repeated in similar form to specify the pathname of the log file, its maximum length, and the debug flags to be used.

Controlling debugging output describes the events that can be reported through logging controls.

Setting up file system monitoring describes keywords that monitor the available free space on the file system and also enable system responses to increase system fault tolerance.

Table 20. Log control statements

Daemon/ Process Log File (required)

(See note (PAT))

Max Length (required)

(See note (MXL))

Debug Control (required)

(See note (FLA))

Master MASTER_LOG = path MAX_MASTER_LOG = bytes MASTER_DEBUG = flags
Schedd SCHEDD_LOG = path MAX_SCHEDD_LOG = bytes SCHEDD_DEBUG = flags
Startd STARTD_LOG = path MAX_STARTD_LOG = bytes STARTD_DEBUG = flags
Starter STARTER_LOG = path MAX_STARTER_LOG = bytes STARTER_DEBUG = flags
Negotiator NEGOTIATOR_LOG = path MAX_NEGOTIATOR_LOG = bytes NEGOTIATOR_DEBUG = flags
Kbdd KBDD_LOG = path MAX_KBDD_LOG = bytes KBDD_DEBUG = flags
GSmonitor GSMONITOR_LOG = path MAX_GSMONITOR_LOG = bytes GSMONITOR_DEBUG = flags

Notes:

  1. When coding the path for the log files, it is not necessary that all LoadLeveler daemons keep their log files in the same directory, however, you will probably find it a convenient arrangement.

  2. There is a maximum length, in bytes, beyond which the various log files cannot grow. Each file is allowed to grow to the specified length and is then saved to an .old file. The .old files are overwritten each time the log is saved, thus the maximum space devoted to logging for any one program will be twice the maximum length of its log file. The default length is 64KB. To obtain records over a longer period of time, that don't get overwritten, you can use the SAVELOGS keyword in the local or global configuration files. SeeSaving log files for more information on extended capturing of LoadLeveler logs.

    You can also specify that the log file be started anew with every invocation of the daemon by setting the TRUNC statement to true as follows:

    TRUNC_MASTER_LOG_ON_OPEN = true|false
    TRUNC_STARTD_LOG_ON_OPEN = true|false
    TRUNC_SCHEDD_LOG_ON_OPEN = true|false
    TRUNC_KBDD_LOG_ON_OPEN = true|false
    TRUNC_STARTER_LOG_ON_OPEN = true|false
    TRUNC_NEGOTIATOR_LOG_ON_OPEN = true|false
    TRUNC_GSMONITOR_LOG_ON_OPEN = true|false

  3. LoadLeveler creates temporary log files used by the starter daemon. These files are used for synchronization purposes. When a job starts, a StarterLog.pid file is created. When the job ends, this file is appended to the StarterLog file.

  4. Normally, only those who are installing or debugging LoadLeveler will need to use the debug flags, described in Controlling debugging output The default error logging, obtained by leaving the right side of the debug control statement null, will be sufficient for most installations.

Controlling debugging output

You can control the level of debugging output logged by LoadLeveler programs. The following flags are presented here for your information, though they are used primarily by IBM personnel for debugging purposes:

D_ACCOUNT
Logs accounting information about processes. If used, it may slow down the network.
D_AFS
Logs information related to AFS credentials.
D_CKPT
Logs information related to checkpoint and restart
D_DAEMON
Logs information regarding basic daemon set up and operation, including information on the communication between daemons.
D_DBX
Bypasses certain signal settings to permit debugging of the processes as they execute in certain critical regions.
D_DCE
Logs information related to DCE credentials.
D_EXPR
Logs steps in parsing and evaluating control expressions.
D_FULLDEBUG
Logs details about most actions performed by each daemon but doesn't log as much activity as setting all the flags.
D_HIERARCHICAL
Used to enable messages relating to problems related to the transmission of hierarchical messages.
Note:A hierarchical message is sent from an originating node to lower ranked receiving nodes.
D_JOB
Logs job requirements and preferences when making decisions regarding whether a particular job should run on a particular machine.
D_KERNEL
Activates diagnostics for errors involving the process tracking kernel extension.
D_LOAD
Displays the load average on the startd machine.
D_LOCKING
Logs requests to acquire and release locks.
D_MACHINE
Logs machine control functions and variables when making decisions regarding starting, suspending, resuming, and aborting remote jobs.
D_NEGOTIATE
Displays the process of looking for a job to run in the negotiator. It only pertains to this daemon.
D_NQS
Provides more information regarding the processing of NQS files.
D_PROC
Logs information about jobs being started remotely such as the number of bytes fetched and stored for each job.
D_QUEUE
Logs changes to the job queue.
D_STANZAS
Displays internal information about the parsing of the administration file.
D_SCHEDD
Displays how the schedd works internally.
D_STARTD
Displays how the startd works internally.
D_STARTER
Displays how the starter works internally.
D_THREAD
Displays the ID of the thread producing the log message. The thread ID is displayed immediately following the date and time. This flag is useful for debugging threaded daemons.
D_XDR
Logs information regarding External Data Representation (XDR) communication protocols.

For example, 

SCHEDD_DEBUG = D_CKPT  D_XDR

Causes the scheduler to log information about checkpointing user jobs and exchange xdr messages with other LoadLeveler daemons. These flags will primarily be of interest to LoadLeveler implementers and debuggers.

Saving log files

By default, LoadLeveler stores only the two most recent iterations of a daemon's log file (<daemon name>_Log, and <daemon name>_Log.old). Occasionally, for problem diagnosing, users will need to capture LoadLeveler logs over an extended period. Users can specify that all log files be saved to a particular directory by using the SAVELOGS keyword in a local or global configuration file. Be aware that LoadLeveler does not provide any way to manage and clean out all of those log files, so users must be sure to specify a directory in a file system with enough space to accommodate them. This file system should be separate from the one used for the LoadLeveler log, spool, and execute directories. The syntax is: 

SAVELOGS = <directory>

Where <directory> is the directory in which log files will be archived.

Each log file is represented by the name of the daemon that generated it, the exact time the file was generated, and the name of the machine on which the daemon is running. When you list the contents of the SAVELOGS directory, the list of log file names looks like this: 

NegotiatorLogNov02.16:10:39c163n10.ppd.pok.ibm.com
NegotiatorLogNov02.16:10:42c163n10.ppd.pok.ibm.com
NegotiatorLogNov02.16:10:46c163n10.ppd.pok.ibm.com
NegotiatorLogNov02.16:10:48c163n10.ppd.pok.ibm.com
NegotiatorLogNov02.16:10:51c163n10.ppd.pok.ibm.com
NegotiatorLogNov02.16:10:53c163n10.ppd.pok.ibm.com
StarterLogNov02.16:09:19c163n10.ppd.pok.ibm.com
StarterLogNov02.16:09:51c163n10.ppd.pok.ibm.com
StarterLogNov02.16:10:30c163n10.ppd.pok.ibm.com
SchedLogNov02.16:09:05c163n10.ppd.pok.ibm.com
SchedLogNov02.16:09:26c163n10.ppd.pok.ibm.com
SchedLogNov02.16:09:47c163n10.ppd.pok.ibm.com
SchedLogNov02.16:10:12c163n10.ppd.pok.ibm.com
SchedLogNov02.16:10:37c163n10.ppd.pok.ibm.com
StartLogNov02.16:09:05c163n10.ppd.pok.ibm.com
StartLogNov02.16:09:26c163n10.ppd.pok.ibm.com
StartLogNov02.16:09:47c163n10.ppd.pok.ibm.com
StartLogNov02.16:10:12c163n10.ppd.pok.ibm.com
StartLogNov02.16:10:37c163n10.ppd.pok.ibm.com

[ Top of Page | Previous Page | Next Page | Table of Contents | Index ]