IBM LoadLeveler for AIX 5L: Using and Administering

The startd daemon

The startd daemon monitors jobs and machine resources on the local machine and forwards this information to the negotiator daemon. The startd also receives and executes job requests originating from remote machines. The master daemon starts, restarts, signals, and stops the startd daemon.

The startd daemon can be in any one of the following states:

Busy

The maximum number of jobs are running on this machine.

Down

The daemon is not running on this machine. The startd daemon enters this state when it has not reported its status to the negotiator. This can occur when the machine is actually down, or because there is a network failure.

Drained

The startd machine will not accept any new jobs. However, any jobs that are already running on the startd machine will be allowed to complete.

Draining

The startd daemon has been drained by the administrator, but some jobs are still running. The machine remains in the draining state until all of the running jobs have completed, at which time the machine status changes to drained. The startd daemon will not accept any new jobs while in the draining state.

Flush

Any running jobs have been vacated (terminated and returned to the queue to be redispatched). The startd daemon will not accept any new jobs.

Idle

The machine is not running any jobs.

None

LoadLeveler is running on this machine, but no jobs can run here.

Running

The machine is running one or more jobs and is capable of running more.

Suspend

All LoadLeveler jobs running on this machine are stopped (cease processing), but remain in virtual memory. The startd daemon will not accept any new jobs.

The startd daemon performs these functions:

• Runs a time-out procedure that includes building a snapshot of the state of the machine that includes static and dynamic data. This time-out procedure is run at the following times:
  • After a job completes.
  • According to the definition of the POLLING_FREQUENCY keyword in the configuration file.
• Records the following information in LoadLeveler variables and sends the information to the negotiator. These variables are described in LoadLeveler variables.
  • State (of the startd daemon)
  • EnteredCurrentState
  • Memory
  • Disk
  • KeyboardIdle
  • Cpus
  • LoadAvg
  • Machine
  • Adapter
  • AvailableClasses
• Calculates the SUSPEND, RESUME, CONTINUE, and VACATE expressions. These are described in Step 8: Manage a job's status using control expressions.
• Receives job requests from the schedd daemon to:
  • Start a job
  • Vacate a job
  • Cancel

  When the schedd daemon tells the startd to start a job, the startd determines whether its own state permits a new job to run:
  

  If:	Then this happens:
  Yes, it can start a new job	The startd forks a starter process.
  No, it cannot start a new job	The startd rejects the request for one of the following reasons: Jobs have been suspended, flushed, or drained The job limit set for the MAX_STARTERS keyword has been reached There are not enough classes available for the designated job class

• Receives requests from the master (via llctl) to do one of the following:
  • Drain
  • Flush
  • Suspend
  • Resume.
• For each request, startd marks its own new state, forwards its new state to the negotiator daemon, and then performs the appropriate action for any jobs that are active.
• Receives notification of keyboard and mouse activity from the kbdd daemon
• Periodically examines the process table for LoadLeveler jobs and accumulates resources consumed by those jobs. This resource data is used to determine if a job has exceeded its job limit and for recording in the history file.
• Send accounting information to schedd.

The starter process

The startd daemon spawns a starter process after the schedd daemon tells the startd to start a job. The starter process manages all the processes associated with a job step. The starter process is responsible for running the job and reporting status back to startd.

The starter process performs these functions:

• Processes the prolog and epilog programs as defined by the JOB_PROLOG and JOB_EPILOG keywords in the configuration file. The job will not run if the prolog program exits with a return code other than zero.
• Handles authentication. This includes:
  • Authenticates AFS, if necessary
  • Verifies that the submitting user is not root
  • Verifies that the submitting user has access to the appropriate directories in the local file system.
• Runs the job by forking a child process that runs with the user id and all groups of the submitting user. The starter child creates a new process group of which it is the process group leader, and executes the user's program or a shell. The starter parent is responsible for detecting the termination of the starter child. LoadLeveler does not monitor the children of the parent.
• Responds to vacate and suspend orders from the startd.