Fullauto Utility
Fullauto is a command line utility that automatically generates job scripts for Master Control Program (MCP), an automatic resource selection utility. Fullauto simplifies the process of submitting a job to multiple clusters through MCP by allowing users to specify and store job and machine attributes. After learning those attributes, it selects the appropriate clusters for the job and generates job scripts for each cluster. It then uses MCP to run the job on the earliest available cluster. The user defined attributes are kept in the autojob.py module
Locations
Currently fullauto is installed at: IA-64 clusters at NCSA (Mercury) and SDSC.
Fullauto Workflow
-
User runs grid-proxy-init or myproxy-get-delegation to establish grid credential.
grid-proxy-init
OR
myproxy-get-delegation * -
User constructs an appropriate autojob.py module file
with personalized settings. (See example
autojob.py files) User should create a working directory and place autojob.py in it.
vi autojob.py
- User runs fullauto.py with autojob.py as the input.
fullauto.py <directory containing autojob.py>
fullauto.py --attributes **
- Fullauto imports the autojob.py module to learn which attributes to it should use to select remote clusters. It then creates job scripts for each selected cluster.
- Fullauto uses MCP to run the scripts.
Notes
* Use myproxy-get-delegation to establish your grid credential using TeraGrid Single Sign-On with your TeraGrid-Wide (TeraGrid User Portal) password.
** The name of the module must be autojob.py.
The fullauto program launches the job on any and all available queues. For example, at NCSA it might submit three jobs (fastqueue, highqueue, compute) depending on the requirements. When a job in one queue starts, the others will be killed off. This behaviour is a consequence of the heterogenous nature of the clusters. NCSA IA-64 is a mix of 1300 and 1500 MHz nodes; if a user specifies a particular MHz, then only jobs appropriate to that queue would be submitted.
Fullauto calls MCP; the user does not have to run MCP separately; however, MCP can be used to select resources automatically using manually generated scripts. See the MCP page for more information.
Since there may be a long delay before job start, use the screen utility to run MCP sessions in the background (See example commands).
To see all attributes of autojob.py, including machines and queues where fullauto/MCP will run, use --attributes.
fullauto.py --attributes
