NCSA HPC Systems: Transition to TeraGrid

UPDATE March 27 2006:
The installation of new TeraGrid software has been moved back. Therefore, there will be no changes in the current environment on the NCSA HPC systems at this time. This means that details of Grid Services (including File Transfer) outlined in Section 3 below is not yet in place.

Information about the status and testing of new Grid Services on NCSA HPC systems is being made available at Globus 4.0 Software and Service Testing as they become ready.

Overview
Impact for current users of copper and tungsten
New TeraGrid Features and Capabilities
Useful Links

1. Overview

As announced in NCSA Access News (12.20.05), all NCSA HPC systems will be available as TeraGrid resources as of April 1, 2006. This will add NCSA's IBM p690 (copper), Xeon Cluster (tungsten), and Condor Pool (radium) to TeraGrid. NCSA's IA-64 Linux Cluster (mercury) and SGI Altix (cobalt) have already been available as TeraGrid resources.

All existing software and functionality will remain intact on copper and tungsten (with the exception of Grid Services, which will be replaced by a new version; see the Grid Services section for details). Access to the NCSA Mass Storage System (UniTree) will remain the same. See Section 2 for the impact for current users of copper and tungsten. The systems gain some new features as they become part of the TeraGrid. These are outlined in Section 3.

There will be no changes on radium on April 1, 2006.

2. Impact for current users of copper and tungsten

2.1 Allocations and Accounting

Allocation Balance Change

All allocations that continue after April 1, 2006 will have their Service Unit (SU) balances transferred to a new TeraGrid version of the machine, and this new machine will have that balance as the new initial allocation, rounded up to the next thousand SU. Under TeraGrid, machine copper will be represented as tg_copper and machine tungsten will be represented as tg_tungsten. The 3-letter NCSA project (PSN) will remain the same. This will be reflected in the NCSA usage command.

IMPORTANT NOTE: The transfer of balances will occur over a 3 day period starting Wednesday March 29, 2006 through Friday March 31, 2006. Users will see the new TeraGrid machine and balances in usage as their projects are processed over this period.

As a hypothetical example, Principal Investigator John Smith was awarded an allocation of 100,000 SUs on copper in January 2006, of which 27,820 SUs have been used. When the balances are transferred, the usage command will show:

$ usage -p xyz
Project_title: Simulations of Black Holes
Principal_Investigator: Smith, John

 Proj     Mach    Login   Usage  Status P_alloc P_usage P_expire Name 
  xyz   copper   jsmith  5301.38 Deleted 100000 27820.12 12/31/06 Smith, John
  xyz   copper    lahti  8173.31 Deleted 100000 27820.12 12/31/06 Lahti, Tracy 
  xyz   copper   hjiang 14345.43 Deleted 100000 27820.12 12/31/06 Jiang, Hong
  xyz tg_copper  jsmith     0.00  Active  73000     0.00 12/31/06 Smith, John  
  xyz tg_copper   lahti     0.00  Active  73000     0.00 12/31/06 Lahti, Tracy
  xyz tg_copper  hjiang     0.00  Active  73000     0.00 12/31/06 Jiang, Hong

Impact for current users on copper and tungsten whose allocations are up for renewal April 1, 2006

All renewal allocations starting April 1, 2006 will be considered new TeraGrid allocations, which will include a courtesy login (referred to as a Roaming Allocation) on all TeraGrid systems for access to Data Resources in the TeraGrid.

Checking your Allocation Balances

The NCSA usage command will continue to be available for account and usage information on NCSA systems only. The TeraGrid usage command tgusage will list usage and account information for copper and tungsten beginning April 1.

2.2 Environment

The default environment on copper and tungsten will remain the same. You will be able to continue computing on these systems without any changes. The default environment provides what you need to use the local system, and does not include TeraGrid specific tools and services needed to use multiple systems. To add TeraGrid tools and services, see the SoftEnv section below.

2.3 User Support

The User Support structure remains the same. As before, please contact NCSA Support Services for NCSA-specific issues. For TeraGrid-wide or cross-system issues, please contact TeraGrid Support.

NCSA Consulting Services

NCSA Consulting Services provides software and applications support for users of NCSA's high-performance systems. To contact Consulting Services, send electronic mail to consult@ncsa.uiuc.edu anytime or phone (217) 244-1144 during the hours of 8:30 a.m. - 5 p.m. CT, Monday through Friday.

NCSA Helpdesk

The NCSA Helpdesk provides resolution to all system-related problems within the Center's computational environment. To reach the Helpdesk, send electronic mail to help@ncsa.uiuc.edu. In an emergency, telephone the Helpdesk at (217) 244-0709. Helpdesk email and phones are monitored 24 hours a day, 365 days a year.

TeraGrid Support

TeraGrid wide support is available via the TeraGrid Helpdesk at help@teragrid.org; details are at TeraGrid Helpdesk.

3. New TeraGrid Features and Capabilities

3.1 System Names

The switch to TeraGrid necessitates defining new TeraGrid hostnames for the systems. The current NCSA hostnames will continue to work.

System	NCSA Hostname	new TeraGrid Hostname
IBM p690 (copper)	cu.ncsa.uiuc.edu	login-cu.ncsa.teragrid.org
Xeon Cluster (tungsten)	tun.ncsa.uiuc.edu	login-w.ncsa.teragrid.org
Condor Pool (radium)	radium.ncsa.uiuc.edu	login-ra.ncsa.teragrid.org

3.2 Grid Services

The Globus Toolkit™ (version 2.4.3) deployment will be upgraded to GT™ 4.0.1 as of April 1, 2006. The following services will be affected by the upgrade.

GRAM (Globus Gatekeeper - Job submission) service:
- Version 2.4.3 will be replaced with a newer version of the same implementation. The user interface will be backward compatible.
- A new Web Services implementation will also be available with a new user interface
GridFTP service:
- The existing GridFTP server will be replaced with new servers on each system each providing 3rd party transfer capability, and other enhancements such as striping, and parallel streams.
- The user interface will be backward compatible
- A server will be running on a set of dedicated hosts for tungsten.
  - Use gridftp-w.ncsa.teragrid.org to target GridFTP servers for tungsten
- A server will be running on a set of dedicated ports for copper.
  - Use gridftp-cu.ncsa.teragrid.org to target GridFTP servers for copper
MDS service:
- The existing implementation will no longer be supported and is being replaced with a new implementation called MDS4
- The new implementation will require use of the new command wsrf-query to replace the grid-info-search command.

All existing GT™ commands will be replaced with those distributed with the GT4.0.1.

To access the new Grid Services, refer to the SoftEnv section below.

3.3 File Transfer

As noted in the Grid Services section above, copper and tungsten (along with mercury and cobalt) will each be outfitted with a set of dedicated GridFTP servers and the full compliment of accompanying TeraGrid tools including tgcp; a GridFTP file-transfer utility. The GridFTP protocol enables many performance enhancements such as parallel streams and striped transfers. Having a dedicated hardware resource on each system will enable NCSA to optimize local filesystem access and improve network throughput to/from the HPC system.

File transfer using the GridFTP servers is particularly beneficial when moving large files from one scratch file system to another. The tgcp utility integrates the Globus command line GridFTP client (globus-url-copy) with TeraGrid-specific optimization parameters and optionally, the database-powered Reliable File Transfer (RFT) file-tracking service.

See the document Moving Data Fast on the TeraGrid for details on tgcp.

To access tgcp, refer to the SoftEnv section below.

3.4 SoftEnv

All NCSA systems use the SoftEnv system for users to manage their environment. The default environment on NCSA systems is designated by the @default macro entry in a user's $HOME/.soft file.

The basic Common TeraGrid Software Stack (CTSS) will be available April 1 via the @teragrid-basic macro in SoftEnv. All softenv keys associated with Grid services (including file transfer tools) will be defined as part of the @globus macro and are intended to be a set of supplemental keys to add grid capabilities as defined by the CTSS.

See the SoftEnv documentation for details on use of SoftEnv. The SoftEnv command

soft-dbq @macro

will provide the environment set by @macro.

Users interested in using CTSS can modify their $HOME/.soft file below the @default as follows:

     #
     # This is the .soft file.
     # It is used to customize your environment by setting up environment
     # variables such as PATH and MANPATH.
     # To learn what can be in this file, use 'man softenv'.
     #
     #
     @default
     @teragrid-basic
     @globus

Individual keys for CTSS will also be available for anyone interested in using only a subset of CTSS components in addition to their default environment.