NCSA NEWS

Terascale Computing System Installed at Pittsburgh Supercomputing Center

released October 2, 2001

Contacts
Michael Schneider
Sean Fulton
Pittsburgh Supercomputing Center
schneider@psc.edu
sfulton@psc.edu
412.268.4960

The most powerful computer system in the world for open research is up and running.

PITTSBURGH — The Terascale Computing System (TCS), the most powerful system in the world committed to unclassified research, is installed on schedule. Developed and implemented by the Pittsburgh Supercomputing Center in collaboration with Compaq Computer Corporation, with funding from the National Science Foundation, the TCS provides computational capability to scientists and engineers nationwide. They will use it in many areas of research that have wide social impact, including earthquake modeling, storm-scale weather forecasting, global climate change, and protein genomics, modeling that's integral to the development of new drug therapies.

A joint project of Carnegie Mellon University, the University of Pittsburgh and Westinghouse Electric Company, the Pittsburgh Supercomputing Center (PSC) has developed the TCS to fill a gap in U.S. basic research capability highlighted in a 1999 presidential report. Terascale means computational power beyond a "teraflop"—a trillion calculations per second. With peak capability of six teraflops, the new system is now by far the most powerful available as an open resource for researchers attacking a wide range of problems.

"The TCS system at Pittsburgh Supercomputing Center continues a history of National Science Foundation support for high-performance computing," said Robert Borchers, director of NSF's Division of Advanced Computational Infrastructure and Research. "Through the NSF's Partnerships for Advanced Computational Infrastructure (PACI) program, this system will increase long-term, fundamental research across all science and engineering disciplines."

"In scale alone, the TCS pushes beyond where open-resource supercomputing technology has been before or would have gone without the NSF PACI program," said PSC scientific directors Michael Levine and Ralph Roskies in a joint statement. "Compaq committed themselves to the success of the TCS and worked side-by-side with us to make it available on schedule. With storage capacity that's 100,000 times that of most PCs and with 10 million times the communications capability, this system brings significant new research capability to bear on many important problems. While the immediate, direct beneficiaries will be academic scientists, the benefits will flow to the country as a whole, in practical ways we can't forecast."

"Compaq is proud to join with the Pittsburgh Supercomputing Center and the National Science Foundation in delivering TCS," said Bill Blake, Compaq's Vice President of High Performance Technical Computing. "And we're excited that Compaq's AlphaServer SC supercomputing architecture will make such a significant contribution in opening new frontiers in scientific computing."

The TCS represents an unprecedented synthesis of "off-the-shelf" components integrated with an advanced interconnect—from Quadrics Supercomputers World—and other technologies to provide a very large-scale system for scientific computing. It comprises 3,000 Compaq Alpha EV68 microprocessors, housed in 750 four-processor AlphaServer systems running Tru64 UNIX. The latest evolution of the widely used Alpha microchip technology, the EV68 has peak floating-point capability of two gigaflops (two billion calculations per second).

Along with six teraflops of processing power, the TCS features 3.0 terabytes of memory, high-bandwidth, low-latency interconnections and remarkable capabilities for large-scale data handling, including the ability to write the entire memory to disk in under 40 seconds. This extremely short system-write time, developed through PSC systems and software engineering, is critical to efficient checkpointing, needed to preserve research data in the event of component failure.

Preparation for the TCS began in October 2000 with installation of a 256-processor prototype system. In August 2001, the first of the new AlphaServer systems arrived at the PSC computer room at Westinghouse Energy Center in Monroeville, Pennsylvania. System components came in multiple deliveries from Compaq facilities in Texas and Scotland. An on-site team of Compaq, PSC and Westinghouse engineers and technicians—supported by expert teams at Compaq locations in the United States, Bristol, England and Galway, Ireland—worked aggressively to meet the Oct. 1 installation date.

"PSC's success in deploying this unprecedented, very large-scale system right on time is a fine achievement," said Jared L. Cohon, president of Carnegie Mellon University. "This is another important chapter in the center's outstanding record of providing the nation's scientists with the most advanced computational tools. This world-class computing system reflects Pittsburgh's international leadership in technology development and is a key component of our region's technology future."

"This computing system is an important advance in assuring the continuation of our nation's leadership in basic research," said University of Pittsburgh Chancellor Mark A. Nordenberg. "There is growing national and international recognition that university research is at the heart of most commercial innovation and much of our recent economic prosperity. And it is significant for our region that Western Pennsylvania is the home of this great national resource—attesting to the strength of our institutions. When great research universities such as Pitt and CMU partner with industry—in this case Westinghouse and Compaq—and the federal government, we are poised to achieve the next great breakthroughs of this new century."

"PSC is to be congratulated on bringing this powerful new technology into being," said Charlie Pryor, president and CEO of Westinghouse Electric Company. "Westinghouse is proud to add its internationally recognized expertise in management excellence and technology leadership to the team. Once again, the Pittsburgh Supercomputing Center has demonstrated its leadership in high-performance computing."

The TCS installation marks the first operation of AlphaServer SC, the system software that ties AlphaServer systems together, on this scale and the first large-scale, multi-level Quadrics switch structure that supports thousands of processors while achieving sustained operation across the system. Standard benchmark software has measured system performance over three teraflops. The TCS will next go through a period of "friendly user" testing, and by early 2002 it will become available to researchers nationwide through the peer-review process of the NSF PACI program.

PSC and Compaq collaborated on numerous machine enhancements to improve the performance of the TCS, changes that range from the disk controller and file system to wiring optimizations. By careful site planning and redesign of the AlphaServer configurations, PSC engineers reduced the distance between processors, thereby also reducing cabling and minimizing network latency.

Total TCS floor space is roughly that of a basketball court. It uses 14 miles of high-bandwidth interconnect cable to maintain communication among its 3,000 processors. Another seven miles of serial, copper cable and a mile of fiber-optic cable provide for data handling.

The TCS requires 664 kilowatts of power, enough to power 500 homes. It produces heat equivalent to burning 169 pounds of coal an hour, much of which is used in heating the Westinghouse Energy Center. To cool the computer room, more than 600 feet of eight-inch cooling pipe, weighing 12 tons, circulate up to 900 gallons of water per minute, and twelve 30-ton air-handling units provide cooling capacity equivalent to 375 room air conditioners.

For background information, see http://www.psc.edu/publicinfo/terascale/bigiron.html.

Compaq and AlphaServer are trademarks of Compaq Information Technologies Group, L.P.

The Pittsburgh Supercomputing Center was established in 1986 and is supported by several federal agencies, the Commonwealth of Pennsylvania and private industry.

 

Releases Archive