Petascale Application Collaboration Teams

Collaborative Petascale Application Collaboration Teams (PACTs) will ensure that a range of applications can take full advantage of Blue Waters when it comes online in 2011. Through the PACTs, scientists and engineers who develop and use these applications are working closely with computing experts to address science and engineering challenges outlined by the National Science Foundation (NSF) in its initial request for proposals, as well as other computational problems at this large scale.

Together, these teams optimize, scale, and re-engineer applications that will run on Blue Waters once it comes online.

New PACTs will be formed throughout Blue Waters' lifespan.

While each PACT functions independently, some members of the teams meet periodically with experts from IBM and NCSA to discuss challenges and potential solutions. Lessons learned in optimizing one application will be applied to others as appropriate.

NSF Challenge: Lattice-gauge QCD

Lattice quantum chromodynamics (QCD) is a method for studying quarks and gluons, subatomic particles that comprise some of the basic building blocks of our universe. In lattice QCD theory, physicists envision space-time as a crystalline lattice where quarks can be found only on vertices and gluons can travel only along lines connecting quarks. By simulating the evolution of the lattice system as the spacing between vertices changes, physicists gain understanding of the behavior and interaction of these tiny, mysterious particles.

An application called MILC is being used to address the challenging lattice QCD problem described by NSF. MILC frequently passes many small messages and also requires frequent collective communication, during which all of the processors are communicating simultaneously. This communication load requires a network with very low latency. The lattice QCD PACT is working to increase MILC's scalability by increasing the overlap between communication and computation operations. The PACT is also improving the efficiency of the conjugate gradient algorithm that is part of MILC in order to reduce the number of required iterations and will be exploring alternatives to this algorithm in order to make the code operate more efficiently.

NSF Challenge: Molecular dynamics

University of Illinois researcher Klaus Schulten describes biophysical modeling as a "computational microscope." While light microscopes gave scientists their first glimpse of cells, computational methods now provide more dynamic insight into the basics of life at the molecular level. Today's high-performance computers enable scientists to simulate biomolecular systems as large as several million atoms, but this still falls short of capturing the full complexity and extended timescale of important biological processes. NSF has outlined a molecular dynamics problem including 100 million atoms, more than 10 times more than has been done before. At this level, biophysicists will be able to simulate the interactions of viruses with cell membranes and embedded proteins, ribosomes and the mechanism of protein construction, and the protein-folding problems that contribute to many diseases.

NAMD, the widely used parallel molecular dynamics program developed by a team at Illinois, is being used to solve the NSF molecular dynamics problem. NAMD, written in C++ using the Charm++ parallel programming model, divides the computational domain into box-shaped patches, each containing varying numbers of atoms. Since the set of patches affected by pair-wise and bond interactions changes as the atoms move about, the dynamic load-balancing capability of Charm++ is crucial to scalability, and improving load balancing is one focus of the NAMD PACT. The PACT is also working to overlap communication and computational work, reduce the code's memory usage, and fully exploit the capabilities of the Blue Waters interconnect hardware.

NSF Challenge: Turbulence

Turbulence problems are everywhere—from designing aircraft and automobiles to predicting the weather to understanding the lifecycle of a star.

The NSF-specified turbulence problem requires the use of the pseudospectral method within a periodic box. Simulations at a higher resolution than ever used before in this simple setting can be used to validate turbulence approximations made in other more complex simulation codes.

Various approaches and codes are being explored within the context of the specified problem, which requires over 30,000 one-dimensional Fast Fourier Transforms (FFTs) and transposes of multiple three-dimensional arrays for each of 10,000 time-steps. In addition to the need for high-performance transforms on a single core, high-performance data communication across the whole system is needed. The PACT also is looking at performance gains through overlapping FFTs and communication.

NSF Challenge: Material Science

PARATEC is the initial materials science code receiving support through the Blue Waters PACT program. The code performs ab-initio quantum-mechanical total energy calculations using pseudopotentials and a plane wave basis set. Forces and stress can be calculated with PARATEC and used to relax the atoms into their equilibrium positions.

The NSF-specified PARATEC problem includes a seven-by-seven-by-seven supercell of silicon with two atoms per cell and two bands per atom.

NSF Challenge: Weather research

Researchers have been investigating the weather on NCSA supercomputers for as long as there have been NCSA supercomputers. For Blue Waters, a PACT is working with researchers at the National Center for Atmospheric Research to improve and model the performance of WRF, a next-generation mesocale numerical weather prediction system that is used for both weather forecasting and atmospheric research. The code is also used for regional climate modeling, air chemistry and air quality research, large-eddy turbulence simulations, and cloud and storm modeling.

The PACT is comparing strong and weak scaling approaches to using WRF, as well as strategies for routing communications among processors and memory and means of mapping computational tasks to available processors.

For more information on Blue Waters PACTs, send email to the Blue Waters Consulting Office at bwconsult@ncsa.illinois.edu.