Preparing for the big one

released 04.21.10

Download a PDF of this story

Email this story

Subscribe to our newsletter

Cumulative peak ground velocities for a southeast-to-northwest rupture Mw8.0 scenario on the San Andreas fault. The simulation computed 350s of wave propagation in a 800km x 400km x 100km subset, dividing the region into 32 billion cubes 100 meters on a side, and up to a maximum frequency of 1 Hz. The simulation used 96,000 cores on NICS Kraken supercomputer and took 2.6 hours to compute. The proposed Blue Waters simulation of this kind is 256 times larger in terms of computational requirement.

Image credits: Kim Olsen, San Diego State University, Yifeng Cui and Amit Chourasia, San Diego Supercomputer Center

Below are snapshots of velocity magnitude at different intervals. Along the fault, the Coachella Valley experiences particularly strong shaking, and large amplitude waves are generated in the Los Angeles and Ventura basins. Strong near-fault ground motion is noticed further north beyond Parkfield.

Image credits: Kim Olsen, San Diego State University, Yifeng Cui and Amit Chourasia, San Diego Supercomputer Center

PRAC winners represent a range of disciplines

Petascale Computing Resource Allocations (PRAC awards) from the National Science Foundation allow research teams to work closely with the Blue Waters project team in preparing their codes. The codes and projects address key challenges faced by our society and explore fundamental scientific and engineering problems.

These multiyear collaborations include help porting and re-engineering existing applications. In some cases, the teams will build entirely new applications based on new programming models.

Current projects—18 representing about 30 institutions—represent a wide range of scientific disciplines. They will drive scientific discovery for years to come. Some of the projects are:

Testing hypotheses about climate prediction at unprecedented resolutions on the NSF Blue Waters system
A team from the University Corporation for Atmospheric Research, Colorado State University, the University of Miami, and the Institute for Global Environment and Society plan to test two hypotheses about the Earth's climate system using Blue Waters. The first is that the transport fluxes and other effects associated with cloud processes and ocean large-scale eddy mixing are significantly different from the theoretically derived averages embodied in current-generation climate models, and that these differences explain a large portion of the errors in these models. The second is that a more faithful representation of these eddy-scale processes will increase the predictability of the climates generated by climate models. The project team plans to perform three sets of numerical experiments using three cutting-edge climate models: the Community Climate System Model, a new version of the Community Climate System Model that includes an innovative treatment of cloud processes, and the Colorado State University Global Cloud-Resolving Model.


Petascale simulations of complex biological behavior in fluctuating environments
Change an organism's environment, and it adapts. That's true of complex creatures, and it's true of unicellular organisms. A research team led by the University of California at Davis' Ilias Tagkopoulos plans to use Blue Waters to model multi-scale biological systems where processes ranging from gene expression and intracellular biochemistry to ecosystem dynamics are in play. The study will look at the influence of nutrient concentrations and mutation on adaptation, compare different strategies for survival in static and fluctuating environments, and examine how unicellular organisms modify their internal networks to facilitate such changes. This modeling approach has been used on today's supercomputers, but it requires a number of simplifications. The team anticipates that Blue Waters will allow some of these simplifications to be relaxed, so that more biological processes can be included.

Computational relativity and gravitation at petascale: Simulating and visualizing astrophysically realistic compact binaries
A team working on an astrophysics code for Blue Waters is led by the Rochester Institute of Technology's Manuela Campanelli. With it, they hope to explore pairs of very dense astrophysical objects, including the merger of asymmetric black hole binaries, of neutron star binaries, and the merger of a black hole and a neutron star. Through more accurate simulations, they will be able to characterize the events' gravity wave signatures, which will ultimately be detected by instruments like the Laser Interferometer Gravitational-Wave Observatory.

Computational chemistry at the petascale
This project will port two established chemistry codes—GAMESS and NWChem—to Blue Waters. Led by Iowa State University's Monica Lamm, the team hopes to use these codes to simulate the molecular dynamics of water, aerosols in the atmosphere, and dendrimer-ligand binding. These topics have applications in many areas of science and engineering. The GAMESS and NWChem are used by a large number of other research groups, which may make it possible for other teams to take advantage of Blue Waters and other extreme-scale computing systems.

Lattice QCD on Blue Waters
Lattice quantum chromodynamics (QCD) is a method for studying quarks and gluons, subatomic particles that comprise some of the basic building blocks of our universe. In lattice QCD theory, physicists envision space-time as a crystalline lattice where quarks can be found only on vertices and gluons can travel only along lines connecting quarks. By simulating the evolution of the lattice system as the spacing between vertices changes, physicists gain understanding of the behavior and interaction of these tiny, mysterious particles.

The code being developed for Blue Waters by a team led by the University of California, Santa Barbara's Robert Sugar will be used to determine fundamental parameters of the Standard Model of nuclear physics and will help to determine its range of validity. They will also be used to calculate the masses, internal structure and interactions of strongly interacting particles, including the masses of the up and down quarks and the values of the weak transition couplings between quarks. The combination of calculations such as these and experiments in facilities such as the Large Hadron Collider and Relativistic Heavy Ion Collider offer a way of probing for new physics in the behavior of sub-atomic particles.

A complete list of PRAC winners and their projects may be viewed online at: http://www.ncsa.illinois.edu/BlueWaters/prac-teams.html.

A Blue Waters team works with earthquake engineers to build and combine codes that will provide better seismic hazard assessments and inform safer building codes.

by J. William Bell

The northern reaches of the San Andreas Fault have seen their share of major earthquakes in the last century. The 1906 San Francisco killed more than 3,000 people, and a 1989 quake near Santa Cruz postponed the World Series. The other end of the fault near Los Angeles, meanwhile, hasn't seen a major earthquake since 1680. But there is a high probability of a rupture over the next two decades.

A team of more than 30 earthquake scientists, computer scientists, and other specialists are very interested in that next earthquake—what it might look like, what sort of damage it might cause, and what might be done to mitigate the damage. Led by the Southern California Earthquake Center (SCEC), they plan to use the Blue Waters sustained-petascale supercomputer at NCSA to model it.

"SCEC researchers are developing the ability to conduct end-to-end, rupture-to-rafters simulations that will extend our understanding of seismic hazards," says Thomas Jordan, SCEC's director. "The integration of these research applications represents an unprecedented opportunity to advance our ability to characterize seismic hazard and risk."

From today's biggest to tomorrow's

SCEC's Community Modeling Environment (CME) allows the researchers to model fault ruptures and seismic wave propagation. Parts of the tool can run across all the available processors in some of the world's largest supercomputers—60,000 cores on the Texas Advanced Computing Center's Ranger, 96,000 cores on the National Institute for Computational Sciences' Kraken, and 130,000 cores on Argonne National Lab's Intrepid.

Even at that scale, they don't yet reveal some important aspects of earthquake behavior. Current simulations, for example, only describe earthquakes with a maximum frequency of 1.0 hertz, which provides spectral acceleration information of one to two seconds.

"Frequencies in this range are of interest for medium-height buildings to high-rise—10 to 30 stories. However, the number of areas with such tall buildings is limited," says Yifeng Cui, a computational scientist at the San Diego Supercomputer Center who is working on the project. But the impact on shorter buildings, including many homes, must also be understood.

With the Blue Waters supercomputer, the team expects to get to two hertz. That will be on simulations that model an 8.1 magnitude earthquake, which is "a worst-case scenario on the San Andreas Fault," according to Cui. Those simulations will look at a section of the earth 800-by-400 kilometers and 100 kilometers deep, covering it with more than two trillion mesh points, or one every 25 meters. That makes for a simulation more than 256 times more computationally taxing than their current work.

"There's huge demand for high frequency. The engineering interest is all the way up to 10 hertz. If we want to simulate on the 10 hertz scale, not even Blue Waters will be big enough," Cui says. The two hertz range will "provide useful data of engineering interest and greatly expand the usefulness of the simulations to the engineering community," nonetheless.

It will also provide the code base that will allow researchers to ultimately scale up to those incredibly high frequencies. Preparation is key—whether you're facing the big one in the real world or modeling it on a supercomputer.

To increase the frequency that earthquake engineers can model, the team applied for—and won—a Petascale Computing Resource Allocation (PRAC). Through these awards from the National Science Foundation, the Blue Waters team is working with almost 20 teams around the country to prepare their codes to run on Blue Waters and other computing systems like it. The multiyear collaborations include help porting and re-engineering existing applications.

The PRAC team led by SCEC will focus on a set of three seismic and engineering modeling codes. The codes model fault rupture, propagate seismic energy through a detailed structural model of Southern California, predict ground motion, and model buildings responses to earthquakes. Work on interaction between the soil, building foundations, and large collections of buildings, bridges, and other infrastructure—as well as the interactions among structures in densely built areas—will be led by a team from Carnegie-Mellon University.

The goal is to combine the use of these tools to understand building damage likely to result from strong earthquakes and to run them on petascale supercomputers. As a collaborative, interdisciplinary research group, SCEC's computational science group specializes in designing and performing large-scale simulations that involve multiple disciplinary groups such as the ground motion and building modelers involved in this PRAC research.

'We know there is a due date'

SCEC's Information Technology Architect, Philip Maechling says, "When the SCEC science community identifies an important geophysical simulation that, due to its scale or complexity, exceeds the capabilities of individual research groups, SCEC's Community Modeling Environment collaboration may take on the challenge. Due to the broad range of scientific expertise within this group, and our close collaboration with computer scientists and HPC experts, the SCEC CME can work on some of the largest and most complex problems in our field such as the 2-hertz wall-to-wall scenario earthquake simulation we are preparing to run on Blue Waters."

"At this large scale and this level of complexity, you require a lot of expertise from different areas," Cui says. "In particular we require an in-depth understanding of the computational and storage hardware on Blue Waters as well as information on optimal software designs on this new architecture. Without support from NCSA, we would not be able to produce such large simulations. Our collaboration with the NCSA PRAC team enables SCEC to focus most of our efforts on the geophysical aspects of our research and reduces the time we must spend on scaling-up our codes."

SCEC's efforts to achieve sustained petascale computing on Blue Waters are expected to contribute to a broad range of SCEC's seismological research over the next few years. "Code improvements developed in support our largest simulations, such as this 2Hz simulation on Blue Waters, are rapidly integrated into more common, less demanding, seismic research calculations," says Maechling. "Our Blue Waters work represents a technological driver for SCEC that we believe will lead to significant improvements in a broad range of existing SCEC seismological research."

"We know there is a due date for our research. We just don't know when that due date is."

This work is funded by the National Science Foundation

Team members
Thomas H. Jordan - USC
Jacobo Bielak - CMU
Kim Olsen - SDSU
Yifeng Cui - SDSC
Steve Day - SDSU
Amit Chourasia - SDSC
Philip Maechling - USC
Scott Callaghan - USC
John Urbanic - PSC
Geoff Ely - USC
Po Chen - U of Wyoming
Dongju Choi - SDSC
Zizhong Chen - Colorado School of Mines
Daniel Roten - SDSU

Tweet this story Share on Facebook del.icio.us Slashdot Digg this