...And the sky is blue | National Center for Supercomputing Applications at the University of Illinois

...And the sky is blue

11.01.10 -

by J. William Bell

Computing codes improved as part of the National Science Foundation's Petascale Computing Resource Allocations program will transform fundamental chemistry with wide-ranging impact.

Water is wet, but the chemistry of water sounds like a dry subject, doesn't it? Don't we have water—ubiquitous, elemental water—figured out by now? Two hydrogens and an oxygen. Simple enough, right? Wrong.

"In fact, water isn't understood really well," explains Mark Gordon, director of Ames Laboratory's Applied Mathematics and Computational Science Program and a chemistry professor at Iowa State University.

It's poorly understood but critically important, impacting our bodies, our world's climate, and the products we buy.

Gordon and Iowa State's Monica Lamm and Theresa Windus are working with NCSA to improve a pair of computational codes that will further illuminate the chemistry behind water and many, many other molecules that make life possible. The codes, called GAMESS and NWChem, are being prepared to run on Blue Waters as part of the National Science Foundation's Petascale Computing Resource Allocations program. Blue Waters is the sustained-petascale supercomputer coming to NCSA.

"Blue Waters is really going to take us to the next level," Windus says. "We're going to be able to look at more complex chemical systems. We're going to be able to look at grand challenge problems, such as catalysis, drug delivery, and solvent effects on chemical processes in a very detailed way and get very accurate answers."

GAMESS has been developed by Gordon's research group since the mid-1980s. Windus, meanwhile, was lead developer of NWChem at Pacific Northwest National Laboratory before coming to Iowa State.

On the water front (and beyond)

The researchers plan to use Blue Waters to explore essential problems like liquid water and the way it vaporizes, as well as the way it solidifies to form ice, with unprecedented precision. Those behaviors in turn influence, well, just about everything else.

"All biological processes occur in water, so it's incredibly important for understanding biology, biochemistry. And the solvent most commonly used for materials development is also water. So the breadth of importance of understanding water and understanding what happens in water is very, very high," Gordon says.

But GAMESS and NWChem are applied to many other fields of research as well. Windus, for example, uses NWChem to study aerosols—how molecules in the air form droplets. "We really don't understand at the molecular scale how that's happening," she says. But the process impacts everything from weather and global climate change to the design of new consumer products like deodorants or cars' fuel injection systems.

"It's going to be a long time before we're at the level of really being able to incorporate [the basic science of aerosols] into weather modeling. On the other hand, what we can do is inform the classical models they now use for forming droplets," Windus says. "The chemicals going into clouds affect how the clouds, and thus the weather, behave by impacting features like the clouds' radiative behavior.

"Today we can do things where we can look at, say, three or four or five waters in a cluster and be able to look at the nucleation events. What we're really looking to do is incorporate things like sulfuric acid, nitric acid. Look at the reactions that are happening inside the cluster, as well as looking at the growth. That's really impossible today with the current computing capability. That's what Blue Waters is really going to enable us to do."

Lamm, who is principal investigator on the Petascale Computing Resource Allocations project, is interested in what are known as dendrimers. She would like to better understand how these long, branching polymers bind to smaller molecules.

"Dendrimers have some unique properties that set them apart from a typical polymer that is like a straight chain with two ends. For one, despite the fact that they are a large molecule, they can be made relatively precisely with a consistent size and low defects. The many branches in their structure mean that they have several 'end groups'—not just two like a typical polymer—and these end groups are where smaller molecules can be bound and captured, so to speak," Lamm explains.

"This property has important implications for health technologies, such as drug delivery and medical imaging, and we are also exploring opportunities for using dendrimers in environmental remediation applications where contaminants are removed from water or soil."

Combined, GAMESS and NWChem have well more than 100,000 users in about 100 countries, according to Gordon. Many companies use these applications as well, exploring material science, biology, and chemical engineering, for example. "And they run on every computing platform you can imagine," he adds.

Scaling up

Getting the codes to run on the newest platforms like Blue Waters is still a challenge, though. In order to get breakthrough results and explore new areas of science, the simulation codes have to take advantage of as large a portion of these systems as possible and run efficiently on them despite that size. In the case of Blue Waters, that means running on hundreds of thousands of processing cores simultaneously. Those challenges are where the collaboration with NCSA through the Petascale Computing Resource Allocations project come in.

"When you start scaling up from 1,000 or 2,000 processors to 100,000 processors, there's all sorts of problems that arise that never would have arisen with smaller numbers. And we're really going to rely on the Blue Waters team to give us some help on how to get around roadblocks that occur with that many processors. It's very important," Gordon says.

"We have been taking full advantage of the expertise of the Blue Waters team to help us modify our codes," says Lamm. "Without the PRAC grant, we would not have access to the critical information we need to make our codes run successfully on Blue Waters."

Those problems include tuning the code to the particulars of the IBM POWER7 architecture that Blue Waters is based on, as well as debugging the code.

What is known as load imbalance is also an issue when the code runs. That is, they're inventing ways of managing the fact that little bits of the code are running on many, many processors and that some of those processors finish their work early while others finish late. When working on relatively few processors, those conditions are often simply ignored—the completed task waits for its processor or the idle processor waits for its next task. When working at petascale and beyond, however, those delays pile up into unacceptable snarls.

"We really think of the PRAC relationship as a collaboration," Windus says. "We have great in-house expertise as well, but what NCSA helps us with is to get a broader base for that expertise and be able to really apply what we know to this machine."

This project is funded by the National Science Foundation.

Team members
Mark Gordon
Seung Ha Kim
Monica Lamm
Spencer Pruitt
Theresa Windus