It's all in the cells

released 08.23.07

Astrophysicist Paul Woodward is director of the Laboratory for Computational Science and Engineering (LCSE) at the University of Minnesota. LCSE tests and applies innovative hardware and software solutions to problems in science and engineering. This year Woodward is spending half his summer at NCSA as part of the summer fellows program. He is working on modifying his code that models stellar convection cells to see if it will simulate thunderstorm convection cells as well as run on the center’s newest machine, Abe. He took some time to chat with Access' Barbara Jewett about computational science in a petascale age.

Tell me about your current work.

I do fluid dynamics in astrophysics. The large problem I'm working on now has to do with what goes on inside the central region of a star near the end of its life.

The region of the star near the interior that I'm interested in and that I'm seeking to describe on the computer turns out to be fairly exotic. It is like an atmosphere in that there are convection motions and so the material, which is a gas, is rising where it's buoyant and descending where it's cooler. [At the bottom] nuclear fusion is going on and helium is burning and forming carbon. At the top of the layer is unburned hydrogen. If some of that gas could be pulled in and pulled down with the moving gas, down toward the helium burning shell, as it's called in these stars, then it would react very violently...it would set off multiple reactions and would affect what heavy elements that star would produce ultimately. From my description it may not sound very hard...but we have to focus a lot of attention right at the top of that layer, the details of what goes on there.

Is there anything else that's difficult about this problem?

The second thing that's harder is the aspect that is also hard about describing the Earth's weather. The problem is that the motions of this material are fast but not too fast, about Mach 1/30th. That's like a 20-25 mph wind on the Earth.

The simple way of looking at it—if you take the numerical methods that I and others have used for years, and you just apply them to this problem without modification, you need to take about 20 times as many time steps in order for the same amount of [fluid motion] to happen as you would in some other problems. As a result, to a certain extent for many years I have avoided problems of this nature as they're more expensive. It costs 20 times as much to do the calculation. It turns out also that once you drag some of this unburned hydrogen down and you start burning it then the calculation gets much more expensive also because you have to follow all the reaction productions and make sure you know where they went and what they turned into, and then where that went, and so forth.

And there's no way to reduce that cost?

You can develop different numerical methods to try and get that cost down. [NCSA chief science officer] Bob Wilhelmson has done some of that [with weather simulations] and I'm working on some of that, but it never gets down by that factor of 20. You might be able to bring it down maybe a factor of five, but you are still left with these being relatively expensive calculations to do.

What does the potential of petascale computing mean to your work?

This is just the kind of development that we need to make it possible at a reasonable cost to do simulations like this. Two summers ago I talked to a post doc at Los Alamos, Falk Herwig. He said he had talked to others and it was generally believed these problems were impossible, and they were impossible because of this extra cost factor of 20. ...We figured out that it might run for a while on 256 processors but you really could get an answer, so he talked me into doing it. I wrote a special version of my computer program to attack these problems. He did one run, then moved to Keele University in England.

The point is that two years ago this sort of problem was considered impossible. Now there are several places, including NCSA, that have several thousand processors available, and in the intervening time I've gotten my program to run on these kinds of systems and it's running very well. And it turns out that changes things by about another factor of 40. I have been investigating the IBM Cell processor so I've reworked my program again in order to run on the Cell processor and that made it run much faster on everything. It takes that factor of 40 and makes it into about a factor of 100.

So with the right combination of software and machine, studying the helium shell flash in 3D is possible?

It was possible two summers ago to conceive of taking a piece of this convection zone and doing it, and actually Falk Herwig did do that [The Astrophysical Journal, May 2006]. With 100 times the speed available it is possible to contemplate one of two things, and with petascale computing—which would give us another factor beyond that of another probably 20 to 40—we'd be able to think about both of those things together. One of them is that with the factor of 100 you could run 100 times longer. ...[The other is that] instead of taking a chunk of this layer you could take the whole layer. ...If you go to the petascale level you can do the whole [layer] for a [long time]. Not for the whole two years duration [of the helium flash], obviously.

For a petascale machine I imagined a calculation where...it ran at the speed [of] 2.4 PF, so it would really be a petascale calculation. So, you ask, what could it do? It could simulate an hour of the whole star, of that whole shell, doing its thing at very high resolution, so that I should hope to get the answer kind of right. And it's affordable. And I'll just be able to solve it! That's kind of a neat thing.

You have frequently referred to "my program." Can you tell me a little bit about the application you are using?

It's an explicit compressible gas dynamics code. The algorithm it uses is called PPM, and that is an algorithm I've been using for a long time. It's found in quite a few codes that are used by many people so it's standard in that sort of sense, but this conversion that I went through to make it run on the Cell processor, that is unique at the moment. My interest in the Cell processor was stimulated through my work with collaborators at Los Alamos, particularly Karl-Heinz Winkler, Mike Steinkamp, and Rob Lowrie.

While I'm here I'm working with NCSA staff to get my program to run on your new machine, Abe. I'm also working with Bob Wilhelmson to see if we can get the program to simulate a tornado. [My work is] similar to weather in that you're in the same Mach number region and this convection process which is very important [in my problem] is also important in weather. A thunderstorm is a convection cell but it has water vapor that we don't have in the star. But from many perspectives the computational challenge is similar; there's about as many things to keep track of in each case—if you are doing weather on the Earth you'd be keeping track of all the different forms of water, whether vapor, or ice, or liquid droplets, and we're instead keeping track of different isotopes of elements. And we would have these burning reactions which you wouldn't have with weather.

You'll be able to quickly solve your helium shell flash problem when a petascale machine is in production; do you foresee any other changes in your work with the use of petascale machines?

There's something else that is really special that is happening now that we are having these high computer performance levels and that is it becomes possible to do these scientific investigations interactively. When I started out doing computational fluid dynamics in the early '70s I was doing problems that were described in just 1D. It was like you had a tube of gas and were describing what happens inside the tube. But you didn't describe anything about what happens in the cross section, just what is going on along the length. And those kinds of runs I ran on a Control Data 7600 and they took 10 minutes. You could set one going and you'd go to lunch—because you shared the machine; you'd go to lunch and you'd come back and your 10-minute problem would be done if you were important enough. And if you weren't important enough, well, your 10-minute problem would never be done because you'd never have enough priority. So part of the game was figuring out how to be important enough to get your work done. But in a one-hour slot you could get a single 1D problem done. That was not interactive, that was a batch sort of job.

After a few years, when the Cray 1 came, we got a factor of 10 increase over the 7600 and so the 10 minutes became one-minute. So you didn't go to lunch to wait for your one-minute job to run. You're still sharing the machine so your one-minute job might take four minutes, but you would sort of log in and you would run it. We had then a state-of-the art system—we had televisions that sat next to our teletype terminal and the line plots could be sent to the television set if you were important enough to get one of the 16 available channels in our laboratory. If you could grab a channel and you could send a job, you could do a run interactively and you could see it, but for 1D. By the early '80s, which was when I left that laboratory to go to the university, I was doing 2D calculations but it was back to batch. You could not do them interactively.

Since the late '80s I've been doing 3D problems but the 3D problems have not been interactive until this year. Which means we can now always do fluid dynamics calculations interactively because it's not going to get any more complicated—we are not planning any research in 4D fluid flow.

On a petascale machine, if you could use that interactively, I could be sitting at the controls, which need not be where the petascale machine is but at some controls that nonetheless control it, and I could be interacting with this calculation as it unfolds. I could watch it, I could stop it if necessary, I could explore the parameter space of these kinds of calculations. With interactive petascale computing we're going to be able to do that for a whole variety of 3D flows and what it's going to do, I believe, is bring [more] people into the practice of fluid dynamics. Because if it is possible for me to sit down and interactively explore, it would also be possible for somebody else.