XSEDE and Blue Waters go supernova

07.02.14 -

by Liz Murray

If you were to go back far enough into the Earth’s cosmic ancestry, you might be surprised to find it all started with a supernova explosion. These explosive cosmic events are like laboratories in space, generating elements that enable the creation of life later on; in fact, most of what makes up the Earth, including us humans, evolved from these fundamental elements. This is why simulating the process of a star going supernova is so important—it could potentially be the key to unlocking some of the bigger mysteries of how we came to be in the universe.

Philipp Mösta, postdoctoral scholar at Caltech, Christian D. Ott, professor of astrophysics at Caltech, and fellow researchers working with Peter Diener, research professor at the Center for Computation and Technology of Louisiana State University, are studying extreme core-collapse supernovae. These events make up only one percent of all supernovae that are observed but are the most extreme in terms of the energy emitted into the universe.

In the past, simulation work within their field was mostly done in two dimensions (2D), and codes did not always include all the physics; for example, neglecting general relativistic effects. For the first time ever, Mösta and his fellow researchers are running fully general relativistic three dimensional (3D) simulations, and new details are emerging. The researchers are finding what was believed to be the driving mechanisms behind these explosions might not work as anticipated from previous 2D simulation work. Mösta and Ott’s simulations could change the landscape of computational research in their field moving forward.

“It will probably indicate to other groups who, so far, have focused on performing simulations with symmetries imposed, that they will have to move to full 3D simulations as well, which will ultimately strengthen our community,” says Mösta.

This journey through dimensions began with an Extreme Science Engineering Discovery Environments (XSEDE) allocation on the Stampede supercomputer at the Texas Advanced Computing Center (TACC) at The University of Texas at Austin back in 2013.

Working with Lars Koesterke, research associate in the 
High Performance Computing Group at the TACC, along with XSEDE’s Extended Collaborative Support Services (ECSS), the team got assistance in optimizing the performance of their code. “We had to pair fast individual compute core performance with excellent communication throughput through a fast network,” explains Ott. “This required a thorough optimization of our code towards the strength of modern architectures such as Stampede and Blue Waters.”

For the team, this meant larger simulations were possible without fear of burning through their allocation too quickly. “We were able to perform the first fully general relativistic 3D simulations without any symmetries and the difference in comparison to 2D was drastic,” Ott continues. “We now know if we want to predict what the signature of these extreme supernova explosions might look like, we need to do it in full 3D.”

At first, the team had to cut out a part of the inner star. “At the end of the life of one of these massive stars that we're simulating there's an iron core in the center, about 1.5 times as massive as the sun, just consisting of iron,” explains Ott. “That’s what is undergoing the collapse to a neutron star, and it’s this collapse that provides the energy that then ultimately powers the explosion.”

For the other 99-percent of observed supernova events, the main proposed mechanism driving the explosions is energy deposition by neutrinos behind the shockwave. However, it couldn’t possibly deliver the amount of energy that is observed for these hyper-energetic supernovae. The mechanism studied by Mösta and Ott is based on a combination of magnetic fields and strong rotation building up extra magnetic field behind the shockwave by extracting rotation from the collapsed core of the star. This magnetic field exerts extra pressure, which leads to a bipolar explosion in the form of two jets that are pinching out along the rotation axis of the star—or so it was anticipated on the basis of 2D simulations.

These initial general-relativistic magnetohydrodynamics (GRMHD) simulations were performed on Stampede, but when the shockwave launched in the explosion propagates further out into the core, the computational demand of the simulations increases dramatically. The team decided to move from Stampede to the Blue Waters supercomputer at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign to utilize a 30,000-core count.

While running their simulations on Blue Waters, the team worked with Robert Sisneros, visualization programmer at NCSA, to improve the performance of their analysis scripts to read and operate their data. This included integrating the launch of VisIt—an open source, interactive, scalable, visualization, animation, and analysis tool. “While the analysis itself is no different when deploying a parallel job to the batch system on Blue Waters, there is less flexibility in the ways components can be launched,” explains Sisneros. “Their codes needed a few updates to correctly manage launching parallel components, as well as a few other changes due to subtle differences among various versions of VisIt.”

Mösta and Ott credit their success to having computational time on both of these powerful, petaflop supercomputers, along with access to their knowledgeable support staff. “Both Stampede and Blue Waters provided the computational power we needed to prove we could successfully perform 3D simulations, and more importantly, to prove the importance and impact of the simulations,” says Mösta.

“What we’ve shown is that the jets that appear stable in 2D are actually unstable in 3D. They twist, rotate and become unstable due to a phenomenon that is called the magneto-hydrodynamic kink instability. This instability of the magnetic field itself is the same that is also seen in fusion reactors that are using magnetic fields to confine the plasma,” says Mösta.

Mösta and Ott are using their new simulations to predict how these extreme explosions will work in full 3D. “This will allow us to take the next step, connecting our results to observations by predicting which elements are produced and how are they distributed when we look at the remnant of the supernova,” says Mösta. With the hope to ultimately connect these simulations to actual observable signals from one of the NASA satellite telescopes, one example being the Hubble Space Telescope.

The team believes these results will generate needed momentum to move forward in developing the next generation codes dedicated to performing fully 3D simulations. Soon including all of the relevant physics while running on a large number of compute cores will become the norm for simulation work in their field of research.

National Science Foundation

Blue Waters is supported by the National Science Foundation through awards ACI-0725070 and ACI-1238993.

National Science Foundation

XSEDE is supported by National Science Foundation through award ACI-1053575.