Simulating supernovae with Blue Waters

Chris Malone and Andy Nonaka explains how their UC Santa Cruz/Lawrence Berkeley National Laboratory team is using the Blue Waters Early Science System to study how carbon and oxygen burn to iron during the initial stages of thermonuclear runaway as a white dwarf goes supernova.

What research are you conducting on the Blue Waters Early Science System?

Cosmologists have used the light curves of Type Ia supernovae (SN Ia) as tools for surveying vast distances. However, great uncertainty still exists regarding the underlying physics of a SN Ia explosion. While it is generally accepted that the exploding star is a white dwarf driven to thermonuclear runaway by the accretion of mass from a binary companion, the nature of the progenitor star and how it ignites and burns are debated.

Just how carbon and oxygen burn to iron during the initial stages of the runaway is one of the most uncertain aspects of models for SN Ia. Previous simulations have used coarse resolution and artificial initial conditions that substantially influenced their outcome. Here, we have the unique advantage of being able to import the results from previous simulations of convection leading to ignition from our low Mach number code, MAESTRO, directly into our compressible code, CASTRO. These initial conditions include the location of ignition and the turbulence on the grid. CASTRO is a finite-volume, adaptive mesh refinement (AMR) code. We are using multiple levels of grid refinement to capture the early post-ignition dynamics at unprecedented resolution, revealing the essential character of the burning.

What additional research do you have planned for the full Blue Waters system?

Although our proposed ESS calculation will generate publication-quality results, follow-up studies with a full Blue Waters allocation will be extremely insightful. By following up on the proposed simulation as the burning breaks through the surface and spreads over the star, we can check for locations where shearing or converging burning fronts might trigger a detonation, which is required to explain observations.

What is particularly computationally demanding about this research, and why is Blue Waters a good system for meeting your simulation needs?

In our CASTRO simulation, we divide up the entire star into 10,000 to 20,000 grids. Each grid contains between 10,000 to 100,000 computational zones, each carrying information about the fluid velocity, density, temperature, pressure, composition, etc. in that particular zone. The grid structure dynamically changes over time to ensure that we are tracking the flame front using zones at the highest resolution. Due to the size of the Blue Waters machine, we are able to distribute these grids among tens to hundreds of thousands of processors for efficient parallelization. Blue Waters is also able to handle the complex communication patterns between grids of different sizes containing zones at different spatial resolution.

How have you benefited from working with NCSA staff through your PRAC allocation?

We have been in regular contact with NCSA staff to test the CASTRO code on the emerging Blue Waters configuration. Our PRAC representative, Kalyana Chadalavada, has been instrumental in the configuration, benchmarking, and debugging processes. With Kalyana’s guidance, we have been able to run CASTRO efficiently on the entire ESS machine using a hybrid MPI/OpenMP approach to parallelism. We have also discovered a bug in Cray’s newest Fortran compiler, and have worked with staff from NCSA and Cray to resolve the issue in the latest compiler. We have worked with David Semeraro to test the VisIt visualization software, and are now able to use VisIt to visualize our datasets using an arbitrary number of processors.

Disclaimer: Due to changes in website systems, we've adjusted archived content to fit the present-day site and the articles will not appear in their original published format. Formatting, header information, photographs and other illustrations are not available in archived articles.