At the center of the Milky Way, the galaxy our planet is in, resides a supermassive black hole (SMBH) called Sagittarius A* (pronounced A-star). While this might fill you with existential dread, Sagittarius A* (Sgr A*) is a surprisingly quiet black hole, almost dormant, so it won’t be consuming our solar system any time soon. Researchers at the University of Illinois Urbana-Champaign (U. of I.) have been using the National Center for Supercomputing Applications (NCSA) Delta supercomputer to run simulations of Sgr A* that would be nearly impossible to complete without the assistance of high-performance computing resources like those at NCSA.
Vedant Dhruv is a graduate student fellow at U. of I. working with data and images from the Event Horizon Telescope (EHT). This telescope is actually an array that links eight telescopes around the world to create one powerful virtual telescope. One of the major successes of the EHT was capturing the first image of Sgr A*, a black hole 4 million times larger than our sun.
New Data, New Models
Dhruv worked with his advisor and other collaborators on this project, and they published their findings in The Astrophysical Journal Letters. He’s been able to gain a number of insights from using this data to simulate Sgr A*.
“In recent years, the EHT has produced high-resolution images of two supermassive black holes: Sgr A* at the center of the Milky Way and the black hole in Messier 87 (M87*), a giant elliptical galaxy in the Virgo constellation,” explained Dhruv. “These images provide insight into the plasma accreting onto the black hole.”
One of the images that Dhruv mentions might be one you’ve already seen. When M87* was first captured in an image, it made global news, especially because it was the first image of a black hole. Sgr A* took longer to get a clear image, but scientists have theorized for a while that it was there, even capturing a time-lapse of stars orbiting Sgr A* at the center of the Milky Way.
It’s impossible to take direct physical measurements of something as far away as the center of our galaxy, but fluid simulations of black hole accretion on machines like NCSA’s Delta can bring the sky down to the earth for researchers to study.
“The first horizon-scale image of Sgr A* revealed a bright ring surrounding a central brightness depression, the black hole ‘shadow.’ Comparison with theoretical models suggests that these observations are consistent with a hot, dilute plasma accreting onto a spinning black hole with a mass of roughly 4 million times the mass of the sun,” said Dhruv.
These theoretical models had to make some assumptions because designing and simulating highly realistic models for these phenomena is challenging. As Dhruv explains, most contemporary studies assume the gas behaves like an “ideal” fluid, where particles crash into each other constantly. But reality is more complex. “In reality, however, the hot plasma accreting onto a black hole is largely collisionless,” said Dhruv. “Charged particles can travel long distances before significantly interacting with one another through collisions.”
These older models also predicted that the black hole’s light should “flicker” in a violent and chaotic way. But in reality, the photo of the black hole indicates something different: a steadiness in the light curve.
“In our paper, we attempt to address these issues using ‘weakly collisional’ fluid models that include the leading-order corrections associated with particles having long mean free paths, namely viscosity and heat conduction,” said Dhruv. “In particular, we ask whether these ‘nonideal’ effects play a significant role in the synthetic observations produced by the model. Answering this helps determine whether such effects should be included in future modeling efforts.”
Refining the Simulation
If you wanted to simulate everything about the gas around a black hole, in a one-to-one simulation, you’d likely want to simulate every single atomic particle. As it stands currently, running a model like that would be prohibitively computationally expensive. Researchers also want their work to be reproducible, and creating a more efficient model meant that future researchers would be able to build on the team’s efforts.
To meet their requirement for an accurate model that could run on the available resources, the authors treated the plasma as a magnetized fluid rather than tracking individual particles.
“Our group has developed a fluid model for accreting plasma that includes nonideal processes such as viscosity and heat conduction,” said Dhruv. “These effects are expected to become important when the plasma is effectively collisionless. In that regime, the system is often described as weakly collisional, or, equivalently, as a dissipative fluid. We implemented this model in our group’s open source black hole accretion code, KHARMA, which is designed to run efficiently on modern GPUs and CPUs. We generated synthetic images and spectral energy distributions (SEDs) from simulations of this model by performing radiative transfer calculations and compared the results with those from an otherwise equivalent ideal model. It is worth emphasizing that this work presents the first images of Sgr A* from fluid simulations that explicitly include low-collisionality physics.”
NCSA’s Delta system provided the GPU capability, scale and support we needed to survey physically realistic simulations of the galactic center. Access to these resources made an otherwise computationally prohibitive study feasible.
Graduate Research Fellow, University of Illinois
“We found that, when viewed in a time-averaged sense, the weakly collisional fluid models are remarkably similar to the ideal models,” said Dhruv. “However, the low-collisionality physics included in our model tends to make the simulations ‘quieter,’ reducing variability and bringing them closer to observations of the galactic center.”
Using Black Hole Research to Find Your Way to Albuquerque
If black holes can capture humanity’s imagination, supermassive black holes must inspire awe. But studying these phenomena brings us more than just pretty pictures and interesting settings for science fiction cinema. The science derived from accurate modeling of fluid mechanics eventually works its way into many other domains that benefit from the research.
“The same modeling ideas we use here, namely viscosity and heat conduction, also show up in many familiar settings,” explains Dhruv. “Viscosity is what makes honey flow more ‘thickly’ than water, and heat conduction is what spreads heat through a metal spoon or a cooking pan. In a plasma, these processes control how momentum and heat are transported through a hot, magnetized gas. In this work, we ask how these effects change how the flow evolves over time, which is exactly the kind of physics that becomes increasingly important when the goal is to interpret the light curves of astronomical sources.”
Imagine you’re planning a trip to Albuquerque. You check the weather forecast, the road conditions and construction alerts. But what if you also had a “Space Weather” alert on your phone? Dhruv’s discoveries could inspire someone to approach their research into solar flares differently. There is a great deal of research that builds off the results of work in closely related fields. While Dhruv’s work doesn’t directly lead to a project that would prevent GPS satellite outages that power your map systems, every bit of science that better refines plasma physics models makes related science more accurate.
“These transport effects also show up in other important plasma environments where particle collisions are rare,” said Dhruv. “One example is the solar wind, the stream of plasma flowing out from the sun that drives space weather and can influence satellites and communications.”
Finally, because of the way Dhruv’s team designed their model, it can be used in numerous other fields of research.
“To include more realistic plasma physics, we developed a new solver within an open source GPU code,” said Dhruv. “The mathematical structure of the weakly collisional equations requires an implicit method, which iteratively solves for the fluid state at each time step. We designed the solver to be physics-agnostic, so others in the community can reuse the same framework to incorporate and evolve additional physical effects of their own.”
Continued Refinement
Researchers usually continue to innovate and refine their work. This case is no exception. While Dhruv’s team created a model that aligns more closely with observed black hole behavior, they still feel they can make it even more accurate.
There is a tiny element in this study that still requires substantial guesswork: the electron. The gas around a black hole is made up of both ions and electrons, but only one of these is accurately modeled.
“A major uncertainty in most simulations of black hole accretion is the electron physics,” Dhruv explained. “The plasma in these systems is a collisionless, ionized gas, and because ions and electrons differ largely in mass (by a factor of nearly 2,000!), they do not necessarily behave the same way or share energy in the same manner. As a result, the electrons that produce much of the observed radiation can be governed by physical processes that are distinct from those affecting the ions. This makes it important to model the electrons as accurately as possible.”
There is a reason most researchers don’t model electrons: it’s incredibly expensive and difficult to calculate electron movement in real time.
“Developing a fully relativistic, two-component model of an electron-ion plasma is theoretically challenging, and it also comes with substantial computational cost,” said Dhruv. “As a result, most simulations to date treat the plasma as a single fluid that is effectively ion-dominated because the ions carry most of the mass and inertia. The electron physics is then incorporated afterward, using physically motivated prescriptions applied in post-processing.”
Dhruv plans to address this issue by carrying out particle-in-cell (PIC) simulations of a small patch of the accretion disk. PIC simulations track the motion of individual charged particles – electrons and protons – and the electromagnetic fields they generate, capturing plasma behavior from first principles. Once the electron-scale behavior is understood, that physics can be folded into larger-scale simulations of the full accretion flow.
“A key challenge is that the mechanisms that heat and accelerate electrons are often not captured reliably by a fluid description. Kinetic approaches, such as PIC simulations, are needed to understand how energy is transferred and partitioned between ions and electrons, especially in a collisionless plasma. To address this, I am performing localized PIC simulations that model a small region of the accretion flow. These simulations allow me to study the particle energization mechanisms, how energy is divided between species, what sets the electron-to-ion temperature ratio and which plasma instabilities control electron evolution. The overarching goal is to use these kinetic results to build a subgrid prescription that can be incorporated into global fluid simulations, enabling them to evolve electron physics more realistically.”
Dhruv’s work benefits greatly from the resources available at NCSA. Through his allocation from the U.S. National Science Foundation’s ACCESS program, Dhruv’s team had access to NCSA’s robust portfolio of HPC resources.
“Because of the structure of the governing equations, weakly collisional simulations are, on average, about 10 times more computationally expensive than otherwise-equivalent ideal simulations,” said Dhruv. “In addition, EHT-style modeling is inherently a parameter-survey problem. To evaluate the impact of the new physics, we needed to run multiple weakly collisional simulations alongside matched ideal counterparts, and then compare both sets of results with observations. In this context, GPU resources on Delta were invaluable. A single weakly collisional simulation that takes on the order of a month on A100s would likely take more than a year on comparable CPU-only resources.”
In addition to the research paper linked above, you can find more context for this research in the following publications:
An Extended Magnetohydrodynamics Model For Relativistic Weakly Collisional Plasmas in The Astrophysical JournalFirst Sagittarius A* Event Horizon Telescope Results. V. Testing Astrophysical Models of the Galactic Center Black Hole in The Astrophysical Journal Letters
ABOUT DELTA AND DELTAAI
NCSA’s Delta and DeltaAI are part of the national cyberinfrastructure ecosystem through the U.S. National Science FoundationACCESS program. Delta (OAC 2005572) is a powerful computing and data-analysis resource combining next-generation processor architectures and NVIDIA graphics processors with forward-looking user interfaces and file systems. The Delta project partners with the Science Gateways Community Institute to empower broad communities of researchers to easily access Delta and with the University of Illinois Division of Disability Resources & Educational Services and the School of Information Sciences to explore and reduce barriers to access. DeltaAI (OAC 2320345) maximizes the output of artificial intelligence and machine learning (AI/ML) research. Tripling NCSA’s AI-focused computing capacity and greatly expanding the capacity available within ACCESS, DeltaAI enables researchers to address the world’s most challenging problems by accelerating complex AI/ML and high-performance computing applications running terabytes of data. Additional funding for DeltaAI comes from the State of Illinois.