A New Era of Astrophysics Discovery

Rubin Observatory in March 2024. Credit: Rubin Observatory/NSF/AURA/A. Pizarro D.

You might be surprised to find out that most profound breakthroughs in modern astrophysics no longer happen solely at the lens of a telescope, but within the world’s most powerful supercomputers. With the help of the National Center for Supercomputing Applications (NCSA) and the Center for AstroPhysical Surveys (CAPS), we have entered a new era of discovery where the practical impact of space science is measured by our ability to transform petabytes of raw data into actionable knowledge. Today, in addition to looking at the stars through massively powerful telescopes, research is also about the computational resources required to decode the data collected from these observations.

At the University of Illinois Urbana-Champaign (U. of I.), this shift is driven by NCSA and CAPS. In many ways, the universe is the ultimate big data challenge. Through a partnership that has yielded numerous breakthroughs, NCSA and CAPS prove that studying the furthest reaches of space drives the very hardware and AI innovations that define our technological future. “This is literally a new era in astronomy,” said Joaquin Vieira, director of Astronomy and CAPS.

The power of these partnerships is showcased in events like the annual AstroFest conference. Hosted at NCSA, the conference brings together researchers from around campus to discuss their progress and achievements. This year, one of the presenters was Britt Lundgren, a U. of I alumnus (Ph.D. Astronomy 2009) and Philip G. Carson Distinguished Professor in the Sciences at the University of North Carolina Asheville (UNC Asheville). Lundgren is also notable for being a member of the Astronomy and Astrophysics Advisory Committee (AAAC). The committee recently presented its annual report to Congress, U.S. National Science Foundation (NSF) leadership, NASA and the Department of Energy. The report assesses the federal astronomy and astrophysics research portfolio and progress toward the priorities outlined in the Astro2020 decadal survey, a blueprint for federal investment in space science, that highlights NCSA and CAPS as key contributors to the nation’s progress in handling massive survey datasets. Specifically, the AAAC report identifies the processing of massive survey datasets, like those from the Dark Energy Survey (DES), the Legacy Survey of Space and Time (LSST) and the South Pole Telescope (SPT), as a critical achievement in the nation’s progress toward understanding the fundamental laws of the universe.

“We’re currently experiencing a golden age for astronomy, in which large surveys like SDSS and Rubin/LSST are producing vast datasets that enable astronomers to answer big questions with new precision and also discover and pursue rare objects and phenomena,” said Lundgren. “What excites me the most about the current moment is the relatively new paradigm of making these large, science-ready datasets publicly accessible through the web – a transition that has truly democratized exploration and discovery in our field. Data from these massive surveys can be accessed and visualized by anyone with a web browser, enabling students and the public to explore cutting-edge professional astronomy data while building transferrable skills in coding, big data analysis and visualization. Federal investment has been critical to developing this modern survey technology and data infrastructure, which directly supports the education of the next generation of astronomers and the development of the STEM workforce more broadly.”

The following three projects serve as evidence of this computational revolution, demonstrating how high-performance computing (HPC) is powering the next wave of astrophysical breakthroughs.

The Dark Energy Survey (DES) is a U.S.-led international project that has mapped large portions of the sky at optical wavelengths, surveying everything from galaxies to supernovae. The main goal of the project is to better understand dark energy and why it seems to accelerate cosmic expansion. In the first six years of operation, the DES recorded information about 550 million galaxies, giving researchers an unprecedented amount of data to study. NCSA, along with Fermilab and NOIRLab, is a founding partner of the DES project. The DES project is funded by the NSF and DOE.

DES recently published results that combine all six years of data collected from weak lensing and galaxy clustering probes – a first for the international collaboration that is mapping hundreds of millions of galaxies, detecting thousands of supernovae and analyzing patterns of cosmic structure that could reveal what is accelerating the expansion of the universe.

NCSA led end-to-end data processing and archival for DES using the DES Data Management System. NCSA operated the data management and computing infrastructure that processed, quality-controlled and served the full DES imaging and catalog data set, enabling the creation of science-ready sky maps and cosmological measurements.

“Dark energy and the universe’s accelerating expansion sit at the boundary between what we can measure precisely and what we can explain,” said Vieira. “Pinning down what is driving that acceleration would reshape our understanding of the universe’s fate and force revisions to the deepest laws that describe space, time and matter.”

There are many trillions of objects in observable space. The NSF–DOE Rubin Observatory began tracking a sampling of these trillions of objects last year and released its first images in June, 2025. The AAAC report highlights the Rubin Observatory as a top national priority for the coming decade. NCSA has been a partner in the Rubin project since its inception, ensuring the infrastructure is ready for this massive influx of data.

The Rubin Observatory project aims to conduct a 10-year optical survey of the visible sky. Such an enormous undertaking requires decades of research and work. This spring, after meticulous planning, the Legacy Survey of Space and Time (LSST) camera was installed on the Simonyi Survey Telescope at Rubin. The camera is the world’s largest digital camera, and it’s expected to capture 500 petabytes of image data over the course of the project.

Stephen Pietrowicz, a principal research software engineer at NCSA, is part of the CAPS team, and his recent work has been with the NSF–DOE Vera C. Rubin Observatory, funded by the U.S. National Science Foundation (NSF) and the U.S. Department of Energy’s (DOE) Office of Science.

He’s now part of the middleware team for Vera C. Rubin’s Data Management group. The NCSA team’s work encompasses many different parts of the project’s data management. They’re responsible for gathering data used to construct the images, data movement between sites and orchestration of image processing campaigns. Pietrowicz manages several different tasks, including the Observatory Operations Data Service, or OODS. “I wrote the OODS, which handles images sent by the Simonyi Survey Telescope. My software quickly ingests those images at the summit in Chile, so they can immediately be used by scientists.”

Support from NCSA has helped Rubin make such breakthroughs as their recent First Alert system, a “near-real-time alert system” that will “enable scientists around the world to coordinate follow-up observations like never before,” according to the recent First Alerts press release posted by the Vera C. Rubin Observatory.

“These projects are large and complex,” said Vieira. “They require sustained partnerships and coordination among national laboratories, researchers, and institutions such as NCSA. CAPS plays a crucial role on campus by enabling the University of Illinois to act as more than a collection of individual faculty. It allows us to operate as a peer institution with national labs and to contribute meaningfully to major big-science efforts, including large cosmological surveys that cost nearly a billion dollars and span more than a decade.”

Funded by a five-year, $20 million grant from the U.S. National Science Foundation (NSF) and the Simons Foundation, in 2024, NCSA partnered with other academic institutions and federal laboratories in the Midwest to develop new artificial intelligence (AI) tools to advance astrophysics research and exploration of the universe.

Led by Northwestern University, the collaboration established the NSF-Simons AI Institute for the Sky or SkAI (pronounced “sky”), one of two AI research centers that will help astronomers better understand the cosmos. This move directly aligns with the AAAC’s emphasis on the ‘computational revolution’ in astronomy, where AI is no longer just a tool but a foundational infrastructure for interpreting the next generation of celestial data.

“Our mission at the Center for AstroPhysical Surveys (CAPS) in NCSA has been to bring together innovative software and cutting-edge hardware to tackle the most pressing questions in the universe,” said SkAI co-principal investigator and CAPS Deputy Director Gautham Narayan. “We’re very excited to have our students, postdocs, faculty and staff deepen our involvement with our colleagues at Northwestern and U of Chicago, provide the entire SkAI community access to NSF’s Delta and DeltaAI supercomputers here at NCSA and build tools and services that lead to AI methods becoming more interpretable and reliable. Our goal is to democratize AI and make it more trustworthy – not just for astrophysics and cosmology, or our campus, but for everyone. This is a big leap forward, and Illinois will lead the way.”A number of CAPS personnel hold leadership positions within the SkAI Institute Project, and several SkAI-funded projects led by CAPS members are underway.

---

ABOUT CAPS
NCSA’s Center for AstroPhysical Surveys (CAPS) brings cohesion and advancement to astrophysical survey science efforts across the University of Illinois Urbana-Champaign. CAPS offers annual postdoctoral and graduate fellowship programs, publishes vast amounts of astronomical data from a broad portfolio of astrophysical surveys and collaborates across disciplines to enable trailblazing research.