Skip to main content

Meet HOLL-I: NCSA, Cerebras Systems Unveil New Artificial Intelligence Machine


Cerebras front pattern on their systems can be seen overlaid in the sky of NPCF in hues of orange/purple.

The National Center for Supercomputing Applications (NCSA) has officially launched a new supercomputer targeted at the training of large artificial intelligence (AI) models on extreme scale datasets. HOLL-I, or Highly Optimized Logical Learning Instrument, uses the new Cerebras Systems CS-2 Wafer Scale Engine, providing an at-cost resource for machine-learning training algorithms for researchers and industry. For a limited time, NCSA is offering a discretionary trial-use period for new applicants.

At base, the unit is centered on the CS-2, a top-of-the-line, high-performance single wafer processor with a supporting compute cluster. Deep learning research needs are growing ever larger with high loads of detailed and unwieldy files. Given these massive datasets, often heavy with imagery or text, the CS-2 is structured to specifically reduce the load times and distance to memory for computationally taxing programs. While NCSA has other compute resources, such as Delta, Hal, DGX, Campus Cluster and Radiant, HOLL-I’s architecture is built to handle machine learning jobs at intense speeds that will cut down on compute time and lower overall costs while obtaining excellent performance results.

The HOLL-I system is a great example of the evolution of computing to specialized systems for important classes of applications.

William Gropp, NCSA director

Part of the creation of HOLL-I is about the story of NCSA’s Industry Partners program. For more than 30 years, a central part of NCSA’s work has been to create and reinforce a bond between academics and industry. Foundational research is difficult and risky in the private sector, requiring massive investment that is not always borne out with identifiable profit. Allowing some of that risk to pass to academics whose goal is purely to advance science, creates an ideal environment for experimentation. Then adding in the targeted needs of NCSA’s industrial partners provides a needed impetus for innovation. 

“Our innovative industry partnership has made this amazing system for machine learning and other applications available to Illinois researchers, and I am looking forward to seeing how this accelerates research at Illinois,” said Gropp.

Brendan McGinty, associate director of the Industry Partners program, played a key role in getting the CS-2 onto the University of Illinois Urbana-Champaign campus. NCSA maintains a close relationship with our corporate players, and McGinty, working closely with Seid Koric, technical assistant director, keeps a finger on the pulse of technological innovation. When Cerebras announced the CS-1, NCSA started getting feedback that having the raw AI computing power of such a unit would be a significant advancement. 

Said McGinty, “The CS-2 will be part of the NCSA legacy in HPC, a real next-generation story.” As plans began to take shape, it was clear what the pitch for the Industry Partners group would be: “Compute expertise and the people who know what to do with it.” A research study was commissioned, a proposal drafted – all while McGinty kept close contact with NCSA’s industry partners, some of whom had already begun trial uses of the CS-1. With a clear path forward, NCSA began working with Cerebras Systems to get its latest upgrade to the CS architecture, the CS-2.

We founded Cerebras Systems with the audacious goal to forever change the AI compute landscape. Not only are we seeking to accelerate AI workloads orders of magnitude over what is possible on legacy hardware, but we also want to put this extraordinary capability in the hands of academics and researchers. Partnering with NCSA ensures that academics and researchers will have access to the world’s fastest solution for AI and HPC.

 Andrew Feldman, CEO and co-founder of Cerebras Systems

Cerebras Systems’s Wafer-Scale-Engine (WSE) is remarkable. Understanding that much of the slowdown in systems architecture has been in the space between nodes, and the space between nodes and stored data, the enormous WSE is designed to make both of these types of input/outputs almost instantaneous.

Image of the WSE, Wafer-Scale Engine, courtesy of Cerebras Systems
Image of the WSE, Wafer-Scale Engine, courtesy of Cerebras Systems

According to Cerebras Systems, the Wafer-Scale Engine (WSE-2) is the largest chip ever built and powers the CS-2 system. The WSE-2 is 56 times larger than the largest GPU, has 123 times more compute cores, and 1000 times more high performance on-chip memory. The only wafer-scale processor ever produced, it contains 2.6 trillion transistors, 850,000 AI-optimized cores, and 40 gigabytes of high performance on-wafer memory. Purpose built for AI, the CS-2 system replaces an entire cluster of graphics processing units (GPUs). Gone are the challenges of parallel programming, distributed training, and cluster management. From chip to system to software – every aspect of the CS-2 is optimized to accelerate and to simplify AI work.

This diagram shows the layout of the processing elements, or PEs. With 40 gigs of RAM spread out among each chip, each node can operate independently or work together with other nodes on a project. The accelerator partitions the WSE for the job at hand. Unlike some supercomputers, like BlueWaters and Delta, HOLL-I is structured to perform one job at a time – and though it can perform more jobs, optimizing the computational space for a single job is part of the intended workflow. Using Slurm, a common cluster management and job scheduling system, users submit their jobs to the system with one of the two most competitive ML APIs, TensorFlow or PyTorch frameworks.

What makes the CS-2 so different? For years, desktops, and mobile devices have continued to get smaller, so why is the wafer so big? Computer scientists, developers and hardware engineers are familiar with the concept of Dennard Scaling, that is, the idea that every few years, the hardware gets smaller, but capacity and speed gets larger. To be more precise, as the size of the transistors gets smaller, the corresponding voltage is lowered, and the circuits can operate at higher frequencies at the same power. There’s even an equation for it. And while this notion seemed to have held for many years, by 2006, such gains were nullified by leakage current, and threshold of voltage, causing an increased power density that could cause the chips to overheat. And so, for a while, hardware manufacturers were left with the idea of scaling up the transistors and attempting to reduce or eliminate leakage. Cerebras CS-1, unveiled at SC 2019 and now the CS-2, took a different tactic: “reinvigorating innovation in architecture, particularly in using specialization to particular computing needs to achieve more performance.” said Director Gropp. 

There’s another aspect of the CS-2 that has the potential to make waves: the standard supercomputer has traditionally filled up entire rooms. Powering those computers, and keeping them at the right temperature in the cleanest, dust-free environments used megawatts of power from both the computers and the associated cooling devices. But with the CS-2 unit being relatively contained, its own massive cooling tank, and the specialized node-to-memory architecture, huge power savings are expected per unit of productivity. Not only does the CS-2 perform jobs more quickly, it is just a physically smaller footprint than the racks upon racks of blades that take up supercomputing facilities around the globe.

We’re thrilled to have the Cerebras CS-2 system up and running in our HOLL-I supercomputer. HOLL-I is unique in the NCSA’s AI computing space in that we will have multiple clusters at NCSA that address the various levels of AI and machine-learning needs. Delta and HAL, our DGX providing resources, and now HOLL-I with the CS-2 as the go-to resource for AI jobs. Each system is at the correct scale for the various types of usage and all systems having access to our shared center-wide TAIGA filesystem eliminating delays and slowdowns caused by data migration as users move up the ladder of more intense machine learning computations.

Vlad Kindratenko, director of the Center for Artificial Intelligence Innovation at NCSA

NCSA, acutely aware of limited compute resources and the process by which allotments are distributed, offers HOLL-I as a machine-learning supercomputer available to all. Following in the footsteps of previous systems that NCSA has deployed for industry and for special research needs HOLL-I is available to researchers, public or private, at-cost. 

“Cerebras is an ‘appliance’ that is part of a larger computing ecosystem, said Gropp. While the CS-2 can be programmed to perform other computations, training for machine learning is the core application, and one that, because of the broad and growing use of machine-learning, is important enough to tune a system to that specific task. By integrating HOLL-I into our center-wide file system, users can take advantage of other systems for today’s complex workflows.”

HOLL-I and NCSA aren’t in this “computing ecosystem” alone. Having HOLL-I at NCSA is a net benefit to the university and the state as well. Not only do local corporate partners benefit, but the CS-2 has the speed to reduce R&D expense for companies nationwide. The type of machine learning for which HOLL-I will be dedicated has already accomplished much for NCSA corporate partners, creating models and simulations that enhance efficiency and reduce waste, saving millions of dollars. Per McGinty, NCSA brings “faster, more accurate, deeper solutions and results to the world’s grandest challenges. What that translates to is return on investment.” That investment can also translate to money invested in the University of Illinois, and the state itself. When Senator Richard Durbin visited the NCSA recently, he noted similarly, “university research has a ripple effect that helps power regional economies.” He went on to note that such investment  “creates new companies, industries, opportunities and jobs, and the tech-savvy educated workforce needed to fill them. In the state of Illinois, the impact is significant.”

Learn More About HOLL-I

  • See our resources page and our project highlights page.
  • Email Volodymyr Kindratenko for more information
  • Email NCSA Help to start the onboarding process
  • Visit the HOLL-I site as tutorials and documentation become available
Back to top