NCSA’s Private Sector Program pushes LS-DYNA code to extreme scale on Blue Waters sup

LS-DYNA, an explicit finite element code used for simulations in the auto, aerospace, manufacturing, and bioengineering industries, was recently scaled to 15,000 cores on NCSA’s Blue Waters supercomputer—a world record for scaling of any commercial engineering code.

Both software developers and end-users face constraints when it comes to testing the limits of commercial codes. They often don’t have access to truly massive supercomputers, and their resources and staff are focused on daily business needs—they can’t spare the time and manpower to attempt extreme scaling studies. NCSA’s Private Sector Program (PSP) is able to bring all of the key components together: LS-DYNA developer LSTC; the petascale Blue Waters supercomputer and its hardware manufacturer, Cray; the industrial users with real challenges; and the expertise of PSP’s staff.

“Once Blue Waters was in production, we looked for test cases to run at extreme scale,” says Seid Koric, a senior computational resources coordinator with NCSA’s PSP and a University of Illinois adjunct professor of Mechanical Science and Engineering. Software company LSTC provided a large license pool for LS-DYNA for this benchmarking. This enabled the collaborative team to run the software across as many cores as possible. LS-DYNA was quickly scaled to 1,000 cores on Blue Waters; Koric continued to run larger and larger real-world problems provided by a PSP partner on Blue Waters, pushing the code to 8,000 cores.

Progress was iterative, with repeated analysis of bottlenecks addressed by the software development team. For example, Koric worked with Cray and LSTC to distribute the problem across the system’s memory efficiently. Performance was also boosted when the team switched from an MPI solver to a hybrid MPI/OpenMP solver with lower communication overhead and a smaller memory requirement.

With his expertise in both high-performance computing and mechanical engineering, Koric was able to examine the physics of the problem and suggest specific algorithms that might benefit from further parallelization. Cray profiled the code, confirming Koric’s assessment, and LSTC removed those algorithmic bottlenecks.

“Cray was very helpful in understanding the system,” Koric says. “They know all the tricks of the hardware, particularly when it comes to load balancing and profiling the code, while the same applies to LSTC and their LS-Dyna code.”

Intrigued by the performance gains, a second PSP partner provided a complex engineering problem, which the team was able to scale to a world-record 15,000 cores on Blue Waters in January 2014.

For more details on the project, contact Seid Koric at skoric@ncsa.illinois.edu or 217-265-8410.

Disclaimer: Due to changes in website systems, we've adjusted archived content to fit the present-day site and the articles will not appear in their original published format. Formatting, header information, photographs and other illustrations are not available in archived articles.