Preparing for breakthrough geology | National Center for Supercomputing Applications at the University of Illinois

Preparing for breakthrough geology

05.02.14 -

By Elizabeth Murray

Normal procedure for running a job on Blue Waters goes something like this: compile the code; create an executable; write a submission script; submit the script; and the scheduler runs the script. The script specifies how many nodes to run on, and the scheduler allocates resources on the machine. Once the job is launched, it will run on whatever nodes are set aside for that job.

When it comes to running jobs using community codes—which are complex in their structure but all-purpose in their construction and execution—things can get tricky, especially when using a code for the first time on a new machine.

For Lijun Liu, self-proclaimed geodynamicist in training and computational geologist at the University of Illinois at Urbana-Champaign, this is the very challenge he ran into when first attempting to use a community code on the Blue Waters supercomputer—which he hopes to use in the study of large-scale deformation of the earth through numerical simulations.

Liu specializes in developing geological inverse models. “We take vast amounts of data as inputs and then we run the model backward in time for about 100 million years. This way, we effectively reverse the convection process of mantle,” he explains. “By doing that, we're trying to make a connection between the temporal evolution of the earth's interior with the surface deformation.” With access to such a behemoth system as Blue Waters, he intends to further develop those models, enabling his team to look past the large-scale convection, or the deformation of the earth's interior, and focus more at the fine-scale deformation of the surface structure itself.

For Liu, Blue Waters was uniquely suited for these research objectives. The supercomputer is especially beneficial in terms of resolving this type of multi-scale physics, especially models with a huge amount of unknowns. “Simulating the earth's evolution, including both the mantle and the lithosphere, is not an easy task,” he explains. “This requires a significant amount of computational power plus memory. Blue Waters is by far the best tool we have found.”

First though he had to be able to successfully run a job on Blue Waters, and to do that, Liu needed to make his community code, known as CitcomS, more compatible with the configuration of this particular supercomputer. This is where NCSA senior research programmer and Blue Waters team member Ryan Mokos came in to help. While this code had successfully run on supercomputers across the country, Mokos is intimately familiar with the uniqueness of the Blue Waters system. “While there are many similarities between supercomputing systems, each has its own distinct characteristics. The compilers, libraries, and other components of the Blue Waters software environment have a different structure from most other systems,” Mokos clarifies. “Oftentimes, there are several layers of software and drivers between the user and the computing hardware. I had to dig down a few layers to get to the heart of Lijun's issue.”

As is the case for most community codes, they are created over time with the help of many authors. Often they are complex in their infrastructure, layers deep with many languages. For the code Liu is using, the wide range of languages includes Python, C and C++ with Pyre serving as the submission wrapper.

Lijun's code needed two configuration changes to compile and run on Blue Waters according to Mokos. “First, the programming environment had to be set correctly so the compiler could find all the libraries and all the parts of the code it needed. Then Pyre—which is basically a software framework that simplifies job submission by hiding many of the machine-specific details from the user—had to be changed to make it compatible with Blue Waters,” Mokos continues.

Both Mokos and Liu call these configuration changes a learning experience, but for Liu, it gave him a better understanding of the depth and complexity of the code itself. He is hoping this will enable him to advance his modeling research in ways he hadn’t thought possible before. “Initially, that part of the code was like a black box to me because I never touched it before. It's a big step forward for how I will approach running my jobs in the future.”

Since this code breakthrough, both Mokos and Liu have had multiple successful test runs on Blue Waters and hope to see some breakthrough modeling results follow soon.

National Science Foundation

Blue Waters is supported by the National Science Foundation through awards ACI-0725070 and ACI-1238993.