As scientific output becomes more complex and high-performance computers crunch enormous datasets, the need to help people make sense of numbers continues.
Scientific visualization is a technique of showing complicated output in an easy-to-understand, data-driven visual representation. Scientists and artists team up to mold a series of 3D pictures of data, such as the graphic representation of a severe storm. In the visualization, weather balloons change colors and float up or down within clouds to represent changing air flow. Through sonification, sounds could represent air pattern changes.
"Sonification is the next logical follow-on to visualization," says Donna Cox, NCSA's pioneer of the Renaissance Teams approach to scientific visualization. "Sonification is necessary--especially for high-dimensional datasets."
"Sound is complementary to images in everyday life, at the movies, and in computing as well. It reinforces what we see and tells us things we cannot see," says Robin Bargar, a composer of music and research programmer at NCSA. Bargar leads a group of computer scientists, engineers, and composers who are creating ways to enhance virtual environments by adding real- time interactive sound [see access, Summer 1994]. Group members include Ulrike Axen (computer science), Sumit Dasa and Kelly Fitz (electrical engineering), Camille Goudeseune (music), and Insook Choi (fine and applied arts).
The Audio Development Group recently produced the NCSA Sound Server, a prototype audio software pipeline used for virtual reality. Bargar says this pipeline does for sonification what SGI's Graphics Pipeline did for visualization. With it, data- driven audio can become an integral component of any software application. "Putting sound and graphics together and showing that sound really made a difference opened the door for the work we are doing today," he says.
He began at NCSA assisting Cox in the production of animations and visualizations, including the well-known Venus and Milo, which was featured at SIGGRAPH two years in a row (1990, 1991) in the Computer Graphics and Animation Screening Rooms. Bargar continued producing videos, but his musical background drew him into developing a way for sound to become a major part of scientific research.
Sound is "one more sensory input in which one can quantify or qualify changes in data," says Cox, UIUC art and design professor and NCSA research scientist. "It acts like our visual perception system in giving us information."
Navigation in the nebulous space of virtual reality is a problem, Cox continues. "It is very difficult to find out where you are in this dataset, or artificial space that is created by the computer," she says. "Using audio to figure out where you are in a space is quite valuable. You can use the changing amplitude of a sound to navigate, like a fish in water."
Five years ago, auditory display research had little support in the scientific community. Bargar says the vision and imagination of a handful of early supporters like Cox; David Curtis, creative director of NCSA's scientific communications; and Joseph Hardin, associate director of NCSA's Software Development Group, paved the way for today's success. At this early stage, sonification is just beginning to generate acceptance, but it is a long way from the recognition enjoyed by visualization. For example, computing platforms are far behind in providing necessary hardware and software to create high-fidelity audio signals. "Our first research has been to design and build the tools we need to conduct our work," says Bargar. "Now that the server is in place, we can explore methods to create sound patterns that can convey patterns in data, changing over time."
"It's such an early field," says Adrian Freed of Berkeley's Center for New Music and Audio Technology (CNMAT). "There's lots of promise, but the thing that's missing is some very compelling examples of where it works." Examples are beginning to surface, however. Sonification was showcased at SIGGRAPH 94. All 43 VROOM applications included audio provided by the NCSA Sound Server.
To produce data-driven sounds, researchers need a computer that contains software to manage sound-files and hardware to convert digital data to analog audio signals. It is important to have an audio system attached to the computer that reproduces high- fidelity signals. Since the software to turn data into sound is not standard on most platforms, the Audio Development Group drew on existing computer music software and adapted it to scientific research.
"The computer music community is an untapped resource for computational science and engineering" Bargar says. "Computer musicians--mostly composers--and their engineering collaborators have been inventing hardware and software for producing data-controlled sound for over 30 years."
The Institute for Research and Coordination of Acoustics and Music (IRCAM) in Paris is one of the original centers for collaborations between computational science and music composition and performance. Xavier Rodet, an audioengineer from IRCAM, assisted Bargar in adapting chaotic circuit equations into a real-time synthesis environment for Sounds from Chaos in Chua's Circuit [see access, Summer 1994]. One of the world's leading audioengineering researchers in signal processing and physically based models for sound synthesis and musical applications, Rodet has developed important techniques for understanding and creating sounds using spectral representations. These techniques are similar to those used in astrophysics and other sciences.
"Sound is a supplementary information channel that has different properties than the eye. You don't have to face the sound source to hear it; it can wake you up even if you are sleeping," Rodet says. "So by this other channel, more information can be transmitted to a user. This is crucial for very complex information such as highly dimensional structures."
As an example, Rodet points to his study of a version of Chua's circuit called the Time-Delayed Chua's circuit. "The different behaviors--different periods and chaos--are extremely easy to discriminate by ear according to parameter values, whereas this is not so easy by other means."
When asked what sound contributes to science, Rodet replied: "In medicine, the Doppler effect is used to get some information about blood circulation. [The Doppler effect is the change in the observed frequency of an acoustic wave caused by the motion of a sound's source and one who observes it.] The corresponding sound is a very convenient way to carry this information to the doctor since he has to look at the measuring device in order to orient it precisely. He could not be looking at a screen at the same time.
"So sonification is a way to help in the exploration of complex data, which appear more and more in modern scientific research. Think of biology or population dynamics, for example."
"These collaborations are important for the transfer of software technology," Bargar adds, "including algorithms and rendering engines, from highly specialized and sometimes inaccessible fields such as computer music and digital sound synthesis, into a much larger scientific user community."
Through another collaboration, Bargar was able to get prototype software to NCSA's server that was developed at CNMAT. The Audio Group adapted the software to NCSA's needs. "We are not working on sonification, but we build tools," says Freed, CNMAT's director of Software and Systems. "Our tools work on SGI machines, and the CAVE is driven by SGI machines. [The NCSA Audio Development Group] took our sound synthesis tools and built layers on top to make them accessible to their audience."
Interactions with science research groups have also been fruitful. In a project involving UIUC's Department of Computer Science and NCSA's Software Development Group, sound is generated by traversing geometric structures called alpha shapes, to study features that may be hidden in graphical representations [see access, Summer 1994]. "Support from Herbert Edelsbrunner [UIUC computer science professor] in terms of ideas and models as well as assistantships for audio personnel really uplifted the Audio Development Group during our formative period," Bargar reported. "We are delighted that one of Herbert's students is now a regular member of the Audio Development Group and is considering sound as part of her doctoral thesis."
Ongoing interactions between the CCSR (Center for Complex Systems Research) and the Audio Development Group have led to a number of exciting projects involving the sonification and visualization of nonlinear dynamical systems and "deterministic chaos." In particular, these interactions formed the basis for the use of the chaotic behavior of the Chua circuit in musical composition [see access, Summer 1994], provided an important test case for development of the CAVE interface, and are currently suggesting more effective and appealing ways of rendering sound from EEG patterns and of producing acoustic stimuli for inducing EEG responses in laboratory tests.
"We look forward in the near future," adds Bargar, "to establishing a dedicated audio facility where researchers can collaborate with NCSA to produce sound from their computational models."
"We hope that by adding sound to our virtual reality CAVE environment, we can create a more realistic virtual prototype of our vehicles," says Kem Ahlers, manager of University Relations for Caterpillar on-site at NCSA. "By making a more realistic virtual prototype of Caterpillar tractors, we can reduce the amount of costly iron prototypes we must build, and we are able to evaluate many more potential designs. This creates a product that has more value to our customers and means more profit. We can also bring those designs to market quicker, which we need to do to compete in this new global marketplace. All of these hopefully translate into more profit for Caterpillar."
"Sonification is one corner of the field of auditory display that is becoming increasingly important for many segments of the computer community," he says. "In the past, computers were only accessible by numerals and by text. More recently, graphics revolutionized the ability of the computer and user to interact. With multimedia computing upon us, computers are being built to reproduce sound with the same quality as a home compactdisc player and stereo system. This means that audio tools and computed sounds now have more ways to be heard than ever before.
"Sound--produced and controlled by computed data--will play a larger role in these environments as people learn how to approach sound from a computational perspective and as tools become established in the form of stable software."
