Audio + Visual

released 09.18.07

By Tracy Culumber, NCSA

So you thought the visualizers on iTunes were entertaining? Check out a live musical performance where digital artwork goes well beyond oscilloscopes and actually creates music in a real-time collaborative, interactive virtual world.

Following the International Computer Music Conference in Copenhagen, Denmark, in August, Ben Smith sat at his laptop on a stage in a darkened night club. As he and another performer began to work at their computers, flashes, streams of light, geometric shapes, and explosions appeared on the black screen behind them. As these visualizations danced and floated in the virtual space, their interactions created what Smith calls "MusiVerse," an improvised, collaborative online symphony of images and sounds. Eerie and modern, the notes resonated throughout the room and the audience responded with applause. Reminiscent of visualizations that appear on music applications such as iTunes and online worlds such as SecondLife.com, Smith's performance changed with every keystroke.

Smith, a graduate student at the University of Illinois at Urbana-Champaign School of Music, has developed an algorithmic computer music program that allows users to produce perfectly synchronized sounds and images with the ease of creating a website. Smith's faculty advisor for the project is Guy Garnett, a professor of music and computer science and an NCSA Faculty Fellow; Smith also consulted on the project with NCSA researcher Robert McGrath.

Smith's software draws upon the same technology that makes elaborate multi-person online games and virtual worlds possible. Smith will showcase his work during a live performance of "MusiVerse" at the Krannert Center for the Performing Arts at 5 p.m. Friday (Sept. 21).

"In the MusiVerse I go beyond superficial connections and try to create deeper relationships between the visuals and the music," Smith said. "Music itself is a result of carefully structured sounds and it is my hope that by carefully structuring the simultaneous generation of the graphics and music from a single source dataset, I can portray deeper connections and bring the audience into a new aesthetic experience."

The Krannert performance will be similar to the performance in Denmark, but with the addition of geographically distributed performers; participants will log in to the MusiVerse portal from New York, Indiana, and Chicago as well as at Krannert Center. Smith said that although only 10 performers will contribute to the Krannert performance, the server can theoretically support a few dozen performers (musical programmers) in the same online environment from anywhere in the world.

"I am very interested in reaching a bigger audience than this sort of music software usually reaches—making it a networked thing so people can share what they are creating," Smith said.

The idea of using computers in performance arts is not new, but in the past, most computer-generated music performances have lacked visuals, leaving the audience to stare at the performer sitting at a computer while the music played. In the case of music program visualizers, like on iTunes or Windows Media Player, the only connection between the sound and the visuals is the oscilloscope—the bouncing line that represents the audio waveform—so any actual alignment between them is coincidental and won't repeat if you play the same song over and over. Unsatisfied with this, Smith experimented with different online environments—like SecondLife—and found that visuals with music vastly increased his audience.

"I came to this idea after seeing results of what everyone else was doing, then rebelling," Smith said. "What I see missing is the visual connections between the performer and the music, and how it is being played and produced; my focus is on the interaction between performers and computers."

Although audience members have mistaken Smith's smooth improvisation for prerecorded material—a mistake that he considers a great compliment—he encountered several challenges in the development of MusiVerse. Smith explains that although complications associated with large-scale, high-resolution graphics and computing pose the majority of challenges to his research, the settings engrained into the code for computer game software—such as gravity and terrain—are not optimal for his visualizations and difficult to remove. Also, game websites, or "toy simulations," do not involve immediate interactions and they are not about collaborative immediate responses.

In September 2006, he began to research ways to avoid these problems and provide "online world-users" with fewer limitations on how they use computers to create art. Although he still needs to walk others through the MusiVerse software, Smith's goal for the technology is to make the creation of these visual-musical performances a reality for anyone with a desktop and a word processor.

NCSA's McGrath agrees that Smith's research has great potential to become a household application.

"I have visions of this being something that is easily available so people can sit down in the (Department of Computer Science) Siebel Center and start jamming on the network," he said.

However, Smith explained that his project is not the culmination of the technology, but rather a milestone that shows how online environment technology works. As a participant in the Cultural Computing Program (CCP)—a campuswide initiative to foster creative activities, innovation, collaboration and research that spans technology and the arts—Smith has worked on MusiVerse and other projects with faculty and researchers from the School of Music, NCSA and the Department of Computer Science for over a year.

"There is a shortage of people who are both interested in computing and talented in the arts, so we need to look for ways to change that situation and break down the barrier between the scientific side of campus and the arts sides," said Garnett, who is co-director of the CCP's mWorlds initiative.

mWorlds are synthetic worlds focused on testing the limits of creativity, scalability, security, and flexibility in online, collaborative environments through art, science, research, education and entertainment. The goal of the mWorlds initiative is to make the creation of and participation in these online worlds as easy as creating websites or browsing the Web. Garnett said MusiVerse is a good representation of the endless possibilities of projects users can tackle in a synthetic world.

"We want to simplify and generalize the process of creating a virtual world and make it more flexible," Garnett said. "The idea is to make this technology accessible on a desktop and ensure that the environment is collaborative and distributive—like Ben's performance."

Smith said in the near future he would like to take his interface and develop music that is not necessarily modern or "hard-to-listen-to" but music that people hear at a concert or on the radio.

"I want it to be music people are used to listening to in the clubs," Smith said. "They would really freak out if they saw this with that music."