Audio + Visual

By Tracy Culumber

So you thought the visualizers on iTunes were entertaining? How about a live musical performance where digital artwork goes well beyond oscilloscopes and actually creates music in a real-time collaborative, interactive virtual world.

Following the International Computer Music Conference in Copenhagen last August, Ben Smith sat at his laptop on a stage in a darkened night club. As he and another performer began to work at their computers, flashes, streams of light, geometric shapes, and explosions appeared on the black screen behind them. As these visualizations danced and floated in the virtual space, their interactions created what Smith calls “MusiVerse,” an improvised, collaborative online symphony of images and sounds. Smith’s performance changed with every keystroke. Eerie and modern, the notes resonated throughout the room and the audience responded with applause.

Smith, a graduate student in the University of Illinois at Urbana-Champaign School of Music, developed an algorithmic computer music program that allows users to produce perfectly synchronized sounds and images with the ease of creating a website. Drawing upon the same technology that makes elaborate multi-person online games and virtual worlds possible, Musiverse creates visualizations.

“In the MusiVerse I go beyond superficial connections and try to create deeper relationships between the visuals and the music,” Smith explains. “Music itself is a result of carefully structured sounds and it is my hope that by carefully structuring the simultaneous generation of the graphics and music from a single source dataset, I can portray deeper connections and bring the audience into a new aesthetic experience.”

Smith recently took his music experience another step further with a performance at the university’s Krannert Center for the Performing Arts. The performance was similar to the Copenhagen debut, but with the addition of several other performers logged in to the MusiVerse portal from New York, Indiana, and Chicago. Smith says that although only 10 performers contributed to the Krannert performance, the server can theoretically support a few dozen performers (musical programmers) in the same online environment from anywhere in the world.

“I am very interested in reaching a bigger audience than this sort of music software usually reaches, making it a networked thing so people can share what they are creating,” he says.

Perfecting the art

The idea of using computers in performance arts is not new, but in the past most computer-generated music performances lacked visuals, leaving the audience to stare at the performer sitting at a computer while the music played. In the case of music program visualizers, the only connection between the sound and the visuals is the oscilloscope (the bouncing line that represents the audio waveform) so any actual alignment between them is coincidental and won’t repeat if you play the same song over and over.

Unsatisfied with this, Smith experimented with different online environments, like Secondlife, and found that visuals with music vastly increased his audience.

“I came to this idea after seeing results of what everyone else was doing, then rebelling,” Smith says. “What I see missing is the visual connection between the performer and the music, and how it is being played and produced; my focus is on the interaction between performers and computers.”

Audience members have mistaken Smith’s smooth improvisation for prerecorded material a mistake that he considers a great compliment. But the development of MusiVerse is not without challenges. Although complications associated with large-scale, high-resolution graphics and computing pose the majority of challenges to his research, the settings engrained into the code for computer game software, such as gravity and terrain, are not optimal for his visualizations and difficult to remove. Also, game websites, or “toy simulations,” do not involve immediate interactions and they are not about collaborative immediate responses. He has spent the past year researching ways to avoid these problems and provide “online world-users” with fewer limitations on how they use computers to create art.

Although he still needs to walk others through the MusiVerse software initially, audience participants from the Krannert performance say they found the program easy to use once mastered. Smith’s goal for the technology is to make the creation of these visual-musical performances a reality for anyone with a computer.

Smith’s faculty advisor for the project is Guy Garnett, a professor of music and computer science and an NCSA Faculty Fellow; Smith also consulted on the project with NCSA researcher Robert McGrath. McGrath agrees that Smith’s research has great potential to become a household application. “I have visions of this being something that is easily available so people can sit down …and start jamming on the network,” he said.

Cyberenvironment in action

Smith says that his project is not the culmination of the technology, but rather a milestone that shows how online environment technology works.

“There is a shortage of people who are both interested in computing and talented in the arts, so we need to look for ways to change that situation and break down the barrier between the scientific side of campus and the arts side,” says Garnett, who is co-director of the Cultural Computing Program’s mWorlds initiative. CCP is a campuswide initiative at Illinois to foster creative activities, innovation, collaboration, and research that spans technology and the arts.

mWorlds are synthetic worlds focused on testing the limits of creativity, scalability, security, and flexibility in online, collaborative environments through art, science, research, education, and entertainment. The goal of the mWorlds initiative is to make the creation of and participation in these online worlds as easy as creating websites or browsing the Web. MusiVerse is a good representation of the endless possibilities of projects users can tackle in a synthetic world.

“We want to simplify and generalize the process of creating a virtual world and make it more flexible,” Garnett says. “The idea is to make this technology accessible on a desktop and ensure that the environment is collaborative and distributive, like Ben’s performance.”

Smith said in the near future he would like to use this interface to develop music that is not necessarily modern or “hard-to-listen-to” but music that people hear at a concert or on the radio.

“I want it to be music people are used to listening to in the clubs,” Smith says. “They would really freak out if they saw this with that music.”

Disclaimer: Due to changes in website systems, we've adjusted archived content to fit the present-day site and the articles will not appear in their original published format. Formatting, header information, photographs and other illustrations are not available in archived articles.