When Technology and Art Unite

For years John Toenjes has been on a mission to blur the lines between audience and performers in theatrical, dance and musical performances.

“If you give a performance, or show somebody something, that is one thing,” said Toenjes, a professor at the University of Illinois Urbana-Champaign department of dance, and musical director for the department. “But if you make them have agency in how it goes or take an action on how things proceed, that is gamification.”

Toenjes has earned a reputation for weaving technology and interactivity into performances since producing a performance in 2014 in which dancers carried networked tablets onstage and the audience saw an enhanced version of the action onstage with an augmented-reality app. That performance, called Kama Begata Nihilum, a phrase Toenjes made up as a tip of the hat to the classic sci-fi film The Day the Earth Stood Still, also allowed audience members to use the smartphone app to participate in the lighting design for the performance and to point their phones at the stage at various times to initiate augmented reality animations.

Toenjes became a Faculty Fellow with NCSA in 2015 and used his fellowship funds to launch the Laboratory for Audience Interactive Technologies (LAIT) as a way to explore integrating technology into performances and to continue the development of a mobile app that could continue to bring audiences into performances.

That app – now called Mosho – was unveiled in March 2015, first at the University of Illinois and one week later at the American College Dance Festival (ACDA), where dancers held cell phones and used the app to guide an improvisational performance. Since then, Mosho has been refined and improved and used in a variety of settings. For the 2016 performance Interface: Public Figure, the audience used their smartphones to “like” various dancers to help their favorite characters win a small game show within the dance. 

Another performance, called Critical Mass, allowed audience members to participate in the performance in a variety of ways using Mosho. The audience was able to form an online “community” by texting and sending selfies to a large cube onstage, which became a character itself, communicating directly with the audiences’ phones. Dancers shared the ideas behind their dance solos through the app and audience members were encouraged to come onstage. Audience members voted on their favorite duets and the dance changed based on the vote totals. The interactivity meant the audience had a role in determining how the dance progressed. Both performances were collaborations with Los Angeles-based choreographer Chad Michael Hall.

Toenjes said art and technology have always been intertwined (the harpsichord was high-tech in its day, he pointed out) and that artists need to explore how new technologies can help create new types of performances and expression.

In my work, I’m very involved with young people and I see where their interests are. They are into their phones and they are used to having agency over what they see and what they do. If we don’t keep up with that, we won’t have art anymore.

John Toenjes, NCSA faculty fellow

Toenjes’s most recent efforts include working with William Patterson, an associate professor of fine arts and engineering and creator of the Hip Hop Express mobile unit, which teaches kids in underserved communities about music, technology and engineering by giving them opportunities to create their own hip hop tracks. Toenjes will bring motion capture suits used to create dance animations to the Hip Hop Express, exposing young people to the possibilities of new technologies and how they can be used for expression in a culturally relevant way. That project won support from the University of Illinois Research Board, its Grainger Engineering Library IDEA Lab, and its Seibel Center for Design.

He is also delving into virtual reality through a project called Master Dancer. The project, still in the early stages of development and funded by a grant from the Research Board, is an artistic game in which players (the audience) learn dance movements by participating in a series of games. The game centers around famous dancer Loïe Fuller (1862-1928), who requests help from players in restoring a theatre and her dancing abilities. As players complete the mini-games and earn points, Fuller becomes younger and the theatre gets brighter.

“We’re trying to resurrect Loïe Fuller in this game and teach creative movement through the actions the players have to take to succeed in the game,” explained Toenjes. “Virtual reality takes a lot of people and a lot of work, but there are so many possibilities for creating, teaching and interacting.”

The Mosho app has been incorporated into performances by theater companies in Chicago and Cincinnati and by the West Hollywood Dance Festival. While Toenjes sees a natural connection between the arts and technology, he acknowledged that his approach to dance and performance isn’t for everyone.

“I think there is a place for people to just dance. Not everyone has a programming brain,” he said. “But those with a proclivity to logical scientific-type thinking should try it.

“Artists are generally curious people,” he continued. “I am curious about what these technologies can do. I’m always asking, “How can I make art with it?”