T. rex on your desk!

10.07.12 -

by Nicole Schiffer

NCSA's Alan Craig describes how augmented reality brings a new dimension to information.

Augmented reality, or AR, digitally superimposes sights and sounds on the images captured by the camera on your smartphone, iPad, or other electronic device, giving the sensation they are really there. Projectors can bring AR to a larger audience, and other devices can appeal to your senses of smell, touch, and taste in addition to sight and hearing. Possibilities are myriad.

Reality of AR

NCSA's Alan Craig works at the forefront of this new technology. When he asked me if I had seen AR, I figured I had seen something similar enough that the novelty would quickly pass—a common reaction, says Craig.

He laid a picture of a T. rex on regular printer paper on the desk in front of me, handed me an iPad, and told me to point the camera at the picture. In less than a second, a little three-dimensional T. rex popped up. "Tilt the iPad up," he instructed.

The dinosaur stood on the desk like the little Hungarian Horntail dragon Harry Potter drew from a bag!

I reached behind the iPad and poked at the miniature. It didn't react and my finger disappeared behind the 3D image, but this is only a prototype.

"Things seem fake until we interact with them the way we do in the real world," says Craig.

How does AR work?

It starts with an idea, something to communicate, says Craig. He and his team plan how to display the content and then find an existing application to implement their plan or write their own app.

Craig started experimenting several years ago with BlackMagic, an application based on the freely available, open-source ARToolKit. BlackMagic creates what is called a fiducial marker, which is some symbol that a device's camera will recognize, and associates it with a 3D image.

One drawback of ARtoolKit and ARToolworks is that they require a customized version for each operating system. Craig said that three different Android phones would require three different implementations because of small variations in the open-source Android operating system.

"This is not a shortcoming of that specific product," says Bob McGrath, a retired NCSA software developer who is working with Craig on the project. "AR is extremely closely tied to the video input and output of the device, and these are quite different on each platform, at least up until now."

AR in use

Where might AR be used? Games, advertising, education, news—the list goes on. Even in its infancy, AR is making inroads in many areas.

For instance, Starbucks has run AR holiday ad campaigns using their cups as the trigger image for the AR app. Marvel used AR to activate videos with sound over the comic book pages of Avengers vs. X-Men, including animating the evolution of some pages from pencil to full-color panels. Even more staid publications like newspapers are getting into the AR game. The Arkansas Democrat-Gazette printed its first enhanced edition on July 1, peppering it with icons that indicate interactive material. Several game systems have taken a shot at incorporating AR. Nintendo 3DS, for example, comes with the AR Games suite.

AR in education

Craig's main interest is in education. He uses AR to make molecules pop out of a chemistry textbook, simulate an archeological dig for students in a classroom, and create an interactive 3D display of anatomical systems in the human body.

An early project for Craig's team was working with Maurice Godfrey at the University of Nebraska Medical Center to create DNA models for "DNA Days" at Omaha's Henry Doorly Zoo and Aquarium. More than 15,000 people used markers in their programs to see 3D images of DNA at a station set up at the zoo. Godfrey says "there was a lot of wow factor there."

After their first booming success, the collaboration continued for a new project, adding color 3D depictions of wildflowers to an ethnobotany workbook. The workbook is designed to educate Native American high school students in Nebraska and South Dakota about native flora and originally included text and black-and-white sketches of the plants.

The first step for the NCSA portion of the team was to talk to botanists about how to accurately portray the plants. Student employee Ryan Rocha created the images using a 3D image and animation software package called Maya, which is also used in scientific visualization at NCSA and in some Hollywood movies.

Close collaboration between the developers and artists was essential throughout the creation of graphics, says McGrath. Better graphics take more memory, sometimes more than smaller devices have. The team aimed for images that were as good as they could be without sacrificing usability.

"Of course, 'good enough' depends on the goals of the user...We will have to adjust graphics to satisfy user communities," McGrath says.

Then they imported the models into the application and used the application to create fiducial markers for each plant. Students used iPads with the custom app installed to access the AR features.

Preliminary feedback is overwhelmingly positive. Classrooms on several reservations, like the Yankton and Rosebud reservations in South Dakota and Winnebago and Omaha reservations in Nebraska, began using the AR version of the workbook this fall, says Godfrey. And he says teachers in Arizona have expressed interest in adapting the workbook to their region. In addition, he exhibited the AR workbook at the National Indian Education Association's annual convention in October.

Still developing

Both Craig and McGrath say one of their primary concerns in AR development is latency—how long does it take for the digital augmentation to appear once the camera sees the marker and how well does the digital image keep up when the user moves the viewing device?

"This is a significant issue for AR because you are trying to do visual tracking and 3D graphics in 'real time,' updating at least 25 frames per second," says McGrath.

Even the amount of memory makes a huge difference. McGrath and Craig found that the application worked much better on McGrath's MacBook than Craig's, which is identical except for the amount of memory. Some mobile devices also have trouble running the AR app, one more technical detail to work around.

For his current projects, Craig is collaborating with a new company called daqri. daqri provides a platform that is compatible with multiple devices like smartphones, iPads, and iPhones, which will allow Craig to reach a wider audience more easily. McGrath says that the development of graphics is the bottleneck, and daqri aims to make that easier too. daqri promises the potential to create and distribute AR applications very quickly across a wide variety of platforms. This issue of Access was a collaboration with daqri and shows many different (though not all) of the capabilities of augmented reality.

Where is AR going?

AR is growing. Craig expects that people will first encounter AR through education or gaming. He says he thinks it will spread quickly through mobile devices and eventually become embedded in glasses and even contact lenses.

"Getting power to contacts is the hard part," says Craig, "but they have working one-pixel displays now. I think AR will become just a part of our world and sort of expected that it's there. AR is a new medium, and as such, it remains to be seen all the ways it will be adopted and realized."

Craig is in the final stages of authoring a book titled "Understanding Augmented Reality" that will be published by Morgan Kaufmann Publishers in early 2013.

Project at a glance

Team members
Alan B. Craig
Robert E. McGrath
Ryan Rocha
Michael Vila
Gaia Dempsey (daqri)
Brian Mullins (daqri)
Jonathan Ross (daqri)

Funding
National Science Foundation
daqri
I-CHASS

For more information
http://site.daqri.com/