Virtual learning space

released 03.25.09

By Erika Strebel

Thanks to NCSA's Image Spatial Data Analysis group, you may one day be able to literally insert yourself into situations even though you are far, far away.

There are many ways of communicating with people who aren't in the same area. You can use a telephone or e-mail to talk to a relative living thousands of miles away. Talk show hosts sometimes use Skype to chat with interviewees who couldn't make it to the set. You can listen to a live lecture or seminar online. But learning physical skills, like basketball, is more difficult since these technologies do not allow an instructor and a student who are geographically separated to physically interact.

NCSA's Peter Bajcsy wants to push those limits. He and his team are collaborating with various units at the University of Illinois at Urbana-Champaign—the computer science and electrical and computer engineering departments, as well as disability resources and educational services—and University of California, Berkeley to develop a way to digitally clone people in remote areas into the same virtual space and allow them to physically interact in real time.

Putting reality into a virtual space

The team constructed a system of cameras and displays that capture, transmit, and display three-dimensional movement in real time. Several digital camera clusters consisting of visible and thermal spectrum cameras, and multiple large LCD displays surround a defined physical space where someone will be learning complex movements or physical activities, like wheelchair basketball.

The camera clusters are placed on tripods and the LCD displays are mounted on TV carts so the system is portable. Three-dimensional information is extracted from the images those cameras acquire and rendered in a virtual space.

With the help of various software, the system makes 3D data in real time available to remote machines through the Internet. LCD displays allow users to see their own digital clones and the clones from remote sites together, and move their bodies in response to the images on the screen.

Bajcsy, the team's leader and head of the image spatial data analysis group at NCSA, calls this a "tele-immersive environment" because a user can be immersed with another user at a remote site and communicate in 3D and in real time. Unlike the virtual reality people see in video games or in digitally animated films, these virtual environments record real-time actions.

"It's a virtual environment that is the product of real-time imaging, not the result of programming 3D CAD models." Bajcsy says. "Nobody has to be supplied with equipment to enable imaging and 3D reconstruction. The only thing you might have is some kind of controller, like a Wii controller, so you can change the view angle of the data you see."

In 2007, Bajcsy and his team started working with groups of wheelchair basketball players to test the tele-immersive system. During these experiments, wheelchair basketball coach Michael Frogley and his students work on basketball moves and wheelchair maneuvers.

"I really have to praise the wheelchair basketball players," said Bajcsy. "They are just really fun to work with! They are always interested in trying the new technology, although the technology might be frustrating."

Deploying a tele-immersive system

Before deploying the system, a user would need recommendations where to place clusters of cameras. Bajcsy's team designed a simulation framework that allows users to input the amount of money available, the type of activity they want to learn, the dimensions of the space they want to use, and information about lighting in that space. The framework will determine the number of cameras the group should purchase and where to position those cameras. The NCSA team also devised a special code that allows the camera clusters to be calibrated in terms of color consistency and geometric locations/orientations once the camera clusters are placed.

Tweaking the system

Over the course of the project, Bajcsy and his team have made various adjustments to the system.

"As we were starting to build the system, we immediately noticed the system had some robustness issues with respect to illumination," Bajcsy says. "The system would not perform well when illumination would change or objects would not be different from the background."

In order to fix this issue, the team introduced thermal-infrared cameras to the system, allowing the system to identify things that are warmer or cooler than the ambient temperature in the environment. As a result, the system can detect the foreground regardless of what subjects are wearing or whether they cast shadows, said Bajcsy.

But there are more challenges ahead.

"The system is very dynamic," said Bajcsy. "Bottlenecks are shifting to different places in the system as you replace technology or technology works better."

The team is now trying to fix a networking and data transmission problem. A single camera cluster generates about 460 megabytes of data for every second of real-time footage, which is just under half of a gigabyte of information. But only one gigabyte per second of total bandwidth is available. This poses a problem because a system that employs 10-20 cameras would easily surpass that bandwidth, says Bajcsy.

"We are working very closely with Professor Klara Nahrstedt's group in the computer science department," says Bajcsy. "Their expertise is in networking, so we'll be relying on them for the right protocol."

He says the NCSA team is also collaborating with electrical and computer engineering departments at the University of Illinois and UC Berkeley to compress the 3D information, and to consider alternative ways of representing data collected from the cameras.

Moving forward

Bajcsy and his team are continuing to improve their system to meet the needs of the wheelchair basketball community.

"If we could build a system so that it works robustly and can be deployed in the gym where they practice every day, then it would have tremendous value for them," says Bajcsy.

The NCSA team has also been working on adding a replay feature that would allow a user to replay a previous session while in the virtual space.

"You can actually exercise next to yourself and say, 'Oh, I see. I was making that mistake,'" said Bajcsy.

The ultimate goal is to construct a system that costs less than $50,000. Current models on the market cost at least half a million dollars and focus primarily on head movements. However, Bajcsy says commercially available technology is still not advanced enough to make the system more portable and resilient.

He also hopes to be able to connect four to five remote sites at NCSA and the computer science department so that his team can demonstrate the scalability of the system.

"As we demonstrate that the system is really working," he says. "I'm looking for other communities who can take advantage of the technology."

For more information, see: isda.ncsa.uiuc.edu.

This project was funded by the National Science Foundation.

Team members
Peter Bajcsy
Rob Kooper
Suk Kyu Lee
Rahul Malik