U of I VTD program demonstrates how to blend animation with virtual reality

In an effort to spend more time outside during the COVID-19 summer of 2020, Jean-Marc Gauthier spent many hours riding his e-bike around Paradise Creek between Moscow and Pullman, filming the landscape and a pair of herons he ran into on a daily basis.

He later finished his passion project, Moments in Nature, and turned it into a short video production that was recognized by SIGGRAPH Asia RTL as one of the 11 best submissions worldwide in the Real-Time Live category.

Gauthier, an associate professor and head of the Virtual Technology & Design department in U of I’s College of Art and Architecture (CAA), shot all of the footage with his cell phone camera. Then he and VTD graduate student Payton Finney ‘22 used video game controllers and software that allowed them to manipulate the characters in real time – a relatively new use of that technology for virtual reality (VR) applications.

In doing so, they are highlighting a faster and simpler way to use VR to create animation – a method that will enable users to create more content with fewer people involved. It might also be the key for any interested department at U of I to get their foot in the door of the Metaverse.

“This is a way to collaborate by building stories using this technology,” said Gauthier. “Like with any newer technology or application, we are finding different ways to apply it. Those that are tech savvy will see that it’s an easier way to tell stories.”

Tilda and the herons

Gauthier’s story began by scanning portions of Paradise Creek with his phone by using LiDAR, or light detection and ranging. He built his virtual world starting with the creek, then inserted the herons. After creating his world, he developed a story: a woman comes down to the creek every day to watch the herons. After she notices the red heron has not been showing up for a few days, the woman hops on the back of the blue heron to fly around the area, hoping to find the red one.

Finney, who began collaborating with Gauthier in May 2022, moved the project forward by using 3-D software to animate the characters. The two worked together on setting up the connection between the characters by setting up iPads, which are used to move the characters, and having the iPads link with a laptop, where the animation lives.  

“It’s built in a video game engine, so the action is very much how you would progress through scenarios while gaming,” said Finney. “Plus there is the real-time aspect, which makes it like VR because you are manipulating characters in real time.”

Other U of I team members who assisted on the project include 2021 VTD graduates Clayton Christensen and Randall Erickson, as well as current VTD student Emma Ferguson.

Technology meets performance art

Although this is a newer use of this application, most users familiar with VR or animation would likely agree this kind of production wouldn’t really be considered high-tech. Which is exactly the idea according to Gauthier.

“Using this technology allows you to capture images quickly but you aren’t using a lot of time-consuming or costly production tools to make your film,” he said. “The emphasis is on speed and collaboration, not on production.”

Finney agreed, saying this product is a hybrid between pure animation and VR.

“It’s kind of a novel approach because it’s not old fashioned VR, which emphasizes the technology piece, and it’s not pure animation, because it’s being displayed in real-time,” he said. “It’s a live technical showcase, which makes it more like a performance piece.”

It could also make a lot of the older, bulkier and more confining pieces of technology for animation and VR, like helmets, motion-capture suits and even big computers, obsolete.

“This is a new way to do this type of production and like any other technology, the more it gets used, the better it will get and the old ways will not be utilized as much,” said Gauthier. “We are showing everyone what can be done with just cell phones, laptops and iPads.”

David vs. Goliath

Gauthier and Finney flew to South Korea last December, with their cell phones, laptops and iPads in tow, to attend the SIGGRAPH Asia 2022 awards ceremony. While they were unpacking their relatively small displays, they watched as their bigger competitors set up complex productions worthy of large industrial trade shows.

Gauthier knew they did not have the same resources as most of their competition, but noted that his team was one of only 11 from around the world who made it that far. They were there to prove the point that it wasn’t about using the best technology, it was about the best use of the technology they had.

“Of course it looks like it’s made quickly, because it is, and it’s not as visually appealing as others, but the jury said there was real production value in our model,” he said. “These projects were all peer reviewed. You can’t pay your way in to being recognized like this, you earn it.”

And although the U of I team did not win the Real-Time Live category, Gauthier said the feedback they received and connections they established made the trip worthwhile. After leaving South Korea, Gauthier and Finney were invited to University of Tampa, where they produced a live demonstration of the technology to students and faculty at Tampa’s virtual production studio, which is similar to the studio used to create The Mandalorian, a Star Wars spin-off created for the Disney Plus streaming service.

Entering the Metaverse

As with any new technology, it’s almost impossible to predict where it will go, how fast it will get there and how many turns it will take along the way. But based on the success of collaborative storytelling by Gauthier’s team, this technology seems primed to help U of I groups tell their stories to anyone around the world.

The term Metaverse is a mostly conceptual one – it’s the idea of a VR world within the internet where, for example, a virtual professor could lecture a class full of virtual students, all in real time and with the ability to interact with each other.

Gauthier has discussed this concept with many U of I faculty members, as well as with Vice Provost for Digital Learning Ken Udas. The result of those discussions is VR Classroom, which Gauthier is using to create virtual classrooms for future U of I projects.

“We used this technology to tell our story about herons but it can also transform easily into an educational tool,” said Gauthier. “We plan to make this available to anyone at U of I who wants to take advantage of the technology.”