Real experience is three-dimensional, while video is “flat” and 2D? Hardly! With interactive volumetric video, people captured on film appear to be standing live in the room right in front of the viewer, eye contact included. This is a particularly good idea in the case of reports by eyewitnesses to historical events.
If someone tells you their life story while standing right in front of you, looking you right in the eye, the narrative moves and touches you in a completely different way than if you were to hear it from an image on a flat screen. And when it comes to eyewitnesses from the time of the Holocaust, such personal accounts are naturally becoming increasingly rare. However, Fraunhofer HHI’s interactive volumetric video technology can help preserve life stories – narrated by the witnesses themselves as they stand face to face with you. Not live, of course, even though it may feel that way to viewers. Using VR goggles, they see the eyewitness – such as Eva Umlauf, one of the youngest survivors of the Auschwitz concentration camp – right in front of them in 3D. They can even walk around her. What’s special about this medium is that Eva looks at the listener and follows them with her gaze as they move.
“To do this, we recorded Eva Umlauf from every angle simultaneously using 32 cameras, all arranged in stereo pairs,” explains Dr. Oliver Schreer, group manager at Fraunhofer HHI. The Fraunhofer researchers carried out the recordings as part of the VoviRex project, a collaboration with the Ludwig Maximilian University of Munich. The stereo pairs of cameras not only capture a “normal” image, but also estimate the distance to the person being filmed – in other words, they create a depth map. Having obtained 16 such depth maps, the researchers first combine them into a point cloud and then convert that into meshes. “This lets us subsequently animate the recorded person so they can, for example, maintain eye contact with the users,” says Wieland Morgenstern, a scientist at Fraunhofer HHI.
Eva’s life story can also be incorporated into school lessons, although equipping each of the students with a gaming PC and VR goggles is somewhat complicated. “For this and similar applications, we can render the three-dimensional scene in the edge cloud – basically, on a computer located close to the classroom,” explains Dr. Cornelius Hellge, group manager at Fraunhofer HHI. From there, the user’s current view of the scene is transmitted to their goggles as a 2D video. Each student has to have a headset, as the eyewitness’s gaze is supposed to be adjusted to each viewer. In other words, every single student sees an individual volumetric video in their VR goggles. An alternative to this VR application is augmented reality, which eliminates the need for expensive VR headsets. Here, each student can look at their tablet to see both the classroom setting and the virtual eyewitness up at the front, sharing her moving life story.
Fraunhofer-Gesellschaft