iMinds immersive experience session

iMinds break-out session: Immersive Experience

Advances in various domains – including networking, human-computer interaction, visual computing, virtual worlds and display technology – are paving the way towards an Immersive Internet.  The Immersive Internet will support real-time capture, distribution, delivery and display of ultra-high resolution virtual and real-world audio-visual (multi-view) media.

These new technologies for communication, collaboration and media consumption will fundamentally influence the way we organize education, business, entertainment, healthcare, etc. 

The “Immersive Experiences” session will discuss challenges and technical breakthroughs for realizing this vision, and will highlight ongoing R&D initiatives on the above topics. 

Program for this track:

  • Olivier Schreer (HHI) - Solutions for Immersive 3D Video Communication

3D video communication is one relevant application scenario in future 3D services. A new concept for immersive 3D videoconferencing is presented, which targets the existing challenges beyond the state of the art. One of the key challenges is providing direct eye contact. A real-time solution for multi-view video analysis and virtual view rendering at high resolution and high quality is presented, which provides direct eye contact between the user and its remote counterpart.

  • Philippe Bekaert (IBBT) - Immersive video: bridging the gap between computer graphics and video,” Philippe Bekaert

Immersive experiences can be based on either computer graphics (CG) or on video. CG offers tremendous flexibility when it comes to creating synthetic virtual environments, as it allows to compose such environments at will by adding and modifying virtual objects, editing their appearance, adding illumination. Objects can be animated, and interaction and navigation modes defined. In spite of thousands of man-years or R&D efforts in the area, creating a convincingly realistic synthetic environment is a painstakingly costly task still today. This is in contrast with video or film, which preserves realism from scene to screen, unless equipment and processing is modified for the purpose. Video however does not nearly offer the tremendous flexibility of CG in creating environments for immersive experiences, and in particular navigation and interaction possibilities are rather limited. As a result, todays immersive experiences are usually limited in interaction and navigation modes, or they are cartoon like. This talk will highlight some recent developments that aim at bridging the gap between the complementary media of CG and video, illustrating with omnidirectional video as a maturing case. As some of the main problems are related with the creative use of the resulting new media, the importance of collaborative research with creatives in this area will be pointed out.

  • Jean-Francois Macq (Bell Labs) – Towards Immersive and Interactive Video entertainment – the coding and delivery challenges

Watching football on a wide-angle display and feeling like you are amid the cheering spectators in the stadium, or freely navigating with your smart phone into a panoramic video of your favorite rock concert: these are examples of truly immersive and interactive TV services that may soon be delivered to you, at home or on the move.

In addition to the enormous data rates required by very high-fidelity and panoramic A/V scenes, the network will have to support fine-grained levels of personalization and interactivity. To solve these challenges, some aspects of the way A/V content is represented, coded and delivered need to be revisited.

In this talk, we will present the end-to-end technological chain, from the A/V sensors till the end-user experience, being currently researched in the EC-funded project FascinatE to support ultra-high resolution and interactive television services.

  • Harm Belt & Kees Janse (Philips Research) – Lifelike Communication – front-end audio and video technologies

Communication is a basic human need. In our modern society however, families have split schedules or are living apart, and struggle to find the right balance between work, other activities, and relationships. They seek quality time together and opportunities for rich interactions. They want to keep in touch through tele-communication technology, and they want a more lifelike interaction. For tele healthcare and remote patient monitoring applications, lifelike interaction between the physician, the patient, and potentially a family member, is something that is highly desired.

Lifelike communication implies that the people using the technology enjoy full freedom in their movements, and the communication technology itself is hidden. It also implies large displays in order to be able to display remote people at sufficient size. Consequently, with such systems the distance between the people and the audio/video sensors and actuators is large. On top of that, the environment in which the technology is used can be severe in the sense that the acoustic and light conditions are bad. This often leads to bad audio and video quality, while for lifelike communication a high quality is required. Another important aspect in video communication is eye contact, which is not automatically preserved due to the misalignment between camera and display.

At Philips Research, over the past two decades much work was done on audio and video signal processing improving the quality of the microphone and camera signals before transmission and the loudspeaker and display signals after receiving. In this talk we address many of these technologies and provide several audio and video demonstration examples, enabling lifelike communication in the living room with the TV, and lifelike communication for healthcare. We present solutions for the communication of spatial sound and 3D video, the latter being important also for eye contact.

Leave a comment

Comments are closed.