Format-agnostic Approach for Production, Delivery and Rendering of Immersive Media

The research leading to these results has received funding from the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 248138

Format-agnostic Approach for Production, Delivery and Rendering

of Immersive Media

Felix Lee

HPC I28.10.2013

© FascinatE Consortium - 2 - www.fascinate-project.eu

Real-time Live Panoramic VideoOMNICAM

real-time acquisition

warping & stitching

blending and rendering of 6 HD video streams

one panoramic video of 7k x 2k resolution

on a single PC exploiting GPU processing

© FascinatE Consortium - 3 - www.fascinate-project.eu

Scene Acquisition (1)

OMNICAM ARRI Alexa

HD Broadcast camera

© FascinatE Consortium - 4 - www.fascinate-project.eu

Scene Acquisition (2)

Soundfield® Stereo pair

Gun micsEigenmike®

© FascinatE Consortium - 5 - www.fascinate-project.eu

Football Premier League Game

Chelsea FC – Wolverhampton Wanderers (London, October 2010)

Dance Performance Carmen Suite

Berlin Philharmonic Orchestra conducted by Sir Simon Rattle & Dance Compagnie Sasha Waltz (Berlin, May 2012)

Concert BBC Proms (London, August 2012)

Three Demo Productions (so far)

© FascinatE Consortium - 6 - www.fascinate-project.eu

Saliency Map Center-surround model

inspired from biological models of human visual sensory system Center-surround features

Pixel intensity

Color chromatic opponents RG and BY

Saliency map for still images

© FascinatE Consortium - 7 - www.fascinate-project.eu

Saliency Detection Image pyramid based saliency map

Create image pyramid: blur and shrink the image

Build the image differences Add the image differences

© FascinatE Consortium - 8 - www.fascinate-project.eu

Spatio-temporal Saliency Histograms Spatiotemporal saliency model

Split image into grid cells (cell size: e.g. 20x20 pixels for HD) Compute histogram of pyramid differences for each cell Compare histograms

Current histogram vs. average of histograms from past second Distance function: earth mover distance

Region of interest (ROI) Convert high salient regions into rectangles

Application: automatic view selection

Demo video