3D Stereo Video Coding Heejune AHN Embedded Communications Laboratory Seoul National Univ. of Technology Fall 2013 Last updated 2013. 9. 31.

3D Stereo Video Coding

Heejune AHNEmbedded Communications Laboratory

Seoul National Univ. of TechnologyFall 2013

Last updated 2013. 9. 31

Heejune AHN: Image and Video Compression p. 2

Outline

Stereo video display Stereo video principle Multi-View Coding

H.264 amendment HEVC extension

Heejune AHN: Image and Video Compression p. 3

3D display

Stereo (existing) Anaglyphic (passive red-cyan glasses) Polarized (passive polarized glasses) Shutter/Synchronized (alternative frame sequencing) Auto stereographic (No glasses)

Hologram

Heejune AHN: Image and Video Compression p. 4

Anaglyph

Anaglyph

Heejune AHN: Image and Video Compression p. 5

Stereo Display

Polarization type micro-polarizer

(patterned retarder)

polarized glass

Active retarder

½ ver. resol.

full ver. resol.

Heejune AHN: Image and Video Compression p. 6

Active Shutter glasses type

2 times frame rate (i.e. 120 Hz)

synchronization

• cross-talk due to incomplete sync.• eye fatigue• more expensive glasses than polarized

L R

Heejune AHN: Image and Video Compression p. 7

No glasses Lenti-cular - Array of

lenslets in front of the pixel raster; relies on refraction of light through lenslets.

Parallax barrier - A barrier mask is placed in front of the pixel raster to control the light emitted to certain viewing positions

Limitation on distance

Heejune AHN: Image and Video Compression p. 8

Display Industry trends

3D TV

Heejune AHN: Image and Video Compression p. 9

TV market trends

Heejune AHN: Image and Video Compression p. 10

The Principle of Stereo Vision

Disparity between two eyes

Heejune AHN: Image and Video Compression p. 11

Human can understands 3D without disparity.

Heejune AHN: Image and Video Compression p. 12

Disparity D is in reverse proportional to depth (Z).

Heejune AHN: Image and Video Compression p. 13

Stereo Video Coding

Frame Compatible Stereo Spatial: side-by-side, top-bottom, line/row interleaved Time: views interleaved as alternating frames or fields +: Leverage existing infrastructure (2D coding) - : Reduced-resolution, coding in-efficiency Currently in TV broadcast service

Multiview Video Coding (MVC) H.264 amendment (2011)

• 3D Video and Free Viewpoint Video (FVV) Exploits correlation among views + : Full-resolution Adopted as format for 3D Blue-ray Disc

Heejune AHN: Image and Video Compression p. 14

Frame-compatible Stereo Video Top-bottom, side-by-side, line-row interleaved Example

Heejune AHN: Image and Video Compression p. 15

H.264/MVC

statistical dependence in temporal and inter-views temporal, spatial, plus inter-view prediction Backwards compatible to H.264 single view profiles

Heejune AHN: Image and Video Compression p. 16

MVD Coding

Multi-View(2D)+Depth Coding HEVC extension Better for auto-stereoscopic with DIBR (Depth Information Based

Rendering)

Depth maps : distance from the cameras/view points • inverse proportional to disparity

Heejune AHN: Image and Video Compression p. 17

Block diagram MV capturing, coding, decoding, and rendering

Heejune AHN: Image and Video Compression p. 18

Coding algorithm 1 base + N-1 dependent views N associate depth maps

Heejune AHN: Image and Video Compression p. 19

Interview Motion prediction

time

view

Heejune AHN: Image and Video Compression p. 20

Interview residual signal prediction Predict ‘residual’ of a current block using a coded residual block in a reference view Determined disparity using the estimated depth maps (same depth as for

inter-view motion parameter prediction) Disparity-compensated prediction of the current residual Bilinear filtering

for sub-sample interpolation use after motion compensation

residual signal depthmap

Heejune AHN: Image and Video Compression p. 21

Depth map coding

Depth map properties1. Sharp edges representing object

borders

2. Large areas of slowly varying values representing object areas

3. Edges in depth maps are correlated with edges in video pictures

Depth map coding ideas 1. Representation of depth edges

2. Partition block into two regions with constant sample values

3. Prediction based on co-located texture block

4. Optional transform coding of residual

Heejune AHN: Image and Video Compression p. 22

Non-rectangular block partitions Approximating the signal of a depth block by a model that partitions

the area into two non-rectangular regions Each region is represented by a constant value

Heejune AHN: Image and Video Compression p. 23

Motion parameter inheritance Get parameter info for depth map from those from view data