Digital Visual Effects, Spring 2007 Yung-Yu Chuang 2007/5/1

Matting and Compositing

Digital Visual Effects, Spring 2007Yung-Yu Chuang2007/5/1

Traditional matting and composting

Photomontage

The Two Ways of Life, 1857, Oscar Gustav RejlanderPrinted from the original 32 wet collodion negatives.

Photographic compositions

Lang Ching-shan

Use of mattes for compositing

The Great Train Robbery (1903) matte shot

Use of mattes for compositing

The Great Train Robbery (1903) matte shot

Optical compositing

King Kong (1933) Stop-motion + optical compositing

Digital matting and compositing

The lost world (1925) The lost world (1997)

Miniature, stop-motion Computer-generated images

Digital matting and composting

King Kong (1933) Jurassic Park III (2001)

Optical compositingBlue-screen matting, digital composition,

digital matte painting

Digital matting: bluescreen matting

Forrest Gump (1994)

• The most common approach for films.• Expensive, studio setup.• Not a simple one-step process.

Titanic

Matting and Compositing Matting and Compositing

backgroundreplacement

backgroundediting

Color difference method (Ultimatte)

Blue-screenphotograph

C=F+αB

Spill suppressionif B>G then B=G

F

Matte creationα=B-max(G,R)

α

Chroma-keying (Primatte)

Compositing

BFC )α(α −+= 1

αF B

C

foreground color alpha matte background plate

composite

compositingequationB

F

C

α=0

Compositing

BFC )α(α −+= 1

αF B

Ccomposite

compositingequationB

F

Cα=1

Compositing

BFC )α(α −+= 1

αF B

Ccomposite

compositingequationB

FC

α=0.6

Matting

CobservationBFC )α(α −+= 1

compositingequation

αF B

Matting

CBFC )α(α −+= 1

compositingequation

αF B

Three approaches:

1 reduce #unknowns

2 add observations

3 add priors

Matting (reduce #unknowns)

C

F BB

BFC )α(α −+= 1

differencematting

α

Matting (reduce #unknowns)

C

F

BFC )α(α −+= 1B

blue screenmatting

α

Matting (add observations)

F

BFC )α(α −+= 1

triangulation

α

BFC )α(α −+= 1

BC

Natural image matting

BC

Matting (add priors)

F

BFC )α(α −+= 1

α B

rotoscopingRuzon-Tomasi

FG

BG

unknown

Bayesian image matting

Bayesian framework

posterior probability

likelihood priors

Priors

Optimization

repeat

until converge

1. fix alpha

2. fix F and B

inputtrimapalpha

Results

input composite

Results

Comparisons

input imagetrimap

Comparisons

Bayesian Ruzon-Tomasi

Comparisons

Bayesian Ruzon-Tomasi

Comparisons

input image

Comparisons

Bayesian Mishima

Comparisons

Bayesian Mishima

Video matting

inputvideo

Video matting

inputkeytrimaps

inputvideo

Video matting

interpo-latedtrimaps

inputvideo

Video matting

interpo-latedtrimaps

inputvideo

outputalpha

Video matting

Compo-site

interpo-latedtrimaps

inputvideo

outputalpha

Video matting

optical flow

optical flow

Sample composite

Garbage mattes Garbage mattes

Background estimation Background estimation

Alpha matte Comparison

without background

withbackground

C

B

P(F)

More on matting

Recent progresses on matting• Poisson matting• Two-camera matting methods• Flash matting

Poisson matting Poisson matting

Two-camera matting methods• Invisible lights

– Polarized lights– Infrared

• Thermo-key• Depth Keying (ZCam)

Invisible lights (Infared)

Invisible lights (Infared) Invisible lights (Infared)

Invisible lights (Infared) Invisible lights (Infared)

Invisible lights (Infared) Invisible lights (Polarized)

Invisible lights (Polarized) Thermo-Key

Thermo-Key ZCam

ZCam Flash matting

flash no flash matte

Flash matting

Background is much further than foreground and receives almost no flash light

Flash matting

Foreground flash matting equation

Generate a trimap and directly apply Bayesian matting.

Foreground flash matting Joint Bayesian flash matting

Joint Bayesian flash matting Comparison

flash no flash

Comparison

foreground flash matting

ioint Bayesian flash matting

Flash matting

Shadow mattingand composting

target backgroundsource scene

target backgroundblue screen image target backgroundblue screen composite

photographblue screen composite photographblue screen composite

Geometric errors

photographblue screen composite

Photometric errors

S L

C

β

S L

C

β

shadowcompositingequation

Shadow compositing equation

C=βL+(1-β)S

C

β

shadowcompositingequation

Shadow matting

C=βL+(1-β)S

S L

C

β

shadowcompositingequation

Shadow matting

C=βL+(1-β)S

S L

C

β

shadowcompositingequation

Shadow matting

β

C=βL+(1-β)S

C

S Lβ

shadowcompositingequation

Shadow compositing

C=βL+(1-β)S

Geometric errors

target backgroundsource scene target backgroundsource scene

Requirement #1

target backgroundsource scene

Requirement #2

Environment matting

photographblue screen matting

F

foreground

Ccomposite

Tbackground

1-αB

traditional compositing equation

F

foreground

Ccomposite

Tbackground

1-αB

AR

environment compositing equation [Zongker’99]

Environment matting [Zongker’99]

O(k) images

Problem: color dispersion

photographZongker et al.

Problem: glossy surface

photographZongker et al.

Problem: multiple mappings

photographZongker et al.

C Tweightingfunction

W

backgroundcomposite

Arbitrary weighting functionMultimodal oriented Gaussian

Problem: color dispersion

photographZongker et al.high accuracyalgorithm

Glossy surface

photographZongker et al.high accuracyalgorithm

Oriented Gaussian

photograph

withoutorientation

withoutorientation

withoutorientation

withorientation

Problem: multiple mappings

photographZongker et al.high accuracyalgorithm

F

foreground

Ccomposite

Tbackground

1-αB

W

weightingfunction

AR

3

3 observations

3 1 4

11 variables

• A, R• α• F

3

4

7 variables

• A, R• α• F

33

3 observations

4

5 variables

• A, R• α• F

13

3 observations

A, ρcolorless

3 variables

• A, R• α• F

13

3 observations

A, ρcolorless

T(cx , cy)11

cx , cy , ρspecularlyrefractive

T

M(T , A)A

x

y

(cx, cy)

≈ T(cx, cy)

Stimulus functionStimulus function

x

y

T(x,y)

Ideal plane in RGB cubeIdeal plane in RGB cube

T(x,y)

x

y

(cx ,cy)C′

Calibrated manifold in RGB cubeCalibrated manifold in RGB cube

T

backgroundmanifold

K

W

C′

C′ → (cx ,cy)

Cρρ = KC

KC′

Estimate cx ,cy and ρEstimate cx ,cy and ρ Problem: noisy matte

with filteringwithout filtering

Edge-preserving filteringEdge-preserving filtering Input image

Difference thresholding Morphological operation

Feathering Heuristics for specular highlights

T

backgroundmanifold

K

W

Cρ >1+ε C′

Heuristics for specular highlightsHeuristics for specular highlights

C=ρT(cx, cy)

ρ cx cy

=

input estimation foreground(highlights)

Heuristics for specular highlightsHeuristics for specular highlights Composite with highlights

mattingmethod

compositingmodel

BFC )α(α −+= 1color

blending

shadow

refractionreflection

blue-screenBayesian

Shadowmattingβ)L(βSC −+= 1

High-accuracyenv. matting∫+= WBFC