Spectral Matting
A. Levin D. Lischinski and Y. Weiss. A Closed Form Solution to Natural Image Matting. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), June 2006, New York
A. Levin, A. Rav-Acha, D. Lischinski. Spectral Matting. Best paper award runner up. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), Minneapolis, June 2007
A. Levin1,2, A. Rav-Acha1, D. Lischinski1. Spectral Matting. IEEE Trans. Pattern Analysis and Machine Intelligence, Oct 2008.
1School of CS&Eng The Hebrew University2CSAIL MIT
1
Previous approaches to segmentation and matting
Unsupervised
Input Hard output Matte output
Spectral segmentation:Spectral segmentation: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Zelnik and Perona 05 Tolliver and Miller 06
3
Previous approaches to segmentation and matting
Unsupervised
Input Hard output Matte output
Supervised
0
1
July and Boykov01 Rother et al 04 Li et al 04
4
Previous approaches to segmentation and matting
Unsupervised
Input Hard output Matte output
Supervised
0
1
Trimap interfaceTrimap interface: Bayesian Matting (Chuang et al 01) Poisson Matting (Sun et al 04) Random Walk (Grady et al 05)Scribbles interface:Scribbles interface: Wang&Cohen 05 Levin et al 06 Easy matting (Guan et al 06)
?
5
Generalized compositing equation
iiiii BFI )1( 2 layers compositing
= x x+ 1 2L1L
Ki
Kiii LLLI
iii ...2211
K layers compositing
= x x+
+ x x+3 4 4L3L
1 2 2L1L
Matting components
8
Generalized compositing equation
1...21 K
iii
“Sparse” layers- 0/1 for most image pixels
Matting components:
Ki
Kiii LLLI
iii ...2211
K layers compositing
= x x+
+ x x+
10 ki
1
3 4
2 2L
4L3L
1L
9
Spectral segmentation
22/
),(ji CC
ejiW
WDL
j
jiWiiD ),(),(
Spectral segmentation: Analyzing smallest eigenvectors of a graph Laplacian L
E.g.: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Maila and shi 01 Zelnik and Perona 05 Tolliver and Miller 0612
The matting Laplacian
LJ T )(
• semidefinite sparse matrix
• local function of the image:),( jiL
L
16
Spectral segmentationFully separated classes: class indicator vectors belong to Laplacian nullspace
General case: class indicators approximated as linear combinations of smallest eigenvectors
Null
Binary indicating
vectors
Laplacian matrix
21
Spectral segmentation
Fully separated classes: class indicator vectors belong to Laplacian nullspace
General case: class indicators approximated as linear combinations of smallest eigenvectors
Smallest eigenvectors- class indicators only up to linear transformation
33
RZero eigenvectors
Binary indicating
vectors
Laplacian matrix
Smallest eigenvecto
rs
Linear transformati
on
22
From eigenvectors to matting components
Smallest eigenvectors
Projection into eigs space kCTk mEE
....
K-means
..
kCmle
1) Initialization: projection of hard segments
2) Non linear optimization for sparse components26
Brief Summary
LJ T )(
Construct Matting Laplacian
Smallest eigenvectors
Linear Transformation
Matting components
28
User-guided matting Graph cut method
Energy function
Unary term Pairwise termConstrained components
33
Components with the scribble interface
Components (our
approach)
Levin et al cvpr06
Wang&Cohen 05
Random Walk
Poisson 34
Components with the scribble interface
Components (our
approach)
Levin et al cvpr06
Wang&Cohen 05
Random Walk
Poisson 35
Limitations Number of eigenvectors
Ground truth matte Matte from 70 eigenvectors
Matte from 400 eigenvectors40