Transcript
Sparse Coding and Dictionary Learningfor Image Analysis
Part II: Dictionary Learning for signal reconstruction
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro
ICCV’09 tutorial, Kyoto, 28th September 2009
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 1/43
What this part is about
The learning of compact representations of imagesadapted to restoration tasks.
A fast online algorithm for learning dictionaries andfactorizing matrices in general.
Various formulations for image and video processing.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 2/43
The Image Denoising Problem
y︸︷︷︸measurements
= xorig︸︷︷︸original image
+ w︸︷︷︸noise
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 3/43
Sparse representations for image restoration
y︸︷︷︸measurements
= xorig︸︷︷︸original image
+ w︸︷︷︸noise
Energy minimization problem - MAP estimation
E (x) = ||y − x||22︸ ︷︷ ︸relation to measurements
+ Pr(x)︸ ︷︷ ︸prior
Some classical priors
Smoothness λ||Lx||22Total variation λ||∇x||21Wavelet sparsity λ||Wx||1. . .
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 4/43
Sparse representations for image restoration
Sparsity and redundancy
Pr(x) = λ||α||0 for x ≈ Dαx
︸ ︷︷ ︸x∈Rm
=
d1 d2 · · · dp
︸ ︷︷ ︸
D∈Rm×p
α[1]α[2]
...α[p]
︸ ︷︷ ︸α∈Rp,sparse
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 5/43
Sparse representations for image restoration
Designed dictionaries
[Haar, 1910], [Zweig, Morlet, Grossman ∼70s], [Meyer, Mallat,Daubechies, Coifman, Donoho, Candes ∼80s-today]. . . (see [Mallat,1999])Wavelets, Curvelets, Wedgelets, Bandlets, . . . lets
Learned dictionaries of patches
[Olshausen and Field, 1997], [Engan et al., 1999], [Lewicki andSejnowski, 2000], [Aharon et al., 2006] , [Roth and Black, 2005], [Leeet al., 2007]
minαi ,D∈C
∑i
1
2||xi −Dαi ||22︸ ︷︷ ︸reconstruction
+λψ(αi )︸ ︷︷ ︸sparsity
ψ(α) = ||α||0 (“`0 pseudo-norm”)
ψ(α) = ||α||1 (`1 norm)
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 6/43
Sparse representations for image restoration
Solving the denoising problem
[Elad and Aharon, 2006]
Extract all overlapping 8× 8 patches yi .
Solve a matrix factorization problem:
minαi ,D∈C
n∑i=1
1
2||yi −Dαi ||22︸ ︷︷ ︸reconstruction
+λψ(αi)︸ ︷︷ ︸sparsity
,
with n > 100, 000
Average the reconstruction of each patch.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 7/43
Sparse representations for image restorationK-SVD: [Elad and Aharon, 2006]
Figure: Dictionary trained on a noisy version of the imageboat.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 8/43
Sparse representations for image restoration
Inpainting, Demosaicking
minD∈C,α
∑i
1
2||βi ⊗ (yi −Dαi )||22 + λiψ(αi )
RAW Image Processing (see our poster)
Whitebalance.
Blacksubstraction.
Denoising
Demosaicking
Conversionto sRGB.Gamma
correction.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 9/43
Sparse representations for image restoration[Mairal, Bach, Ponce, Sapiro, and Zisserman, 2009c]
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 10/43
Sparse representations for image restoration[Mairal, Sapiro, and Elad, 2008b]
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 11/43
Sparse representations for image restorationInpainting, [Mairal, Elad, and Sapiro, 2008a]
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 12/43
Sparse representations for image restorationInpainting, [Mairal, Elad, and Sapiro, 2008a]
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 13/43
Sparse representations for video restoration
Key ideas for video processing
[Protter and Elad, 2009]
Using a 3D dictionary.
Processing of many frames at the same time.
Dictionary propagation.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 14/43
Sparse representations for image restorationInpainting, [Mairal, Sapiro, and Elad, 2008b]
Figure: Inpainting results.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 15/43
Sparse representations for image restorationInpainting, [Mairal, Sapiro, and Elad, 2008b]
Figure: Inpainting results.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 16/43
Sparse representations for image restorationInpainting, [Mairal, Sapiro, and Elad, 2008b]
Figure: Inpainting results.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 17/43
Sparse representations for image restorationInpainting, [Mairal, Sapiro, and Elad, 2008b]
Figure: Inpainting results.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 18/43
Sparse representations for image restorationInpainting, [Mairal, Sapiro, and Elad, 2008b]
Figure: Inpainting results.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 19/43
Sparse representations for image restorationColor video denoising, [Mairal, Sapiro, and Elad, 2008b]
Figure: Denoising results. σ = 25
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 20/43
Sparse representations for image restorationColor video denoising, [Mairal, Sapiro, and Elad, 2008b]
Figure: Denoising results. σ = 25
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 21/43
Sparse representations for image restorationColor video denoising, [Mairal, Sapiro, and Elad, 2008b]
Figure: Denoising results. σ = 25
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 22/43
Sparse representations for image restorationColor video denoising, [Mairal, Sapiro, and Elad, 2008b]
Figure: Denoising results. σ = 25
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 23/43
Sparse representations for image restorationColor video denoising, [Mairal, Sapiro, and Elad, 2008b]
Figure: Denoising results. σ = 25
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 24/43
Optimization for Dictionary Learning
minα∈Rp×n
D∈C
n∑i=1
1
2||xi −Dαi ||22 + λ||αi ||1
C M= {D ∈ Rm×p s.t. ∀j = 1, . . . , p, ||dj ||2 ≤ 1}.
Classical optimization alternates between D and α.
Good results, but very slow!
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 25/43
Optimization for Dictionary Learning[Mairal, Bach, Ponce, and Sapiro, 2009a]
Classical formulation of dictionary learning
minD∈C
fn(D) = minD∈C
1
n
n∑i=1
l(xi ,D),
where
l(x,D)M= min
α∈Rp
1
2||x−Dα||22 + λ||α||1.
Which formulation are we interested in?
minD∈C
{f (D) = Ex [l(x,D)] ≈ lim
n→+∞
1
n
n∑i=1
l(xi ,D)}
[Bottou and Bousquet, 2008]: Online learning can
handle potentially infinite or dynamic datasets,
be dramatically faster than batch algorithms.Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 26/43
Optimization for Dictionary Learning
Require: D0 ∈ Rm×p (initial dictionary); λ ∈ R1: A0 = 0, B0 = 0.2: for t=1,. . . ,T do3: Draw xt
4: Sparse Coding
αt ← arg minα∈Rp
1
2||xt −Dt−1α||22 + λ||α||1,
5: Aggregate sufficient statisticsAt ← At−1 + αtα
Tt , Bt ← Bt−1 + xtα
Tt
6: Dictionary Update (block-coordinate descent)
Dt ← arg minD∈C
1
t
t∑i=1
(1
2||xi −Dαi ||22 + λ||αi ||1
).
7: end for
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 27/43
Optimization for Dictionary Learning
Which guarantees do we have?
Under a few reasonable assumptions,
we build a surrogate function ft of the expected cost f verifying
limt→+∞
ft(Dt)− f (Dt) = 0,
Dt is asymptotically close to a stationary point.
Extensions (all implemented in SPAMS)
non-negative matrix decompositions.
sparse PCA (sparse dictionaries).
fused-lasso regularizations (piecewise constant dictionaries)
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 28/43
Optimization for Dictionary LearningExperimental results, batch vs online
m = 8× 8, p = 256Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 29/43
Optimization for Dictionary LearningExperimental results, batch vs online
m = 12× 12× 3, p = 512Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 30/43
Optimization for Dictionary LearningInpainting a 12-Mpixel photograph
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 31/43
Optimization for Dictionary LearningInpainting a 12-Mpixel photograph
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 32/43
Optimization for Dictionary LearningInpainting a 12-Mpixel photograph
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 33/43
Optimization for Dictionary LearningInpainting a 12-Mpixel photograph
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 34/43
Extension to NMF and sparse PCA[Mairal, Bach, Ponce, and Sapiro, 2009b]
NMF extension
minα∈Rp×n
D∈C
n∑i=1
1
2||xi −Dαi ||22 s.t. αi ≥ 0, D ≥ 0.
SPCA extension
minα∈Rp×n
D∈C′
n∑i=1
1
2||xi −Dαi ||22 + λ||α1||1
C′ M= {D ∈ Rm×p s.t. ∀j ||dj ||22 + γ||dj ||1 ≤ 1}.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 35/43
Extension to NMF and sparse PCAFaces: Extended Yale Database B
(a) PCA (b) NNMF (c) DL
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 36/43
Extension to NMF and sparse PCAFaces: Extended Yale Database B
(d) SPCA, τ = 70% (e) SPCA, τ = 30% (f) SPCA, τ = 10%
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 37/43
Extension to NMF and sparse PCANatural Patches
(a) PCA (b) NNMF (c) DL
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 38/43
Extension to NMF and sparse PCANatural Patches
(d) SPCA, τ = 70% (e) SPCA, τ = 30% (f) SPCA, τ = 10%
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 39/43
Summary of this part
The dictionary learning framework leads tostate-of-the-art results for many image . . .
. . . and video processing tasks.
Online learning techniques are well-suited for thisproblem and allows training sets with millions ofpatches.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 40/43
References I
M. Aharon, M. Elad, and A. M. Bruckstein. The K-SVD: An algorithm for designingof overcomplete dictionaries for sparse representations. IEEE Transactions onSignal Processing, 54(11):4311–4322, November 2006.
L. Bottou and O. Bousquet. The trade-offs of large scale learning. In J.C. Platt,D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural InformationProcessing Systems, volume 20, pages 161–168. MIT Press, Cambridge, MA, 2008.
M. Elad and M. Aharon. Image denoising via sparse and redundant representationsover learned dictionaries. IEEE Transactions on Image Processing, 54(12):3736–3745, December 2006.
K. Engan, S. O. Aase, and J. H. Husoy. Frame based signal compression usingmethod of optimal directions (MOD). In Proceedings of the 1999 IEEEInternational Symposium on Circuits Systems, volume 4, 1999.
A. Haar. Zur theorie der orthogonalen funktionensysteme. Mathematische Annalen,69:331–371, 1910.
H. Lee, A. Battle, R. Raina, and A. Y. Ng. Efficient sparse coding algorithms. InB. Scholkopf, J. Platt, and T. Hoffman, editors, Advances in Neural InformationProcessing Systems, volume 19, pages 801–808. MIT Press, Cambridge, MA, 2007.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 41/43
References IIM. S. Lewicki and T. J. Sejnowski. Learning overcomplete representations. Neural
Computation, 12(2):337–365, 2000.
J. Mairal, M. Elad, and G. Sapiro. Sparse representation for color image restoration.IEEE Transactions on Image Processing, 17(1):53–69, January 2008a.
J. Mairal, G. Sapiro, and M. Elad. Learning multiscale sparse representations forimage and video restoration. SIAM Multiscale Modelling and Simulation, 7(1):214–241, April 2008b.
J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online dictionary learning for sparsecoding. In Proceedings of the International Conference on Machine Learning(ICML), 2009a.
J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online learning for matrix factorizationand sparse coding. ArXiv:0908.0050v1, 2009b. submitted.
J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Non-local sparse modelsfor image restoration. In Proceedings of the IEEE International Conference onComputer Vision (ICCV), 2009c.
S. Mallat. A Wavelet Tour of Signal Processing, Second Edition. Academic Press,New York, September 1999.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 42/43
References IIIB. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A
strategy employed by V1? Vision Research, 37:3311–3325, 1997.
M. Protter and M. Elad. Image sequence denoising via sparse and redundantrepresentations. IEEE Transactions on Image Processing, 18(1):27–36, 2009.
S. Roth and M. J. Black. Fields of experts: A framework for learning image priors. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2005.
Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 43/43
top related