Dictionary Learning for Sparse Learning
based Image Classification
Dr. Meng Yang
Smart LV Lab, Computer Vision Institute, Shenzhen University
OUTLINE
Introduction of sparse learning
Discriminative dictionary learning
Outlook
OUTLINE
Introduction of sparse learning
Discriminative dictionary learning
Outlook
DATA REPRESENTATION
Massive High-Dimensional Data
0
0
Low-dimensional structures
SPARSE TRANSFORMATION
x is a sparse vector.
Most energy concentrated in a small number of features
SPARSE SIGNAL PROCESSING
Signal Processing
Compressive
sensing
Sparse Learning
N
M M
y
=
N f x
Measurement matrix
SPARSE NEURAL CODES
Population sparseness Lifetime sparseness
B. Willmore and D. J. Tolhurst. Characterizing the sparseness of neural codes. Network, 12:255–270, 2001
W. E. Vinje and J. L. Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. SCIENCE, 287(5456):1273–1276, 2000.
Visual Cortex
Neural
codes
Sparse Learning
SRC
0 20 40 60 80 100-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Atom index
Co
din
g c
oe
ffici
en
t va
lue
Test image Coefficient Training set corruption/occlusion
8
Coding coefficient x and residual e are sparse!
J. Wright, Y. Ma, et al. Robust face recognition via sparse representation, 2009.
1 2 3 … N
subject i subject 1… subject n subject i
1 2 3 … N
Classification criterion: Identity = argmini{ri}.
DICTIONARY LEARNING FOR CLASSIFIER
2000
Sparse HVS[1][2]
2006 2007 2013 2014 2009 2011 2012 2010
DKSVD[15] LC-KSVD[16] Task DL[17]
Pair DL[8]
Metaface[18] DLSI[19]
CS-DL[20] FDDL[21]
SVDL[22]
LatentDL[23]
KSVD[5] Ana & Syn [6] Ana-KSVD[7] A-S DL[26]
LASSO[3] CS[4]
Hybrid [27][28]
REFERENCE
1. B. Willmore and D. J. Tolhurst. Characterizing the sparseness of neural codes. Network, 12:255–270, 2001
2. W. E. Vinje and J. L. Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. SCIENCE, 287(5456):1273–1276, 2000.
3. M. Osborne, B. Presnell, and B. Turlach, “A new approach to variable selection in least squares problems,” IMA Journal of Numerical Analysis, vol. 20, 2000
4. Emmanuel Candès, Compressive Sampling. ((Int. Congress of Mathematics, 3, pp. 1433-1452, Madrid, Spain, 2006
5. M. Aharon, M. Elad, and A.M. Bruckstein, "The K-SVD: An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation", the IEEE Trans. On Signal
Processing, Vol. 54, no. 11, pp. 4311-4322, November 2006.
6. M. Elad, P. Milanfar, and R. Rubinstein, “Analysis versus synthesis in signal priors,” Inverse Problems, vol. 23, no. 3, pp. 947–968, June 2007.
7. R. Rubinstein, T. Peleg and M. Elad, Analysis K-SVD: A Dictionary-Learning Algorithm for the Analysis Sparse Model, IEEE TSP, March 2013.
8. S. Gu, L. Zhang, W. Zuo, and X. Feng, “Projective Dictionary Pair Learning for Pattern Classification,” In NIPS 2014.
9. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma. Robust face recognition via sparse representation. IEEE PAMI, 31(2):210–227, 2009.
10. M. Yang, L. Zhang, J. Yang, and D. Zhang, “Robust sparse coding for face recognition,” in Proc. IEEE Conf. CVPR, Jun. 2011, pp. 625–632.
11. R. He, W. S. Zheng, and B. G. Hu, “Maximum correntropy criterion for robust face recognition,” IEEE Trans. PAMI, vol. 33, no. 8, pp. 1561–1576,2011
12. J. Yang, et all, Nuclear norm based matrix regression with application to face recognition with occlusion and illumination changes, 2014
13. Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching uisng sparse coding for image classification. CVPR, 2009
14. S.H, Gao, et al. Laplacian Sparse Coding, Hypergraph Laplacian Sparse Coding, and Applications, PAMI, 2012
15. Zhang, Q., & Li, B. X. (2010). Discriminative K-SVD for dictionary learning in face recognition. In:Proceedings of the IEEE CVPR
16. Jiang, Z. L., Lin, Z., & Davis, L. S. (2013). abel consistent K-SVD: Learning a discriminative dictionary for recognition. IEEE TPAMI,34, 533.
17. Mairal, J., Bach, F., & Ponce, J. (2012). Task-driven dictionary learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4), 791–804.
18. Yang, M., Zhang, L., Yang, J., & Zhang, D. (2010c). Metaface learning for sparse representation based face recognition. In:Proceedings of the ICIP
19. Ramirez, I., Sprechmann, P., & Sapiro, G. (2010). Classification and clustering via dictionary learning with structured incoherence and shared features. In:Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition
20. Wang, H. R., Yuan, C. F., Hu, W. M., & Sun, C. Y. (2012). Supervised class-specific dictionary learning for sparse modeling in action recognition. PR,45(11), 3902–3911
21. Yang, M., Zhang, L., Feng, X. C., & Zhang, D. (2011b). Fisher discrimination dictionary learning for sparse representatio. In:Proceedings of ICCV
22. Yang, M., et al. Sparse Variation Dictionary Learning for Face Recognition with A Single Training Sample Per Person, ICCV 2013
23. Yang, M. , et al. Latent Dictionary Learning for Sparse Representation based Classification, CVPR 2014
24. Zhang, L., Yang, M. et al. Sparse Representation orCollaborative Representation: Which Helps Face Recognition? ICCV 2011
25. Yang. Meng, et al. Relaxed Collaborative Representation for Pattern Classification, CVPR 2012
26. Ron Rubinstein, Member, IEEE, and Michael Elad, Dictionary Learning for Analysis-Synthesis Thresholding, TSP, 2014
27. Zhou, N., & Fan, J. P. (2012). Learning inter-related visual dictionary for object recognition. In:Proceedings of the IEEE CVPR
28. Kong, S., & Wang, D. H. (2012).A dictionary learning approach for classification: Separating the particularity and the commonality.In: ECCV
OUTLINE
Introduction of sparse learning
Discriminative dictionary learning
Outlook
DISCRIMINATIVE DICTIONARY LEARNING
Class-specific DL
Hybrid DL
Latent DL
min y2
2 1
“The choice of the dictionary that sparsifies the signals is crucia
l for the success of this model.” M. Elad et al. Proc. IEEE 10.
?
CLASS-SPECIFIC DL
Metaface[18], DLSI[19],CS-DL[21], FDDL[20]…
FDDL
Predefined bases (e.g., wavelet, DCT) too general
Training data matrix may have a big size (e.g., SRC)
Discriminative Dictionary Learning
Discriminative sparse coding coefficients
Discriminative class-specific sub-dictionary
0 20 40 60 80 100-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Atom index
Co
din
g c
oe
ffic
ien
t va
lue
MAIN IDEA
Sparse
Coefficient
Discrimination of representation residual and coding coefficient.
GOOD FOR
BAD FOR Fisher
criterion
SMALL within- class scatter
BIG between- class scatter
D=[D1,D2,…,Dc]
Di Xi
Xj
FDDL MODEL
A=[A1, A2, …, AK], Ai :training samples from class i.
D=[D1,D2,…,DK], Di :sub-dictionary of the ith class.
X=[X1, X2, …, XK], Xi : coding coefficients of Ai over D.
1 211,min , ,
K
i iir f
D XA D X X X
Discriminative data fidelity term
Discriminative coefficient term
2
W B Ff tr X S X S X X
2 22
1, ,Ki jji i i i i i i j iF F Fj i
r
A D X A DX A D X D X
SIMPLIFIED-FDDL MODEL
A=[A1, A2, …, AK], Ai :training samples from class i.
D=[D1,D2,…,DK], Di :sub-dictionary of the ith class.
X=[X1, X2, …, XK], Xi : coding coefficients of Ai over D.
1 21 1,min , , s.t .
K i j
i i i i iir f
0
D XA D X X X X
Discriminative data fidelity term
Discriminative coefficient term
2
i i Ff X X M
2
, , i
i i i i i Fr A D X A D X
CLASSIFICATION MODEL
Global classifier
Local classifier for i-th class
DIGIT RECOGNITION (USPS)
Algorithms FDDL SRSC REC-L REC-BL SDL-G
Error rate (%) 2.89 6.05 6.83 4.38 6.67
TDDL SDL-D DLSI KNN SVM COPAR
2.84 3.54 3.98 5.2 4.2 3.61
Learned dictionary atoms
LATENT DL
Imagenet Vehicle
general
specific
ADAPTIVE HIERARCHY DL TO LATENT DL
. . . . . .
. . . . . . . . .
LATENT DICTIONARY LEARNING
Latent dictionary
atom dm
Training
data
?
Class
1
Class
2
Class
3
Class
4
? ? ? ?
Latent vector for dm
Latent Dictionary Learning (LDL)
w1,m
w2,m
w3,m
w4,m
wj,m indicates the relationship between atom dm and jth class label.
LATENT DICTIONARY LEARNING MODEL
2 2
1 21, ,1
2
3 , , ,1 1
,
min diag
s.t. 0 , ;
, ;
C
j j j j j j FFj
C N T
j m m n l n j mj l j n m n
j mm
w w w j m
w m
D X WA D w X X X M
d d
Latent sparse representation Discriminative coefficient
Latent dictionary incoherence
The latent vector could be efficiently solved by an iterative
procedure, and there is an analytical solution in each
iteration.
LATENT CLASSIFICATION MODEL
Global classifier
Local classifier for j-th class
ACT RECOGNITION (UCF SPORTS ACTION)
OUTLINE
Introduction of sparse learning
Discriminative dictionary learning
Outlook
DICTIONARY LEARING FOR BIG DATA
Beyond small data
DICTIONARY LEARNING
Beyond shallow dictionary learning
非常感谢各位!
Question?