Discussion Ying Nian Wu UCLA Department of Statistics JSM 2011 Population value decomposition
Jan 03, 2016
Discussion
Ying Nian WuUCLA Department of Statistics
JSM 2011
Population value decomposition
Latent variable models
Hidden
Observed
Learning: Examples
Inference:
Latent variable models Mixture model
Factor analysis
Computational neural science
Z: Internal representation by neurons
Y: Sensory data from outside environment
Hidden
Observed
Connection weights
Hierarchical extension: modeling Z by another layer of hidden variables explaining Y instead of Z
Inference / explaining away
Source: Scientific American, 1999
Visual cortex: layered hierarchical architecture
V1: primary visual cortex simple cells complex cells
bottom-up/top-down
Independent Component Analysis Bell and Sejnowski, 1996
CBcBcI NN B ...11
Nicpci ,...,1tly independen )(~
)dim(IN
IIC AB 1
mNNmmm CBcBcI B ,11, ...
mmm IIC AB 1
Laplacian/Cauchy
Hyvarinen, 2000
Sparse coding Olshausen and Field, 1996
Laplacian/Cauchy/mixture Gaussians
Nicpci ,...,1tly independen )(~
NNBcBcI ...11
mNNmmm BcBcI ,11, ...)dim(IN
Inference: sparsification, non-linear lasso/basis pursuit/matching pursuit mode and uncertainty of p(C|I) explaining-away, lateral inhibition
Nicpci ,...,1tly independen )(~
Sparse coding / variable selection
Learning: mNNmmm BcBcI ,11, ...
)dim(IN
A dictionary of representational elements (regressors)
NNBcBcI ...11
Olshausen and Field, 1996
}exp{)(
1),(
,, j
jiiji vhW
WZVHp
Nihi ,...,1 ,
V
Restricted Boltzmann Machine Hinton, Osindero and Teh, 2006
P(V|H)P(H|V): factorized no-explaining away
hidden, binary
visible
Source: Scientific American, 1999
Visual cortex: layered hierarchical architecture
bottom-up/top-down
What is beyond V1?Hierarchical model?
P(V,H) = P(H)P(V|H) P(H) P(V’,H)
I
H
V
V’
Discriminative correction by back-propagation
Unfolding, untying, re-learning
Hierarchical RBM Hinton, Osindero and Teh, 2006
Hierarchical sparse coding
NNBcBcI ...11
,,sxB
Attributed sparse coding elements transformation group topological neighborhood system
UBcIii sx
n
ii
,,
1
Layer above : further coding of the attributes of selected sparse coding elements
Active basis modelWu, Si, Gong, Zhu, 10Zhu, Guo, Wang, Xu, 05
n-stroke templaten = 40 to 60, box= 100x100
Learning and Inference
Finding n strokes to sketch M images simultaneouslyn = 60, M = 9
Scan over multiple resolutions
Scan over multiple resolutions and orientations (rotating template)
Learning active basis models from non-aligned imageEM-type maximum likelihood learning, Initialized by single image learning
Learning active basis models from non-aligned image
Learning active basis models from non-aligned image
Hierarchical active basis
Lowlog-likelihood
Highlog-like
Model based clustering
MNIST500 total