Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 1 Deep -Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders Generative Adversarial Networks Pr. Fabien MOUTARDE Center for Robotics MINES ParisTech PSL Université Paris [email protected]http://people.mines-paristech.fr/fabien.moutarde Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 2 Acknowledgements During preparation of these slides, I got inspiration and borrowed some slide content from several sources, in particular: • Fei-Fei Li & J. Johnson & S. Yeung: course on Generative Models http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture13.pdf • I. Kokkinos: slides of a CentraleParis course on Deep Belief Networks http:// cvn.ecp.fr/personnel/iasonas/course/DL5.pdf • I. Goodfellow: NIPS’2016 tutorial on Generative Adversarial Networks (GANs) https:// media.nips.cc/Conferences/2016/Slides/6202-Slides.pdf • Binglin, Shashank & Bhargav: A short tutorial on Generative Adversarial Networks (GANs) http ://slazebni.cs.illinois.edu/spring17/lec11_gan.pdf
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 1
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 3
Outline
• Unsupervised Learning and Generative Models
• Deep Belief Networks (DBN)
and Deep Boltzman Machine (DBM)
• Autoencoders
• Generative Adversarial Networks (GAN)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 4
Deep vs Shallow Learning techniques overview
DEEPSHALLOW
GAN
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 5
Supervised vs Unsupervised
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 6
Unsupervised Learning
Examples:
General framework:
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 7
Generative models
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 8
Why Generative?
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 9
Why generative?
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 10
Taxonomy of Generative Models
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 11
Outline
• Unsupervised Learning and Generative Models
• Deep Belief Networks (DBN)
and Deep Boltzman Machine (DBM)
• Autoencoders
• Generative Adversarial Networks (GAN)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 12
Deep Belief Networks (DBN)
• One of first Deep-Learning models• Proposed by G. Hinton in 2006• Generative probabilistic model (mostly UNSUPERVISED)
For capturing high-order correlations of observed/visible data (à pattern analysis, or synthesis); and/or characterizingjoint statistical distributions of visible data
Greedy successive UNSUPERVISED learning of layersof Restricted Boltzmann Machine (RBM)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 13
Restricted Boltzmann Machine (RBM)
h, hidden
(~ latent variables)
v, observed
Modelling probability distribution as:
with « Energy » E given by
NB: connections are
BI-DIRECTIONAL
(with same weight)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 14
Training RBM
Finding q=(W,a,b) maximizing likelihood !"#$ %&(v) of dataset S
ó minimize NegLogLikelihood '*+#, log %&(-)
So objective = find ./ = argMin&
'0+#,
01log %&(-2)
Algo: Contrastive Divergence
» Gibbs sampling used inside a gradient descent procedure
Independance within layers è % - 5) = A3% -3 5 % 5 -) = A
2% 52 -and
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 15
Repeat:
1. Take a training sample v, compute B C1 = D +) = E F1 8G1;:+and sample a vector h from this probability distribution
2. Compute positive gradient as outer product HI = +JC = +CK3. From h, compute B +LN = D C) = E ON 8G:;NC and sample reconstructed v',
then resample h' using B CL1 = D +L) = E F1 8G1;:+L[Gibbs sampling single step; should theoretically be repeated until convergence]
4. Compute negative gradient as outer product HP = +LJCL = +LCLK5. Update weight matrix by QG = R HI ' HP = R +CK ' +SCLK6. Update biases a and b analogously: QO = R + ' +L and QF = R C ' CL
Contrastive Divergence algo
Gibbs sampling
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 16
Use of trained RBM
• Input data "completion" : set some vi thencompute h, and generate compatible full samples
• Generating representative samples
• Classification if trainedwith inputs=data+label
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 17
Modeling of input data distribution from trained RBM
Initial data is in blue, reconstructed in red (and green line connects each data point with
reconstructed one).
Learnt energy function:
minima created where data points are
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 18
Interpretation of trained RBM hidden layer
• Look at weights of hidden nodes à low-level features
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 19
Why go deeper with DBN ?
DBN: upper layers à more « abstract » features
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 20
Learning of DBN
Greedy learning of successive layers
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 21
Using low-dim final featuresfor clustering
Much better results than clustering in input space
or using other dimension reduction (PCA, etc…)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 22
Example application of DBN:Clustering of documents in database
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 23
Image Retrievalapplication example of DBN
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 24
DBN supervised tuning
UNSUPERVISED SUPERVISED
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 25
Outline
• Unsupervised Learning and Generative Models
• Deep Belief Networks (DBN)
and Deep Boltzman Machine (DBM)
• Autoencoders
• Generative Adversarial Networks (GAN)
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 26
Autoencoders
Learn qq
and pF
in order to minimize reconstruction cost:
à unsupervised learning of latent variables,
and of a generative model
T =0UVWU 'WU X =0
UBY Z[ WU 'WU X
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 27
Variants of autoencoders
• Denoising autoencoders
• Sparse autoencoders
• Stochastic autoencoders
• Contractive autoencoders
• VARIATIONAL autoencoders
• …
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 28
Deep Stacked Autoencoders
Proposed by Yoshua Bengio in 2007
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 29
Training of StackedAutoencoers
Greedy layerwise training:
for each layer k, use backpropagation to minimize
|| Ak(h(k))-h(k) ||2 (+ regularization cost l Sij |Wij|
2)
possibly + additional term for "sparsity"
etc…
Unsupervised Generative Deep-Learning: DBN+DSA+GAN, Pr F.MOUTARDE, Center for Robotics, MINES ParisTech, PSL, March2019 30