Deep Unsupervised Learning using Nonequlibrium Thermodynamics Tran Quoc Hoan @k09ht haduonght.wordpress.com/ 14 December 2015, Paper Alert, Hasegawa lab., Tokyo The University of Tokyo Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli Proceedings of the 32 nd International Conference on Machine Learning, 2015
30
Embed
007 20151214 Deep Unsupervised Learning using Nonequlibrium Thermodynamics
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Deep Unsupervised Learning using Nonequlibrium Thermodynamics
Tran Quoc Hoan
@k09ht haduonght.wordpress.com/
14 December 2015, Paper Alert, Hasegawa lab., Tokyo
The University of Tokyo
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli Proceedings of the 32nd International Conference on Machine Learning, 2015
Abstract
Deep Unsupervised Learning using Nonequilibrium Thermodynamics 2
“…The essential idea, inspired by non-equilibrium statistical
physics, is to systematically and slowly destroy structure in
a data distribution through an iterative forward diffusion
process. We then learn a reverse diffusion process
that restores structure in data, yielding a highly flexible
and tractable generative model of the data…”
Outline
3
- The promise of deep unsupervised learning
• Motivation
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- Diffusion processes and time reversal
• Physical intuition
- Derivation and experimental results
• Diffusion probabilistic model
Deep Unsupervised Learning
4
- Novel modalities
• Unknown features/labels
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- Ex. disease part in medical image
• Expensive labels
• Unpredictable tasks / one shot learning
- Exploratory data analysis
https://www.ceessentials.net/article40.html
Physical Intuition
5
- Destroy structure in data
• Diffusion processes and time reversal
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- Carefully characterize the destruction
- Learn how to reverse time
Observation 1: Diffusion Destroy Structure
6
Data distribution
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Uniform distribution
Uniform distributionData distribution
(Observation)Diffusion destroys structure
(Recover structure)Recover data distribution by starting from uniform
distribution and running dynamics backwards
Observation 2: Microscopic Diffusion
7
• Time reversible
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
https://www.youtube.com/watch?v=cDcprgWiQEY
• Brownian motion
• Position updates are small Gaussians (both forwards and backwards in time)
Diffusion-based Probabilistic Models
8
• Destroy all structure in data distribution using diffusion process
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Learn reversal of diffusion process
- Estimate function for mean and covariance of each step in the reverse diffusion process (Ex. binomial rate for binary data)
• Reverse diffusion process is the model of the data
Diffusion-based Probabilistic Models
9
• Algorithm
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Deep convolutional network: universal function approximatior