Deep Belief Networks (D2L1 Deep Learning for Speech and Language UPC 2017)

Day 2 Lecture 1

Deep Belief Networks (DBN)

Elisa Sayrol

Restrictive Boltzmann Machine (RBM)

Training. Contrastive Divergence (CD)

Deep Belief Networks (DBN)

Overview

3

Restricted Boltzmann Machine (RBM)

Figure: Geoffrey Hinton (2013)

Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. "Restricted Boltzmann machines for collaborative filtering." Proceedings of the 24th international conference on Machine learning. ACM, 2007.

● Shallow two-layer net.● Restricted=No two nodes in a layer share a

connection● Bipartite graph.● Bidirectional graph

○ Shared weights.○ Different biases.

4

Restricted Boltzmann Machine (RBM)

Figure: Geoffrey Hinton (2013)

Salakhutdinov, Ruslan, Andriy Mnih, and Geoffrey Hinton. "Restricted Boltzmann machines for collaborative filtering." Proceedings of the 24th international conference on Machine learning. ACM, 2007.

5

Restricted Boltzmann Machine (RBM)

DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”.

Forward pass

6

Restricted Boltzmann Machine (RBM)

DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”.

Backwardpass

c

c

c

c

7

Restricted Boltzmann Machine (RBM)

DeepLearning4j, “A Beginner’s Tutorial for Restricted Boltzmann Machines”.

Backwardpass

The reconstructed values at the visible layer are compared with the actual ones with the KL Divergence.

8

What are the Maths behind RBMs? (Estimation of the parameters)

Geoffrey Hinton, "Introduction to Deep Learning & Deep Belief Nets” (2012)Geoorey Hinton, “Tutorial on Deep Belief Networks”. NIPS 2007.

9

What are the Maths behind RBMs?

Other references:

Deeplearning.net: Restricted Boltzmann Machines (with Theano functions and concepts)Hugo Larochelle: Course on NN

Let’s take a look at some of his slides on RBM….

10

What are the Maths behind RBMs?

Hugo Larochelle Slides

Hugo Larochelle Slides

Hugo Larochelle Slides

Hugo Larochelle Slides

Hugo Larochelle Slides

Hugo Larochelle Slides

Hugo Larochelle Slides

Hugo Larochelle Slides

Hugo Larochelle Slides

19

Deep Belief Networks (DBN)

Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.

● Architecture like an MLP.● Training as a stack of

RBMs.

20

Deep Belief Networks (DBN)

Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.

● Architecture like an MLP.● Training as a stack of

RBMs.

21

Deep Belief Networks (DBN)

Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.

● Architecture like an MLP.● Training as a stack of

RBMs.

22

Deep Belief Networks (DBN)

Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.

● Architecture like an MLP.● Training as a stack of

RBMs.

23

Deep Belief Networks (DBN)

Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.

● Architecture like an MLP.● Training as a stack of

RBMs…● ...so they do not need

labels:

Unsupervisedlearning

24

Deep Belief Networks (DBN)

Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18, no. 7 (2006): 1527-1554.

After the DBN is trained, it can be fine-tuned with a reduced amount of labels to solve a supervised task with superior performance.

Supervisedlearning

Softm

ax

Thank You!