Machine Learning Lecture 11 Deep Learning RNN & GAN Dr. Patrick Chan [email protected]South China University of Technology, China 1 Dr. Patrick Chan @ SCUT Agenda Recurrent Neural Network Structure Backpropagation Through Time Types Example: Image Captioning Generative Adversarial Network Generative and Discriminative models Training Process and Loss Function Lecture 11: DL - RNN & GAN 2
21
Embed
Lecture11 - DL - RNN & GAN · Dr. Patrick Chan @ SCUT Problem of traditional ML Traditional ML assumes Sample format is identical Decision of sample is independent Cannotdeal with
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Sequence learningLearn from and handle the sequential data
Application Example:
Language
Video
Recurrent Neural Network learn the information of sequence
Lecture 11: DL - RNN & GAN4
Dr. Patrick Chan @ SCUT
RNN
Structure
Lecture 11: DL - RNN & GAN5
��
�� (�) �� (�) �� (��)
(�) �� (�)
Feed Forward
Neural Network
Recurrent
Neural Network
(�) ��
(�)
��
(�)(��)
��
��
��
tanh is used
Dr. Patrick Chan @ SCUT
RNN
Structure
Lecture 11: DL - RNN & GAN6
��� � �
�
���
� �
�
Feed Forward
Neural Network
(�)
����(�)�
� ����(��)�
�
(�)
���
�(�)�
�
Recurrent
Neural Network
(�)
(��)
�(�)
���
ℎ(�)
����
���� ��
(�)
��
ℎ�(�) ℎ�
(�)
���
������
��
(�) ��
� ���
ℎ
����
���� ��
��
ℎ� ℎ� �
��
������
��
��� ���� ����
ℎ(��)
�
� − �ℎ�
(��) ℎ�(��)
��
Dr. Patrick Chan @ SCUT
RNN
Structure
RNN is multiple copies of the same network, each passing a message to a successor
, , and are
shared (Do not change)
Lecture 11: DL - RNN & GAN7
������ ���x(t) h(t) ��(t)
h(t-1)
Unfold the loop
x(1) h(1) ��(1)h(1)
x(2) h(2) ��(2)h(2)
x(3) h(3) ��(3)h(3)
x(t) h(t) ��(t)h(t-1)
h(0)
������ ���
������ ���
������ ���
��� ���
…
Dr. Patrick Chan @ SCUT
RNN
Example
Character-level language model
Generate one new character at a time by outputting the probability distribution of the next character in the sequence given a sequence of previous characters
Lecture 11: DL - RNN & GAN8
Dr. Patrick Chan @ SCUT
RNN
Different Types
Lecture 11: DL - RNN & GAN9
One to One Feed-Forward Network
One to Many Image CaptioningImage > Seq. of Words
Many to One Sentiment ClassificationSeq. of Words > Sentiment
Many to Many TranslationSeq. of Words > Seq. of Words
Video Classification (frame Level)Frame > Class
Dr. Patrick Chan @ SCUT
RNN
Backpropagation Through Time
Parameters of RNN ( , and ) are
shared
Different time steps are affected each other
Derivatives are aggregated across time steps
A special Backpropagation Backpropagation through time (BPTT)
Lecture 11: DL - RNN & GAN10
Dr. Patrick Chan @ SCUT
J
RNN
Backpropagation Through Time
Lecture 11: DL - RNN & GAN11
�� ()
�����
��� �� = " �� ()
�� ��
�
= � 12 � () − �� () �
�� ��
= − � () − �� () ��� ()
�� ��
= �%(� ��ℎ (�))�� ��
= �%(� ��ℎ (�))�(� ��ℎ (�))
�(� ��ℎ (�))�� ��
= ℎ (�)
x(1) h(1) ��(1)h(1)
x(2) h(2) ��(2)h(2)
x(3) h(3) ��(3)
h(0)
������ ���
������ ���
��� ���
J(1) y(1)
J(2) y(2)
J(3) y(3)
��� ()
�� ��
�(� ��ℎ (�))�� ��
= %′(� ��ℎ (�)) �(� ��ℎ (�))�� ��
ℎ(�) = %(� ��' � + � ��ℎ (��))
��(�) = %(� ��ℎ (�))
() (�) (�)
() () () �
Dr. Patrick Chan @ SCUT
J
RNN
Backpropagation Through Time
Lecture 11: DL - RNN & GAN12
�� ()
�� ��
��� �� = " �� ()
�� ��
�
= � 12 � () − �� () �
�� ��
= − � () − �� () ��� ()
�� ��
= �%(� ��ℎ (�))�� ��
= �%(� ��ℎ (�))�(� ��ℎ (�))
�(� ��ℎ (�))�� �� x(1) h(1) ��(1)
h(1)
x(2) h(2) ��(2)h(2)
x(3) h(3) ��(3)
h(0)
������ ���
������ ���
��� ���
J(1) y(1)
J(2) y(2)
J(3) y(3)
��� ()
�� ��
= %′(� ��ℎ (�)) �(� ��ℎ (�))�� ��
= � �� �(ℎ (�))�� ��
�(� ��ℎ (�))�� ��
ℎ(�) = %(� ��' � + � ��ℎ (��))
��(�) = %(� ��ℎ (�))
() (�) (�)
() () () �
Dr. Patrick Chan @ SCUT
J
RNN
Backpropagation Through Time
Lecture 11: DL - RNN & GAN13
x(1) h(1) ��(1)h(1)
x(2) h(2) ��(2)h(2)
x(3) h(3) ��(3)
h(0)
������ ���
������ ���
��� ���
J(1) y(1)
J(2) y(2)
J(3) y(3)
= �(� ��' � + � ��ℎ (��))�� ��
�(ℎ (�))�� ��
= ℎ (��) + � �� �(ℎ (��))�� ��
Recursive function
�(ℎ ())�� �� = ℎ ()) + � �� �(ℎ ()))
�� ��
= ℎ ())
Base case
= ' � �(� ��)�� �� + � �� �(' � )
�� ��
+ℎ (��) �(� ��)�� �� + � �� �(ℎ (��))
�� ��
ℎ(�) = %(� ��' � + � ��ℎ (��))
��(�) = %(� ��ℎ (�))
() (�) (�)
() () () �
Dr. Patrick Chan @ SCUT
J
RNN
Backpropagation Through Time
Lecture 11: DL - RNN & GAN14
�� ()
�� ��
��� �� = " �� ()
�� ��
�
= � 12 � () − �� () �
�� ��
= − � () − �� () ��� ()
�� ��
= �%(� ��ℎ (�))�� ��
= �%(� ��ℎ (�))�(� ��ℎ (�))
�(� ��ℎ (�))�� ��
x(1) h(1) ��(1)h(1)
x(2) h(2) ��(2)h(2)
x(3) h(3) ��(3)
h(0)
������ ���
������ ���
��� ���
J(1) y(1)
J(2) y(2)
J(3) y(3)
��� ()
�� ��
= %′(� ��ℎ (�)) �(� ��ℎ (�))�� ��
= � �� �(ℎ (�))�� ��
�(� ��ℎ (�))�� ��
ℎ(�) = %(� ��' � + � ��ℎ (��))
��(�) = %(� ��ℎ (�))
() (�) (�)
() () () �
Dr. Patrick Chan @ SCUT
J
RNN
Backpropagation Through Time
Lecture 11: DL - RNN & GAN15
x(1) h(1) ��(1)h(1)
x(2) h(2) ��(2)h(2)
x(3) h(3) ��(3)
h(0)
������ ���
������ ���
��� ���
J(1) y(1)
J(2) y(2)
J(3) y(3)
= �(� ��' � + � ��ℎ (��))�� ��
�(ℎ (�))�� ��
�(' ())�� �� = ' () + � �� �(ℎ ()))
�� ��
= ' ()
= ' � �(� ��)�� �� + � �� �(' � )
�� ��
+ℎ (��) �(� ��)�� �� + � �� �(ℎ (��))
�� ��
= ' � + � �� �(ℎ (��))�� ��
ℎ(�) = %(� ��' � + � ��ℎ (��))
��(�) = %(� ��ℎ (�))
() (�) (�)
() () () �
Recursive function
Base case
Dr. Patrick Chan @ SCUT
RNN
Other Structures
RNNs with multiple hidden layers
16
(�)
(�)
()
()
(�)
(�)
(�)
(�)
(*)
(*)
(+)
(+)
Lecture 11: DL - RNN & GAN
Dr. Patrick Chan @ SCUT
RNN
Other Structures
Bi-directional RNN
process the input sequence in forward and in the reverse direction
Popular in speech recognition
17
(�)
(�)
(�,)
(��)
()
()
(�)
(�)
(�)
(�)
(*)
(*)
(+)
(+)
Lecture 11: DL - RNN & GAN
Dr. Patrick Chan @ SCUT
RNN
Example: Image Captioning
Lecture 11: DL - RNN & GAN18
straw
()
<START>
()straw
(�)hat
(�)
<END>
(�)hat
(�)
Dr. Patrick Chan @ SCUT
RNN
Example: Image Captioning
Lecture 11: DL - RNN & GAN19
Dr. Patrick Chan @ SCUT
RNN
Long-Term Dependency
RNN performs badly when a task requires long-term dependency
Example:
A language model trying to predict the next word based on the previous ones
I grew up in France… I speak fluent French
“I speak fluent” suggests the next word is a language
For narrowing down which language, the context of France is required, which need long dependency