Junhua Mao [email protected]UCLA 10/18/2016 Multimodal Learning Using Recurrent Neural Networks This talk follows from joint work with Alan Yuille, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, Haoyuan Gao, Jonathan Huang, Kevin Murphy, Jiajing Xu, and Kevin Jing, among others.
57
Embed
Multimodal Learning Using Recurrent Neural Networksayuille/JHUcourses/Probabilistic...Junhua Mao. [email protected] UCLA. 10/18/2016. Multimodal Learning Using Recurrent Neural Networks
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Multimodal Learning Using Recurrent Neural Networks
This talk follows from joint work with Alan Yuille, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, Haoyuan Gao, Jonathan Huang, Kevin Murphy, Jiajing Xu, and Kevin Jing, among others.
• The m-RNN image captioning model Image caption generation
Image retrieval (given query sentence)
Sentence retrieval (given query image)
• Extensions Incremental novel concept captioning
Multimodal word embedding learning
Referring expressions
Visual Question Answering
The m-RNN Image Captioning Model
a close up of a bowl of food on a table
a train is traveling down the tracks in a city
a pizza sitting on top of a table next to a box of pizza
http://www.stat.ucla.edu/~junhua.mao/m-RNN.html
Mao, J., Xu, W., Yang, Y., Wang, J., Z. Huang & Yuille, A. Deep captioning with multimodal recurrent neural networks (m-rnn). In Proc. ICLR 2015.
a cat laying on a bed with a stuffed animal
Abstract• Three Tasks:
Image caption generation
Image retrieval (given query sentence)
Sentence retrieval (given query image)
• One model (m-RNN): A deep Recurrent NN (RNN) for the sentences
A deep Convolutional NN (CNN) for the images
A multimodal layer connects the first two components
• State-of-the-art Performance: For three tasks
On four datasets: IAPR TC-12 [Grubinger et al. 06’], Flickr 8K [Rashtchian et al. 10’], Flickr 30K [Young et
al. 14’] and MS COCO [Lin et al. 14’]
The m-RNN Model
𝑤𝑤1,𝑤𝑤2, … ,𝑤𝑤𝐿𝐿 is the sentence description of the image𝑤𝑤𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠, 𝑤𝑤𝑒𝑒𝑒𝑒𝑒𝑒 is the start and end sign of the sentence
Embedding I Embedding II Recurrent Multimodal SoftMaxLayers:
The m-RNN model for one word
128 256 256 512 Predict
Predict
Predict
CNN
CNN
CNN
Image
Image
Image
wstart w1
w1 w2
wendwL
The m-RNN ModelEmbedding I Embedding II Recurrent Multimodal SoftMaxLayers:
The m-RNN model for one word
128 256 256 512 Predict
Predict
Predict
CNN
CNN
CNN
Image
Image
Image
wstart w1
w1 w2
wendwL
The output of the trained model: 𝑃𝑃(𝑤𝑤𝑒𝑒|𝑤𝑤1:𝑒𝑒−1, 𝐈𝐈)
Application• Image caption generation:
o Begin with the start sign 𝑤𝑤𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠o Sample next word from 𝑃𝑃 𝑤𝑤𝑒𝑒 𝑤𝑤1:𝑒𝑒−1, 𝐈𝐈o Repeat until generating 𝑤𝑤𝑒𝑒𝑒𝑒𝑒𝑒
Application• Image caption generation:
o Begin with the start sign 𝑤𝑤𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠o Sample next word from 𝑃𝑃 𝑤𝑤𝑒𝑒 𝑤𝑤1:𝑒𝑒−1, 𝐈𝐈o Repeat until generating 𝑤𝑤𝑒𝑒𝑒𝑒𝑒𝑒
• Image retrieval given query sentence:o Ranking score: 𝑃𝑃 𝑤𝑤1:𝐿𝐿
𝑄𝑄 𝐈𝐈𝐷𝐷 = ∏𝑒𝑒=2𝐿𝐿 𝑃𝑃 𝑤𝑤𝑒𝑒
𝑄𝑄 𝑤𝑤1:𝑒𝑒−1𝑄𝑄 , 𝐈𝐈𝐷𝐷
o Output the top ranked images
Application• Image caption generation:
o Begin with the start sign 𝑤𝑤𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠o Sample next word from 𝑃𝑃 𝑤𝑤𝑒𝑒 𝑤𝑤1:𝑒𝑒−1, 𝐈𝐈o Repeat until generating 𝑤𝑤𝑒𝑒𝑒𝑒𝑒𝑒
• Image retrieval given query sentence:o Ranking score: 𝑃𝑃 𝑤𝑤1:𝐿𝐿
𝑄𝑄 𝐈𝐈𝐷𝐷 = ∏𝑒𝑒=2𝐿𝐿 𝑃𝑃 𝑤𝑤𝑒𝑒
𝑄𝑄 𝑤𝑤1:𝑒𝑒−1𝑄𝑄 , 𝐈𝐈𝐷𝐷
o Output the top ranked images
• Sentence retrieval given query image:o Challenge: Some sentences have high probability for any image queryo Solution: Normalize the probability. 𝐈𝐈′ are sampled images:
𝑃𝑃 𝑤𝑤1:𝐿𝐿𝐷𝐷 𝐈𝐈𝑄𝑄
𝑃𝑃 𝑤𝑤1:𝐿𝐿𝐷𝐷 𝑃𝑃 𝑤𝑤1:𝐿𝐿
𝐷𝐷 = �𝐈𝐈′𝑃𝑃 𝑤𝑤1:𝐿𝐿
𝐷𝐷 𝐈𝐈′ ⋅ 𝑃𝑃(𝐈𝐈′)
Application
𝑃𝑃 𝐈𝐈𝑄𝑄 𝑤𝑤1:𝐿𝐿𝐷𝐷 =
𝑃𝑃 𝑤𝑤1:𝐿𝐿𝐷𝐷 𝐈𝐈𝑄𝑄 ⋅ 𝑃𝑃(𝐈𝐈𝑄𝑄)𝑃𝑃 𝑤𝑤1:𝐿𝐿
𝐷𝐷
• Image caption generation:o Begin with the start sign 𝑤𝑤𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠o Sample next word from 𝑃𝑃 𝑤𝑤𝑒𝑒 𝑤𝑤1:𝑒𝑒−1, 𝐈𝐈o Repeat until generating 𝑤𝑤𝑒𝑒𝑒𝑒𝑒𝑒
• Image retrieval given query sentence:o Ranking score: 𝑃𝑃 𝑤𝑤1:𝐿𝐿
𝑄𝑄 𝐈𝐈𝐷𝐷 = ∏𝑒𝑒=2𝐿𝐿 𝑃𝑃 𝑤𝑤𝑒𝑒
𝑄𝑄 𝑤𝑤1:𝑒𝑒−1𝑄𝑄 , 𝐈𝐈𝐷𝐷
o Output the top ranked images
• Sentence retrieval given query image:o Challenge: Some sentences have high probability for any image queryo Solution: Normalize the probability. 𝐈𝐈′ are sampled images:
𝑃𝑃 𝑤𝑤1:𝐿𝐿𝐷𝐷 𝐈𝐈𝑄𝑄
𝑃𝑃 𝑤𝑤1:𝐿𝐿𝐷𝐷 𝑃𝑃 𝑤𝑤1:𝐿𝐿
𝐷𝐷 = �𝐈𝐈′𝑃𝑃 𝑤𝑤1:𝐿𝐿
𝐷𝐷 𝐈𝐈′ ⋅ 𝑃𝑃(𝐈𝐈′)
Equivalent
Experiment: Retrieval
Sentence Retrival (Image to Text) Image Retrival (Text to Image)R@1 R@5 R@10 Med r R@1 R@5 R@10 Med r
R@K: The recall rate of the groundtruth among the top K retrieved candidatesMed r: Median rank of the top-ranked retrieved groundtruth
Results on the MS COCO test set
(**) Provided in https://www.codalab.org/competitions/3221#results(***) We evaluate it on the MS COCO evaluation server: https://www.codalab.org/competitions/3221
c5 and c40: evaluated using 5 and 40 reference sentences respectively.
“-beam” means that we generate a set of candidate sentences, and then selects the best one. (beam search)
Experiment: Captioning
一辆双层巴士停在一个城市街道上。
一列火车在轨道上行驶。一个年轻的男孩坐在长椅上。
We acknowledge Haoyuan Gao and Zhiheng Huang from Baidu Research for designing the Chinese image captioning system
A young boy sitting on a bench. A train running on the track. A double decker bus stop on a city street.
Discussion: Chinese captions
Novel Visual Concept Learning
Mao, J., Xu, W., Yang, Y., Wang, J., Z. Huang & Yuille, A. Learning like a Child: Fast Novel Visual Concept Learning from Sentences Descriptions. In Proc. ICCV 2015
Novel Visual Concept Learning
Train
Captioning Model
…
Testing Image
A group of people in red and green are playing soccer.
Using a few examples
Our NVCS Model A group of people is playing quidditchwith a red ball.
Meet with Novel Concepts, E.g. Quiddich?
The learning Novel Visual Concept from Sentences (NVCS) Task:• Hard but important
o Slow if retrain the whole model
o Lack of training samples
o Overfit easily
Our contributions to this new task:• Datasets [Released]
• A novel frameworko Fast, simple and effective
o No extensive retraining
A group of people playing a game of soccer.
The Novel Visual Concept Dataset• Released on project page: http://www.stat.ucla.edu/~junhua.mao/projects/child_learning.html
• Contains 11 novel visual conceptso 100 images per concept, 5 sentences per image
A quidditch team is takinga group photo withbrooms in their hands.
A side view of a t-rex fossilskeleton in a museum withmany visitors.
Quidditch T-rex Samisen Kiss
A man in grey kimono isplaying samisen with abrown background.
A couple kisses on a field ofgrass while forming a heartin front of them with theirhands.
A man in a white uniform leads a group of people in red and black uniforms in practicing tai-ji.
The rocket gun on the backof a green truck fires amissile making a lot ofsmoke
Tai-ji Rocket Gun Waterfall Wedding Dress
On the mountain a steepcliff holds a waterfall thatpours down through theleftovers of trees
Two women wear weddingdresses , one with a long veiland the other strapless.
A short windmill with abrown wooden base standson the top of the hill amongflowers and trees.
Four pieces of shrimptempura are arranged withflowers and vegetables on awhite plate.
Windmill Tempura
Huangmei Opera
A huangmei opera actresswearing green and anotheractress wearing pink with acloak.
• Baseline Probability Fixationo Avoid overfitting to novel concepts
o Do not disturb previously learned concepts
o Only need a few samples
Before NVCS Training
a window with awindow
a man in blue shirt is riding a horse
After NVCS Training
a cat standing in front of a window
a t-rex is standing in a room with a man
Example results:
Novel Concepts: Cat T-tex
Transposed Weight Sharing
Transposed Weight Sharing
Decompose UM
…
Transposed Weight Sharing
Discussion: Word Embeddings
Will learned word embedding benefit from the transposed weight sharing strategy?
…
Leaning Multimodal Word Embeddings
Mao, J., Xu, J., Jing, Y., & Yuille, A. Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images. In Proc. NIPS 2016.
The Pinterest40M training dataset
White and gold ornate library with decorated ceiling, iron-work balcony, crystal chandelier, and glass-covered shelves. (I don't know if you're allowed to read a beat-up paperback in this room.)
This strawberry limeade cake is fruity, refreshing, and gorgeous! Those lovely layers are impossible to resist.
Make two small fishtail braids on each side, then put them together with a ponytail.
This is the place I will be going (hopefully) on my first date with Prince Stephen. It's the palace gardens, and they are gorgeous. I cannot wait to get to know him and exchange photography ideas!
This flopsy-wopsywho just wants a break from his walk. | 18 German Shepherd Puppies Who Need To Be Snuggled Immediately
The Pinterest40M training dataset
The Pinterest related phrase testing datasetUser Query
hair styles
Most Clicked Items
Annotations:
hair tutorial
prom night ideas
long hair
beautiful
…
Annotations:
pony tails
unique hairstyles
hair tutorial
picture inspiration
…
…
Positive Phrases for the Query
hair tutorialpony tails
makeup…
inspirationideas
Remove overlapping words
pony tails
makeup…
Sorted by Click Final List
Related Phrase 10M dataset (RP10M)
Crowdsourcing cleaned up
Gold Related Phrase 10K dataset (Gold RP10K)
The Multimodal Word Embedding Models
Image CNNOne Hot Embedding GRU FC SoftMaxwt wt+1
Uw UM
128 512 128
Sample 1024 negatives
Model B: Supervision on the final GRU state
Model C: Supervision on the embeddings
Baselines (No transposed weight sharing):
Model A
Experiments
Experiments
Content
• The m-RNN image captioning model Image caption generation
A dark brown horse with a white stripe wearing a black studded harness.
A dark brown horse with a white stripe wearing a black studded harness.
A white horse carrying a man.
A white horse carrying a man.
A dark horse carrying a woman
A dark horse carrying a woman
A woman on the dark horse.
A woman on the dark horse.
A cat laying on the left.A black cat laying on the right.
Our Full Model
A cat laying on a bed.A black and white cat.
Baseline
Speaker Results
Experiments: 4% improvement for precision@1
Baseline Our full model Improvement
Listener Task 40.6 44.6 4.0
End-to-End(Speaker & Listener)
48.5 51.3 2.8
Human Evaluation(Speaker Task) 15.9 20.4 4.5
Take Home Message
To be a better communicator, need
to be a better listener!!
Content
• The m-RNN image captioning model Image caption generation
Image retrieval (given query sentence)
Sentence retrieval (given query image)
• Extensions Incremental novel concept captioning
Multimodal word embedding learning
Referring expressions
Visual Question Answering
Visual Question Answering
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering. In Proc. NIPS 2015.
Image
Question
Answer
公共汽车是什么颜色的?What is the color of the bus?
公共汽车是红色的。The bus is red.
草地上除了人以外还有什么动物?What is there on the grass, except the person?
羊。Sheep.
观察一下说出食物里任意一种蔬菜的名字 ?Please look carefully and tell me what is the name of the vegetables in the plate?
西兰花 。Broccoli.
猫咪在哪里?Where is the kitty?
在椅子上 。On the chair.
黄色的是什么?What is there in yellow?
香蕉。Bananas.
The results of the mQA model on FM-IQA dataset.
Visual Question Answering
Discussion: Nearest Image SearchEmbedding I Embedding II Recurrent Multimodal SoftMaxLayers: