Top Banner
Richard Zemel Learning Generative Models of Sentences and Images
40

Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

May 26, 2018

Download

Documents

lamnhu
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Richard Zemel

Learning Generative Models

of Sentences and Images

Page 2: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Learning Generative Models of Sentences and Images

Richard ZemelJuly 8, 2015

Microsoft Faculty Summit

Page 3: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Current successes of deep networks: classification problems (object recognition, speech recognition)

Standard supervised learning scenario with single correct response (class) for given input example

Building Strong Models

Page 4: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Current Models are Brittle

Szegedy, et al., ICLR, 2014

Page 5: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Current successes of deep networks: classification problems (object recognition, speech recognition)

Key aim: learn high quality generic representations, of images and text

Devise new objectives, based on image/text statistics, co-occurrence

Building Strong Models

Page 6: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

When input consists of pairs (sets) of items, a sensible objective is to predict one item from the other

Standard setup:

encoder maps first input to a vector

decoder maps vector to second input

Example: each word predicts the two words before and two words after it in a sentence (skip-gram [Mikolov et al., 2013])

Objective 1: Predict Context

Page 7: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Skip-Thought Vectors

Abstract the encoder-decoder model to whole sentences

Decode by predicting next word given generated words

[Kiros et al, 2015]

Page 8: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Skip-Thought Vectors

Train on sentence triplets extracted from books

Demonstrate utility on 5 different NLP tasks (semantic relatedness, paraphrase detection)

[Kiros et al, 2015]

Page 9: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Image Captioning as Context Prediction

A castle and

reflecting water

A ship

sailing

in the ocean

Joint

space

Minimize the following

objective:

images

text [Kiros et al, 2014]

Page 10: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Objective 2: Learning Generative Models

Another objective is to construct a model that can generate realistic inputs – ideally generalize beyond the training set

Difficult to formulate: cannot just directly match the training examples (over-fitting)

Page 11: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Learning Adversarial Models

One recently popular option: train model to fool adversary

The adversary attempts to discriminate samples from the model from data samples

[MacKay 1995, 1996; Magdon-Ismail and Atiya, 1998; Goodfellow et al. Generative Adversarial Nets. 2014]

Problem: min-max formulation makes optimization difficult

Page 12: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Make model codes close to data codes

Generative Moment Matching Networks

h

Samples Data

MMD

Uniform Prior

[Li, Swersky, Zemel, 2015]

Page 13: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

• Suppose we have access to samples from two probability distributions X ~ PA and Y ~ PB, how can we tell if PA = PB?

• Maximum Mean Discrepancy (MMD) is a measure of distance between two distributions given only samples from each. [Gretton2010]

• Our idea: learn to make two distributions indistinguishable small MMD!

MMD

Page 14: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Direct backpropagation through MMD, no adversary required!

Generative Moment Matching Networks

h

Uniform Prior

MMD

Samples

GM

MN

Dec

od

er

Data

Enco

der

Page 15: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

GMNN: Experiments

Page 16: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Independent Samples

Generated faces vs. Nearest Neighbors in Training Set

GMNN: Generalizing?

Page 17: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Interpolating between 5 random points (highlighted in red)

Exploring Latent Space

Page 18: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Interpolating between 5 random points (highlighted in red)

Exploring Latent Space

Page 19: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Objective 3: Learning One:Many Problems

Interesting tasks are often inherently ambiguous:

Segmenting image into coherent regions: What level of granularity?

Page 20: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Objective 3: Learning One:Many ProblemsInteresting tasks are often inherently ambiguous:

Segmenting image into coherent regions: What level of granularity?Generating caption for an image: What is relevant content?

Can think of problem as one of diversity – what is the appropriate level of diversity in one many mapping?

Luckily, data becoming available – can learn appropriate level of diversity for given input

A car on a beach with a boat in the background . generat

e Two hilly islands in the water.

Page 21: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Conditional Generative Moment Matching Networks

Include input as bias during generation

• Makes generation image-dependent

• Apply MMD on model/data samples per input

Idea: Generate outputs whose statistics match the statistics of multiple outputs for given input

[Li, et al , 2015]

Page 22: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

CGMMN: Image CaptioningTrain joint embedding on Flickr8K dataset:

– 8000 images, 5 captions each

– 6000 training, 1000 each validate/test

– Images & sentences encoded in sentence space (skip-thought vectors)

Projected down to 300 dimensional space

– CGMMN: 10-256-256-1024-300

– Minimize multiple kernel MMD loss

Aim: capture multi-modal distribution in code space

Page 23: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

CGMMN: Image Captioning

Page 24: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

CGMMN: Image Captioning

Page 25: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

CGMMN: Image Captioning

Page 26: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

CGMMN: Image Captioning

Page 27: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Conclusions & Open ProblemsClaim: Strong representation is not only capable of recognizing objects and words, but can model inputs themselves and their context

Developed 3 objectives for learning strong representations: Predict context (sentence-sentence; sentence-image)Generate distribution of imagesGenerate distribution of sentences, specific to an image

Leverage generative models?Gain insight into model behaviorImprove standard classification tasks, esp. when labels scarceGeneralize beyond static images to video

Page 28: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Generating Captions with Attention

[Xu et al, 2015]

Page 29: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Thanks!

Page 30: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Extra Slides

Page 31: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

CGMMN: Image Segmentation• Ambiguity in segmentation important

– Different attentional foci, granularity– Valuable in interactive setting: user can choose from candidate

segmentations to suit need

• Formulate problem: generate edge maps– CGMMN produces distribution over edge maps, sample to get different

maps– Post-processing system constructs region hierarchy, threshold to form

output regions

• Compare to strong baselines that produce single edge map – sample to get diverse map, sampling distribution optimized– apply same post-processing

Page 32: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

CGMMN: Image Segmentation

Image Ground-Truth Segmentations

Page 33: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

CGMMN: Image Segmentation

CGMMN gPb Boykov-Jolly

Page 34: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

CGMMN: Image Segmentation

Page 35: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Image Q&ADeveloped new Question/Answer dataset:

– Based on descriptions in COCO (COCO-QA)

– Use parse tree to make coherent questions

– Single word answers

– ~80K Q&A pairs (Object, Number, Color, Location)

• Developed a variety of models, baselines

[Ren, Kiros, Zemel, 2015]

Page 36: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Image Q&A

Page 37: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Image Q&A

Page 38: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Multiple Ground Truths: Caption Generation

Page 39: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Multiple Ground Truths: Image Segmentation

Page 40: Learning Generative Models of Sentences and Images Generative Models of Sentences and Images Learning Generative Models of Sentences and Images Richard Zemel July 8, 2015 Microsoft

Multiple Ground Truths: Image Segmentation


Related Documents
Learning Deep Generative Models - MIT

Learning Deep Generative Models - MIT

Category: Documents
Generative Models (other than GANs)alinlab.kaist.ac.kr/resource/AI602_Lec09_Explicit_generative_models.pdf · Generative Models (other than GANs) Algorithmic Intelligence Lab 1. Introduction

Generative Models (other than...

Category: Documents
Generative Models for Crowdsourced Data

Generative Models for Crowdsourced Data

Category: Documents
Generative Models Utilized for Superior Design Developmentssi.wdfiles.com/local--files/about-us/110415 submit.pdf · Generative Models Utilized for Superior Design ... Generative

Generative Models Utilized for Superior Design...

Category: Documents
Introduction to generative models - Mihaela Roscaelarosca.net/intro_generative_models.pdf · Introduction to generative models— Mihaela Rosca Generate new dataWhat do intelligent

Introduction to generative models - Mihaela...

Category: Documents
Machine Learning: Generative and Discriminative Models

Machine Learning: Generative and Discriminative Models

Category: Documents
Exploring generative models with Python - Not a Numberjjorge.es/.../05/exploring-generative-models-pycon2018.pdf · 2018. 5. 13. · Exploring generative models with Python Javier

Exploring generative models with Python - Not a...

Category: Documents
Deep Generative Models · BOWMAN, Samuel R., et al. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015. How to formulate a generative model? Modeling

Deep Generative Models · BOWMAN, Samuel R., et al....

Category: Documents
Models of Generative Grammar Smriti Singh. Generative Grammar A Generative Grammar is a set of formal rules that can generate an infinite set of sentences.

Models of Generative Grammar Smriti Singh. Generative...

Category: Documents
generative models - arXiv

generative models - arXiv

Category: Documents
Generative Models

Generative Models

Category: Documents
Generative Topic Models for Community Analysis

Generative Topic Models for Community Analysis

Category: Documents