Top Banner
AI & Smart Tech: Words, words, words Confidential and proprietary. Any use of this material without specific permission of Rasa is strictly prohibited.
49

Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

May 30, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

AI & Smart Tech:Words, words, words

Confidential and proprietary.Any use of this material without specific permission of Rasa is strictly prohibited.

Page 2: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Thanks for having me!

Tom MetcalfeMachine learning engineer at [email protected]

@MetcalfeTom on Github

Page 3: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Rasa is the leading open source machine learning toolkit for developers to extend bots beyond answering simple questions

RASA STACK - OPEN SOURCE COMMUNITY

Downloads

100k+ 1,000+

Community Support

100+

Contributors

Page 4: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Structure of this talkIntroduction

I. Why words?II. Teaching words

III. Teaching meaningIV. Research from RasaV. Where will words go?

Page 5: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Why words?

How and why do we use words as data?

Page 6: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Machine learning is all about informationWhy words?

We have a wealth of information to gather from words and literature

Google’s Talk to Books

Whether text or speech, people use words to communicate, so good word representation facilitates AI’s integration

Page 7: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

This is not an easy task

Page 8: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Words are nonsenseWhy words?

“Uksrukuaktak”{Iñupiat Eskimo dictionary}

Page 9: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Vocabularies are vectorsWhy words?

Banana : [1 0 0 ... 0 0 0]

Bubblegum: [0 1 0 ... 0 0 0]

Caramel : [0 0 1 ... 0 0 0]

Vanilla : [0 0 0 ... 0 0 1]Each word is a feature

Page 10: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Using word vectorsWhy words?

[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0]

“Which flavour ice cream?”

Page 11: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Why words?Why words?

● How do words even work anyway?

They are the smallest element of language that can be uttered in isolation with objective or practical meaning.

● Why use them?

They are convenient representations which we use to externalise our thoughts. There is a wealth of knowledge to be acquired from literature

● How do we use words as data?

Separate at whitespace and assign each word a linearly-independent vector

Page 12: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Teaching words

Why word vectors are bad

Page 13: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

One-hot vectors are badTeaching words

Rome : [1 0 0]

Sofia : [0 1 0]

Dog : [0 0 1]Rome

Sofia

Dog

Complete the sentence: “A holiday in ___”

Page 14: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

One-hot vectors are badTeaching words

Rome : [1 0 0 0]

Sofia : [0 1 0 0]

London : [0 0 0 1]

Dog : [0 0 1 0]

Language Words in the Dictionary

Korean 1,100,373

Japanese 500,000

Italian 260,000

English 171,476

Page 15: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Cosine similarityTeaching words

sim(Rome, Sofia) = 0

sim(A, B) = cos(θ) =A · B

||A|| ||B||

A

Page 16: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Learning word embeddingsTeaching words

“Bacon and ___ with toast”x1

x2

x3

x4

Embedding layer

egg?

[0 0.05 0.8 0.1 0 0.05]

Page 17: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Using learned embeddingsTeaching words

Embedding layer

Embedding layerMan : [0 0 ... 1 0]

King: [0 1 ... 0 0]

E(Man) : [0.1 0.67 ... 0.2 0]

E(King): [0.2 0.59 ... 0.1 0]

sim(E(Man), E(King)) = 0.7

Page 18: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Properties of word embeddingsTeaching words

Embedding space captures relations

Page 19: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

OK, but does this make things better?

Page 20: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Intent classification with Rasa NLU

Goal: classify user messages to a list of “intents”

I’m hungry for sushi

Teaching Words

greetgoodbyethank_yourestaurant_searchbook_tableconfirm

Example intent list Example user message

Page 21: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Intent Classification with word vectors

I’m hungry for sushi

Bag of word

embeddingsSVM/RNN/

LinReg

greetgoodbyethank_yourestaurant_searchbook_tableconfirm

Embedded words sentence representation:

Teaching Words

This works embarrassingly well, and generalises from little data

Page 22: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Teaching words

● Why one-hot vectors are bad

Being sparse causes slow learning, related words share no association

● What are word embeddings?

Word embeddings are a mapping from one-hots to a multi-dimensional space where synonymous words occupy similar regions

● What do word embeddings do?

Capture analogous information, increase task accuracy with low amounts of data

Page 23: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Teaching Meaning

Why word embeddings are bad

Page 24: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Word embeddings are bad

“No I don’t want sushi”

“I’d go hungry before eating sushi”

Teaching Meaning

Page 25: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Sentence EmbeddingsTeaching Meaning

Universal Sentence Encoder

Infersent encoder architecture{Facebook Research, 2017}

Universal Sentence Encoder{Google Research, 2018}

Page 26: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

How good are sentence embeddings?Teaching Meaning

Universal Sentence Encoder{Google Research, 2018}

They don’t fall into the saliency trap

They capture meaning rather than word similarity

Page 27: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

How good are sentence embeddings?Teaching Meaning

Infersent embedding tasks{Facebook Research, 2017}

They have interesting task specialisation

They can capture information on a larger scope than word embeddings alone

Page 28: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Teaching meaning

● Why word embeddings are bad

Don’t capture information from sentence structure like negation. Spelling errors and out-of-vocabulary words still cause problems

● How do we embed whole sentences?

Google use a deep averaging network, Facebook AI use multi-level representations, Skipthought takes the hidden representation of a whole sentence

● What do sentence embeddings do?

Allow for sentiments to be captured rather than synonymous relevance

Page 29: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Time for a break 😅

Page 30: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Research from Rasa

Embedding sentences for FAQ detection

Page 31: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

FAQ detection: hypothesisResearch from Rasa

Can we use sentence embeddings to classify duplicate questions?

Question 1 Question 2 is_duplicate

How has your life changed since you became HIV positive?

How did your life change after you were diagnosed with HIV positive?

1

What is the defence mechanism in plants? Is there any immune system in plants? 1

How long does WhatsApp keep the delivered data like images or etc. in the

server?

How do I share GIF images in WhatsApp without converting the GIF into a video?

0

Within the context of dispute resolution, what is natural justice?

You earn a living to live well not to kill yourself with stress. What thoughts went through your mind after reading this sentence?

0

QUORA duplicate questions dataset

Page 32: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

FAQ detection: methodResearch from Rasa

1. Embed the questions using Google’s pre-trained Universal Sentence Encoder

2. Measure cosine similarity between duplicate and non-duplicate questions

3. Determine if classification is possible

Page 33: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

FAQ detection: resultsResearch from Rasa

Similarity between word embeddings Similarity between sentence embeddings

Page 34: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

If you try ‘em out, let us know

Page 35: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Scalable minds

Page 36: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Rasa is open source

Page 37: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Where will words go?

...are words bad?

Page 38: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

UksrukuaktakWhere will words go?

Page 39: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

The Eskimo libraryWhere will words go?

Page 40: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

The Eskimo libraryWhere will words go?

Page 41: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Grounding words in realityWhere will words go?

“Go grab my lunch”

Page 42: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Untapped languagesWhere will words go?

Deception Detection in Videos{University of Maryland, 2018}

DensePose: Dense Human Pose Estimation in the Wild{Facebook, 2018}

Page 43: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Words are beautiful

Page 44: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Words are beautifulWhere will words go?

Page 45: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Words are disastrous

Page 46: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Words are disastrousWhere will words go?

Aphasia

Dyslexia

Speech impediments

Language barriers

Misunderstandings

Page 47: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Text, type, talkWhere will words go?

Page 48: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Words are emissaries

Page 49: Words, words, words AI & Smart Tech - 2018.java2days.com · Teaching words III. Teaching meaning IV. Research from Rasa V. Where will words go? Why words? How and why do we use words

Thanks for listening to my words!

Tom MetcalfeMachine Learning [email protected]

Rasa Technologies GmbH, Schönhauser Allee 175, 10119 Berlin, Germany