Top Banner
A Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC San Diego Department of Linguistics COGS 1 guest lecture February 2, 2010
82

A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

May 08, 2018

Download

Documents

vukhue
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

A Brief and Friendly Introduction toComputational Psycholinguistics

Roger Levy

UC San DiegoDepartment of Linguistics

COGS 1 guest lectureFebruary 2, 2010

Page 2: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

What is “computational psycholinguistics”?

◮ Inherently, linguistic communication involves the resolutionof uncertainty over a potentially unbounded set of possiblesignals and meanings.

◮ How can a fixed set of knowledge and resources bedeployed to manage this uncertainty?

This is the study of language processing.

◮ And how cansuch knowledge and resources be learned from finite input?

This is the study of language acquisition.

Computational psycholinguistics studies these problems byconstructing explicit mathematical models and testing themwith experiments.

Page 3: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

What is “computational psycholinguistics”?

◮ Inherently, linguistic communication involves the resolutionof uncertainty over a potentially unbounded set of possiblesignals and meanings.

◮ How can a fixed set of knowledge and resources bedeployed to manage this uncertainty?

This is the study of language processing.

◮ And how cansuch knowledge and resources be learned from finite input?

This is the study of language acquisition.

Computational psycholinguistics studies these problems byconstructing explicit mathematical models and testing themwith experiments.

Page 4: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

What is “computational psycholinguistics”?

◮ Inherently, linguistic communication involves the resolutionof uncertainty over a potentially unbounded set of possiblesignals and meanings.

◮ How can a fixed set of knowledge and resources bedeployed to manage this uncertainty?

This is the study of language processing.

◮ And how cansuch knowledge and resources be learned from finite input?

This is the study of language acquisition.

Computational psycholinguistics studies these problems byconstructing explicit mathematical models and testing themwith experiments.

Page 5: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

What is “computational psycholinguistics”?

◮ Inherently, linguistic communication involves the resolutionof uncertainty over a potentially unbounded set of possiblesignals and meanings.

◮ How can a fixed set of knowledge and resources bedeployed to manage this uncertainty?

This is the study of language processing.

◮ And how cansuch knowledge and resources be learned from finite input?

This is the study of language acquisition.

Computational psycholinguistics studies these problems byconstructing explicit mathematical models and testing themwith experiments.

Page 6: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

What is “computational psycholinguistics”?

◮ Inherently, linguistic communication involves the resolutionof uncertainty over a potentially unbounded set of possiblesignals and meanings.

◮ How can a fixed set of knowledge and resources bedeployed to manage this uncertainty?

This is the study of language processing.

◮ And how cansuch knowledge and resources be learned from finite input?

This is the study of language acquisition.

Computational psycholinguistics studies these problems byconstructing explicit mathematical models and testing themwith experiments.

Page 7: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

What is “computational psycholinguistics”?

◮ Inherently, linguistic communication involves the resolutionof uncertainty over a potentially unbounded set of possiblesignals and meanings.

◮ How can a fixed set of knowledge and resources bedeployed to manage this uncertainty?

This is the study of language processing.

◮ And how cansuch knowledge and resources be learned from finite input?

This is the study of language acquisition.

Computational psycholinguistics studies these problems byconstructing explicit mathematical models and testing themwith experiments.

Page 8: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

What is “language processing”?

◮ Language processing is the study of how humanscomprehend and produce language (sentences, wordswithin sentences, and sequences of sentences, etc.) inreal time.

◮ We can divide this into language comprehension(understanding what is spoken and what is written) andlanguage production (choosing what to say or write basedon what you want to “mean”)

Page 9: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

What is “language processing”?

◮ Language processing is the study of how humanscomprehend and produce language (sentences, wordswithin sentences, and sequences of sentences, etc.) inreal time.

◮ We can divide this into language comprehension(understanding what is spoken and what is written) andlanguage production (choosing what to say or write basedon what you want to “mean”)

Page 10: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

What is “language acquisition”?

◮ Language acquisition is the study of how humans acquireknowledge of their native language (as infants and aschildren)

Page 11: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Today

In this talk I’ll focus on language comprehension, and thendiscuss a bit about language production.

Page 12: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Theoretical Desiderata

Realistic models of human sentence comprehension mustaccount for:

◮ Language has structure◮ Robustness to arbitrary input◮ Accurate disambiguation◮ Inference on basis of incomplete input (Tanenhaus et al.,

1995; Altmann and Kamide, 1999; Kaiser and Trueswell,2004)

◮ Processing difficulty is differential and localized

Page 13: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Language has structure

The colored word sequences all have something in common:

◮ The girl gave the dog a big sloppy kiss.◮ I gave the dog a big sloppy kiss.◮ Every boy on the left side of the room gave the dog a big

sloppy kiss.◮ The teacher of this class gave the dog a big sloppy kiss.

In linguistics, this commonality is that the colored wordsequences are all of the same phrase type.In this case, the phrase type is called a noun phrase.Languages have many different phrase types, and we candescribe the grammar of a languages in how its phrase typescome together.

Page 14: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Language has structure

The colored word sequences all have something in common:

◮ The girl gave the dog a big sloppy kiss.◮ I gave the dog a big sloppy kiss.◮ Every boy on the left side of the room gave the dog a big

sloppy kiss.◮ The teacher of this class gave the dog a big sloppy kiss.

In linguistics, this commonality is that the colored wordsequences are all of the same phrase type.In this case, the phrase type is called a noun phrase.Languages have many different phrase types, and we candescribe the grammar of a languages in how its phrase typescome together.

Page 15: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Language has structure

The colored word sequences all have something in common:

◮ The girl gave the dog a big sloppy kiss.◮ I gave the dog a big sloppy kiss.◮ Every boy on the left side of the room gave the dog a big

sloppy kiss.◮ The teacher of this class gave the dog a big sloppy kiss.

In linguistics, this commonality is that the colored wordsequences are all of the same phrase type.In this case, the phrase type is called a noun phrase.Languages have many different phrase types, and we candescribe the grammar of a languages in how its phrase typescome together.

Page 16: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Language has structure

The colored word sequences all have something in common:

◮ The girl gave the dog a big sloppy kiss.◮ I gave the dog a big sloppy kiss.◮ Every boy on the left side of the room gave the dog a big

sloppy kiss.◮ The teacher of this class gave the dog a big sloppy kiss.

In linguistics, this commonality is that the colored wordsequences are all of the same phrase type.In this case, the phrase type is called a noun phrase.Languages have many different phrase types, and we candescribe the grammar of a languages in how its phrase typescome together.

Page 17: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Language has structure

The colored word sequences all have something in common:

◮ The girl gave the dog a big sloppy kiss.◮ I gave the dog a big sloppy kiss.◮ Every boy on the left side of the room gave the dog a big

sloppy kiss.◮ The teacher of this class gave the dog a big sloppy kiss.

In linguistics, this commonality is that the colored wordsequences are all of the same phrase type.In this case, the phrase type is called a noun phrase.Languages have many different phrase types, and we candescribe the grammar of a languages in how its phrase typescome together.

Page 18: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Language has structure

The colored word sequences all have something in common:

◮ The girl gave the dog a big sloppy kiss.◮ I gave the dog a big sloppy kiss.◮ Every boy on the left side of the room gave the dog a big

sloppy kiss.◮ The teacher of this class gave the dog a big sloppy kiss.

In linguistics, this commonality is that the colored wordsequences are all of the same phrase type.In this case, the phrase type is called a noun phrase.Languages have many different phrase types, and we candescribe the grammar of a languages in how its phrase typescome together.

Page 19: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Language has structure

The colored word sequences all have something in common:

◮ The girl gave the dog a big sloppy kiss.◮ I gave the dog a big sloppy kiss.◮ Every boy on the left side of the room gave the dog a big

sloppy kiss.◮ The teacher of this class gave the dog a big sloppy kiss.

In linguistics, this commonality is that the colored wordsequences are all of the same phrase type.In this case, the phrase type is called a noun phrase.Languages have many different phrase types, and we candescribe the grammar of a languages in how its phrase typescome together.

Page 20: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Language has structure

The colored word sequences all have something in common:

◮ The girl gave the dog a big sloppy kiss.◮ I gave the dog a big sloppy kiss.◮ Every boy on the left side of the room gave the dog a big

sloppy kiss.◮ The teacher of this class gave the dog a big sloppy kiss.

In linguistics, this commonality is that the colored wordsequences are all of the same phrase type.In this case, the phrase type is called a noun phrase.Languages have many different phrase types, and we candescribe the grammar of a languages in how its phrase typescome together.

Page 21: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Robustness

Real linguistic input is not always totally well-formed. . .

I think when she finally came to the realization that,you know, no, I can not, I can not take care of myself.. . .I mean, for somebody who is, you know, for most oftheir life has, has, uh, not just merely had a farm buthad ten children had a farm, ran everything becauseher husband was away in the coal mines.And, you know, facing that situation, it’s, it’s quite adilemma.

. . . but usually we come to understand it pretty well anyway.

Page 22: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Robustness

Real linguistic input is not always totally well-formed. . .

I think when she finally came to the realization that,you know, no, I can not, I can not take care of myself.. . .I mean, for somebody who is, you know, for most oftheir life has, has, uh, not just merely had a farm buthad ten children had a farm, ran everything becauseher husband was away in the coal mines.And, you know, facing that situation, it’s, it’s quite adilemma.(The woman is facing being put in a resting home.)

. . . but usually we come to understand it pretty well anyway.

Page 23: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Robustness

Real linguistic input is not always totally well-formed. . .

I think when she finally came to the realization that,you know, no, I can not, I can not take care of myself.. . .I mean, for somebody who is, you know, for most oftheir life has, has, uh, not just merely had a farm buthad ten children had a farm, ran everything becauseher husband was away in the coal mines.And, you know, facing that situation, it’s, it’s quite adilemma.(The woman is facing being put in a resting home.)

. . . but usually we come to understand it pretty well anyway.

Page 24: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Accurate disambiguation

Most sentences are ambiguous in ways we do not even notice:

Mary forgot the pitcher. . .

Page 25: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Accurate disambiguation

Most sentences are ambiguous in ways we do not even notice:

Mary forgot the pitcher. . .

Page 26: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Accurate disambiguation

Most sentences are ambiguous in ways we do not even notice:

Mary forgot the pitcher of water sitting near the stove.

Page 27: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Accurate disambiguation

Most sentences are ambiguous in ways we do not even notice:

Mary forgot the pitcher of water sitting near the stove.

Page 28: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Accurate disambiguation

Most sentences are ambiguous in ways we do not even notice:

Mary forgot the pitcher of water sitting near the stove.

That’s probably not what you were thinking of...

Page 29: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Inference on the basis of incomplete input

Comprehenders do not wait until the whole sentence has beenheard to make inferences about what it means or will wind upmeaning:

(Altmann and Kamide, 1999)

Page 30: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Inference on the basis of incomplete input

Comprehenders do not wait until the whole sentence has beenheard to make inferences about what it means or will wind upmeaning:

(Altmann and Kamide, 1999)

Page 31: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Inference on the basis of incomplete input

Comprehenders do not wait until the whole sentence has beenheard to make inferences about what it means or will wind upmeaning:

“The boy will eat/move the cake. . . ”

(Altmann and Kamide, 1999)

Page 32: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Inference on the basis of incomplete input

Comprehenders do not wait until the whole sentence has beenheard to make inferences about what it means or will wind upmeaning:

“The boy will eat/move the cake. . . ”

(Altmann and Kamide, 1999)

Page 33: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Inference on the basis of incomplete input

Comprehenders do not wait until the whole sentence has beenheard to make inferences about what it means or will wind upmeaning:

“The boy will eat/move the cake. . . ”

That is, comprehension is incremental

(Altmann and Kamide, 1999)

Page 34: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Using multiple relative clauses in a sentence can makeprocessing difficult:

This is the malt that the rat that the cat that the dogworried killed ate.

It’s not the meaning of the sentence, or the use of relativeclauses, that makes it hard:

This is the malt that was eaten by the rat that waskilled by the cat that was worried by the dog.

Page 35: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Using multiple relative clauses in a sentence can makeprocessing difficult:

This is the malt that the rat that the cat that the dogworried killed ate.

It’s not the meaning of the sentence, or the use of relativeclauses, that makes it hard:

This is the malt that was eaten by the rat that waskilled by the cat that was worried by the dog.

Page 36: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Using multiple relative clauses in a sentence can makeprocessing difficult:

This is the malt that the rat that the cat that the dogworried killed ate.

It’s not the meaning of the sentence, or the use of relativeclauses, that makes it hard:

This is the malt that was eaten by the rat that waskilled by the cat that was worried by the dog.

Page 37: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Did you believe that this sentence was English?

This is the malt that the rat that the cat that the dogworried killed ate.

◮ Consider this simple example:

This is the cat that the dog worried.

◮ And this one:This is the rat that the cat killed.

◮ Which cat did the killing? Suppose it was the cat that thedog worried.

This is the rat that the cat that the dog worriedkilled.

Page 38: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Did you believe that this sentence was English?

This is the malt that the rat that the cat that the dogworried killed ate.

◮ Consider this simple example:

This is the cat that the dog worried.

◮ And this one:This is the rat that the cat killed.

◮ Which cat did the killing? Suppose it was the cat that thedog worried.

This is the rat that the cat that the dog worriedkilled.

Page 39: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Did you believe that this sentence was English?

This is the malt that the rat that the cat that the dogworried killed ate.

◮ Consider this simple example:

This is the cat that the dog worried.

◮ And this one:This is the rat that the cat killed.

◮ Which cat did the killing? Suppose it was the cat that thedog worried.

This is the rat that the cat that the dog worriedkilled.

Page 40: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Did you believe that this sentence was English?

This is the malt that the rat that the cat that the dogworried killed ate.

◮ Consider this simple example:

This is the cat that the dog worried.

◮ And this one:This is the rat that the cat killed.

◮ Which cat did the killing? Suppose it was the cat that thedog worried.

This is the rat that the cat that the dog worriedkilled.

Page 41: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Did you believe that this sentence was English?

This is the malt that the rat that the cat that the dogworried killed ate.

◮ Consider this simple example:

This is the cat that the dog worried.

◮ And this one:This is the rat that the cat killed.

◮ Which cat did the killing? Suppose it was the cat that thedog worried.

This is the rat that the cat that the dog worriedkilled.

Page 42: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Did you believe that this sentence was English?

This is the malt that the rat that the cat that the dogworried killed ate.

◮ Consider this simple example:

This is the cat that the dog worried.

◮ And this one:This is the rat that the cat killed.

◮ Which cat did the killing? Suppose it was the cat that thedog worried.

This is the rat that the cat that the dog worriedkilled.

Page 43: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Did you believe that this sentence was English?

This is the malt that the rat that the cat that the dogworried killed ate.

◮ Consider this simple example:

This is the cat that the dog worried.

◮ And this one:This is the rat that the cat killed.

◮ Which cat did the killing? Suppose it was the cat that thedog worried.

This is the rat that the cat that the dog worriedkilled.

Page 44: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is differential

Did you believe that this sentence was English?

This is the malt that the rat that the cat that the dogworried killed ate.

◮ Consider this simple example:

This is the cat that the dog worried.

◮ And this one:This is the rat that the cat killed.

◮ Which cat did the killing? Suppose it was the cat that thedog worried.

This is the rat that the cat that the dog worriedkilled.

Page 45: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is localized

[self-paced reading demo, Example1]

(Grodner and Gibson, 2005)

Page 46: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Processing difficulty is localized

[self-paced reading demo, Example1]

(Grodner and Gibson, 2005)

Page 47: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Try to guess the next word in the sentence

◮ Empirically, it’s been shown that more highly predictablewords are read more quickly (Ehrlich and Rayner, 1981)

◮ Why would this be the case?

Page 48: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Try to guess the next word in the sentence

My brother came inside to. . .

◮ Empirically, it’s been shown that more highly predictablewords are read more quickly (Ehrlich and Rayner, 1981)

◮ Why would this be the case?

Page 49: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Try to guess the next word in the sentence

My brother came inside to. . . chat? get warm? talk? eat? rest?

◮ Empirically, it’s been shown that more highly predictablewords are read more quickly (Ehrlich and Rayner, 1981)

◮ Why would this be the case?

Page 50: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Try to guess the next word in the sentence

My brother came inside to. . . chat? get warm? talk? eat? rest?The children went outside to. . .

◮ Empirically, it’s been shown that more highly predictablewords are read more quickly (Ehrlich and Rayner, 1981)

◮ Why would this be the case?

Page 51: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Try to guess the next word in the sentence

My brother came inside to. . . chat? get warm? talk? eat? rest?The children went outside to. . . play

◮ Empirically, it’s been shown that more highly predictablewords are read more quickly (Ehrlich and Rayner, 1981)

◮ Why would this be the case?

Page 52: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Try to guess the next word in the sentence

My brother came inside to. . . chat? get warm? talk? eat? rest?The children went outside to. . . play

◮ Empirically, it’s been shown that more highly predictablewords are read more quickly (Ehrlich and Rayner, 1981)

◮ Why would this be the case?

Page 53: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Try to guess the next word in the sentence

My brother came inside to. . . chat? get warm? talk? eat? rest?The children went outside to. . . play

◮ Empirically, it’s been shown that more highly predictablewords are read more quickly (Ehrlich and Rayner, 1981)

◮ Why would this be the case?

Page 54: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Describing the hierarchical structure of sentences

◮ Sentences are not just sequences of words◮ Some words are closely associated with other words into

PHRASES

◮ These phrases are in turn associated with other words orphrases to form larger phrases

◮ The sentence is the largest phrase◮ We use FORMAL GRAMMARS to describe these phrasal

arrangements◮ The formal grammatical description of a sentence gives us

considerable inroads into understanding its meaning

Page 55: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Describing the hierarchical structure of sentences

◮ Sentences are not just sequences of words◮ Some words are closely associated with other words into

PHRASES

◮ These phrases are in turn associated with other words orphrases to form larger phrases

◮ The sentence is the largest phrase◮ We use FORMAL GRAMMARS to describe these phrasal

arrangements◮ The formal grammatical description of a sentence gives us

considerable inroads into understanding its meaning

Page 56: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Describing the hierarchical structure of sentences

◮ Sentences are not just sequences of words◮ Some words are closely associated with other words into

PHRASES

◮ These phrases are in turn associated with other words orphrases to form larger phrases

◮ The sentence is the largest phrase◮ We use FORMAL GRAMMARS to describe these phrasal

arrangements◮ The formal grammatical description of a sentence gives us

considerable inroads into understanding its meaning

Page 57: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Describing the hierarchical structure of sentences

◮ Sentences are not just sequences of words◮ Some words are closely associated with other words into

PHRASES

◮ These phrases are in turn associated with other words orphrases to form larger phrases

◮ The sentence is the largest phrase◮ We use FORMAL GRAMMARS to describe these phrasal

arrangements◮ The formal grammatical description of a sentence gives us

considerable inroads into understanding its meaning

Page 58: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Describing the hierarchical structure of sentences

◮ Sentences are not just sequences of words◮ Some words are closely associated with other words into

PHRASES

◮ These phrases are in turn associated with other words orphrases to form larger phrases

◮ The sentence is the largest phrase◮ We use FORMAL GRAMMARS to describe these phrasal

arrangements◮ The formal grammatical description of a sentence gives us

considerable inroads into understanding its meaning

Page 59: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Describing the hierarchical structure of sentences

◮ Sentences are not just sequences of words◮ Some words are closely associated with other words into

PHRASES

◮ These phrases are in turn associated with other words orphrases to form larger phrases

◮ The sentence is the largest phrase◮ We use FORMAL GRAMMARS to describe these phrasal

arrangements◮ The formal grammatical description of a sentence gives us

considerable inroads into understanding its meaning

Page 60: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars

A context-free grammar (CFG) consists of a tuple (N, V , S, R)such that:

◮ N is a finite set of non-terminal symbols;◮ V is a finite set of terminal symbols;◮ S is the start symbol;◮ R is a finite set of rules of the form X → α where X ∈ N

and α is a sequence of symbols drawn from N ∪ V .

A CFG derivation is the recursive expansion of non-terminalsymbols in a string by rules in R, starting with S, and aderivation tree T is the history of those rule applications.

Page 61: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example

Let our grammar (the rule-set R) be

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

The nonterminal set N is {S, NP, VP, Det , N, P, V}, theterminal set V is {the, dog, cat , near , growled}, and our startsymbol S is S.

Page 62: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 63: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 64: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 65: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 66: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 67: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 68: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 69: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 70: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 71: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 72: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 73: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 74: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Context-free Grammars: an example II

S →NP VPNP→Det NNP→NP PPPP→P NPVP→V

Det→ theN → dogN → catP → nearV → growled

Here is a derivation and the resulting derivation tree:

S

NP

NP

Det

the

N

dog

PP

P

near

NP

Det

the

N

cat

VP

V

growled

Page 75: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Grammar and structural ambiguity◮ Most sentences are ambiguous in ways we don’t even

noticeThe a are of I (Abney, 1996)

◮ are can be a noun: “there are a hundred ares in a hectare”◮ a can be a descriptor (“the a students”)◮ I can be a descriptor that stands in as a full proper noun

◮ Some sentences are am-biguous in ways that we don’t notice without some reflection

I ate the cake with a spoon.

◮ Other sentences are ambiguous in ways that are prettyobvious

The son of the colonel who shot himself was dearlyloved.

◮ One goal of computational psycholinguistics is to give aprecise statement of how the alternative interpretations areconstructed and chosen between

Page 76: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Grammar and structural ambiguity◮ Most sentences are ambiguous in ways we don’t even

noticeThe a are of I (Abney, 1996)

◮ are can be a noun: “there are a hundred ares in a hectare”◮ a can be a descriptor (“the a students”)◮ I can be a descriptor that stands in as a full proper noun

◮ Some sentences are am-biguous in ways that we don’t notice without some reflection

I ate the cake with a spoon.

◮ Other sentences are ambiguous in ways that are prettyobvious

The son of the colonel who shot himself was dearlyloved.

◮ One goal of computational psycholinguistics is to give aprecise statement of how the alternative interpretations areconstructed and chosen between

Page 77: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Grammar and structural ambiguity◮ Most sentences are ambiguous in ways we don’t even

noticeThe a are of I (Abney, 1996)

◮ are can be a noun: “there are a hundred ares in a hectare”◮ a can be a descriptor (“the a students”)◮ I can be a descriptor that stands in as a full proper noun

◮ Some sentences are am-biguous in ways that we don’t notice without some reflection

I ate the cake with a spoon.

◮ Other sentences are ambiguous in ways that are prettyobvious

The son of the colonel who shot himself was dearlyloved.

◮ One goal of computational psycholinguistics is to give aprecise statement of how the alternative interpretations areconstructed and chosen between

Page 78: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Grammar and structural ambiguity◮ Most sentences are ambiguous in ways we don’t even

noticeThe a are of I (Abney, 1996)

◮ are can be a noun: “there are a hundred ares in a hectare”◮ a can be a descriptor (“the a students”)◮ I can be a descriptor that stands in as a full proper noun

◮ Some sentences are am-biguous in ways that we don’t notice without some reflection

I ate the cake with a spoon.

◮ Other sentences are ambiguous in ways that are prettyobvious

The son of the colonel who shot himself was dearlyloved.

◮ One goal of computational psycholinguistics is to give aprecise statement of how the alternative interpretations areconstructed and chosen between

Page 79: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Grammar and structural ambiguity◮ Most sentences are ambiguous in ways we don’t even

noticeThe a are of I (Abney, 1996)

◮ are can be a noun: “there are a hundred ares in a hectare”◮ a can be a descriptor (“the a students”)◮ I can be a descriptor that stands in as a full proper noun

◮ Some sentences are am-biguous in ways that we don’t notice without some reflection

I ate the cake with a spoon.

◮ Other sentences are ambiguous in ways that are prettyobvious

The son of the colonel who shot himself was dearlyloved.

◮ One goal of computational psycholinguistics is to give aprecise statement of how the alternative interpretations areconstructed and chosen between

Page 80: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Grammar and structural ambiguity◮ Most sentences are ambiguous in ways we don’t even

noticeThe a are of I (Abney, 1996)

◮ are can be a noun: “there are a hundred ares in a hectare”◮ a can be a descriptor (“the a students”)◮ I can be a descriptor that stands in as a full proper noun

◮ Some sentences are am-biguous in ways that we don’t notice without some reflection

I ate the cake with a spoon.

◮ Other sentences are ambiguous in ways that are prettyobvious

The son of the colonel who shot himself was dearlyloved.

◮ One goal of computational psycholinguistics is to give aprecise statement of how the alternative interpretations areconstructed and chosen between

Page 81: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

Grammar and structural ambiguity◮ Most sentences are ambiguous in ways we don’t even

noticeThe a are of I (Abney, 1996)

◮ are can be a noun: “there are a hundred ares in a hectare”◮ a can be a descriptor (“the a students”)◮ I can be a descriptor that stands in as a full proper noun

◮ Some sentences are am-biguous in ways that we don’t notice without some reflection

I ate the cake with a spoon.

◮ Other sentences are ambiguous in ways that are prettyobvious

The son of the colonel who shot himself was dearlyloved.

◮ One goal of computational psycholinguistics is to give aprecise statement of how the alternative interpretations areconstructed and chosen between

Page 82: A Brief and Friendly Introduction to Computational ...rik/courses/cogs1_w10/slides/levy_100202.pdfA Brief and Friendly Introduction to Computational Psycholinguistics Roger Levy UC

References I

Abney, S. (1996). Statistical methods and linguistics. In Klavans, J.and Resnik, P., editors, The Balancing Act: Combining Symbolicand Statistical Approaches to Language. Cambridge, MA: MITPress.

Altmann, G. T. and Kamide, Y. (1999). Incremental interpretation atverbs: restricting the domain of subsequent reference. Cognition,73(3):247–264.

Ehrlich, S. F. and Rayner, K. (1981). Contextual effects on wordperception and eye movements during reading. Journal of VerbalLearning and Verbal Behavior, 20:641–655.

Grodner, D. and Gibson, E. (2005). Some consequences of the serialnature of linguistic input. Cognitive Science, 29(2):261–290.

Kaiser, E. and Trueswell, J. C. (2004). The role of discourse contextin the processing of a flexible word-order language. Cognition,94:113–147.

Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K., and Sedivy,J. C. (1995). Integration of visual and linguistic information inspoken language comprehension. Science, 268:1632–1634.