What is the Jeopardy Model? A Quasi-Synchronous Grammar for Question Answering Mengqiu Wang, Noah A. Smith and Teruko Mitamura Language Technology Institute.

What is the Jeopardy Model?A Quasi-Synchronous Grammar

for Question Answering

Mengqiu Wang, Noah A. Smith and Teruko Mitamura

Language Technology InstituteCarnegie Mellon University

2

The task

High-efficiency document retrieval

High-precision answer ranking

Who is the leader of France?

1. Bush later met with French president Jacques Chirac. 2. Henri Hadjenberg, who is the leader of France ’s Jewish community, …3. …

1. Henri Hadjenberg, who is the leader of France ’s Jewish community, …2. Bush later met with French president Jacques Chirac. (as of May 16 2007)

3. …

3

Challenges

High-efficiency document retrieval

High-precision answer ranking

Who is the leader of France?

1. Bush later met with French president Jacques Chirac.

2. Henri Hadjenberg, who is the leader of France ’s Jewish community, …3. …

4

Semantic Tranformations

Q:“Who is the leader of France?”

A: Bush later met with French president Jacques Chirac.

5

Syntactic Transformations

Who leaderthe Franceofis ?

Bush met Frenchwith president Jacques Chirac

mod mod

mod

6

Syntactic Variations

Who leaderthe Franceofis ?

Henri Hadjenberb , who leaderis the of France ’s Jewish community

mod mod

mod

mod

7

Two key phenomena in QA

Semantic transformation leader president

Syntactic transformation leader of France French president

Q A)|( QAP

8

Existing work in QA

Semantics Use WordNet as thesaurus for expansion

Syntax Use dependency parse trees, but merely

transform the feature space into dependency parse feature space. No fundamental changes in the algorithms (edit-distance, classifier, similarity measure).

9

Where else have we seen these transformations?

Machine Translation (especially in syntax-based MT)

Paraphrasing Sentence compression Textual entailment

F E)|( FEP

10

Noisy-channel

Machine Translation

Question Answering

S E)()|()|( EPEFPFEP

Q A)()|()|( APAQPQAP

Language modelTranslation model

retrieval modelJeopardy model

11

From wikipedia.org: Jeopardy! is a popular international television

quiz game show (#2 of the 50 Greatest Game Show of All

Times). 3 contestants select clues in the form of an

answer, to which they must supply correct responses in the form of a question.

The concept of "questioning answers" is original to Jeopardy!.

What is Jeopardy! ?

)|( AQP

12

Jeopardy Model

We make use of a formalism called quasi-synchronous grammar [D. Smith

& Eisner ’06], originally developed for MT

13

Quasi-Synchronous Grammars Based on key observations in MT:

translated sentences often have some isomorphic syntactic structure, but not usually in entirety.

the strictness of the isomorphism may vary across words or syntactic rules.

Key idea: Unlike some synchronous grammars (e.g. SCFG,

which is more strict and rigid), QG defines a monolingual grammar for the target tree, “inspired” by the source tree.

14

Quasi-Synchronous Grammars In other words, we model the generation of

the target tree, influenced by the source tree (and their alignment)

QA can be thought of as extremely free translation within the same language.

The linkage between question and answer trees in QA is looser than in MT, which gives a bigger edge to QG.

15

Jeopardy Model Works on labeled dependency parse trees Learn the hidden structure (alignment between Q and

A trees) by summing out ALL possible alignments

One particular alignment tells us both the syntactic configurations and the word-to-word semantic correspondences

An example…

question answer

answerparse tree

questionparse tree

an alignment

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

theDT

FranceNNP

location

Q: A:$

root$

root

root

subj obj

det of

root

subj with

nmod

nmod

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

theDT

FranceNNP

location

Q: A:$

root$

root

root

subj obj

det of

root

subj with

nmod

nmod

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

isVB

Q: A:$

root$

root

root root

subj with

nmod

nmod

root)|P(root

noNE)|P(noNE

VBD)| P(VB

Our model makes local Markov assumptions to allow efficient computation via Dynamic Programming (details in paper)

given its parent, a word is independent of all other words (including siblings).

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

isVB

Q: A:$

root$

root

root

subj

root

subj with

nmod

nmod

child)-parent|P(subj

person)|P(qword

NNP)|P(WP

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

Q: A:$

root$

root

root

subj obj

root

subj with

nmod

nmod

child)-tgrandparen|P(obj

noNE)|P(noNE

NN)|P(NN

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

theDT

Q: A:$

root$

root

root

subj obj

det

root

subj with

nmod

nmod

)word-same|P(det

noNE)|P(noNE

N)|P(DT

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

theDT

FranceNNP

location

Q: A:$

root$

root

root

subj obj

det of

root

subj with

nmod

nmod

)child-parent|P(of

location)|P(location

JJ)|P(NNP

23

6 types of syntactic configurations

Parent-child

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

theDT

FranceNNP

location

Q: A:$

root$

root

root

subj obj

det of

root

subj with

nmod

nmod

Parent-child configuration

26

6 types of syntactic configurations

Parent-child Same-word

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

theDT

FranceNNP

location

Q: A:$

root$

root

root

subj obj

det of

root

subj with

nmod

nmod

Same-word configuration

Parent-child configuration

29

6 types of syntactic configurations

Parent-child Same-word Grandparent-child

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

theDT

FranceNNP

location

Q: A:$

root$

root

root

subj obj

det of

root

subj with

nmod

nmod

Parent-child configuration Same-word configuration

Grandparent-child configuration

32

6 types of syntactic configurations

Parent-child Same-word Grandparent-child Child-parent Siblings C-command(Same as [D. Smith & Eisner ’06])

Parent-child configuration Same-word configuration Grandparent-child configuration

Child-parent configuration Siblings configuration C-command configuration

34

Modeling alignment Base model

)child-parent|P(of

location)|P(location

N)|P(N

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

theDT

FranceNNP

location

Q: A:$

root$

root

root

subj obj

det of

root

subj with

nmod

nmod

BushNNP

person

metVBD

FrenchJJ

location

presidentNN

Jacques ChiracNNP

person

whoWP

qword

leaderNN

isVB

theDT

FranceNNP

location

Q: A:$

root$

root

root

subj obj

det of

root

subj with

nmod

nmod

37

Modeling alignment cont.

Base model

Log-linear modelLexical-semantic features from WordNet,Identity, hypernym, synonym, entailment, etc.

Mixture model

38

Parameter estimation

Things to be learnt Multinomial distributions in base model Log-linear model feature weights Mixture coefficient

Training involves summing out hidden structures, thus non-convex.

Solved using conditional Expectation-Maximization

39

Experiments

Trec8-12 data set for training Trec13 questions for development

and testing

40

Candidate answer generation

For each question, we take all documents from the TREC doc pool, and extract sentences that contain at least one non-stop keywords from the question.

For computational reasons (parsing speed, etc.), we only took answer sentences <= 40 words.

41

Dataset statistics Manually labeled 100 questions for training

Total: 348 positive Q/A pairs 84 questions for dev

Total: 1415 Q/A pairs 3.1+, 17.1-

100 questions for testing Total: 1703 Q/A pairs 3.6+, 20.0-

Automatically labeled another 2193 questions to create a noisy training set, for evaluating model robustness

42

Experiments cont.

Each question and answer sentence is tokenized, POS tagged (MX-POST), parsed (MSTParser) and labeled with named-entity tags (Identifinder)

43

Baseline systems (replications) [Cui et al. SIGIR ‘05]

The algorithm behind one of the best performing systems in TREC evaluations.

It uses a mutual information-inspired score computed over dependency trees and a single fixed alignment between them.

[Punyakanok et al. NLE ’04] measures the similarity between Q and A by

computing tree edit distance. Both baselines are high-performing, syntax-based,

and most straight-forward to replicate We further enhanced the algorithms by augmenting

them with WordNet.

44

ResultsMean Average

PrecisionMean Reciprocal

Rank of Top 1

Statistically significantly better than the 2nd best score in each column

28.2% 23.9% 41.2% 30.3%

45

Summing vs. Max

46

Conclusion We developed a probabilistic model for QA

based on quasi-synchronous grammar Experimental results showed that our model

is more accurate and robust than state-of-the-art syntax-based QA models

The mixture model is shown to be powerful. The log-linear model allows us to use arbitrary features.

Provides a general framework for many other NLP applications (compression, textual entailment, paraphrasing, etc.)

47

Future Work Higher-order Markovization, both horizontally

and vertically, allows us to look at more context, at the expense of higher computational cost.

More features from external resources, e.g. paraphrasing database

Extending it for Cross-lingual QA Avoid the paradigm of translation as pre- of post-

processing We can naturally fit in a lexical or phrase

translation probability table into our model to model the translation inherently

Taking into account parsing uncertainty

48

Thank you!

Questions?