Question Generation with Minimal Recursion Semantics Xuchen Yao European Masters in Language and Communication Technologies Supervisors: Prof. Hans Uszkoreit and Dr. Yi Zhang, Saarland University Co-supervisor: Dr. Gosse Bouma, University of Groningen 28 July 2010, Master Colloquium
50
Embed
QuestionGenerationwithMinimalRecursion Semanticscs.jhu.edu/~xuchen/paper/Yao2010Master.slides.pdfQuestionGenerationwithMinimalRecursion Semantics XuchenYao European Masters in Language
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Question Generation with Minimal RecursionSemantics
Xuchen Yao
European Masters in Language and Communication Technologies
Handleiding gebruik logo’s, versie 2.0, augustus 2007
Handleidinglogobestandenversie 2.0, augustus 2007
Supervisors: Prof. Hans Uszkoreit and Dr. Yi Zhang, Saarland UniversityCo-supervisor: Dr. Gosse Bouma, University of Groningen
28 July 2010, Master Colloquium
Introduction Background System Architecture Evaluation Conclusion
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
Introduction Background System Architecture Evaluation Conclusion
Question Generation (QG)
The task of generating reasonable questions from a text.Deep QG: why, why not, what-if, what-if-not, howShallow QG:.who, what, when, where, which, how many/much,yes/noJackson was born on August 29, 1958 in Gary, Indiana.
• Who was born on August 29 , 1958 in Gary , Indiana?• Which artist was born on August 29 , 1958 in Gary , Indiana?• Where was Jackson born?• When was Jackson born?• Was Jackson born on August 29 , 1958 in Gary , Indiana?
Introduction Background System Architecture Evaluation Conclusion
Question Generation (QG)
The task of generating reasonable questions from a text.Deep QG: why, why not, what-if, what-if-not, howShallow QG:.who, what, when, where, which, how many/much,yes/noJackson was born on August 29, 1958 in Gary, Indiana.
• Who was born on August 29 , 1958 in Gary , Indiana?• Which artist was born on August 29 , 1958 in Gary , Indiana?• Where was Jackson born?• When was Jackson born?• Was Jackson born on August 29 , 1958 in Gary , Indiana?
Introduction Background System Architecture Evaluation Conclusion
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
Introduction Background System Architecture Evaluation Conclusion
Approaches
• Template-based• What did <character> <verb>?
• Syntax-based• John plays football. (S V O)• John plays what? (S V WHNP)• John does play what? (S Aux-V V WHNP)• Does John play what? (Aux-V S V WHNP)• What does John play? (WHNP Aux-V S V)
Introduction Background System Architecture Evaluation Conclusion
Approaches
• Template-based• What did <character> <verb>?
• Syntax-based• John plays football. (S V O)• John plays what? (S V WHNP)• John does play what? (S Aux-V V WHNP)• Does John play what? (Aux-V S V WHNP)• What does John play? (WHNP Aux-V S V)
(TOOL)LKB: Linguistic Knowledge Buildera grammar development environment for grammars in typedfeature structures and unification-based formalisms.(TOOL)PET: a platform for experimentation with efficientHPSG processing techniquesa two-stage parsing model with HPSG rules and PCFG models,balancing between precise linguistic interpretation and robustprobabilistic coverage.
Introduction Background System Architecture Evaluation Conclusion
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
Introduction Background System Architecture Evaluation Conclusion
Why?Question Transformation does not Work Well without Sentence Simplification
Input Sentence:ASC takes a character as input, and returns theinteger giving the ASCII code of the input character.Desired question:(a) What does ASC take as input?(b) What does ASC return?
Actual questions that could have been generated from mrstransformation:(c) What does ASC take as input and returns theinteger giving the ASCII code of the input character?(d) ASC takes a character as input and returns whatgiving the ASCII code of the input character?
Introduction Background System Architecture Evaluation Conclusion
Why?Question Transformation does not Work Well without Sentence Simplification
Input Sentence:ASC takes a character as input, and returns theinteger giving the ASCII code of the input character.Desired question:(a) What does ASC take as input?(b) What does ASC return?
Actual questions that could have been generated from mrstransformation:(c) What does ASC take as input and returns theinteger giving the ASCII code of the input character?(d) ASC takes a character as input and returns whatgiving the ASCII code of the input character?
Introduction Background System Architecture Evaluation Conclusion
Introduction Background System Architecture Evaluation Conclusion
Coordination Decomposer“John likes cats very much but hates dogs a lot.”
but c
like v 1
cat n 1
udef q
rstr/h
arg2/neq
very+much a 1
arg1/eql-hndl/heq
l-index/neq
named(”John”)
proper q
rstr/h
hate v 1
dog n 1
udef q
rstr/h
arg2/neq
a+lot a 1
arg1/eqr-hndl/heq
r-index/neq
arg1/neq arg1/neq
but c
like v 1
cat n 1
udef q
rstr/h
arg2/neq
named(”John”)
proper q
rstr/h
arg1/neq
very+much a 1
arg1/eq
hate v 1
named(”John”)
proper q
rstr/h
arg1/neq
dog n 1
udef q
rstr/h
arg2/neq
a+lot a 1
arg1/eq
1
Introduction Background System Architecture Evaluation Conclusion
Coordination Decomposerleft: “John likes cats very much.“ right: “John hates dogs a lot.”
but c
like v 1
cat n 1
udef q
rstr/h
arg2/neq
very+much a 1
arg1/eql-hndl/heq
l-index/neq
named(”John”)
proper q
rstr/h
hate v 1
dog n 1
udef q
rstr/h
arg2/neq
a+lot a 1
arg1/eqr-hndl/heq
r-index/neq
arg1/neq arg1/neq
but c
like v 1
cat n 1
udef q
rstr/h
arg2/neq
named(”John”)
proper q
rstr/h
arg1/neq
very+much a 1
arg1/eq
hate v 1
named(”John”)
proper q
rstr/h
arg1/neq
dog n 1
udef q
rstr/h
arg2/neq
a+lot a 1
arg1/eq
1
Introduction Background System Architecture Evaluation Conclusion
Subclause Decomposeridentifies the verb, extracts its arguments and reconstructs MRS
be v id
named(”Bart”)
proper q
rstr/h
arg1/neq
cat n 1
the q
rstr/h
chase v 1
dog n 1
the q
rstr/h
arg2/neqarg1/eqarg2/neq
be v id
named(”Bart”)
proper q
rstr/h
arg1/neq
cat n 1
the q
rstr/h
arg2/neq
be v id
cat n 1
the q
rstr/h
chase v 1
dog n 1
the q
rstr/h
arg2/neqarg1/neqarg2/neq
(a): Bart is the cat that chases the dog.
(b): Bart is the cat. (c): The cat chases the dog.
decompose(({ chase v 1},{})decompose(({ be v id},{},keepEQ = 0)
1
Introduction Background System Architecture Evaluation Conclusion
Connected Dependency MRS GraphConnected DMRS GraphA Connected DMRS Graph is a tuple G = (N,E , L, Spre , Spost) of:a set N, whose elements are called nodes;a set E of connected pairs of vertices, called edges;a function L that returns the associated label for edges in E ;a set Spre of pre-slash labels and a set Spost of post-slash labels.Specifically,N is a set of all Elementary Predications (eps) defined in a grammar;Spre contains all pre-slash labels, namely {arg*, rstr, l-index, r-index, l-hndl, r-hndl, null};Spost contains all post-slash labels, namely {eq, neq, h, heq, null};L is defined as: L(x , y) = [pre/post, . . .]. For every node x , y ∈ N,L returns a list of pairs pre/post that pre ∈ Spre , post ∈ Spost . Ifpre 6= null, then the edge between (x , y) is directed: x is the governer,y is the dependant; otherwise the edge between x and y is not directed.If post = null, then y = null, x has no dependant by a pre relation.
Introduction Background System Architecture Evaluation Conclusion
parameters:rEPS: a set of eps for which we want to find related eps.eEPS: a set of exception eps.relaxEQ: a boolean value of whether to relax the post-slash value from eqto neq for verbs and prepositions (optional, default:1).keepEQ: a boolean value of whether to keep verbs and prepositions with apost-slash eq value(optional, default:1).returns:a set of eps that are related to rEPS;; assuming concurrent modification of a set is permitted in a for loopaEPS ← the set of all eps in the dmrs graphretEPS ← ∅ ;; initialize an empty setfor tEP ∈ rEPS and tEP /∈ eEPS do
for ep ∈ aEPS and ep /∈ eEPS and ep /∈ rEPS do
pre/post← L(tEP, ep) ;; ep is the dependant of tEPif pre 6= null then ;; ep exists
if relaxEQ and post = eq and (tEP is a verb ep or (tEP is a prepo-sition ep and pre = arg2)) then
assign ep a new label and change its qeq relation accordinglyend ifretEPS.add(ep)aEPS.remove(ep)
end if
pre/post← L(ep, tEP ) ;; ep is the governor of tEPif pre 6= null then ;; ep exists
if keepEQ = 0 and ep is a (verb ep or preposition ep) and post = eqand ep has no empty arg* then
continue ;; continue the loop without going further belowend ifretEPS.add(ep)aEPS.remove(ep)
end if
end forend for
if retEPS 6= ∅ thenreturn rEPS ∪ decompose(retEPS, eEPS, relaxEQ = 0) ;; the unionof two
elsereturn rEPS
end if
1
Introduction Background System Architecture Evaluation Conclusion
English Sentence Structureand corresponding decomposers
English Sentence Structure
Complex
dependent clause +independent clause
Subordinate Clause
Causal | Non-causalRelative Clause
Compound
coordinationof sentences
Simple
independent &simple clause
Coordinationof phrases
Apposition Others
CoordinationSubclauseSubordinateWhy Apposition
Decomposer Pool
DecomposedSentence
Introduction Background System Architecture Evaluation Conclusion
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
Introduction Background System Architecture Evaluation Conclusion
The ProblemWill the wedding be held next Monday?
MRSXML
Plain text
Termextraction
FSCconstruction
Parsingwith PET
2
3
1
Generationwith LKB
Outputselection
5
6
7
MRSTransformationMRS Decomposition
8
Output toconsole/XML
FSCXML
Apposition Decomposer
Coordination Decomposer
Subclause Decomposer
Subordinate Decomposer
Why Decomposer
MRSXML
4
Unranked Realizations from LKB for“Will the wedding be held next Monday?”
Next Monday the wedding will be held?Next Monday will the wedding be held?Next Monday, the wedding will be held?Next Monday, will the wedding be held?The wedding will be held next Monday?Will the wedding be held next Monday?
Introduction Background System Architecture Evaluation Conclusion
Question RankingWill the wedding be held next Monday?
MaxEnt Model for Declaratives
4.31 The wedding will be held next Monday?1.63 Will the wedding be held next Monday?1.35 Next Monday the wedding will be held?1.14 Will the wedding be held next Monday?0.77 Next Monday, the wedding will be held?0.51 Next Monday will the wedding be held?0.29 Next Monday, will the wedding be held?
Language Model for Interrogatives
1.97 Next Monday will the wedding be held?1.97 Will the wedding be held next Monday?1.97 Will the wedding be held next Monday?1.38 Next Monday, will the wedding be held?1.01 The wedding will be held next Monday?0.95 Next Monday the wedding will be held?0.75 Next Monday, the wedding will be held?
Combined Scores
1.78 Will the wedding be held next Monday?1.64 The wedding will be held next Monday?1.44 Will the wedding be held next Monday?1.11 Next Monday the wedding will be held?0.81 Next Monday will the wedding be held?0.76 Next Monday, the wedding will be held?0.48 Next Monday, will the wedding be held?
Introduction Background System Architecture Evaluation Conclusion
Review of System Architecture
MRSXML
Plain text
Termextraction
FSCconstruction
Parsingwith PET
2
3
1Generationwith LKB
Outputselection
5
6
7
MRSTransformationMRS Decomposition
8Output to
console/XML
FSCXML
Apposition Decomposer
Coordination Decomposer
Subclause Decomposer
Subordinate Decomposer
Why Decomposer
MRSXML
4
Introduction Background System Architecture Evaluation Conclusion
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
Introduction Background System Architecture Evaluation Conclusion
QGSTEC2010The Question Generation Shared Task and Evaluation Challenge (QGSTEC) 2010
Task B: QG from Sentences.Participants are given one complete sentence from which theirsystem must generate questions.1. Relevance. Questions should be relevant to the input
sentence.2. Question type. Questions should be of the specified target
question type.3. Syntactic correctness and fluency. The syntactic
correctness is rated to ensure systems can generate sensibleoutput.
4. Ambiguity. The question should make sense when askedmore or less out of the blue.
5. Variety. Pairs of questions in answer to a single input areevaluated on how different they are from each other.
Introduction Background System Architecture Evaluation Conclusion
QGSTEC2010The Question Generation Shared Task and Evaluation Challenge (QGSTEC) 2010
Task B: QG from Sentences.Participants are given one complete sentence from which theirsystem must generate questions.1. Relevance. Questions should be relevant to the input
sentence.2. Question type. Questions should be of the specified target
question type.3. Syntactic correctness and fluency. The syntactic
correctness is rated to ensure systems can generate sensibleoutput.
4. Ambiguity. The question should make sense when askedmore or less out of the blue.
5. Variety. Pairs of questions in answer to a single input areevaluated on how different they are from each other.
Introduction Background System Architecture Evaluation Conclusion
Test Set
• 360 questions were required to be generated from 90 sentences• 8 question types: yes/no, which, what, when, how many,where, why and who.
Introduction Background System Architecture Evaluation Conclusion
Participants
• Lethbridge, syntax-based, University of Lethbridge, Canada• MrsQG, semantics-based, Saarland University, Germany• JUQGG, rule-based, Jadavpur University, India.• WLV, syntax-based, University of Wolverhampton, UK
Introduction Background System Architecture Evaluation Conclusion
Evaluation Grades
Relevance Question Type
Correctness Ambiguity Variety0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Results per criterion without penalty on missing questions
WLVMrsQGJUQGGLethbridgeWorst
Introduction Background System Architecture Evaluation Conclusion
Generation Coverage
coverage
Page 1
MrsQG WLV JUQGG Lethbridge
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Coverages on input and output(generating 360 questions from 90 sentences)
sentencesquestions
Introduction Background System Architecture Evaluation Conclusion
Evaluation Gradeswith penalty on missing questions
Relevance Question Type
Correctness Ambiguity Variety0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Results per criterion with penalty on missing questions
MrsQGWLVJUQGGLethbridgeWorst
Introduction Background System Architecture Evaluation Conclusion
Evaluation Grades per Question Type
Relevance Question Type Correctness Ambiguity Variety0