Using Automatic Question Generation to Evaluate Questions Generated by Children

Using Automatic Question Generation to Evaluate Questions Generated by Children

Using Automatic Question Generation to Evaluate Questions Generated by Children

Wei Chen, Jack MostowGregory AistProject LISTENSchool of Computer Sciencewww.cs.cmu.edu/~listen Carnegie Mellon UniversityCommunication Studies and Applied Linguistics Department of EnglishIowa State University1or off-task speech?2What are you wondering about now?

Problem: classify speech

Spoken question?

Four Types of ResponsesI wonder how the cool cat survives in the cold and snowy placeYes I cant wait till this book is over please tell me this book is over now1. A complete question2. A partial question3. Off-task speech4. No response3Im wondering how will Tony

Add audio not already played3

Generate QuestionswindI wonderIm wondering

howwhyifwhenwhowhathowmakes electricityI wonderwhatlives on Mars4disfluencyFix fonts; title style doesnt match earlier titles; need more info to motivate segue: Model self-questioning responses: Use FSG to generate questions; 2. Add arcs to model disfluency; 3. Add trigrams to model off-task speech.Use real examples. If cant mix and match N and VP, fix diagram to reflect reality.4Model Self-Questioning Responsesand Im likeback back backcant go onnow Im onof it rightokay lets goalready read thisoh my godReyna come here

I wonderIm wondering

howwhyifwhenwhowhat5Off-task trigramsShow where source text fits in: animate? Text instantiating THING, VP; mix and match any N w/ any VP?5QG in Response ClassificationASRGenerate Questions6

Feature vector for utteranceLanguage model

Avoid gratuitous color.Distinguish data from process?Distinguish decision from data What do Questioning and Off-task boxes represent?Avoid mixing arrows with different meanings (data flow vs. decision tree)6Classifying Responses

I WONDER IF WAS? A NATIONAL YELLOWSTONE DONE? EXISTSVMPitch, intensity, duration, MFCC, % off-task words: 4/9% off-task words with ASR confident: 2/9 % on-task with ASR uncertain: 0QuestioningOff-task

Most informative acoustic features are Use animation to associate words with %7EvaluationInvolves questioningNo questioning involvedFeature vector of utteranceRecall: 0.59Precision: 0.85Recall: 0.83Precision: 0.55vs. 0.55vs. 0.76vs. 0.80vs. 0.50question generation vs. story trigrams250 responses in Reading Tutor by 34 children ages 7-10

Words correctly recognized: 38%

OOV rate: 19%Add story trigrams results as animation.Move R&P below boxes (moved up to make room).QG-based model may need concise self-explanatory name.Mark significant differences if any.8ConclusionGenerated questions to help detect off-task speech

Generate questions to model on-task speech

Interpolate with language model of off-task speech9Use consistent red and blue throughout.Clarify structure of text.Move main point into title?Missing: so what? Future? ? Take-away? Lesson(s)? How generalize?9