AQUAINT Program: Overview Dr. John Prange, Info-X R&D Thrust Director Dr. Lynn Franklin, Dep Info-X R&D Thrust Director [email protected]; [email protected]443-479-8006 (Prange) / 443-479-6604 (Franklin) 301-688-7092 (ARDA Office) http://www.ic-arda.org
44
Embed
AQUAINT Program: Overview Dr. John Prange, Info-X R&D Thrust Director Dr. Lynn Franklin, Dep Info-X R&D Thrust Director [email protected]; [email protected].
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
AQUAINT Program: Overview
Dr. John Prange, Info-X R&D Thrust DirectorDr. Lynn Franklin, Dep Info-X R&D Thrust Director
Move Closerto the Questione.g. QuestionClassi fication
Q&AQ&A
Next Generation Approaches:Question & Answering (Q&A) Systems
“Answer” “Answer”
Move Closerto the Answere.g. P assage
Retrieval
Move Closerto the Answere.g. P assage
Retrieval
ShallowAnalysisShallowAnalysis
Commercial World & Current R&D EffortsAre Addressing the Next GenerationBut Only Selected Content Understanding Barriers Are Being Aggressively Attacked
26QA Workshop - ACL 2001
Overarching Context /Operational Requirement
AdvancedQA
Extract & AnalyzeResults
DeeperAutomated
Understanding
Answers
Interpret Results& Formulate the Answers
Provide Answers in a Form
Analysts Want
Ranked Lists of
“Re levant” Data Objects
System SpecificQueries; Fully Tailoredto S eries of Questions
ExtendTraditionalInformationRetrieval
MultipleHeterogeneous
DataSources
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .. . . . . . . . . .
Mult i-Media Mult i-Media St ructured St ructured
Other Other
Text Text Voice Voice
Inte rpretingComplex
QA Scenario within a
Larger Context FactoidQuest ions?
WhyQuest ions?
InterpretiveQuest ions?
JudgementQuest ions?
OtherQuest ions?
InformationAnalysts
PredictiveQuest ions?
Advanced Question Answering
Multiple KeyBarriers toContentUnderstandingWill Be AggressivelyAttacked
Advanced Question Answering Is Skipping Ahead Two Generations
14
AQUAINTAdvanced QUestion & Answering for INTelligence
• What it is and What it is not . . .
– Question & Answering Aimed at the “Information Professional” --- Not just the Casual User
– Rich, Contextually-based Question Scenarios --- Full Range of Questions --- Not just Isolated, Factoid Questions
– Places much higher premium on knowledge and reasoning across very broad domains
– Open Domain, Multiple Media, Multiple Languages, Multiple Genre, Structured and Unstructured Data --- Not just a Focused Data Environment
15
Increasing Complexity Levels of Questions & Answers
Level 1”Simple
Factual QA’s"
Level 2"Template &
Multi-valued QA’s”
Level 3“Cross Media &
Cross Document QA’s"
Level 4”Context-Based
QA Scenarios”
FULL COMPLEXITY OF QUESTIONS & ANSWERS RANGES:
FROM: TO:
Questions: Simple Facts Questions: Complex; Uses Judgement TermsKnowledge of User Context Needed;Broad Scope
Answers: Simple Answers found in Answers: Search Multiple Sources (in multipleSingle Document Media/languages); Fusion of
information; Resolution of conflictingdata; Multiple Alternatives; AddingInterpretation; Drawing Conclusions
Current Near Term Mid Term Long Term
Advanced QA:Ramping up to the Full Complexity of Questions & Answers
16
Future
Fully Intersected;Automatically
Generated;Variable Structure/Format;
Full Context Responses
Full Context-Based
QuestionScenario
Level III
Full Context-Based
QuestionScenario
Fully Intersected;Automatically
Generated;Variable Structure/Format;
Full Context Responses
Level II
Variable NarrativeSummary;
Multi-Media Presentations;
Simple InterpretedResults
Cross MediaCross Document
Simple Judgement
Level I
Fixed Templatesor
Tabular Lists
Mulit-ValuedFactual QuestionsQuestions
Answers
Today
50/250 BytePassage from
Single TextDocument
SingleFactualIsolated
Questions
Data Chasm
Missing Data
MANY Heterogeneous Data Sources;
All Types, Sizes, Locations
IncreasingVolumes
(Petabyte & up)
Synthesis Across“Documents”/Media
ContradictoryDataReliability
of Data & SourceMultiple
Perspectives
Advanced QA:Attacking the Data Chasm
17
Structured / Semi-Structured
KB’s DB’s“Tagged Data”(e.g. Web Data)
Unstructured
HumanLanguage
Media
Sensor
Economic
Geospatial
VisualData
Video Still Images
Text
Documents
Speech
Multi-Media
Technical /Abstract
Other
Newswire / News Broadcast
Technical
Formal / Informal Communication
Other
Language
English
ForeignLanguage 1
ForeignLanguage 2
ForeignLanguage N
Genre
Advanced QA:Complex QA Across Data Types
18
Advanced QA:Much Deeper Understanding of Human Language is Required
• Some times SMALL differences can produce significantly different results/interpretations:
– Stop Words
• “Books {by; for; about} kids”
– Attachments
• “The man saw the woman in the park with the telescope.”
– Co-reference
• “John {persuaded; promised} Bill to go. He just left.”
• “Mary took the pill from the bottle. She swallowed it.”
• Other times BIG differences can produce the same/similar results:
– “Name the films in which Jude Law starred.”
– “Jude Law played a leading role in which movies?”
– “In what Hollywood productions did Jude Law receive top billing?”
19
Advanced QA:Is Time Our Achilles Heel?
• Real Difficulties Exist in:
– Extracting, correctly interpreting time references & then creating manageable timelines
– Estimating & updating changing reliability of information over time
– Processing information in time sequence e.g. Tracking the details of an evolving event over time -- A whole different set of problems
• And of course:
– We can’t forget all of the issues related to the timeliness of the system’s response to our question(s) -- we’ll need at least “near real time responses”
March April May June July August
20
• Different sources do not report simultaneously on an event.
• Data from different sources may be near real-time or take years to arrive.
• The hypothesis of today may be thrown out by new data arriving next week.
Event
H Hour
Event Planning Aftermath of the Event
H-n H-nH+1H-1
Collectors 1,2,3 observe event
Collector 1 reports
Collector 2 reports Collector
3 reports
Collector 4 observe event-related info &
reports
Collector 5 picks up historical
event planning material in a
raid
Collector 1 observes
event planning &
reports
• Analysis is dependent on a time continuum where data on a future event is found in the historical patterns established in event planning stages. As incoming data is evaluated against historical data, outcomes may change.
Advanced QA:The Challenge of Time in Analysis
21
Advanced QA:The Need for Ever Increasing Knowledge -- Of All Types
** Knowledge Requirement would be better represented with a whole “quiver of arrows” of different sizes, lengths and types
DIMENSIONS OF THE QUESTIONPART OF THE QA PROBLEM
DIMENSIONS OF THE ANSWERPART OF THE QA PROBLEM
Context
Judgement
Scope
Fusion
Interpretation
MultipleSources
QA R&D Program
QA R&D Program
Advanced AdvancedSimpleFactual
Question
SimpleAnswer,SingleSource
Increasing
Knowledge Requirements **
IncreasingKnowledgeRequirements **
22
• A Different Paradigm may be useful when handling QA Scenarios:
• Current Analytic Paradigm:
– Sequentially “Filter Down” to the
final result
Processing & Analysis
Data
Results
– Works when QA’s are independent, isolated activities
– Cast a “wider net” while searching
for “golden nuggets” (Answers)
AnswersSpace of Data Objects and Sources
How Wide to Cast the “Net”?
What Info to Retain? In what form?
For how long?
– Automatically Extract, Represent,
and Preserve “closely related”
background information within
context of the QA Scenario
Background
Discarded
Advanced QA:The Need for a Different Paradigm
23
Overarching Context / Operational Requirement
Who is thisadvisor?
What do weknow about
him/her?
What are his/her views?
What influence does he/she have on FM?
And still more questions ???
In a foreign news broadcast a team of analysts observe a previously unknown individual conferring with the Foreign Minister. They suspect
that he/she is really a new senior advisor.
Does this signal that other
policy changes are coming?
Information Analysts
Advanced QA:Need for Improved Reasoning & Learning
FOCUS
24
Advanced Reasoning:• Use Multi-level Plans• Create and evaluate chains of reasoning• Reason across hetero- geneous data sources• Infer answers from data extracted from multiple sources when the answer is not explicitly stated • Utilize Link Analysis & Evidence Discovery• Plus other strategies
“Views: Past & Present” .….… ….…...……. ….…...……. ….…...……. ….…...……. ….…..
Summarized Results
Collected Views
TV & RadioBroadcasts,Newspapers
& OtherArchives
Raw “Bio”Information
Education
Past Positions
Family
Travels
Other Activities
Summarized Results
Cross Fertilization
Advanced Learning:• Automatically learn new or modify existing reasoning strategies
Advanced QA:Need for Improved Reasoning & Learning
25
Interested External
Stakeholders
ARDA’s Info-X Program Partners
Active IC /Government
Partners
RecentAdditions
• NGIC• DHS
26
QUESTION????
Clarification
Other Analysts
Question & RequirementContext; Analyst Background
Knowledge
Multimedia Examples
Natural Statement ofQuestion;
Use of
QueryAssessment,
Advisor,Collaboration
Question Under- standing andInterpretation
Knowledge Bases;Technical Databases
AQUAINT:R&D Focused on Three Functional Components
Question & Answer Context
•Relevant information extracted and combined where possible;•Accumulation of Knowledge across “Documents”•Cross “Document” Summaries created;•Language/Media Independent Concept Representation•Inconsistencies noted;•Proposed Conclusions and Inferences Generated
HIGHLIGHTS• Dramatic progress on linguistic approach that converts question and
relevant passages into logical forms and then arrives at answer through a powerful combination of an extended “WordNet” and a logic prover
AQUAINTAdvanced QUestion & Answering for INTelligence
39
HIGHLIGHTS• Dramatic progress on linguistic approach that converts question and
relevant passages into logical forms and then arrives at answer through a powerful combination of an extended “WordNet” and a logic prover
• Made significant strides in extending QA from isolated, factoid questions to far more complex “Who is / What is” questions that require combining information from multiple, potentially duplicative or contradictory document sources
AQUAINTAdvanced QUestion & Answering for INTelligence
40
More Complex Question Types
• Definitions– What is Tikrit?
• Biographies – Who is Mahmoud Abbas?
• Events– What happened in Baghdad on Thanksgiving?
• Different Perspectives / Opinions– What people think of Mahmoud Abbas’ resignation?
• Lists– What names of chewing gums are found in the AQUAINT corpus?
• Relationships– The analyst is interested in the line of succession of the Saudi
government, and the relationship between the individuals in their royal family. King Fahd is the current ruler, but is in poor health. Who is next in line, and what is his relationship to King Fahd? Who, if anyone, has been designated as second in line?
41
Example Definition *
What is Tikrit?
Tikrit is a power center for Sunni Arab tribes that may hold out for as long as possible out of fear of losing power to the nation’s Shiite majority (12). Baghdad may be the capital of Iraq, but Tikrit is Saddam country (15). Other experts caution that the years of preferential treatment towards the residents of Tikrit may cause them to stand by Saddam Hussein to the end (4). …
* Reference: Columbia Univ. / Univ. of Colorado AQUAINT Briefing
42
HIGHLIGHTS• Dramatic progress on linguistic approach that converts question and
relevant passages into logical forms and then arrives at answer through a powerful combination of an extended “WordNet” and a logic prover
• Made significant strides in extending QA from isolated, factoid questions to far more complex “Who is / What is” questions that require combining information from multiple, potentially duplicative or contradictory document sources
• Progress made on developing multi-engine QA system that combines linguistic, statistical & KB approaches
AQUAINTAdvanced QUestion & Answering for INTelligence
43
Available Answering Agents
• Predictive Annotation Agent– General-purpose agent, used in almost all cases.
• Statistical Query Agent– Also general-purpose. Courtesy Roukos/Ittycheriah
Multiple QA Agents Approach *What is the largest city in England?
• Text Match– Find text that says “London is the largest city in England” (or
paraphrase). Confidence is confidence of NL parser * confidence of source.
• “Superlative” Search– Find a table of English cities and their populations, and sort.– Find a list of the 10 largest cities in the world, and see which are in
England. • Uses logic: if L > all objects in set R then L > all objects in set E R.
– Find the population of as many individual English cities as possible, and choose the largest.
• Heuristics– London is the capital of England. (Not guaranteed to imply it is the
largest city, but this is very frequently the case.)
• Complex Inference – E.g. “Birmingham is England’s second-largest city”; “Paris is larger
than Birmingham”; “London is larger than Paris”; “London is in England”.
46
HIGHLIGHTS• Dramatic progress on linguistic approach that converts question and
relevant passages into logical forms and then arrives at answer through a powerful combination of an extended “WordNet” and a logic prover
• Made significant strides in extending QA from isolated, factoid questions to far more complex “Who is / What is” questions that require combining information from multiple, potentially duplicative or contradictory document sources
• Progress made on developing multi-engine QA system that combines linguistic, statistical & KB approaches
• Executed Pilot Evaluations for multiple complex QA Types; Developed Metrics for evaluating QA Systems at the Scenario Task Level; Full Evaluation of all End-to-End QA Systems late in Phase 2
AQUAINTAdvanced QUestion & Answering for INTelligence
47
June Sunrise over Kirkwall Bay in the Orkney Islands of Scotland
Your QuestionsYour Questions& Comments& Comments
48
Contact Information
Dr. John Prange, Info-X R&D Thrust Program DirectorDr. Lynn Franklin, Info-X R&D Thrust Program Dep Dir