Scenario Wrapup “Shangri-la” – QA Support in 7-10 Year Dr. John D. Prange AQUAINT Program Manager [email protected] 301-688-7092 .

Scenario Wrapup “Shangri-la” – QA Support in 7-10 Year

Dr. John D. PrangeAQUAINT Program Manager

[email protected]

http://www.ic-arda.org

2

Task

• Two Approaches– Top Down– Bottom Up

• Both Approaches worked simultaneously by Analysts

• Goal is to see if the two approaches can be connected somewhere in the middle; Can we “Marshal the Evidence to Build a Case” that will directly addresses the assigned task.

How many operatives remain in America

& what forms of terror do they have in mind?

3

Top Down

• Top Level Questions:– What are the possible forms of terror?

• Candidate: Man Portables

– Who are the possible perpetrators/operatives?• Suspected Al-Qaeda operatives• Members of God’s Aryan Militia

– What are the possible modes of operations?• Weapons theft from military• Surplus weapons purchases• Suspicious transactions• Transportation of suspicious materials / movement of suspects

• Task: For each possibility – decompose into sufficiently fine sub-elements; indicators, plan outlines, time lines, and then search for data that supports these elements.

• How does the data that the analyst has in hand or can find fit these finer-grained models?

5

?What recent terrorist

activity/Incidents in Yemen?

Against US Militaryworld-wide?

What are the potential threats?

?How secure is the

Aden Harbor?

Are our security plans adequate?

?What is the

Current Political/Social/Cultural climate

in Yemen?In the region?

?What is the current

state of US relations

with Yemen Other countries

in region?

An Asymmetric Threat Scenario *

How safe is the Muscat harbor for refueling US Navy ships?

POSSIBLE SOURCESCurrent Intel Reports;

Latest Terrorist Assessments;

Historical Incident Reports

Terrorist Activities

How secure is theAden Harbor?

Are our security plans adequate?

POSSIBLE SOURCESImagery of Harbor;

Annotated Maps;Military Attaché Video

of Port Facilities; Naval Orders/SOPS/Plans

Harbor Facilities/Security Plans

POSSIBLE SOURCESLocal Print, Radio,

BroadcastNews ReportsOther Analysts

Current YemenEnvironment

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

……………………….…………………..

Current US /YemenRelations

POSSIBLE SOURCESOther Analysts;

State Department/Country Team Cables;Recent Contact withEmbassy/Consulate

* Reference: SRI International AQUAINT Briefing

6

Bottom Up

• Try to form a more coherent picture of the available data / evidence that has been obtained. Use “Shangri-la” QA system to:

– Create detailed timelines of the available data

– Create multiple Link Diagrams to display different types of relationships between events, people, places, materials

– Create overlays of these timelines and link diagrams

• Use “Shangri-la” QA system to

– Develop detailed profiles, background information on each event, people involved, organizations, materials, weapons, vehicles etc

– Cast a “Wide Net”; Follow Leads

– Create a broad foundation of known / suspected facts that appear to be related to the current investigation

– Identify gaps, missing data What don’t I know? And if I knew it what good would it do me?

7

• A Different Paradigm may be useful when handling QA Scenarios:

• Current Analytic Paradigm:

QA Scenarios: A Different Paradigm?

– Sequentially “Filter Down” to the

final result

Processing & Analysis

Data

Results

– Works when QA’s are independent, isolated activities

– Cast a “wider net” while searching

for “golden nuggets” (Answers)

AnswersSpace of Data Objects and Sources

How Wide to Cast the “Net”?

What Info to Retain? In what form?

For how long?

– Automatically Extract, Represent,

and Preserve “closely related”

background information within

context of the QA Scenario

Background

Discarded

8

Marshal the Evidence to Build a Case

Shangra-La QA System assists Analyst in finding closure between the two approaches

Top DownApproach

Bottom UpApproach

Case

9

Improved Reasoning & Learning

Associates Associates Follow-upLeads

Follow-upLeads

Cross Fertilization

Advanced Reasoning:• Use Multi-level Plans• Create and evaluate chains of reasoning• Reason across hetero- geneous data sources• Infer answers from data extracted from multiple sources when the answer is not explicitly stated • Utilize Link Analysis & Evidence Discovery• Plus other strategies

Advanced Learning:• Automatically learn new or modify existing reasoning strategies

New SeniorAdvisor

“Views: Past & Present” .….… ….…...……. ….…...……. ….…...……. ….…...……. ….…..

Summarized Results

Collected Views

TV & RadioBroadcasts,Newspapers

& OtherArchives

“Bio”………..….……..…….………..….……..…….………..….……..…….…………...

Raw “Bio”Information

Education

Past Positions

Family

Travels

Other Activities

Summarized Results

11

Final Comments

• Shangra La QA System:

– Not really a system but an integrated suite of tools

– Allows access to tools/capabilities that fall outside the scope of Advanced QA

– Learns and adapts as the task unfolds

– Capabilities in the small exist today but they need to be done far more automatically with little or no direct analysts involvement (e.g. the metadata hotlink tags done automatically in the large)

– Knows its limitations and can adequately share those with analysts

– Can provide explanation and maintains understandable audit trails

– Guts of the system are transparent to the analyst

– HCI issues/concerns have been significantly addressed

12

Final Comments

• IC Analysts:

– Understands the capabilities and limitations of the Shangra-la QA system; How the system interfaces with the Analyst’s methods

– Knows how to ask questions that the system is capable of responding to

– Uses “Evidence Marshalling Strategies” and knows how to use the Shangra-la System to:• Formulate hypotheses and then decompose them into

subcomponents that can be researched and investigated

• Develop appropriate arguments and build a “Defendable Case”

• Seek both confirming and disconfirming evidence

• Construct scenarios out of the available evidence

13

Final Comments

• It’s a team effort– IC Analyst + Shangra-la QA System

• Use a Football Analogy

• Shangra-la QA System– “Franchise” Player

BUT …

• IC Analyst is the “Quarterback/Coach”– Knows the scouting report– Develops the game plan– Calls the plays– Make the tough decisions