David C. Wilkins Center for Study of Language and Expertise Stanford University David Fried

1

Multi-agent architectures that facilitate apprenticeship learning for

real-time decision making: Minerva and Gerona

David C. WilkinsCenter for Study of Language and Expertise

Stanford University

David FriedDepartment of Computer Science

University of Illinois at U-C

November 5, 2005Supported by ONR: N00014-00-1-0660, N00014-02-1-0731

2

Outline

• Goal– Expert shells multi-agent capabilities

• Minerva – medical diagnosis (1992-1994)– Apprentice program observes expert, improves agent

• Genona – ship damage control (2002-2005)– Apprentice program observes student, improves student

• Summary and conclusions

3

Expert Shells -> Multi-Agent Capabilities

• Traditional performance capabilities– Correct solution, Efficient problem solving

• Multi-agent capabilities – Critiquing

• Expert agent watches – finds errors omission/commission– Apprenticeship Learning

• Expert agent watches expert, improves expert agent• Expert agent watches student, improves student

• Research philosophy– Critiquing & apprenticeship should be natural artifact of shell architecture– Same apprenticeship method should support both learning and tutoring– Unified arch for dimensions of expertise is approach to cognitive modeling

4

Apprenticeship Learning Paradigm

Problem

Human Expert Problem Solver Agent Actions Learning Actions Program

KN Differences

• Situated Learning: within context of problem solving• Good for knowledge refinement of human or expert agent

5

Apprenticeship Learning Challenges

• Global credit assignment – Does good explanation of human action exist?– Challenge: some explanation usually exists

• Local credit assignment– What KN difference creates good explanation?– Challenge: Many repairs will create explanation

• Variance among human problem solvers– How to distinguish between allowable variations among

human problem solvers (who among other things often disagree) and variations that suggest knowledge errors

• Solution– Minerva shell architecture

6

Minerva-Based Apprenticeship Learning: Domain of Neurology Diagnosis

1. Debra Arbed, a 39 year old black female.2. Chief complaint is headache, nausea, vomiting, stiff neck.3. Headache duration? 6 hours.4. Headache severity? 4 on scale of 0-4.5. Fever? No. 6. Recent seizures? No.7. Visual problems? No. 8. Headache onset? Abrupt.30. Final diagnosis is subarachnoid hemorrhage.31. Secondary dx is acute bacterial meningitis.

7

Evolution of Decision-Making Expert Shells:Separation of Different Knowledge Types

Domain Kn

Task Kn

Sched Kn

Inference

Inference

Inference

Program

Minerva(1992)

Odysseus2(1994)

Neomycin(1982)Guidon2(1987)

Odysseus(1988)Mycin

(1972)GuidonTieresias(1978)

Task Kn

Domain Kn Domain Kn

8

Domain, Task, and Scheduling KN are Distinct

• Domain KN: vocabulary and predicates mention domain• Task KN: no mention of domain (e.g., medicine):

strategy(differentiate-hypotheses(Hyp1, Hyp2) :-active-hypothesis(Hyp1), active-hypothesis(Hyp1), different(Hyp1, Hyp2), evidence-for(Finding1, Hyp1, Rule1, Cf1),evidence-for(Finding1, Hyp2, Rule2, Cf2), same-sign-cfs(Cf1, Cf2),get-premise(Rule1, Finding, Premise1), get-premise( Rule2, Finding, Premiise2),premises-contradicting(Premise1, Premise2), not rule-applied(Rule1), strategy (apply-rule (Rule1))

• Scheduling KN: Chains (GSG…A) created by unification. But which Action A is best?

9

Recursive Classification: Use in Scheduler

Inference Level (Domain BBoard)

Scheduler Level(Recursive HC)

Strategy Level (Hypothesis-Directed)

Domain Level(Medical knowledge)

Minerva-Medicine Minerva-Scheduler

Inference Level(Scheduler BBoard)

Scheduler Level(FIFO)

Strategy Level(Exhaustive-Chaining)

Domain Level(Scheduling knowledge)

10

Recursive Classification:Induction of Embedded Knowledge Base of

Scheduler Rules

• Induction of Scheduling rules: – 10-70 (39 avg.) classes, 42 features– 286 scheduling rules– Disjoint training and validation sets.

• Critiquing evaluation– Expert’s action upper 10% = 52.2%– Expert’s action upper 20% = 67.4%– Expert’s action upper 50% = 84.8%

11

Minerva: Related Research

• Blackboard Architectures (BB1, Hearsay III)– Opaque code or scheduler hardwired: not learnable.

• Classification Shells (Mole, Neomycin, Protos, Internist)– Scheduler is mostly hard-wired.

• Advanced Classification Shells (Ask/Mu)– scheduler knowledge specialized 1 expert.

• Critiquing Systems (Disciple, Oncocin/Protégé)– Classification vs. task reduction vs. therapy plans

12

The Problem of Ship Damage Control

• Ship crises– Fire, smoke, flooding, pipe rupture– Primary and secondary damage

• Damage Control Assistant (DCA)– Responsible for overall crisis management– Makes damage control decisions– Coordinates investigation and repair teams

13

“Damage Control Assistant” ExpertiseHow to get decision-making practice?!

• Expertise requires practice – Time-critical decision-making – High stress, information overload– Uncertain and incomplete information

• “Whole task” practice difficult to acquire– Actual ship crises infrequent – Realistic practice expensive and dangerous – Rotation cycle is 2-3 years

14

The DCA Decision-Making Task:Fires, Smoke, Floods, Ruptures, etc

• Event to DCA: fire observed in compartment 1-174-0-L

• Event to DCA: pipe rupture observed compart 1-191-0-Q

• Action by DCA: send repair party to compart 1-174-0-L

• Action by DCA: go to General Quarters (GQ)

• Action by DCA: start fire pump #3 on port side

• Critique to DCA: Error of omission: must request permission of CO to turn on fire pump during GQ

• Action by DCA: Close firemain valve 3-274-2

• Critique to DCA: Error of commission: valve 3-274-2 does not isolate pipe rupture

15

DC-Train 4.0 Simulation Capabilities

• Physical ship simulation– Primary and secondary damage– Fire, smoke, flooding, rupture, firemain

• Intelligent agent personnel simulation– 67 ship personnel

• Commanding officer• Engineering Officer of the Watch• Investigator Teams, Repair Teams, etc.

16

DC-Train and SCoT-DC:Post-Scenario Spoken Dialogue Tutoring

Expert &CritiquingModules

DCAstudent solves

problempresented

byDC-TrainSimulator

CorrectExpert

Solution+

Critiqueof Student

Actions

SpokenDialogueInterface

+Interactive

VisualizationInterface

Tutoring& Dialogue

Modules

University of Illinois | Stanford University DC-Train 4.0 w/ Critiquing | Spoken Dialogue Tutoring

17

Whole-Task Simulation-Based Training of Crisis Decision Making Skills

DCA Student

DC-Train: Physical

Simulator and

Intelligent Agents

Expert, Critiquing, Explanation Models:

Graph Mod Operators (GMOs, Meta-GMOs)

Causal Story

Graph (CSG)

Events

WorldState

WorldInfo

Actions Text-Based and Spoken

Dialogue Tutors

Event Comm Language (ECL)is used along all arrows

18

Gerona Expert Agent Overview • Goal:

– Agent architecture to support multiple uses: • expert model, critiquing, question-answering,

explanations, spoken dialogue tutoring, etc.• Solution

– Explicit Knowledge Representation• ECL (vocabulary), • GMOs, G-Clauses (expert and student critique models) • Meta-GMOs (question-answering, explanations)• CSGs (structured ECLs that represent all models)

– Good for knowledge acquisition from experts– Gerona representation can be “executed” by an interpreter

19

Event Communication Language (ECL)

• Event Communication Language (ECL) statements encode communication to and from the DCA, and communication about state of world.

• Example– English: Boundaries set: RL5 Talker to DCA: “DCA,

Repair 5 reports fire boundaries set for compartment 4-220-0-E, auxiliary machinery room #2.

– ECL message 6310: Boundaries setECL-6310 ([to], [from], “reports”, [problem], “boundaries set for compartment”, [compartment])

20

Event Communication Language (ECL)• ECL 2000 – WorldInfo (81)

– E.g., Contents of compartments, location of bulkheads• ECL 3000 –WorldState Predicates (29)

– E.g., Boundaries contain compartment• ECL 4000 – WorldState Functions (22)

– E.g., Compartment to Jurisdiction• ECL 5000 – Actions from the DCA (48)

– E.g., Send firefighters, Start fire pump, Request permiss• ECL 6000 – Events reported to DCA (88)

– E.g., Fire alarm, firemain pressure low, desmoking space• ECL 7000 – Goals (36)

– E.g., Identify fire, contain fire, patch pipe rupture, • ECL 8000 – Crises (7)

– E.g, Fire, hot mags, flood, smoke, pipe rupture, low fp

21

Causal Story Graph (CSG)

Crisis:Fire

Active Goal:Control

Fire

SatisfiedGoal:

IdentifyFire

AddressedGoal:

ContainFire

Active Goal:Extinguish

Fire

Active Goal:IsolateSpace

Active Goal:Apply Fire

Suppressant

Event:Fire Report

CorrectAction:Set Fire

Boundaries

Error ofOmission:

Electrically IsolateSpace

Error ofCommission:Fight Fire in

Space

Event:Set Fire

Boundariesin progress

Justification:Why Error ofCommission?

22

Graph Modification Operators (GMO)GMO 5120FOR ECL 5120 “Fight Fire”

compartment -> Compartmenttarget -> Station

RULE 5120.fight-fire.critique.1IF

goal(find, unaddressed, 7118, “Apply fire suppressant”,[compartment = Compartment], _, G)

AND action(find, pending, 5120, “Fight fire in space”, [compartment = Compartment], _, A)

AND goal(find, satisfied, 7116, “Isolate compartment if necessary”,

[compartment = Compartment], _, _),AND goal(find, satisfied, 7117, “Active desmoke if necessary”,

[compartment = Compartment], _, _),AND ship-state(find, _, 4302, “Best repair locker for

compartment”,[compartment = Compartment, station = Station], _, _)

23

Graph Modification Operators (cont)

THENaction(modify, correct, 5120, “Fight fire in space”,

[compartment <- Compartment, station <- Station], _, A)goal(modify, addressed, 7118, “Apply fire suppressant”,

[compartment <- Compartment], _, G)END RULE

…

END GMO

24

Meta-GMO Question Types• About 100 templates cover all past instructor-student QAs

– “Why” questions for justifying CSG nodes (12)• “Why should I have ordered firefighting?”

– “What” questions for retrieving expert recommendations (32)• “What should I have done after I got the fire report?”

– “What if” questions to get critiques on hypothetical actions (4)• “What if I ordered fire boundaries to be set?”

– “When/How” questions to explain domain rules (9)• “How do you determine what repair locker has jurisdiction?”

– “When/What/Is” questions evaluate conditions and relations (26)• “Is there a starboard fire pump on at 3:00?”

– More complex questions involving chaining and inference (14)• “How can I satisfy the preconditions for dewatering?”• “If I ordered smoke boundaries, what could I do then?”

25

Meta-GMO Example• “When is it appropriate to order firefighting?”• Question ECL 9300 “when action”

MGMO 9300FOR ECL 9300 “When Action”

LET action-ecl-number -> ActionECLIF g-clause(find, action(create, pending, ActionECL, _, _, _, _), GClauses) g-clause(justify, GClauses, Justifications)THEN answer(create, _, 9300, “When Action”,

[action-ecl-number <- ActionECL, justification <- Justification], miscellaneous-questions, JustificationNode)

END IFEND MGMO

26

In English(direct translation)

“There are two conditions under which you should order firefighting.

“First, when you receive a report that electrical and mechanical isolation has completed, you still need to extinguish the fire in that compartment, you have either active desmoked the compartment or do not need to active desmoke the compartment, and either there is no halon or halon has failed, find the best repair locker for that compartment, and order that repair locker to fight the fire in the compartment.

“Second, when you receive a report that halon has failed, you have either isolated the compartment or the compartment cannot be isolated, and you have either active desmoked the compartment or do not need to active desmoke the compartment, find the best repair locker for that compartment, and order that repair locker to fight the fire in the compartment.”

27

In English(intelligent translation)

“There are two things that might trigger ordering firefighting. The first is a report of electrical and mechanical isolation achieved, and the second is a report that halon has failed.

“The first case only applies when you need to extinguish a fire. You also need to have active desmoked the compartment, if necessary, and if the compartment has halon, it has to already have failed.

“In the second case, you must have active desmoked if necessary and isolated the compartment if possible.

“In both cases, you should send the best repair locker for the compartment to fight the fire.”

28

Meta-Graph Modification Operators (M-GMOs)

MGMO 9002 FOR ECL 9002 "Why Sub-Optimal Action?"LET action-node -> ActionNodeRULE 9002.1 "Explain why the action isn't correct."IF g-clause( find, action([create, modify], correct,

ActionNode.ecl, _, _, _, _), _, CorrectGClauses)AND roll-back(before, ActionNode, _)AND g-clause(justify-and-evaluate, CorrectGClauses, ActionNode, Justification)

THEN answer(create, _, 9002, "Why Sub-Optimal Action?",[action-node <- ActionNode, justification <- Justification], ActionNode, A)

END RULEEND MGMO

29

Power and Learnability

• A Gerona system responding to an incoming message from an agent can do so using an efficiently parallelizable algorithm.

• Total space complexity is O(n) and time complexity is low-order polynomial.

• GMO rules are PAC-learnable using “learning to take actions” paradigm, given certain constraints on length.

30

Current Research Direction

• Extend SCoT-DC/DC-Train Spoken Tutor to allow user-initiated tutoring.

• Approach is to map user-initiated questions in natural language to Gerona question classes

• QABLE for Story Comprehension Q/A (Grois and Wilkins, IJCAI-05 and ICML-05)

• Use Gerona domain model to constrain interpretations (Fried, et al, 2003)

31

Summary

• Ability to critique and learn is facilitated by agent KR&I – KN factorization, explicitness, modularity, being able to

reason over static and dynamic knowledge• Two examples:

– Minerva: separation of domain, task, and scheduling knowledge; use of Recursive Heuristic Classification for scheduling.

– Gerona: graph operators construct a dynamic task-centered representation