Susan epstein at ibm csig speaker series

FORR A Cognitive Architecture for Expertise

Susan L. Epstein

The Graduate Center and Hunter College of The City University of New York

Executive summary

•  FORR (FOr the Right Reasons) is an architecture •  FORR-based systems develop expertise •  FORR-based systems learn quickly from problem solving experience •  FORR-based systems are built from

§  World knowledge (descriptives) §  Good reasons for making decisions (Advisors)

•  FORR-based systems can restructure their decision process •  FORR confirmed cognitively plausible on human subjects

2 Background • FORR • Applications

People, agents and expertise

•  People are our best model of intelligent agents §  Some human approaches work well on really hard problems §  Their methods are robust to imperfect data §  They pursue multiple goals

•  If an agent is to collaborate with people, it is necessary to understand human decision processes

•  A cognitively plausible agent simulates significant human characteristics •  Expert does things faster and better than the rest of us [D’Andrade 1990]


Characteristics of human experts •  They work in a domain (set of related problem classes) •  They satisfice = make good enough decisions •  They entertain multiple decision-making heuristics [Ratterman

& Epstein 1995] •  They access multiple representations •  They do situation-based reasoning [Klein & Calderwood 1991]

•  Human experts are made, not born

4

Learning is the hallmark of human intelligence

Background • FORR • Applications

Agent architecture •  Postulates general principles •  System shell for diverse domains

•  Requirements for cognitive plausibility §  Display reasonable behavior

•  Make obvious decisions •  Avoid obvious errors •  Solve easy problems quickly

§  Balance accuracy and speed §  Be robust to error §  Tolerate and reason with inconsistent, incomplete, noisy data §  Learn


Do forever Sense the world Select an action Execute that action

Fundamental issues for a learning architecture

•  What is there to learn? •  From whom to learn? •  When to learn? •  How to learn? •  How to use learned knowledge to make decisions? •  How to manage reality and noise?


Cornerstones of FORR’s pragmatic approach

•  Expertise is learned, that is, it develops with experience •  Easy questions should have fast (reactive) answers •  Satisfice = make good enough decisions in a simplified model of a complex

world (and recover if need be) •  Exploit synergy inherent in multiplicity

§  Multiple domain-dependent representations §  Multiple domain-dependent heuristics for decision making §  Multiple learning methods

•  Maintain flexibility §  Decouple data, learning methods, and decision methods §  Restructure its own decision-making process

•  Transparency: explain decisions

FORR's building blocks are descriptives and Advisors


Multiple representations

•  Descriptive = a shared data object §  Value provided on demand §  Defined with functions that determine how and when to update it §  Value may be learned

•  Although a descriptive has a single representation, many descriptives can represent the same world state

•  Examples: X-O-blank empty/occupied lines on the board


Multiple ways to use knowledge

•  Operationalization = how to use a data object •  Although a descriptive has a single representation, it can be

operationalized in many ways Ways to reason about the empty/occupied squares

Calculate possible actions Predict opponent's move

Ways to reason about the lines Report a result Finish a winning line Block your opponent’s winning line Create a fork Plan a win on a specific line


An Advisor operationalizes descriptives •  Implements a class-independent, action-selection rationale •  Limitedly-rational (resource-limited) procedure •  Input: state of the world + descriptives + possible actions •  Output: comments whose strengths express intensity of support or

opposition to individual actions (or sets of actions) •  Domain-specific

< Advisor, action, strength>

Advisor

current state possible actions relevant descriptives


Often, Advisors disagree

11

O X X O Panic

(prevent immediate loss)

Worried (prevent long-range loss)

Victory (win!)

And rely on learned descriptives •  Good openings •  Endgame play •  Strategies

…


More about Advisors

•  Advisors have different properties §  Some are always right §  Some need more time to decide §  Some would like to make a sequence of decisions, not just one

•  Comments are opinions from the perspective of the Advisor's rationale §  On a single action

do x x is better than y don’t do z do x or y x is a 10, y is an 8, but z is a –3

§  On an (unordered or fully or partially ordered) set of actions do x and y do p and then q do p and then do q and r


FORR (FOr the Right Reasons) •  Premise: synergy among domain-specific rationales solves problems •  Descriptives isolate representation from reasoning •  Advisor hierarchy

§  Tier 1: correct, quick, pre-sequenced §  Tier 2: reactive plan rationales §  Tier 3: voting among heuristics based on their comment strengths and learned weights

13

<AdvisorA, action2, 10> <AdvisorA, action4, 8> <AdvisorA, action7, 6> <AdvisorB, action2, 7> <AdvisorB, action3, 9> <AdvisorC, action1, 9> <AdvisorC, action2, 7> <AdvisorC, action3, 9> <AdvisorC, action7, 9> …

Voting

For Advisor i and action j

argmaxj

diwicij∑


The FORR decision cycle

14

take action yes

Tier 1: Reaction from perfect knowledge

Victory T-11 T-1n …

Decision? no


state actions descriptives


15

take action yes



Decision?

begin plan yes

Tier 2: Plans triggered by situation recognition

no

T-21 T-22 T-2m …

Decision?




16

take action yes



Decision?

begin plan yes no

T-32 T-31 T-3k … … Tier 3: Heuristic reactions

Voting take action

Tier 2: Plans triggered by situation recognition

no

T-21 T-22 T-2m …

Decision?



How to develop a problem solver

17

•  Specialize FORR with domain knowledge §  Problem classes §  Advisors §  Descriptives with learning methods

•  To solve a class of problems robustly, FORR learns §  Descriptives’ values §  Rationales’ relative utility §  New Advisors §  How to reorganize tier 3

Domain knowledge

FORR

FORR-based problem solver

Learned problem solver

Problem class

Experience

WARNING: problem solving often provides noisy data


FORR-based single agents

18

•  Hoyle learned to play 19 two-person, perfect-information, finite-board games as well or better than human / machine expert [Epstein, 2001]

•  Ariadne learned to navigate efficiently in grid worlds, despite perceptual limitations and no map [Epstein, 1995]

•  ACE learned to solve constraint satisfaction problems and rediscovered the Brélaz heuristic [ Epstein & Freuder, 2005]

•  SemaFORR: controls an autonomous search-and-rescue robot [Epstein, Schneider, Ozgelen, Munoz, Costantino, Sklar & Parsons, 2012]


Lessons learned

19

•  Reactive plans work well •  Elimination of inaccurate heuristics produces substantial speedup •  Lazy descriptive computation also provides speedup •  Self-awareness supports transparency •  Advisor weights may have problem-stage context •  Weight learning has subtle pitfalls (example extraction) •  Autonomous restructuring must balance accuracy against risk •  Sometimes it is more efficient not to reason at all


FORR-based collaborating agents

20

•  Co-FORR: 5 collaborating agents for 2D park design [Epstein, 1998] •  FORRSooth: learned to conduct a spoken dialogue with a library patron

who orders books [Epstein, Passonneau, Gordon, & Ligorio, 2012] •  SemaFORR: controls autonomous search-and-rescue robot team [Epstein,

Aroor, Evanusa, Sklar & Parsons, 2015]

Each new domain poses new challenges whose solution strengthens FORR


FORR-based results

21

•  PhD theses §  Shih on learning multiple behavior sequences, 2000 §  Lock on learning multiple plans from behavior sequences, 2003 §  Petrovic on weight learning for multiple Advisors, 2008 §  Ligorio on learning to select attributes, 2011 §  Li on representation and exploitation of multiple complex relationships, 2011 §  Yun on parallelization of multiple solvers, 2013 §  Osisek on application of multiple relationships in recommendation (in progress) §  Aroor on reactive planning for multiple robots (in progress)

•  Applications to bioinformatics (with Dr. Lei Xie) §  Protein-protein interaction networks §  Virtual drug screening


Take home message

To develop expertise

FORR learns to harness the synergy of

multiplicity in representation and reasoning


Acknowledgements

We gratefully acknowledge the support of The National Science Foundation CUNY’s High Performance Computing Center

Continued thanks to my collaborators

Gene Freuder Rebecca Passonneau Rick Wallace Lei Xie Elizabeth Sklar Simon Parsons

and a host of undergraduate and graduate students with whom I continue to learn

23

Selected references

•  Epstein, S. L. 2001. Learning to Play Expertly: A Tutorial on Hoyle. In Machine Learning in Game Playing

•  Epstein, S. L. 1998. Pragmatic Navigation: Reactivity, Heuristics, and Search. Artificial Intelligence, 100 (1-2): 275-322.

•  Epstein, S. L., E. C. Freuder and M. Wallace 2005. Learning to Support Constraint Programmers. Computational Intelligence 21(4): 337-371.

•  Epstein, S. L., R. J. Passonneau, T. Ligorio and J. Gordon. 2012. Data Mining to Support Human-Machine Dialogue for Autonomous Agents. In Proceedings of Agents and Data Mining Interaction (ADMI2011).

•  Epstein, S.L., Aroor, A., Evanusa, M., Sklar, E.I., Simon, S. 2015. Navigation with Learned Spatial Affordances. In Proceedings of CogSci 2015.

http://www.cs.hunter.cuny.edu/~epstein/

24

Susan epstein at ibm csig speaker series

Technology

expertise forrbased

decision process forr

experience forrbased

multiple learning methods

executive summary forr

multiple goals

multiple ways

obvious decisions