Pat Langley School of Computing and Informatics Arizona State University Tempe, Arizona USA Extending the I CARUS Cognitive Architecture Thanks to D. Choi,

Pat LangleyPat Langley

School of Computing and InformaticsSchool of Computing and InformaticsArizona State UniversityArizona State University

Tempe, Arizona USATempe, Arizona USA

Extending the IExtending the ICARUSCARUS Cognitive ArchitectureCognitive Architecture

Thanks to D. Choi, T. Konik, U. Kutur, D. Nau, S. Ohlsson, S. Rogers, and D. Shapiro for Thanks to D. Choi, T. Konik, U. Kutur, D. Nau, S. Ohlsson, S. Rogers, and D. Shapiro for their many contributions. This talk reports research partly funded by grants from DARPA their many contributions. This talk reports research partly funded by grants from DARPA IPTO, which is not responsible for its contents. IPTO, which is not responsible for its contents.

The IThe ICARUSCARUS Architecture Architecture

IICARUSCARUS is a theory of the human cognitive architecture that is a theory of the human cognitive architecture that posits: posits:

It shares the assumptions with other cognitive architectures like It shares the assumptions with other cognitive architectures like Soar (Laird et al., 1987) and ACT-R (Anderson, 1993). Soar (Laird et al., 1987) and ACT-R (Anderson, 1993).

1.1. Short-term memories are distinct from long-term stores Short-term memories are distinct from long-term stores

2.2. Memories contain modular elements cast as symbolic structuresMemories contain modular elements cast as symbolic structures

3.3. Long-term structures are accessed through pattern matchingLong-term structures are accessed through pattern matching

4.4. Cognition occurs in retrieval/selection/action cyclesCognition occurs in retrieval/selection/action cycles

5.5. Learning involves monotonic addition of elements to memoryLearning involves monotonic addition of elements to memory

6.6. Learning is incremental and interleaved with performanceLearning is incremental and interleaved with performance

Distinctive Features of IDistinctive Features of ICARUSCARUS

However, IHowever, ICARUSCARUS also makes assumptions that distinguish it from also makes assumptions that distinguish it from these architectures:these architectures:

Some of these tenets also appear in Bonasso et al.’s (2003) 3T, Some of these tenets also appear in Bonasso et al.’s (2003) 3T, Freed’s APEX, and Sun et al.’s (2001) CLARION. Freed’s APEX, and Sun et al.’s (2001) CLARION.

1.1. Cognition is grounded in perception and action Cognition is grounded in perception and action

2.2. Categories and skills are separate cognitive entitiesCategories and skills are separate cognitive entities

3.3. Short-term elements are instances of long-term structuresShort-term elements are instances of long-term structures

4.4. Inference and execution are more basic than problem solvingInference and execution are more basic than problem solving

5.5. Skill hierarchies are learned in a cumulative mannerSkill hierarchies are learned in a cumulative manner

Cascaded Integration in ICascaded Integration in ICARUSCARUS

IICARUSCARUS adopts a adopts a cascadedcascaded approach to integration in which approach to integration in which lower-level modules produce results for higher-level ones. lower-level modules produce results for higher-level ones.

conceptual inference

skill execution

problem solving

learning

Like other unified cognitive architectures, ILike other unified cognitive architectures, ICARUSCARUS incorporates a incorporates a number of distinct modules. number of distinct modules.

IICARUSCARUS’ Memories and Processes’ Memories and Processes

Long-TermLong-TermConceptualConceptual

MemoryMemory

Short-TermShort-TermBeliefBelief

MemoryMemory

Short-TermShort-TermGoal MemoryGoal Memory

ConceptualConceptualInferenceInference

SkillSkillExecutionExecution

PerceptionPerception

EnvironmentEnvironment

PerceptualPerceptualBufferBuffer

Problem SolvingProblem SolvingSkill LearningSkill Learning

MotorMotorBufferBuffer

Skill RetrievalSkill Retrievaland Selectionand Selection

Long-TermLong-TermSkill MemorySkill Memory

Each concept is defined in terms of other concepts and/or percepts.Each concept is defined in terms of other concepts and/or percepts.

Each skill is defined in terms of other skills, concepts, and percepts.Each skill is defined in terms of other skills, concepts, and percepts.

IICARUS CARUS interleaves its long-term memories for concepts and skills.interleaves its long-term memories for concepts and skills.

Hierarchical Structure of MemoryHierarchical Structure of Memory

conceptsconcepts

skillsskills

IICARUS CARUS interleaves its long-term memories for concepts and skills.interleaves its long-term memories for concepts and skills.

Hierarchical Structure of MemoryHierarchical Structure of Memory

Each concept is defined in terms of other concepts and/or percepts.Each concept is defined in terms of other concepts and/or percepts.

Each skill is defined in terms of other skills, concepts, and percepts.Each skill is defined in terms of other skills, concepts, and percepts.

conceptsconcepts

skillsskills

Basic IBasic ICARUSCARUS Processes Processes

conceptsconcepts

skillsskills

Concepts are matched bottom up, starting from percepts.Concepts are matched bottom up, starting from percepts.

Skill paths are matched top down, starting from goals.Skill paths are matched top down, starting from goals.

IICARUS CARUS matches patterns to recognize concepts and select skills.matches patterns to recognize concepts and select skills.

IICARUSCARUS Interleaves Execution and Problem Solving Interleaves Execution and Problem Solving

Executed plan

Problem

??

Skill Hierarchy

Primitive Skills

ReactiveExecution

impasse?

ProblemSolving

yesyes

nono

This organization reflects the psychological distinction between automatized This organization reflects the psychological distinction between automatized and controlled behavior. and controlled behavior.

Means-Ends Problem Solving in IMeans-Ends Problem Solving in ICARUSCARUS

Solve(G)Solve(G) Push the goal literal G onto the empty goal stack GS. Push the goal literal G onto the empty goal stack GS. On each cycle, On each cycle, If the top goal G of the goal stack GS is satisfied, If the top goal G of the goal stack GS is satisfied, Then pop GS. Then pop GS. Else if the goal stack GS does not exceed the depth limit, Else if the goal stack GS does not exceed the depth limit, Let S be the skill instances whose heads unify with G. Let S be the skill instances whose heads unify with G. If any applicable skill paths start from an instance in S, If any applicable skill paths start from an instance in S, Then select one of these paths and execute it. Then select one of these paths and execute it. Else let M be the set of primitive skill instances that have not already failed in which G is an effect. Else let M be the set of primitive skill instances that have not already failed in which G is an effect. If the set M is nonempty, If the set M is nonempty, Then select a skill instance Q from M. Then select a skill instance Q from M.

Push the start condition C of Q onto goal stack GS. Push the start condition C of Q onto goal stack GS. Else if G is a complex concept with the unsatisfied subconcepts H and with satisfied subconcepts F,Else if G is a complex concept with the unsatisfied subconcepts H and with satisfied subconcepts F, Then if there is a subconcept I in H that has not yet failed, Then if there is a subconcept I in H that has not yet failed, Then push I onto the goal stack GS. Then push I onto the goal stack GS. Else pop G from the goal stack GS and store information about failure with G's parent. Else pop G from the goal stack GS and store information about failure with G's parent. Else pop G from the goal stack GS. Else pop G from the goal stack GS. Store information about failure with G's parent. Store information about failure with G's parent.

Previous versions of IPrevious versions of ICARUSCARUS have used means-ends analysis, which has been have used means-ends analysis, which has been observed repeatedly in humans, but it differs from standard variants in that it observed repeatedly in humans, but it differs from standard variants in that it interleaves backward chaining with execution. interleaves backward chaining with execution.

Learning from Problem SolutionsLearning from Problem Solutions

operates whenever problem solving overcomes an impasse;operates whenever problem solving overcomes an impasse;

incorporates only information available from the goal stack;incorporates only information available from the goal stack;

generalizes beyond the specific objects concerned; generalizes beyond the specific objects concerned;

depends on whether chaining involved skills or concepts; depends on whether chaining involved skills or concepts;

supports cumulative learning and within-problem transfersupports cumulative learning and within-problem transfer. .

IICARUSCARUS incorporates a mechanism for learning new skills that: incorporates a mechanism for learning new skills that:

This skill creation process is fully interleaved with means-ends This skill creation process is fully interleaved with means-ends analysis and execution.analysis and execution.

Learned skills carry out forward execution in the environment Learned skills carry out forward execution in the environment rather than backward chaining in the mind. rather than backward chaining in the mind.

Forward Search and Mental SimulationForward Search and Mental Simulation

However, in some domains, humans carry out forward-chaining However, in some domains, humans carry out forward-chaining search with methods like progressive deepening (de Groot, 1978).search with methods like progressive deepening (de Groot, 1978).

In response, we are adding a new module to IIn response, we are adding a new module to ICARUSCARUS that: that:

performs mental simulation of a single trajectory consistent performs mental simulation of a single trajectory consistent with its stored hierarchical skills; with its stored hierarchical skills;

repeats this process to find a number of alternative paths from repeats this process to find a number of alternative paths from the current environmental state; the current environmental state;

selects the path that produces the best outcome to determine selects the path that produces the best outcome to determine the next primitive skill to execute. the next primitive skill to execute.

We refer to this memory-limited search method as We refer to this memory-limited search method as hierarchical hierarchical iterative sampling iterative sampling (Langley, 1992). (Langley, 1992).

A Trace of Iterative SamplingA Trace of Iterative Sampling

A Trace of Iterative SamplingA Trace of Iterative Sampling

A Trace of Iterative SamplingA Trace of Iterative Sampling

A Trace of Iterative SamplingA Trace of Iterative Sampling

A Trace of Iterative SamplingA Trace of Iterative Sampling

A Trace of Iterative SamplingA Trace of Iterative Sampling

Challenges in Lookahead SearchChallenges in Lookahead Search

To support such mental simulation in ITo support such mental simulation in ICARUSCARUS, we must first: , we must first:

extend its representation to associate beliefs with states; extend its representation to associate beliefs with states;

use expected values to guide selection of desirable paths.use expected values to guide selection of desirable paths.

We want a single mechanism that will let IWe want a single mechanism that will let ICARUSCARUS handle all of handle all of these situations. these situations.

This should be easy for some domains, but it must also: This should be easy for some domains, but it must also:

reason about environments that change on their own;reason about environments that change on their own;

operate in settings that involve other goal-directed agents.operate in settings that involve other goal-directed agents.

More on Mental SimulationMore on Mental Simulation

We must address other issues to make this idea operational: We must address other issues to make this idea operational:

determine the depth of lookahead and number of samples; determine the depth of lookahead and number of samples;

ensure reasonable diversity among the sampled paths; ensure reasonable diversity among the sampled paths;

explain when problem solvers chain backward and forward; explain when problem solvers chain backward and forward;

use the results of forward search to drive skill acquisition. use the results of forward search to drive skill acquisition.

Answering these questions will let IAnswering these questions will let ICARUSCARUS provide a more provide a more complete theory of human problem solving. complete theory of human problem solving.

Learning from Undesirable OutcomesLearning from Undesirable Outcomes

Despite their best efforts, humans sometimes take actions that Despite their best efforts, humans sometimes take actions that produce undesired results. produce undesired results.

We plan to model learning from such outcomes in IWe plan to model learning from such outcomes in ICARUSCARUS by: by:

identifying conditions on path that, if violated, avoid result; identifying conditions on path that, if violated, avoid result;

carry out search to find another path that would violate it; carry out search to find another path that would violate it;

analyze the alternative path to learn skills that produce it; analyze the alternative path to learn skills that produce it;

store the new skills so as to mask older, problematic ones. store the new skills so as to mask older, problematic ones.

Learning from such counterfactual reasoning is an important Learning from such counterfactual reasoning is an important human ability. human ability.

Plans for EvaluationPlans for Evaluation

We propose to evaluate these extensions to IWe propose to evaluate these extensions to ICARUSCARUS on two on two different testbeds: different testbeds:

a simulated urban driving environment that contains other a simulated urban driving environment that contains other vehicles and pedestrians; vehicles and pedestrians;

a mobile robot that carries out joint activities with humans a mobile robot that carries out joint activities with humans to achieve shared goals. to achieve shared goals.

Both dynamic environments should illustrate the benefits of Both dynamic environments should illustrate the benefits of mental simulation and counterfactual learning. mental simulation and counterfactual learning.

Concluding RemarksConcluding Remarks

includes hierarchical memories for concepts and skills;includes hierarchical memories for concepts and skills;

interleaves conceptual inference with reactive execution;interleaves conceptual inference with reactive execution;

resorts to problem solving when it lacks routine skills; resorts to problem solving when it lacks routine skills;

learns such skills from successful resolution of impasses. learns such skills from successful resolution of impasses.

IICARUSCARUS is a unified theory of the cognitive architecture that: is a unified theory of the cognitive architecture that:

However, we plan to extend the framework so it can support: However, we plan to extend the framework so it can support:

forward-chaining search via repeated mental simulation; forward-chaining search via repeated mental simulation;

learning new skills through counterfactual reasoning. learning new skills through counterfactual reasoning.

These will let IThese will let ICARUSCARUS more fully model human cognition. more fully model human cognition.

End of PresentationEnd of Presentation

IICARUSCARUS Concepts for In-City Driving Concepts for In-City Driving

((in-rightmost-lane ?self ?clane)((in-rightmost-lane ?self ?clane) :percepts (:percepts ( (self ?self) (segment ?seg) (self ?self) (segment ?seg)

(line ?clane segment ?seg))(line ?clane segment ?seg)) :relations ((driving-well-in-segment ?self ?seg ?clane):relations ((driving-well-in-segment ?self ?seg ?clane)

(last-lane ?clane) (last-lane ?clane) (not (lane-to-right ?clane ?anylane))))(not (lane-to-right ?clane ?anylane))))

((driving-well-in-segment ?self ?seg ?lane)((driving-well-in-segment ?self ?seg ?lane) :percepts ((self ?self) (segment ?seg) (line ?lane segment ?seg)):percepts ((self ?self) (segment ?seg) (line ?lane segment ?seg)) :relations ((in-segment ?self ?seg) (in-lane ?self ?lane):relations ((in-segment ?self ?seg) (in-lane ?self ?lane)

(aligned-with-lane-in-segment ?self ?seg ?lane)(aligned-with-lane-in-segment ?self ?seg ?lane)(centered-in-lane ?self ?seg ?lane)(centered-in-lane ?self ?seg ?lane)(steering-wheel-straight ?self)))(steering-wheel-straight ?self)))

((in-lane ?self ?lane)((in-lane ?self ?lane) :percepts (:percepts ( (self ?self segment ?seg) (line ?lane segment ?seg dist ?dist))(self ?self segment ?seg) (line ?lane segment ?seg dist ?dist)) :tests (:tests ( (> ?dist -10) (<= ?dist 0)))(> ?dist -10) (<= ?dist 0)))

Representing Short-Term Beliefs/GoalsRepresenting Short-Term Beliefs/Goals

(current-street me A)(current-street me A) (current-segment me g550)(current-segment me g550)(lane-to-right g599 g601)(lane-to-right g599 g601) (first-lane g599)(first-lane g599)(last-lane g599)(last-lane g599) (last-lane g601)(last-lane g601)(at-speed-for-u-turn me)(at-speed-for-u-turn me) (slow-for-right-turn me)(slow-for-right-turn me)(steering-wheel-not-straight me)(steering-wheel-not-straight me) (centered-in-lane me g550 g599)(centered-in-lane me g550 g599)(in-lane me g599)(in-lane me g599) (in-segment me g550)(in-segment me g550)(on-right-side-in-segment me)(on-right-side-in-segment me) (intersection-behind g550 g522)(intersection-behind g550 g522)(building-on-left g288)(building-on-left g288) (building-on-left g425)(building-on-left g425)(building-on-left g427)(building-on-left g427) (building-on-left g429)(building-on-left g429)(building-on-left g431)(building-on-left g431) (building-on-left g433)(building-on-left g433)(building-on-right g287)(building-on-right g287) (building-on-right g279)(building-on-right g279)(increasing-direction me)(increasing-direction me) (buildings-on-right g287 g279)(buildings-on-right g287 g279)

((in-rightmost-lane ?self ?line)((in-rightmost-lane ?self ?line) :percepts:percepts ((self ?self) (line ?line))((self ?self) (line ?line)) :start ((last-lane ?line)):start ((last-lane ?line)) :subgoals ((driving-well-in-segment ?self ?seg ?line))) :subgoals ((driving-well-in-segment ?self ?seg ?line)))

((driving-well-in-segment ?self ?seg ?line)((driving-well-in-segment ?self ?seg ?line) :percepts:percepts ((segment ?seg) (line ?line) (self ?self))((segment ?seg) (line ?line) (self ?self)) :start ((steering-wheel-straight ?self)):start ((steering-wheel-straight ?self)) :subgoals ((in-segment ?self ?seg):subgoals ((in-segment ?self ?seg)

(centered-in-lane ?self ?seg ?line)(centered-in-lane ?self ?seg ?line)(aligned-with-lane-in-segment ?self ?seg ?line)(aligned-with-lane-in-segment ?self ?seg ?line)(steering-wheel-straight ?self)))(steering-wheel-straight ?self)))

((in-segment ?self ?endsg)((in-segment ?self ?endsg) :percepts:percepts ((self ?self speed ?speed) (intersection ?int cross ?cross)((self ?self speed ?speed) (intersection ?int cross ?cross)

(segment ?endsg street ?cross angle ?angle))(segment ?endsg street ?cross angle ?angle)) :start ((in-intersection-for-right-turn ?self ?int)):start ((in-intersection-for-right-turn ?self ?int)) :actions:actions ((((steer 1)))steer 1)))

IICARUSCARUS Skills for In-City Driving Skills for In-City Driving

A Successful Means-Ends TraceA Successful Means-Ends Trace

(ontable A T)

(on B A)

(on C B)

(hand-empty)

(clear C)

(unst. C B) (unstack C B) (clear B)

(putdown C T)

(unst. B A) (unstack B A) (clear A)

(holding C) (hand-empty)

(holding B)

A

B

C B

A C

initial stateinitial state

goalgoal