Integrating high- and low-level Expectations in Deliberative Agents Michele Piunti - [email protected]@istc.cnr.it Institute of.

Integrating high- and low-level Expectations in Deliberative Agents

Michele Piunti - [email protected] Institute of Cognitive Sciences and Technologies – ISTC, C.n.r.

João Gonçalves - [email protected] Superior Técnico - IST, Lisbon

1. Towards an integrated architecture

1. Expectations, Emotions, Anticipation

2. High and Low level Expectations

3. From deliberative to anticipatory agents

2. Design

1. Mental States

2. Subjective Expected Utilities

• ISTC: Beliefs and Goal

• IST: Emotivectors

3. Experimental comparision and discussion

4. Future works

Outline

Expectations, Emotions, Anticipation

Agent anticipation in partially observable environments can rely in :

• The ability to adjust quickly to changes (making quick decisions with limited information and bounded resources)

• Catching world dynamics and regularities

• Building representations of future states (Expectations)

• Affective competences (Emotions) via Behavioral and Mental changes:

– Long term : intention reconsideration, attention, resource allocation– Long term: appraisal, belief revision and learning

Explicit Vs. Implicit anticipatory representation:

• Expectations-enabled agents may or not make use of explicit representations of the future world state (and/or of the agent internal state).

• Agent architecture may compute, or not, explicit representation of these states

Anticipatory representations and Computational Models

Cognitive anticipatory agents can be endowed with expectations following different design approaches:

• Statistical learning, prediction mechanisms and component;

• Cognitive (and model driven) architectures:

• top down: Architecture for goal driven, affective and anticipatory agents

• bottom up: distribuited, drives, schema driven design (AKIRA), on line expectations beginning from perception and sensor motor control

Top down approach: from deliberative to anticipatory agents

• Deliberative (Goal directed) agents evaluate and chose between alternative courses of actions and their respective outcomes.

• Anticipatory competences require dealing with uncertainty and bounded knowledge about the future.

We do not introduce Expectations as a new primitive of the architecture but in terms of agent epistemic states (Beliefs) and motivational states (Goals).

Scenario

From deliberative to anticipatory agents

We introduce Expectations at many levels :

1. Weak, low level Expectations as moods and Mental States clustering attitudes in Reasoning.

2. Case based reasoning (means-end reasoning and action-selection processes)

3. Expectation ‘driven’ Deliberation:• Subjective Expected Utility (as a function of the agent’s beliefs and desires

(Bratmann88).• IST: Emotivectors (Matinho 2005)

• Surprise: due to (and signal of) experienced mismatch between ’what is expected’ and ’what is perceived’ (at a given level of representation)

• Expectations are ’prerequisites’ for surprise.• Affective States elicited by Surprise can be described in fuctional terms

beginning from expectations

Clustering Mental States in Reasoning

From the series of local observations of unexpected events stored in a short term memory, an agent controller periodically defines the mental state to adopt through a transition function.

Expectations here have a weak (low level) representation(e.g. negative expectation of risk, threats)

Mental States: fuctional role

• Cautiousness elicits mental and behavioral changes on Short term: alert, to become more vigilant, to look ahead, to check better while and before moving (prudence against threats); Long term augmenting the control or doing the, action in another less risky way, using alternatives in repertoires.

• Excitement: increasing the explorative activity for searching for the ’good’ events.

• Lack of surprises produces a special mood: boredom.

• The persistence of boredom can bring to curiosity, whose outcome is to shift from exploitation to exploration attitudes.

Pay OffsIntention reconsideration is a costly process

Space-Time payoffsResources allocation

To be cautious is advantegeous only in highlythreatful environment (see: energy-time)

Balance of resources is ‘environment dependent’

Subjective Expected UtilityForaging Task: drives and motivations to explore Location of Interest (LOI)are balanced:

DriveToAloi = self_confidence.rule_LOI_a * reward_a.getAvg();DriveToBloi = self_confidence.rule_LOI_b * reward_b.getAvg();DriveToCloi = self_confidence.rule_LOI_b * reward_b.getAvg();

1. Subjective Expected Utility: multiply subjective prevision to find valuables close to LOI and expected reward value (based on a k-history lenght items stored in a working memory).

2. Fully represented in domain of probability.

1. Meta-level planning: ε-Greedy strategies to select ‘best expected’ area (i.e. best SEU) to look for valuables at a.

We integrated 2 mechanismsK-lenght history buffers ◄► emotivector predictors

Analysis of the ISTC Architecture

• Emotivectors model expectations– Prediction– Desired Value– Evaluation

• Expectations identified in 3 levels– Based on beliefs about the world– Associated with mental/emotional states – Associated with goal achievement

Expectations in 3 levels

–Based on beliefs about the world

– Associated with mental/emotional states

– Associated with goal achievement

Modeling Beliefs with the Emotivector – Predictor

• Modeling Food Score– Scenario enhancement

Modeling Beliefs with the Emotivector – Affective Evaluation

• This feeling towards a certain kind of food reflects if its getting better or worst

Action Selection

• Previously the Agent used Subjective Expected Utility (SEU)– Just the predicted energy reward and

probability of success

• Now it as a affective bias on the SEU for a specific kind of food– Affective Subjective Expected Utility

Preliminary Results

• Seasons

• No Seasons

Expectations in 3 levels

– Based on beliefs about the world

– Associated with mental/emotional states

– Associated with goal achievement

Possible Integrations

• Associated with mental/emotional states – Emotivector monitors “rate/number” of positive events

• Anticipates mental state change

– In part done by the weight decay mechanism

• Associated with goal achievement– Very context dependent

References

• [Castelfranchi et al., 2006a] C. Castelfranchi, R. Falcone, and M. Piunti. Agents with anticipatory behaviors: To be cautious in a risky environment. In Proc. of European Conf. on Artificial Intelligence, Trento, Italy., 2006.

• [Castelfranchi et al., 2006b] C. Castelfranchi, R. Falcone, and M. Piunti. Developing anticipatory and affective competences in MAS. In Proceedings of InternationalWorkshop on Multi-agent Systems and Simulation (ISC 2006), Palermo, Italy., 2006.

• [Castelfranchi, 2005] C. Castelfranchi. Mind as an anticipatory device: For a theory of expectations. pages 258–276, 2005.

• [C.Martino and Paiva, 2005] C.Martino and A. Paiva. Synthetic emotivectors. In In proc. of Social Intelligence and Interaction in Animals, Robots and Agents AISB, 2005.

• [C.Martino and Paiva, 2006] C.Martino and A. Paiva. Using anticipation to create believable behaviour. In In proc. Of the AAAI 2006 conference, 2006.

• [Falcone et al., 2007] R. Falcone, M. Piunti, and C. Castelfranchi. Surprise as shortcut for anticipation: clustering mental states in reasoning. In In Proc. of IJCAI-07 (to appear), Hyberadad, India., 2007.