Eric Horvitz, Jack Breese, David Heckerman, Eric Horvitz, Jack Breese, David Heckerman, David Hovel, Koos Rommelse Microsoft Research Redmond, WA 98052.

Eric Horvitz, Jack Breese, David Heckerman, Eric Horvitz, Jack Breese, David Heckerman,

David Hovel, Koos RommelseDavid Hovel, Koos Rommelse

Microsoft Research Redmond, WA 98052Microsoft Research Redmond, WA 98052

Presented by Suman B. Pakala, Feng XuPresented by Suman B. Pakala, Feng XuCSCE 582 Fall 2002CSCE 582 Fall 2002

Instructor: Marco ValtortaInstructor: Marco Valtorta

The LumiThe Lumièère Project: Bayesian User re Project: Bayesian User Modeling for Inferring the Goals and Modeling for Inferring the Goals and

Needs of Software Users.Needs of Software Users.

IntroductionIntroduction

LumiLumièère started in 1993. Components re started in 1993. Components used in office 95, 97.used in office 95, 97.Constructing Bayesian user models for Constructing Bayesian user models for reasoning reasoning Gaining access to a stream of events from Gaining access to a stream of events from software applicationssoftware applicationsDeveloping a languageDeveloping a languageDeveloping persistent user profileDeveloping persistent user profileDevelopment of an overall architectureDevelopment of an overall architecture

Bayesian user modelingBayesian user modeling

Modeling the beliefs, intentions, goals and Modeling the beliefs, intentions, goals and needs of users.needs of users.

Goals are tasks or subtasks.Goals are tasks or subtasks.

Needs are either information or automated Needs are either information or automated actions.actions.

Influence Diagram

An influence diagram for providing intelligent assistance given uncertainty in a User’s background, goals and competency in working with a software application

Classes of evidenceClasses of evidence

SearchSearch

Focus of attentionFocus of attention

IntrospectionIntrospection

Undesired effectsUndesired effects

Inefficient command sequenceInefficient command sequence

Domain-specific syntactic and semantic Domain-specific syntactic and semantic contentcontent

A portion of a Bayesian user Model for inferring

A Model used for inferring the likelihood of the user needing assistance,considering profile information as well as observations of recent activity

Partial Bayesian user model for Partial Bayesian user model for ExcelExcel

Markov Model forMarkov Model for Temporal ReasoningTemporal Reasoning

Markov model for temporal reasoning assuming dependencies among the goalsOf a user in adjacent time periods. A persistent Profile variable influencesGoals and observations in all periods.

Temporal ReasoningTemporal Reasoning

Lag of event from present moment

Formulation of the temporal reasoning problem as a set of single-stageproblems. We directly assess conditional probabilities of actions as afunction of time that passed since actions occurred.

P(E_i_tn | Goal_t0) P(E_i_t-1 | Goal_t0) P(E_i_t0 | Goal_t0)

Bridging the gap between System Events and Bridging the gap between System Events and User ActionsUser Actions

User’s Actions that can be accessed:User’s Actions that can be accessed:Mouse and keyboard actionsMouse and keyboard actionsStatus of data structures in Excel filesStatus of data structures in Excel filesMenus being visitedMenus being visitedDialog boxes being opened and closedDialog boxes being opened and closedSelection of specific objects like charts, etc.Selection of specific objects like charts, etc.

They are modified into System events and They are modified into System events and modeled as:modeled as:Menu SurfingMenu SurfingMouse MeanderingMouse MeanderingMenu jitter, etc.Menu jitter, etc.

LumiLumièère Events Languagere Events Language

Why? – To make modeling flexibleWhy? – To make modeling flexiblePrimitives:Primitives: Rate(Rate(xi,txi,t)) One of(One of({{x1, ….,xnx1, ….,xn},},tt)) AllAll(({{x1, ….,xnx1, ….,xn},},tt)) SeqSeq((x1, ….,xn,tx1, ….,xn,t)) Dwell(Dwell(tt)) Example of an event: User dwelled for atleastExample of an event: User dwelled for atleast

tt seconds, following a scroll seconds, following a scroll

A high level architectural view of Lumière/ Excel

Architecture & Control PoliciesArchitecture & Control Policies

Architecture:Architecture:Events => Time stamped observationsEvents => Time stamped observationsObservations => Bayesian Model => P(user needs)Observations => Bayesian Model => P(user needs)If Query, P(events) + Bayesian term spotting => PP(needs)If Query, P(events) + Bayesian term spotting => PP(needs)Also, L(needs) => Control(automated assistance)Also, L(needs) => Control(automated assistance)

Control Policies:Control Policies:Pulsed strategyPulsed strategyEvent-driven control policy + trigger eventsEvent-driven control policy + trigger eventsAugmented pulsed approachAugmented pulsed approachDeferred analysisDeferred analysis

LumiLumièère / Excel In Operationre / Excel In Operation

Atomic events stream, Probability distribution over needs, AssistanceMonitoring agent, User interface for the prototype

When a query is made:When a query is made:

(a) (b)

(a) Inference based on actions. (b) Revised distribution after query is made

Autonomous display of assistanceAutonomous display of assistance

Actual application, Assistance monitoring agent, Offer of assistance

Beyond Real time assistanceBeyond Real time assistance

Information that is recommended for the user to review offline

LumiLumièère in the real world: Office Assistantre in the real world: Office Assistant

Broader but shallower models Broader but shallower models (compared with Lumi(compared with Lumièère/Excel)re/Excel)

Rich set of context variablesRich set of context variables

No persistent user profiling, No competency reasoningNo persistent user profiling, No competency reasoning

Small set of relatively atomic user actionsSmall set of relatively atomic user actions

Only the most recent events are consideredOnly the most recent events are considered

If words available, context and recent actions are not If words available, context and recent actions are not consideredconsidered

Inference results are available only when user requests Inference results are available only when user requests

(Autonomous assistance not employed)(Autonomous assistance not employed)

Current ResearchCurrent Research

Learning Bayesian models from user log dataLearning Bayesian models from user log data

Integrating new sources of eventsIntegrating new sources of events

Automated dialog for users to express goals and needsAutomated dialog for users to express goals and needs

Integrating vision and gaze trackingIntegrating vision and gaze tracking

Use of Value of Information computations Use of Value of Information computations (to engage user (to engage user in dialog to access costly information about activity and program in dialog to access costly information about activity and program state)state)