CALO Learning Overview AIC Machine Learning Discussion Group 26 October 2004 with material shamelessly pilfered from previous presentations by: • Tom Dietterich/Leslie Kaelbling (Transfer Learning) • Colin Evans (Task Setup) • Lynn Voss (Task Discussion) • David Martin (Task Fulfillment)
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CALO Learning Overview
AIC Machine Learning Discussion Group
26 October 2004
with material shamelessly pilfered from previous presentations by:
• Tom Dietterich/Leslie Kaelbling (Transfer Learning)• Colin Evans (Task Setup)• Lynn Voss (Task Discussion)• David Martin (Task Fulfillment)
The Learning Picture (Terminology)
learningalgorithm
learneddevice
naïve Bayesmaximum entropyC4.5k-meanslearning by being told…
• Task Setup• recognize implications of starting a new task
• information harvesting, scheduling setup, dossier preparation
• Task Discussion• integrate results of interaction between humans
and CALOs into task management• meeting understanding
• Task Fulfillment• support user in performing tasks
• scheduling, procurement
Task Setup: Information Harvesting
Task Setup, Colin Evans
Learning in Task Setup:Information Harvesting
Component Algorithm Input Device Output
SemEx desktop contents
extraction rules
facts about and relations between documents, email, etc.
DEX home pages
information extractor (CRF)
information about people, including contact information, areas of expertise, and social groups
Activity Clusterer
bi-directional clustering
emailbox activity (task) groupings of email
Task Setup: Scheduling Setup
Task Setup, Colin Evans
Learning in Task Setup:Scheduling Setup
Component Algorithm Input Device Output
SpeechAct email message
email type classifier
email type
Activity Classifier
email message
activity classifier activity of message
emailME email message
information extractor
meeting constraints
Task Manager advice policies for determining meeting constraints (e.g., start time constraints, participants)
Task Setup: Dossier Preparation
Task Setup, Colin Evans
Learning in Task Setup:Dossier Preparation
Component Algorithm Input Device Output
TaskTracer desktop operations
logging relations between tasks (activities) and desktop operations
Naive Bayes
desktop operation
task (activity) classifier
Activity Classifier
email message
activity (task) classifier
activity (task) of message
Task Discussion: Meeting Room
CAMEOCAMEO
Frame
Frame
Fram
e
User w/ headset
User w
/ he
adse
t
User w/ headset
SMART Board
Stereo Camera
Frame includes:•Stereo Camera•(IR - Blue Eyes Camera)•Array Microphones•All attached around a user’s laptop
Task Discussion, Lynn Voss
Task Discussion: Architecture
Meeting Recorder Architecture
Meeting Playback System
Meeting Record / MOKB
Body Tracker3D-Gesture
Face TrackerFace Recog.
Activity Recog.
Speakerlocalization
Affect Recog.
Head, eye,gaze tracker
Object Recog.Video & Array Microphone
Classifiers DialogueManager
Suite
FSDB
MS ProjectAgent
Task Setup
Purchase Request
Tracking DataIntegrator
Meeting DossierAgendaParticipant List
Handwriting2D Gesture
Charter
Digital InkRecognizers
NTP
OAAFacilitator
Raw Data Capture
Audio Server
InstrumentedText Notes &Power Point
CAMEOPanoramic
MPEG encoder
Whiteboard’sStereo
CameraFrame
SMARTBoard
Digital Ink
CloseTalkingSpeech
TasksMilestones
OOV Words
OOVAgentSuite
MS ProjectFile
End Pointer
Transcription
Prosody
TopicSupporting Docs
Meeting Room IRIS Data Store
Meeting Room
IRIS Data Store
UserFeedbackLoop
Multi-parser
MSBITS
Offline Analysis Suite
Agenda Topics
Phases Action Items
Roles Rough Sum.
Tracking DataIntegrator &Audio Server
MeetingBrowser
Task Discussion, Lynn Voss
Learning in Task Discussion:Project Plan Capture
Component Algorithm Input Device Output
Agent Sphinx
multi-modal learning
speech, gestures, writing
words (complete model)
instruction chart types
2D gesture recognizer
2D gesture recognizer
(written) symbols
speech recognizer
speech recognizer speaker ID
handwriting recognizer
handwriting recognizer
words
Learning in Task Discussion:Physical Awareness
Component Algorithm Input Device Output
CAMEO face recognizer person ID
coarse activity recognizer
sitting, standing, walking, …
BodyTracker articulated tracker participant movement
3D gesture recognizer
pointing, orientation
object recognizer objects
affect recognizer affect
Frame speaker localizer speaker location
speaker detection speaker ID
high-level activity recognizer
Learning in Task Discussion:Meeting Awareness
Component Algorithm Input Device Output
Meeting Analysis Suite
topic tracker topic shifts
agenda tracker agenda segments
action item identifier
action items
decision identifier decisions
meeting phase segmenter
meeting phases
role tracker participant role
acoustic model speech transcript
Task Discussion:Meeting Record Content
• Raw Streams• raw audio, raw video, whiteboard strokes, text notes, PPT
presentations
• Low Level Events• out-of-vocabulary words• participant locations with torso and body positions• participant activities (coarse)• who spoke to whom• recognized affects• recognized words & symbols on the whiteboard• word transcripts• new participants• new chart types• new 2D & 3D gestures
Task Discussion:Meeting Record Content
• High Level Events• project plan (task names, durations, milestones)
• participants, including entrance/exit
• when each agenda item was discussed
• topics/subtopics and relevance to agenda
• action items, including responsible parties, deadlines
• decisions and proposers; alternative proposals and reasons for/against
advice policies for scheduling, relaxation, reminder
AutoMinder reinforcement learning
reminder strategy
procedural learner
revision procedure
memory-based learner
case-based learning
scheduling procedure
Task Fulfillment: Purchasing
Select Type of Item
LearnVendors
Choose Vendors &Define Requirements
Get Quotes
Get User Selectionsand/or Confirmation
Refine Purchase ProcedureWrap
VendorsAdd
Vendors
Execute Purchase Procedure
Select Type of Item
RelaxQuery
Task Fulfillment, David Martin
Learning in Task Fulfillment:Purchasing
Component Algorithm Input Device Output
KnowIt product vendor sites
product ontologies
Fetch learning by being told
Web site Web site wrapper product information
Prometheus Mediator
model of new source
LOQR C4.5 unsatisfiable query
decision rules relaxed query
Tailor learning by being told
revised procedure
Task Manager
advice policies for procurement
The CALO Test
Main Claim: CALO performs well and, through learning, performs even better.
• The Test• AP-style exam• Administered regularly throughout the year• Must show general improvement overall.• Only learning in the wild counts.
The CALO Test
CALO 2.0
CALO 2.1
CALO 2.2
CALO 2.3
CALO 3.0
Tes
t S
core
improvement due to learning
improvement due to engineering
totalimprovement
due toengineeringand learning
Situated Learning
CALO is a cognitive assistant.
• Task Manager (the heartbeat of CALO)• controls what CALO does• situation assessment• workflow management
• Knowledge Machine/Query-Update Manager• what CALO knows• CALO ontology
Situated Learning
CALO is deployed in the office environment.
• IRIS• suite of integrated desktop applications • ontology-driven architecture• provides instrumentation and automation facilities
Learning Issues
CALO is not (yet) a robust, enduring system.
• much in-the-wild learning is not truly online• concept drift/shift is not addressed• disparate sources are not coordinated• new tasks require human engineering• ontology changes require lobotomies• learning is component-specific