1
L C S
Larry Rudolph
An Oxygenated Presentation Manager
Larry Rudolph
Oxygen Workshop, January, 2002
2
L C S
Larry Rudolph
Goals & Overview
• Integrate Many Oxygen Technologies
• Application Driven– Use an application that we understand
– Personally use often
– Would help if were more human-centric
• Develop Architectural Infrastructure– Exposes new requirements
• Identify some important, basic issues
• Infrastructure
• Critique of Presentation Manager– What is wrong with it
3
L C S
Larry Rudolph
Application Scenario
4
L C S
Larry Rudolph
An Oxygen Application
• Components– Input
* Vision
* Speech
* Touch
– Processing
* Changing configuration
– Output
* Projector
* Handheld
* Archive
5
L C S
Larry Rudolph
Camera watching laser point on screen
* Camera Challenges• Inexpensive ones have wrong focal length
• Alignment issues – use edge of screen, display pattern, figure out from what is known to be visible– We ended up displaying a pattern of concentric circles
• Relative size of laser point depends on distance– Beyond ten feet, had to use only certain types of lasers
– Could slow-down camera and let pixels saturate (too complicated)
* Camera Interface• Click at point (x,y)
– Hold laser at same location for 5 seconds
• Select horizontal line ( (x1,y1) , (x1,y2) )– Sweep laser back and forth, line is diameter of ellipse
• Select object centered at point (x,y)– Sweep laser in circle, point is center of circle
• Previous or Next – Click in left (right) 1/8 of screen
6
L C S
Larry Rudolph
Microphone listening to speaker
• Microphone– Many technologies; Current approach: ipaq
* Push to speak
• Audio server on ipaq– Detects start and stop
* Best results when human pushes to start and releases to stop
– Audio wave file sent to Galaxy speech system
• Galaxy output (HTTP Post cmd) to CGI-script Server
• Action are– Powerpoint
* Next slide, Previous slide, Goto slide #n, Goto slide named <xxx>
* Next item, Previous item, Goto item #n, Goto item named <xxx>
* Next animations, previous animation, goto animation #n
– Presentation* Start presentation <name>, End presentation, Pause presentation
– Session* Initialize Camera, test microphone
7
L C S
Larry Rudolph
Speaker controlling presentation via ipaq• Ipaq output (HTTP Post cmd) to CGI-script Server
– Same actions as from speech server
• Action are– Powerpoint
* Next slide, Previous slide, Goto slide #n, Goto slide named <xxx>
* Next item, Previous item, Goto item #n, Goto item named <xxx>
* Next animations, previous animation, goto animation #n
– Presentation* Start presentation <name>, End presentation, Pause presentation
– Session* Initialize Camera, test microphone
• Handheld (Ipaq) display– GUI generated from speechbuilder grammar
– List of slides, items per slides
* Currently use ad-hoc solution where power-point sends lists to ipaq. Need more automatic solution
8
L C S
Larry Rudolph
Processing – controlling session
• Easy to embed movies into presentation– Add rule that is invoked on slide #n
* Switch output producer from powerpoint to media player
• Remove interrupting technologies– Dynamically disconnect any input / output source
• All done via core language
9
L C S
Larry Rudolph
Output to projector, handheld, archive
• Unlimited number of video / audio output producers – E.g. powerpoint just one producer of output
– At any time, each output device has an associated producer
* This producer can receive input from several producers
• Handheld has proxy – To reduce bandwidth to ipaq
– Current slide, list of slides, list of commands
• Archive– Each slide shown, audio (from a different microphone) sent to archive
* Currently just gif of current slide
CORE: Communication Oriented Routing Environment
Part of ORG (Oxygen Research Group)
AssumptionsActuators / Sensors (I/O) in the environment
Many are shared by apps & users
Many are flaky / faulty
“User” does not know much about them
Environment, application, users desires change over time
11
L C S
Larry Rudolph
An Oxygen Application
• Interconnected Collection of Stuff
• Who specifies the stuff?– I don’t know, but its mostly virtual stuff
– Many layers of abstraction
* “Don’t ask, its turtles all the way down”
• Two main layers of programming– Professionals
– Users, e.g. grandmother
12
L C S
Larry Rudolph
Communications-Oriented Programs
• Connecting the (virtual) stuff done by user– Home stereo / theater analogy
* Plug Stuff together; unplug it if doesn’t work
* Don’t like it, unplug it
• Device drivers, services, clients, don’t know to whom or to what they connect– In client/server model,
* server knows a lot about the client,
* the client knows even more about the server
• Extend Unix Pipes
13
L C S
Larry Rudolph
CORE
Larry Bear’sCORE
Physical Devices
App
App
Programs (Processes)
Larry Bear CORE
Other COREs
14
L C S
Larry Rudolph
Message Flow
• Messages flow between nodes & core– Core is both language and router
• Within Core Router, some messages– are interpreted and may trigger actions
– other messages get routed to other nodes
• Request-Reply message strategy– Even number of messages
– No reply within time period, means error
15
L C S
Larry Rudolph
CORE Language Elements
• Four elements1. Nodes,
2. Links,
3. Messages,
4. Rules
• Features– Interpreted Language
– Statement is a message & reply
– Each element has an inverse
16
L C S
Larry Rudolph
Nodes – Specify via INS
CORE
Cam = [device=web-cam; location=518;…]
PTRvision = [device=process; OS=Linux;File=Laser Vision, ..]Slide
SpeechPresentation
Speech
CommandSpeech
LaserVision
Nodehandler = (nickname, specifier)
17
L C S
Larry Rudolph
Node Statement Handler
• When ‘node’ message arrives– Verified for correctness (statements allowed)
– Routed to Node Manager (just another node)
• Node Manager– INS lookup, verifies if allowed, creates if needed
– Creates core thread to manage communication with node
– Bookkeeping & reply message with handle/error
18
L C S
Larry Rudolph
CORE
SlideSpeech
PresentationSpeech
CommandSpeech
LaserVision
Lcamera,vision = (Cam,PTRvision)
Links
19
L C S
Larry Rudolph
Link Statement Handler
• Message routed to ‘link’ manager
• Two queries to node mng for thread cntl
• Message to thread controller of source node– Specifying destination thread controller
• Message to thread controller of dest node– Specifying source thread controller
• Bookkeeping & reply message handler/error
20
L C S
Larry Rudolph
CORE
SlideSpeech
PresentationSpeech
CommandSpeech
LaserVision
Messages flow over the links
Next Slide!
Messages
21
L C S
Larry Rudolph
Message Handling
• Messages can be encrypted
• Core statement messages have fixed format
• Everything else is data message
• Each node thread has two unbounded buffers– Core to node & Node to core
• Logging, rollback, fault-tolerance
22
L C S
Larry Rudolph
CORE
SlideSpeech
PresentationSpeech
CommandSpeech
LaserVision
RULES: (trigger,action)
Questions
( MESSQuestion , Lslide,lcd -- & Lslide,qlcd )
Rules
Questions
Questions
23
L C S
Larry Rudolph
Rule Statement Handler
• ( trigger , consequence )
• Both are “event sets”
• Eight basic events:+Node, -Node, +Link, -Link
+Message, -Message, +Rule, -Rule
• Event set is a set of events
• Trigger is true when events are true
• Consequence makes events true
24
L C S
Larry Rudolph
Rules – A link is a rule
• A message event is of form (node, message specifier)
( message specifier , node )
– Message came from or going to node
• A link (x,y) is just shorthand for the rule: +( x , m ) ( - (x, m) , +(m , y) )
If a message m arrives at node x, then make that event false (remove the message) and make the event of m arriving at y from core true.
25
L C S
Larry Rudolph
Rules – Access Control Lists
• An access control list is just a rule
• When messages arrive at node, if they arrive from valid node, then allowed to continue to flow.
• Modifying access control lists is just adding or removing rules.
26
L C S
Larry Rudolph
Rules
• Rule statement gets sent to rule manager
• Event set is just another shorthand for rules
• Rule manager sends command to trigger node thread that tells it about the consequence
• Rules are reversible
27
L C S
Larry Rudolph
Reversibility
• Each statement is invertible (reversible)
• If there is an error in the application specification, then can undo it all.
• General debugging is possible with reversible rules and message flow
Critique of Presentation Manager
29
L C S
Larry Rudolph
Vision / Gesture Recognition
• Laser Pointer– Great for drawing attention to content
* Audience is primary consumer
* Secondary use to control presentation
– But it is not a mouse
* Semantics are tied to slide context
* Differs from Intelligent-room use
– Small number of identified gestures
* Gestures easily punctuated
– Low computational overhead
* Soon will be handled with a H21
30
L C S
Larry Rudolph
Critique of Vision / Gesture Recognition
• Laser Pointer– Great for drawing attention to content
* Cheap technology but mostly distracting
* Too shaky, imprecise
– But it is not a mouse
* More awkward to use than mouse
* Another gadget to hold in the hand, button to identify, batteries to maintain
– Small number of identified gestures
* There are better ways of drawing attention to slide content
* I rarely use it and don’t like it when others do
– Low computational overhead
* Dumb vs Intelligent Device Discussion
31
L C S
Larry Rudolph
Speech Recognition
• Initially seems like great idea– Speaker is already speaking, so can use it to control presentation
• Want passive, intelligent listener– Not a dialog
– No “prompt” :: alienating distraction
• Want no mistakes– For dialog, better to guess than ignore
– For us, high cost for incorrect guess
– Most words are not relevant to speech system
• More trouble than it is worth– But may be good for real-time search of content
32
L C S
Larry Rudolph
More useful aspect – Output modalities
• Presenter has put the time and effort into the production– Simplier is better
• Audience has harder task– Understand material being presented
– Record thoughts, impressions, connections
– Filter for later review
– Process in real-time
– Keep-up with presentation
– Do all this with minimal distractions
• Output modalities– Content for live audience
– Content for speaker (superset of audience)
– Content for retrieval
* Correlate notes with content
33
L C S
Larry Rudolph
Record and correlate notes with presentation
