L C SL C S 1 Larry Rudolph An Oxygenated Presentation Manager Larry Rudolph Oxygen Workshop, January, 2002.

1

L C S

Larry Rudolph

An Oxygenated Presentation Manager

Larry Rudolph

Oxygen Workshop, January, 2002

2

L C S

Larry Rudolph

Goals & Overview

• Integrate Many Oxygen Technologies

• Application Driven– Use an application that we understand

– Personally use often

– Would help if were more human-centric

• Develop Architectural Infrastructure– Exposes new requirements

• Identify some important, basic issues

• Infrastructure

• Critique of Presentation Manager– What is wrong with it

3

L C S

Larry Rudolph

Application Scenario

4

L C S

Larry Rudolph

An Oxygen Application

• Components– Input

* Vision

* Speech

* Touch

– Processing

* Changing configuration

– Output

* Projector

* Handheld

* Archive

5

L C S

Larry Rudolph

Camera watching laser point on screen

* Camera Challenges• Inexpensive ones have wrong focal length

• Alignment issues – use edge of screen, display pattern, figure out from what is known to be visible– We ended up displaying a pattern of concentric circles

• Relative size of laser point depends on distance– Beyond ten feet, had to use only certain types of lasers

– Could slow-down camera and let pixels saturate (too complicated)

* Camera Interface• Click at point (x,y)

– Hold laser at same location for 5 seconds

• Select horizontal line ( (x1,y1) , (x1,y2) )– Sweep laser back and forth, line is diameter of ellipse

• Select object centered at point (x,y)– Sweep laser in circle, point is center of circle

• Previous or Next – Click in left (right) 1/8 of screen

6

L C S

Larry Rudolph

Microphone listening to speaker

• Microphone– Many technologies; Current approach: ipaq

* Push to speak

• Audio server on ipaq– Detects start and stop

* Best results when human pushes to start and releases to stop

– Audio wave file sent to Galaxy speech system

• Galaxy output (HTTP Post cmd) to CGI-script Server

• Action are– Powerpoint

* Next slide, Previous slide, Goto slide #n, Goto slide named <xxx>

* Next item, Previous item, Goto item #n, Goto item named <xxx>

* Next animations, previous animation, goto animation #n

– Presentation* Start presentation <name>, End presentation, Pause presentation

– Session* Initialize Camera, test microphone

7

L C S

Larry Rudolph

Speaker controlling presentation via ipaq• Ipaq output (HTTP Post cmd) to CGI-script Server

– Same actions as from speech server

• Action are– Powerpoint

* Next slide, Previous slide, Goto slide #n, Goto slide named <xxx>

* Next item, Previous item, Goto item #n, Goto item named <xxx>

* Next animations, previous animation, goto animation #n

– Presentation* Start presentation <name>, End presentation, Pause presentation

– Session* Initialize Camera, test microphone

• Handheld (Ipaq) display– GUI generated from speechbuilder grammar

– List of slides, items per slides

* Currently use ad-hoc solution where power-point sends lists to ipaq. Need more automatic solution

8

L C S

Larry Rudolph

Processing – controlling session

• Easy to embed movies into presentation– Add rule that is invoked on slide #n

* Switch output producer from powerpoint to media player

• Remove interrupting technologies– Dynamically disconnect any input / output source

• All done via core language

9

L C S

Larry Rudolph

Output to projector, handheld, archive

• Unlimited number of video / audio output producers – E.g. powerpoint just one producer of output

– At any time, each output device has an associated producer

* This producer can receive input from several producers

• Handheld has proxy – To reduce bandwidth to ipaq

– Current slide, list of slides, list of commands

• Archive– Each slide shown, audio (from a different microphone) sent to archive

* Currently just gif of current slide

CORE: Communication Oriented Routing Environment

Part of ORG (Oxygen Research Group)

AssumptionsActuators / Sensors (I/O) in the environment

Many are shared by apps & users

Many are flaky / faulty

“User” does not know much about them

Environment, application, users desires change over time

11

L C S

Larry Rudolph

An Oxygen Application

• Interconnected Collection of Stuff

• Who specifies the stuff?– I don’t know, but its mostly virtual stuff

– Many layers of abstraction

* “Don’t ask, its turtles all the way down”

• Two main layers of programming– Professionals

– Users, e.g. grandmother

12

L C S

Larry Rudolph

Communications-Oriented Programs

• Connecting the (virtual) stuff done by user– Home stereo / theater analogy

* Plug Stuff together; unplug it if doesn’t work

* Don’t like it, unplug it

• Device drivers, services, clients, don’t know to whom or to what they connect– In client/server model,

* server knows a lot about the client,

* the client knows even more about the server

• Extend Unix Pipes

13

L C S

Larry Rudolph

CORE

Larry Bear’sCORE

Physical Devices

App

App

Programs (Processes)

Larry Bear CORE

Other COREs

14

L C S

Larry Rudolph

Message Flow

• Messages flow between nodes & core– Core is both language and router

• Within Core Router, some messages– are interpreted and may trigger actions

– other messages get routed to other nodes

• Request-Reply message strategy– Even number of messages

– No reply within time period, means error

15

L C S

Larry Rudolph

CORE Language Elements

• Four elements1. Nodes,

2. Links,

3. Messages,

4. Rules

• Features– Interpreted Language

– Statement is a message & reply

– Each element has an inverse

16

L C S

Larry Rudolph

Nodes – Specify via INS

CORE

Cam = [device=web-cam; location=518;…]

PTRvision = [device=process; OS=Linux;File=Laser Vision, ..]Slide

SpeechPresentation

Speech

CommandSpeech

LaserVision

Nodehandler = (nickname, specifier)

17

L C S

Larry Rudolph

Node Statement Handler

• When ‘node’ message arrives– Verified for correctness (statements allowed)

– Routed to Node Manager (just another node)

• Node Manager– INS lookup, verifies if allowed, creates if needed

– Creates core thread to manage communication with node

– Bookkeeping & reply message with handle/error

18

L C S

Larry Rudolph

CORE

SlideSpeech

PresentationSpeech

CommandSpeech

LaserVision

Lcamera,vision = (Cam,PTRvision)

Links

19

L C S

Larry Rudolph

Link Statement Handler

• Message routed to ‘link’ manager

• Two queries to node mng for thread cntl

• Message to thread controller of source node– Specifying destination thread controller

• Message to thread controller of dest node– Specifying source thread controller

• Bookkeeping & reply message handler/error

20

L C S

Larry Rudolph

CORE

SlideSpeech

PresentationSpeech

CommandSpeech

LaserVision

Messages flow over the links

Next Slide!

Messages

21

L C S

Larry Rudolph

Message Handling

• Messages can be encrypted

• Core statement messages have fixed format

• Everything else is data message

• Each node thread has two unbounded buffers– Core to node & Node to core

• Logging, rollback, fault-tolerance

22

L C S

Larry Rudolph

CORE

SlideSpeech

PresentationSpeech

CommandSpeech

LaserVision

RULES: (trigger,action)

Questions

( MESSQuestion , Lslide,lcd -- & Lslide,qlcd )

Rules

Questions

Questions

23

L C S

Larry Rudolph

Rule Statement Handler

• ( trigger , consequence )

• Both are “event sets”

• Eight basic events:+Node, -Node, +Link, -Link

+Message, -Message, +Rule, -Rule

• Event set is a set of events

• Trigger is true when events are true

• Consequence makes events true

24

L C S

Larry Rudolph

Rules – A link is a rule

• A message event is of form (node, message specifier)

( message specifier , node )

– Message came from or going to node

• A link (x,y) is just shorthand for the rule: +( x , m ) ( - (x, m) , +(m , y) )

If a message m arrives at node x, then make that event false (remove the message) and make the event of m arriving at y from core true.

25

L C S

Larry Rudolph

Rules – Access Control Lists

• An access control list is just a rule

• When messages arrive at node, if they arrive from valid node, then allowed to continue to flow.

• Modifying access control lists is just adding or removing rules.

26

L C S

Larry Rudolph

Rules

• Rule statement gets sent to rule manager

• Event set is just another shorthand for rules

• Rule manager sends command to trigger node thread that tells it about the consequence

• Rules are reversible

27

L C S

Larry Rudolph

Reversibility

• Each statement is invertible (reversible)

• If there is an error in the application specification, then can undo it all.

• General debugging is possible with reversible rules and message flow

Critique of Presentation Manager

29

L C S

Larry Rudolph

Vision / Gesture Recognition

• Laser Pointer– Great for drawing attention to content

* Audience is primary consumer

* Secondary use to control presentation

– But it is not a mouse

* Semantics are tied to slide context

* Differs from Intelligent-room use

– Small number of identified gestures

* Gestures easily punctuated

– Low computational overhead

* Soon will be handled with a H21

30

L C S

Larry Rudolph

Critique of Vision / Gesture Recognition

• Laser Pointer– Great for drawing attention to content

* Cheap technology but mostly distracting

* Too shaky, imprecise

– But it is not a mouse

* More awkward to use than mouse

* Another gadget to hold in the hand, button to identify, batteries to maintain

– Small number of identified gestures

* There are better ways of drawing attention to slide content

* I rarely use it and don’t like it when others do

– Low computational overhead

* Dumb vs Intelligent Device Discussion

31

L C S

Larry Rudolph

Speech Recognition

• Initially seems like great idea– Speaker is already speaking, so can use it to control presentation

• Want passive, intelligent listener– Not a dialog

– No “prompt” :: alienating distraction

• Want no mistakes– For dialog, better to guess than ignore

– For us, high cost for incorrect guess

– Most words are not relevant to speech system

• More trouble than it is worth– But may be good for real-time search of content

32

L C S

Larry Rudolph

More useful aspect – Output modalities

• Presenter has put the time and effort into the production– Simplier is better

• Audience has harder task– Understand material being presented

– Record thoughts, impressions, connections

– Filter for later review

– Process in real-time

– Keep-up with presentation

– Do all this with minimal distractions

• Output modalities– Content for live audience

– Content for speaker (superset of audience)

– Content for retrieval

* Correlate notes with content

33

L C S

Larry Rudolph

Record and correlate notes with presentation