Transcript
© Ramesh Jain 1
Fall 2008
CS 212
Multimedia Systems and Applications
Instructor: Ramesh Jain
© Ramesh Jain 2
Projects Based on Research in Experiential Computing Laboratory
Based on the concept of EventWeb Two majore research projects
Personal Media Management Environment to Environment Connectivity (E2E)
© Ramesh Jain 3
Tim Berners-Lee thought:
Suppose all the information stored on computers everywhere were linked.
Suppose I could program my computer to create a space in which anything could be linked to anything.
Others – including Bush -- had that idea earlier but the technology was not ready.
© Ramesh Jain 4
That resulted in the Web
DocumentWeb Each node is a ‘Page’ or a document. Pages are linked through explicit referential links
© Ramesh Jain 5
Continuing the Evolution of the Web
Consider a Web in which each node Is an event Has informational as well as experiential data Is connected to other nodes using
Referential links Structural links Relational links Causal links
Explicit links can be created by anybody
This EventWeb is connected to other Webs.
© Ramesh Jain 6
Different Facets of an Event
What When Where Who
Why How
© Ramesh Jain 7
Tim
e
1- dimensional SpaceEvents Happen
© Ramesh Jain 8
Tim
e
1- dimensional SpaceEventWeb
© Ramesh Jain 9
Event Base
Atomic Event Entry
Stage 1: Event Ingestion
© Ramesh Jain 10
Stage 2: Event Clustering/Linking
Event Grouping, Linking, Assimilation
Event Base
Atomic Event Entry
© Ramesh Jain 11
Stage 3: Event Presentation, Navigation
Event Base
Event Presentation/ Navigation
Event Grouping, Linking, Assimilation
Atomic Event Entry
© Ramesh Jain 12
Multimedia Storytelling
Collect information about events All relevant events Appropriate information In right media
Stories are sequence of events. Stories/Novels Drama Movies
Present right event information using right media in right order.
© Ramesh Jain 13
Tim
e
1- dimensional SpaceMultimedia Storytelling
Text
Video
Photo
© Ramesh Jain 14
Experiential Media Management Environment
Event-based Photo-taking is an event Photos are taken of an event Albums represents events and objects Presentations are to share and communicate
experiences
First Photos then other media.
© Ramesh Jain 15
EMME Event Cycle
Event Base
Event Presentation/ Navigation
Event Grouping, Linking, Assimilation
Atomic Event Entry
EXIF
Features
Tags/ Context
© Ramesh Jain 16
Modern Cameras
Are more than ‘Camera Obscura’. Many sensors capture scene context and
store it along with intensity values. EXIF data is all metadata related to the Event.
And we throw away all this valuable information!!!
© Ramesh Jain 17
Examples:Photos are Assigned tagsonly based onEXIF
© Ramesh Jain 18
EMME Event Cycle
Event Base
Event Presentation/ Navigation
Event Grouping, Linking, Assimilation
Atomic Event Entry
EXIF
Features
Tags/ Context
Photo stream Segment.Event
Ontology
UserAnnot-ations
© Ramesh Jain 19
Photo Stream Segmentation
Definition: given a photo stream }{ ipP =
Segment
Event 1 Event 2 Event 3
© Ramesh Jain 20
EMME Event Cycle
Event Base
Event Presentation/ Navigation
Event Grouping, Linking, Assimilation
Atomic Event Entry
EXIF
Features
Tags/ Context
Photo stream Segment.Event
Ontology
UserAnnot-ations
StoryTelling
Search
Explore
© Ramesh Jain 21
Using EMME
Searching for photo I2R visit
Creating Albums: Professional in Singapore Tourism in Summer 2008
Telling stories What did I do in Summer 2008
Scenario: In December 2008, I have 20,000 pictures taken in 2008. How do I (semi-automatically) select 25 to send to My mother The uncle that I hate My personal friend My professional friend …
© Ramesh Jain 22
Sentient EventWeb
Use sensors to create events. This class
Infrastructure to create and link all such events. Index Search Present
Archive events and put them on the Web.
© Ramesh Jain 23
First Version: E2E
Person to Person Device to Device
Telephone Video Conferencing
Environment to Environment (E2E)
Communication Modes:
© Ramesh Jain 24
Event Cycle
Event Base
Event Presentation/ Navigation
Event Grouping, Linking, Assimilation
Atomic Event Entry
© Ramesh Jain 25
E2E
Com
mun
icat
ion
Natural interaction
Semantic interaction
Seamlessinteraction
Bi-directional
connectivity
Not depend on physical similarity
Handle privacy
Event-based architecture
Scalable architecture
Sensor abstraction
Multimodal information
No fixed application
Live and archived modes
© Ramesh Jain 26
Environment
Fixed objects Active Objects Events and Activities
Sensor to capture Objects and Events Cameras, microphones, pressure, RFID, …
Displays and Actuators Projectors, Displays, Speakers, …
© Ramesh Jain 27
Key Idea
Put enough sensors and devices to make mediation (sensors and devices) disappear.
System detects events and objects and sends appropriate data streams and displays/renders incoming streams on appropriate devices
Bring back Person to Person communication
© Ramesh Jain 28
System Architecture
ES 2
ES 1
ES 3
ES 4
ES 5
JSM 2
JSM 1
© Ramesh Jain 29
Node Architecture
Data Acquisition & Analysis
Situation Model
Env. Model
+Security settings
Event ServerMultimedia Database
Presentation
EventBase
Distribution/ Networking
Sensors
© Ramesh Jain 30
Event
e.g. E1_Cam2,E1_Mic2
e.g. Lip-motion=true, Volume=High
e.g. Env.1, Position: 102,145 Visitor Chair
e.g. Telemedicine App., Consultation room
e.g. Patient talking
Data Data Acquisition Acquisition & Analysis& Analysis
Env. Env. ModelModel
Situation Situation ModelModel
Event Detection Stages
© Ramesh Jain 31
Env. 1
Env. 4
JSM 2
JSM 1
Env. 5 Env. 3
Env. 2Joint Situation Model
© Ramesh Jain 32
Telemedicine Application
© Ramesh Jain 33
Telemedicine Application
© Ramesh Jain 34
Many Potential Projects: Photos Features to classify images:
Detect specific objects (sky, grass, mountains, …) and label images. Face detection and recognition. Person recognition – faces to other body information to person. Some simple ‘event’ (like a talk, or a wedding) recognition from
photographs. Using calendar information to assign information to images. Integrating Social Network Information In Personal Media Management Mobile App for uploading Event information. Can use Android APIs Select 1 or a few images from a group to represent the group. This
may be based on quality of images or content of images. Prepare a collage of images to represent a group. Use some theme for
selecting and displaying your images. Voice annotation to photos and use of speech recognition to convert
voice tags to text tags. Summarize an Event from Photos from many sources.
Many people shoot a photo for an event. How to create a good event representation from photos from multiple sources?
© Ramesh Jain 35
Many Potential Projects: E2E Tracking people using multiple sensors. Mobile E2E mode: switching from a mobile
phone to a much richer environment. Creating ‘virtual camera’ for heterogeneous
camera connections. Anywhere projector IP Audio streaming: Send and receive audio
by just specifying the IP address – without knowing the devices connected there.
top related