2004.11.30 - SLIDE 1IS 202 FALL 2004 Prof. Ray Larson Prof. Marc Davis UC Berkeley SIMS Tuesday and Thursday 10:30 am - 12:00 pm Fall 2004

SLIDE 1IS 202 FALL 2004 Prof. Ray Larson & Prof. Marc Davis UC Berkeley SIMS Tuesday and Thursday 10:30 am - 12:00 pm Fall SIMS 202: Information Organization and Retrieval Lecture 25: Mobile and Context-Aware Media SLIDE 2IS 202 FALL 2004 Todays Agenda Phone Check In Review of Last Time Metadata for Video Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 3IS 202 FALL 2004 Phone Check In Usage Problems Bluetooth turnoff Clogged phones Upload problems Other usability issues Next Steps Bluetooth fix Upload fix Roaming over break? SLIDE 4IS 202 FALL 2004 Todays Agenda Phone Check In Review of Last Time Metadata for Video Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 5IS 202 FALL 2004 Representing Video Streams vs. Clips Video syntax and semantics Ontological issues in video representation SLIDE 6IS 202 FALL 2004 Stream-Based Representation Makes annotation pay off The richer the annotation, the more numerous the possible segmentations of the video stream Clips Change from being fixed segmentations of the video stream, to being the results of retrieval queries based on annotations of the video stream Annotations Create representations which make clips, not representations of clips SLIDE 7IS 202 FALL 2004 Video Syntax and Semantics The Kuleshov Effect Video has a dual semantics Sequence-independent invariant semantics of shots Sequence-dependent variable semantics of shots SLIDE 8IS 202 FALL 2004 Ontological Issues for Video Video plays with rules for identity and continuity Space Time Person Action SLIDE 9IS 202 FALL 2004 Signal-Based Parsing Practical problem Parsing unstructured, unknown video is very, very hard Theoretical problem Mismatch between percepts and concepts SLIDE 10IS 202 FALL 2004 Why Keywords Dont Work Are not a semantic representation Do not describe relations between descriptors Do not describe temporal structure Do not converge Do not scale SLIDE 11IS 202 FALL 2004 Visual Language Advantages A language designed as an accurate and readable representation of time-based media For video, especially important for actions, expressions, and spatial relations Enables Gestalt view and quick recognition of descriptors due to designed visual similarities Supports global use of annotations SLIDE 12IS 202 FALL 2004 After Capture: Media Streams SLIDE 13IS 202 FALL 2004 Media Streams Features Key features Stream-based representation (better segmentation) Semantic indexing (what things are similar to) Relational indexing (who is doing what to whom) Temporal indexing (when things happen) Iconic interface (designed visual language) Universal annotation (standardized markup schema) Key benefits More accurate annotation and retrieval Global usability and standardization Reuse of rich media according to content and structure SLIDE 14IS 202 FALL 2004 Standards Overview Why do we need multimedia standards? Reliability Scalability Interoperability Layered architecture De facto standards Not legislated, but widely adopted De jure standards Legislated, but not necessarily widely adopted SLIDE 15IS 202 FALL 2004 MPEG Standards MPEG-1 Coding of moving pictures and associated audio for digital storage media at up to about 1.5 Mbit/sec MPEG-2 Generic coding of moving pictures and associated audio information MPEG Audio Layer-3 (MP3) Audio compression MPEG-4 Standardized technological elements enabling the integration of production, distribution, and content access paradigms SLIDE 16IS 202 FALL 2004 MPEG Standards MPEG-7 Describing the multimedia content data that supports some degree of interpretation of the informations meaning, which can be passed onto, or accessed by, a device or a computer code MPEG-21 A normative open framework for multimedia delivery and consumption for use by all the players in the delivery and consumption chain SLIDE 17IS 202 FALL 2004 Todays Agenda Phone Check In Review of Last Time Metadata for Video Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 18IS 202 FALL 2004 METADATA Traditional Media Production Chain M E T A D A T A PRE-PRODUCTIONPOST-PRODUCTIONPRODUCTIONDISTRIBUTION Metadata-Centric Production Chain SLIDE 19IS 202 FALL 2004 Mobile Media Questions How many of you use a camera? How many of you have your camera with you? How many of you use a cell phone? How many of you have your cell phone with you? SLIDE 20IS 202 FALL 2004 Moores Law for Cameras 2000 Kodak DC40 Nintendo GameBoy Camera $400 $ Kodak DX4900 SiPix StyleCam Blink SLIDE 21IS 202 FALL 2004 Capture+Processing+Interaction+Network SLIDE 22IS 202 FALL 2004 Camera Phones as Platform Media capture (images, video, audio) Programmable processing using open standard operating systems, programming languages, and APIs Wireless networking Personal information management functions Rich user interaction modalities Time, location, and user contextual metadata SLIDE 23IS 202 FALL 2004 Camera Phones as Platform In the first half of 2003, more camera phones were sold worldwide than digital cameras By 2008, the average camera phone is predicted to have 5 megapixel resolution Last month Casio and Samsung introduced 3.2 megapixel camera phones with optical zoom and photo flash There are more cell phone users in China than people in the United States (300 million) For 90% of the world their computer is their cell phone SLIDE 24IS 202 FALL 2004 Todays Agenda Review of Last Time Media Metadata Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 25IS 202 FALL 2004 Context SLIDE 26IS 202 FALL 2004 Todays Agenda Review of Last Time Media Metadata Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 27IS 202 FALL 2004 Time SLIDE 28IS 202 FALL 2004 Todays Agenda Review of Last Time Media Metadata Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 29IS 202 FALL 2004 Space SLIDE 30IS 202 FALL 2004 Todays Agenda Review of Last Time Media Metadata Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 31IS 202 FALL 2004 Social Space SLIDE 32IS 202 FALL 2004 Todays Agenda Phone Check In Review of Last Time Metadata for Video Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 33IS 202 FALL 2004 Campanile Inspiration SLIDE 34IS 202 FALL 2004 Mobile Media Metadata Idea Leverage the spatio-temporal context and social community of media capture in mobile devices Gather all automatically available information at the point of capture (time, spatial location, phone user, etc.) Use metadata similarity and media analysis algorithms to find similar media that has been annotated before Take advantage of this previously annotated media to make educated guesses about the content of the newly captured media Interact in a simple and intuitive way with the phone user to confirm and augment system-supplied metadata for captured media SLIDE 35IS 202 FALL 2004 MMM Demo Video SLIDE 36IS 202 FALL 2004 Campanile Scenario Gathering of Contextual Metadata User Verification Image Capture Gathered Data: Location Data Time Date Username Processing Results: Location City: Berkeley (100%) Day of Week: Saturday (100%) Location Name: Campanile (62%) Setting: Outside (82%) Verified Information: Location Name: Campanile (100%) Setting: Outside (100%) Metadata (and Media) Similarity Processing Metadata and Media Sharing and Reuse SLIDE 37IS 202 FALL 2004 From Context to Content Context When Date and time refined to qualitative semantic time periods (weekday, weekend, night, day, fall, spring, summer, winter) and recurring and special events Where CellID (and GPS and geocoded Bluetooth access points) refined to semantic placenames Who Cameraphone user Bluetooth co-presence Whom shared with Context SLIDE 38IS 202 FALL 2004 From Context to Content Context When Where Who Content Where is the location of the subject of the photo? Who is depicted in the photo? What objects are depicted in the photo? What are the depicted people doing in the photo (i.e., actions, activities, events)? SLIDE 39IS 202 FALL 2004 Space Time - Community Spatial Temporal Social spatial cohorts temporal cohorts familiar strangers SLIDE 40IS 202 FALL 2004 Space Time Social Space SPATIAL SOCIAL TEMPORAL SLIDE 41IS 202 FALL 2004 What is Location? SLIDE 42IS 202 FALL 2004 Camera Location vs. Subject Location Camera Location = Golden Gate Bridge Subject Location = Golden Gate Bridge Camera Location = Albany Marina Subject Location = Golden Gate Bridge SLIDE 43IS 202 FALL 2004 Kodak Picture Spot SLIDE 44IS 202 FALL 2004 Themed Kodak Picture Spot SLIDE 45IS 202 FALL 2004 Location Guesser Concept Calculate weighted sum of features Most recently visited location Most visited location by me in this CellID around this time Most visited location by me in this CellID Most visited location by others in this CellID around this time Most visited location by others in this CellID SLIDE 46IS 202 FALL 2004 Location Guesser Performance Exempting the occasions on which a user first enters a new location into the system, MMM guessed the correct location of the subject of the photo (out of an average of 36.8 possible locations): 100% of the time within the first four guesses 96% of the time within the first three guesses 88% of the time within the first two guesses 69% of the time as the first guess SLIDE 47IS 202 FALL 2004 Future of Mobile Imaging Integrate media capture and analysis at the point of capture and throughout the media lifecycle Leverage contextual metadata (spatial, temporal, social, etc.) about the capture and use of media content Design systems that incorporate human beings as functional components and aggregate user behavior Leveraging the spatio-temporal-social context of media capture and use can enable us to infer media content and solve key problems in mobile media management SLIDE 48IS 202 FALL 2004 Todays Agenda Phone Check In Review of Last Time Metadata for Video Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 49IS 202 FALL 2004 Discussion Questions Helen Kim on Dey on Context How would digitally-captured context information, such as the kind used in Dey's "Context Toolkit" differ from information gleaned through human social processes? What kinds of problems would arise from this difference? SLIDE 50IS 202 FALL 2004 Discussion Questions Helen Kim on Dey on Context The Context Toolkit collects information that is "sensed automatically in a particular situation. This contrasts with Lenat's top- down approach of creating context by using the CYC database to draw common sense conclusions. Which type of context- awareness would most benefit particular types of applications (e.g., tour guide or photo trip applications), and why? SLIDE 51IS 202 FALL 2004 Discussion Questions David Thaw on Graham on Time The use of time-clustering as the primary means of organization for a thumbnail-based navigation scheme implies several assumptions about users' photography behavior. These include: the assumption that photographs with high temporal proximity will necessarily be related that a single photograph within a cluster can be representative of it (specific to this study) that a baseline time differential constant for creating clusters will yield the same results if tested with longer time periods (above 24 hours). Will this assumption come into question more as storage size on digital imagers increases in size (and users may go longer periods before downloading them into a more permanent storage device)? The study conducted at Stanford seems to indicate at least that the first two assumptions can be justified. However, the study population did not include individuals with professions outside the academic sphere. Might this affect the survey response? SLIDE 52IS 202 FALL 2004 Discussion Questions David Thaw on Graham on Time Isn't it an inherent problem in user-testing (for organizational models) that people are accustomed to hierarchical organizational schema, and will therefore be more favorable of it? Would these results change if hierarchical folder structures had not been the primary means of organizing files on a computer? Would the hypothetical success of categorical based organizational structures (e.g., GMail) change users' opinions in the future? SLIDE 53IS 202 FALL 2004 Discussion Questions Kelly Bryant on Naaman on Geo- Referenced Photo Collections The algorithm used in PhotoCompas combines time and location to create a system of organizing digital photographs. In the evaluation it was noted that the precision of this algorithm breaks down when the collection contains images from a road trip which span a large location over a long period of time. Is there a way to improve this algorithm to consider the road trip situation, or flight, or any mode of transportation? SLIDE 54IS 202 FALL 2004 Discussion Questions Kelly Bryant on Naaman on Geo- Referenced Photo Collections In evaluating PhotoCompas, it was found that the location hierarchy used was useful in some cases and confusing in others. This confusion was related to the user's knowledge of a given area. How can the system better account for a user's knowledge of a location? Would a system like this be useful for our phone projects and our metadata frameworks? SLIDE 55IS 202 FALL 2004 Discussion Questions Bruce Rinehart on Davis on Mobile Media Metadata If you were to replace CellID with GPS information what variables might you use in the location guesser? Would that change the weights? SLIDE 56IS 202 FALL 2004 Discussion Questions Bruce Rinehart on Davis on Mobile Media Metadata What other spatial, temporal, or social variables might be useful in determining content? SLIDE 57IS 202 FALL 2004 Discussion Questions Bruce Rinehart on Davis on Mobile Media Metadata "Ask the Researcher: Marc, could you describe the process through which you and your team arrived at the array of subalgorithms for the location guesser? SLIDE 58IS 202 FALL 2004 Todays Agenda Phone Check In Review of Last Time Metadata for Video Mobile Media Context Time Space Social Space Mobile Media Metadata Discussion Questions Action Items for Next Time SLIDE 59IS 202 FALL 2004 Protg Workshop II Today 2:00 pm 4:00 pm SLIDE 60IS 202 FALL 2004 Readings for Next Time Looking Backward Looking Forward: Future of Information Systems Emanuel Goldberg, electronic document retrieval, and Vannevar Bush's Memex (Buckland) As We May Think (Bush) Memex II (Bush) Memex Revisited (Bush)