Contextual Augmentation of Ontology for Recognizing Sub-Events Setareh Rafatirad and Ramesh Jain [email protected] , [email protected] ICSC2011, Sep 19-21, Palo Alto Donald Bren School of Information and Computer Science University of California Irvine 1
Contextual Augmentation of Ontology for Recognizing Sub-Events Setareh Rafatirad and Ramesh Jain [email protected], [email protected] ICSC2011, Sep 19-21,
Dec 16, 2015
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Contextual Augmentation of Ontology for Recognizing Sub-EventsSetareh Rafatirad and Ramesh [email protected] , [email protected], Sep 19-21, Palo AltoDonald Bren School of Information and Computer ScienceUniversity of California Irvine
2
•People query their personal photos very frequently.
University of California Irvine
3
Visual Data
Human Semantics
User Queries are High-Level
University of California Irvine
4
… add more “semantic” tags
SubEvent: Visiting Ghost Town During Trip to Arizona
University of California Irvine
•People are interested in events and SubEvents!•Subevents as recall cues •Very important in retrieval•What is the current trend?
5
Problem
Given photos <p1,…,pn> with EXIF metadata for an event E, we partition them into its subevents <se1,…,sen>.
University of California Irvine
6
But photos contain limited data
Common meta-data include:◦Time◦Camera Params (exposure time,
aperture, focal length, ISO..)◦More recently, GPS Tags.
How to tackle this limitation?◦External Data Sources
Sensory Data Web 2.0 Ontologies
University of California Irvine
7
Visual Data
Human Semantics
high relevance to smart-phone applications.
Contextual model
visual content features only when required
Multi-modal context
University of California Irvine
8
OutlineIntroductionRelated WorkProblem FormulationSystem OverviewExperimental Results (the fun
part )Summary
University of California Irvine
9
From Conceptual Model to Contextual Model
Conceptual Model• Multi-layered abstract domain
model• Flexibility for multiple
Classifications• Avoid replication• Leads to complex processing• Mereological,Relative/Absolute
temporal, spatial RelationsContextual Model
• Visual Semantics • Contextual Semantic
• e.g. time and location of events
University of California Irvine
Hindu
Hindu
10
Inspired by Binford Hierarchical Object Model
University of California Irvine
11
R-OntologyTRIP scenarioIndividual: <domain#event_n> Types: <domain#Staying> Facts: <upper#SUPEREVENT> <domain#Trip-to-Beijing>, <upper#P-POI-LOCATION> <domain#place_k>, <upper#P-T-BEFORE> <domain#Checking-out_j>, <upper#P-T-AFTER> <domain#Checking-in_j> Individual: <domain#place_k> Types: <domain#POI> Facts: <upper#HAS-CATEGORY> “hotel”, {domain} <upper#HAS-LATITUDE> 21.13, {R-Onto} <upper#HAS-LONGITUDE> 79.06 {R-Onto} <upper#HAS-BOUNDARY> <domain#bound-k>
Individual: <domain#bound-k> Types: <upper#Bounding-Box> Facts: <upper#S-CONTAINS> <domain#coor_k1>, <upper#S-CONTAINS> <domain#coor_k2>
Individual: <domain#coor-k1> Types: <upper#Coordinates> Facts: <upper#HAS-LATITUDE> 39.908094, <upper#HAS-LONGITUDE> 116.444083
University of California Irvine
12
Related WorkObject-based systems
◦Describing geometric objects [Kokarand et.al.]
◦Pixel-based ontology◦Description rather than Recognition!
COMM,ABC
Event-based systems◦Only ontology creation e.g. F-Model,E-
Model [Schertp et.al, Westerman et.al.]
◦Activity recognition
University of California Irvine
13
Problem Formulation Given photos P :< p1,...,pn > for an event E , we
partition them into its subevents < se1 , . . . , sem >.
R-Ontology◦
How can Or be employed for partitioning P?
I: set of instances of the classes in domain ontologyCIv: contextR: set of relationships between the instances
University of California Irvine
Ontology Store(upper
and domain)
R-Ontology (-ies)
Ontology instance Modifier
Modification
Particular Context
Ontology Augmentation
Input Context
Instantiation
Content Descriptor Extractor
Imaging Feature
ExtractorMetadata Extractor
(time, coordinate, camera parameters)
Feature Extraction
.
.
.
.
.
.
.
.
.
.
.
.
Agglomerative SpatioTemporal Clustering
WW
W
<place, locType, ambient
condition, etc>
P: <p1,…,pn>
Media
Photos
Filtering(Content and
context features)
14
Request
System Overview
University of California Irvine
15
My friend’s Indian Wedding, [02-12-09 02-13-09], Nagpur-India
University of California Irvine
16
Taking portrait
Wedding party
My friend’s marriage
Groom arrival
Indian Ceremony
My friend’s Indian Wedding, [02-12-09 02-13-09], Nagpur-India
University of California Irvine
17
Visiting forbidden city
Having Dinner
Ordering Dinner
Serving Dinner
My Trip, [07-06-10 07-12-10], Beijing-China
Shopping at twins mall
University of California Irvine
18
SummaryOntologies for Trip,WeddingSub-Events as important TagsFuture work
◦Extend the employment of context and content features of photos
◦Sensors on smart phones
University of California Irvine
19
Q & AAnd Thanks for listening.Contact Information:
◦[email protected]◦[email protected]
University of California Irvine
20
Back up Slides
University of California Irvine
21
Hindu Wedding
Variable Context
Constant ContextWeb and other sources
{srafatir,jain}@ics.uci.edu
22
Vacation-TripConstant Context
Variable Context
Web and other sourcesVacation Trip
Process
Professional
Activity
Eating
Lunch
Shopping
Visiting
has-su
beve
nt
subC
lass
Of
has-subevent
has-
subeve
nt
has-
subeve
nt
has-processingUnit
has-subevent
Dinner
subClassOf
subC
lass
Of
Conference
Process-L
Serving food
has-subevent
Ordering food
beforerestaurant ,cafe
Mall,plaza,shopping center
has-locationType
Process-Schedule
has-processingUnit
hotel,university
Trip to Beijing
Process
Professional Activity
Eating
Lunch
Shopping
Visiting
has-subevent
subC
lass
Of
has-subevent
has-
sub
event
has-
subeve
nt
has-processingUnit
has-subevent
Dinner
subClassOf
subC
lass
Of
ACM Confernce
Process-L
Serving food
Ordering food
before
restaurant ,cafe
Mall,plaza,shopping center
has-locationType
Process-Schedul
ehas-processingUnit
hotel,university
Keynote: 8-9 amSpeaker:……
has-value
m1
m2
started-byfinished-by
3:00 pm
11:59 pmhas-value
has-value
Delta-
dinner
occurs-during
Beijing restaura
nt
occurs-at
bound-1
has-boundary
lowerbound
upperbound
s-contains
‘116.02..’
has-longitude
‘39.02..’
has-latitude
‘116.01..’
has-longitude
‘39.01..’
has-latitude
{srafatir,jain}@ics.uci.edu
23
Agglomerative Clustering
{srafatir,jain}@ics.uci.edu
24
Implementation and ResultsTRIP scenarioIndividual: <domain#event_n> Types: <domain#Staying> Facts: <upper#SUPEREVENT> <domain#Trip-to-Beijing>, <upper#P-POI-LOCATION> <domain#place_k>, <upper#P-T-BEFORE> <domain#Checking-out_j>, <upper#P-T-AFTER> <domain#Checking-in_j> Individual: <domain#place_k> Types: <domain#POI> Facts: <upper#HAS-CATEGORY> “hotel”, {domain} <upper#HAS-LATITUDE> 21.13, {R-Onto} <upper#HAS-LONGITUDE> 79.06 {R-Onto} <upper#HAS-BOUNDARY> <domain#bound-k>
Individual: <domain#bound-k> Types: <upper#Bounding-Box> Facts: <upper#S-CONTAINS> <domain#coor_k1>, <upper#S-CONTAINS> <domain#coor_k2>
Individual: <domain#coor-k1> Types: <upper#Coordinates> Facts: <upper#HAS-LATITUDE> 39.908094, <upper#HAS-LONGITUDE> 116.444083
WEDDING scenarioIndividual: <domain#event_m> Types: <domain#Groom-Arrival> Facts: <upper#SUPEREVENT> <domain#indianWeddingCeremony>, <upper#P-POI-LOCATION> <domain#place_p> <…. ….> <domain#HAS-LOC-QUALITY> “outdoor”, …
Individual: <domain#event_s>
Types: <domain#Taking-Portrait> Facts: <upper#SUPEREVENT> <domain#indianWeddingParty>, <upper#P-POI-LOCATION> <domain#place_p> <…. ….> <domain#HAS-OBJECT> <domain#face_h>, <domain#HAS-Processing-Unit> “http://www.unitX.com”, … Individual: <domain#face_h> Types: <domain#Face> Facts: <domain#HAS-VIS-QUALITY> “smiling”
{srafatir,jain}@ics.uci.edu
25
Upper/Domain Ontology Basic Derivation of E* by A. Gupta, R. Jain. Context of event classes in domain ontology
Temporal model (absolute and relative)
Spatial model◦ Coordinate(lat,lng), boundingbox(a pair of coordinates), place-name (e.g.
Disney Land), locationType
◦ (Perdurant occurs-at Place),(Place has-boundary BoundingBox), (BoundingBox s-contains Coordinates).
Structural model (subevent)◦ entailment rule
{srafatir,jain}@ics.uci.edu
26
Implementation and Results Vacation Trip Indian Wedding
Organize my photos based on the subevents I participated during event-x
Organize my photos based on the subevents I participated during event-y
photos of my stay at “blah” hotel
photos of wedding party
photos of my visits to famous landmarks
photos of groom arriving
photos of shopping photos of serving dinner
photos of having lunch/dinner/breakfast
photos of marriage ceremony
{srafatir,jain}@ics.uci.edu
Sources from Web:◦ For Trip {Landmark finder,
Trip itinerary,location database}
◦ For Wedding {Electronic invitations,face-detector,location database}
•OWL-API•EXIFTOOL•Lab tools•Personal Archives
27
Ontology Augmentation
{srafatir,jain}@ics.uci.edu
28
A Qualitative User Study
{srafatir,jain}@ics.uci.edu
29
Implementation and Results
{srafatir,jain}@ics.uci.edu
Sources from Web:◦ For Trip {Landmark finder, Trip itinerary,location database}
◦ For Wedding {Electronic invitations,face-detector,location database}
OWL-API EXIFTOOL Lab tools Personal Archives
Related Documents
