Personal Ontology Learning Grace Hui Yang Language Technologies Institute, Carnegie Mellon University [email protected]Thesis Committee: Jamie Callan (CMU,Chair) Jaime Carbonell (CMU) Christos Faloutsos (CMU) Eduard Hovy (ISI/USC) Nov 9, 2011 Ph.D. Thesis Defense Talk 1 Notice Comment Rulemaking U.S. regulatory agencies receive and deal with large amount of public comments everyday By law, they need to read each of them A few rules attracts hundreds of thousands emails per year Government employees needs to quickly overview the “lay of the land” 2 Ph.D. Defense, Nov 9, 2011
46
Embed
Personal Ontology Learning - Georgetown Universityinfosense.cs.georgetown.edu/publication/slides/defense-talk.pdf · Ph.D. Thesis Defense Talk 1 Notice Comment Rulemaking U.S. regulatory
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
1. Pantel and Ravichandran 04. 2. Snow Jurafsky, and Ng 06.3. Hearst92.4. Snow et al. 05.5. Pantel et al.04. 6. Roark and Charniak 98. 7. Davidov and Rappoport.06. 8. Kozareva et al. 08.9. Etzioni et al.05.10. Mitchell et al. 10.
Ph.D. Defense, Nov 9, 2011
Clustering vs. Patterns vs. What We Want
14Ph.D. Defense, Nov 9, 2011
Heart AttackCauses
Self-help
Medical Treatment
Blood PressureMood Change
Diabetes
Blood Clots
Vessel Narrowness
Blood Sugar
New Medicine
Surgery
Angioplasty, Bypass
Healthy Food
Enough Sleep
Exercise
Vegetable, Fruit
Surgery
Angioplasty
Bypass
Disease
Heart Disease
Diabetes
Healthy Food
Vegetable
Fruit
Food
Enough Sleep
Heart Attack
Self-help
Medical Treatment
Blood Pressure
Mood Change
Blood Clots
Vessel Narrowness
Blood Sugar
New Medicine
Exercise
Heart Attack
Causes
Self-help
Medical Treatment
Blood Pressure
Mood Change
Diabetes
Blood Clots
Vessel Narrowness
Blood Sugar
New Medicine
Surgery
Angioplasty, Bypass
Healthy Food
Enough Sleep
Exercise
Vegetable, Fruit
?
?
?
How will a human organize concepts?
� Human solution:
� Form small & accurate fragments
� Examine the remaining concepts one by one
� Look for the best place for a concept
� We take a similar approach!
15Ph.D. Defense, Nov 9, 2011
Zooming in: Pair-wise Semantic Distances
� Many techniques … …
16
Context
Co-occurrence
… , and other …
… consists of …
Clustering Pattern
KL Divergence in Google snippets
KL Divergence
in Wikipedia
…, …, or other …
… is a …
…, including …
Edge distance in parse tree
Word Length Difference
Others
Overlaps in Definition
Overlaps in Modifier
� They are all good
� So we decide to use all of them!
� … by providing a general framework
� Transform each technique into a feature
� Weighted combination of the features
� Learning the weights from training data
�WordNet, ODPPh.D. Defense, Nov 9, 2011
Weighted Combination of Feature Function Values as the Pair-wise Distance
17
),( yx ccd
),( ),( 1
yx
T
yx ccfeaturesWccfeatures−
| |
Patterns
Syn. Pars. Tree
Context
Co-occurrence
Definition
Word Length
…
Weight Matrix
Learned from
Training Data
Ph.D. Defense, Nov 9, 2011
Mahalanobis distance
W≥0, positive semi-definite to ensure triangular inequality
cy
cx
Best Possible Position for a Concept
� Connections:
� Minimum evolution principle in biology
� Minimum spanning tree in graph theory
� When a concept arrives,
� Its insertion should give the least increase to the overall semantic distance in the ontology
� Why this is true?
� Correct position = small distances to neighbors
� Wrong position = big distance to neighbors
� Minimize overall semantic distance in the ontology
18
Minimum Evolution
Ph.D. Defense, Nov 9, 2011
Minimum Evolution
The Optimal Ontology is One that Introduces Least
Increase to Overall Semantic Distance
),(minarg '0
'TTT
T∆=
),(minarg '1
'TTT n
T
n∆=
+
19Ph.D. Defense, Nov 9, 2011
Minimum Evolution (An Example)
20
Relation: is-a
e.g., Apple is-a Fruit
Fruit is-not-an Apple
0.3
Ph.D. Defense, Nov 9, 2011
Game Equipment
Overall Distance = 0.3
Minimum Evolution (An Example)
21
Relation: is-a
e.g., Apple is-a Fruit
Fruit is-not-an Apple
dist(“ball”, ) = 0.27
dist(“Game Equipment”,“ball”) = 0.1
dist(“ball”, “Game Equipment”) = 3
dist( , “ball”) = 12
Overall Distance = 12.3
0.3
12
Ph.D. Defense, Nov 9, 2011
Game Equipment
ball
Minimum Evolution (An Example)
22
Relation: is-a
e.g., Apple is-a Fruit
Fruit is-not-an Apple
Overall Distance = 0.4
0.3
0.1
Ph.D. Defense, Nov 9, 2011
dist(“ball”, ) = 0.27
dist(“Game Equipment”,“ball”) = 0.1
dist(“ball”, “Game Equipment”) = 3
dist( , “ball”) = 12Game Equipment
ball
Minimum Evolution (An Example)
23
Relation: is-a
e.g., Apple is-a Fruit
Fruit is-not-an Apple
Overall Distance = 0.370.1
0.27
min
Ph.D. Defense, Nov 9, 2011
dist(“ball”, ) = 0.27
dist(“Game Equipment”,“ball”) = 0.1
dist(“ball”, “Game Equipment”) = 3
dist( , “ball”) = 12Game Equipment
ball
Minimum Evolution (An Example)
24Ph.D. Defense, Nov 9, 2011
table
Game Equipment
ball
Concerns
� Order of the insertions
� Small ontologies: Random restarts
� Big ontologies: Partial random restarts for recent arrivals
� Search space is big
� Constrain the ontology candidates
� Constraints come from a good understanding of the characteristics of a personal ontology
� Concept abstractness
� Long distance concept coherence
25Ph.D. Defense, Nov 9, 2011
Concept Abstractness
26
Mo
re A
bstra
ct
Mo
re C
on
cre
te
things to discuss
global
warming
issues actions
pollution policies
causes
CO2
impact
animal
death
polar bear seal wolf
severe
weather
EPA
rules
DOT
rules
reduce
power plant
reduce
emission
flood
Ph.D. Defense, Nov 9, 2011
Concept Abstractness
27
Each abstraction level has its own distance function
Ph.D. Defense, Nov 9, 2011
Long Distance Coherence
28
car
sportsedan
swim ball
games athletics
football baseball basketball
BMW
things
to buy
things
to work on
tenurepurse
good
teaching
I see myself in 5 years
good
research
great
ideas
hard
work
Ph.D. Defense, Nov 9, 2011
Long Distance Coherence
29
car
sportsedan
swim ball
games athletics
football baseball basketball
BMW
things
to buy
things
to work on
tenure
good
teaching
I see myself in 5 years
good
research
great
ideas
hard
work
purse
Ph.D. Defense, Nov 9, 2011
Long Distance Coherence
30
car
sportsedan
swim ball
games athletics
football baseball basketball
BMW
things
to buy
things
to work on
tenure
good
teaching
I see myself in 5 years
good
research
great
ideas
hard
work
purse
Ph.D. Defense, Nov 9, 2011
Long Distance Coherence
31
car
sportsedan
swim ball
games athletics
football baseball basketball
BMW
things
to buy
things
to work on
tenure
good
teaching
I see myself in 5 years
good
research
great
ideas
hard
work
purse
Ph.D. Defense, Nov 9, 2011
Long Distance Coherence
32
car
sportsedan
swim ball
games athletics
football baseball basketball
BMW
things
to buy
things
to work on
tenure
good
teaching
I see myself in 5 years
good
research
great
ideas
hard
work
Each root-to-leaf path is coherent;
Overall distances in a path should be
minimized.
purse
Ph.D. Defense, Nov 9, 2011
Multi-Criterion Optimization
33
Minimum
Evolution
objective
Coherence
objective
Abstractness
objective
Ph.D. Defense, Nov 9, 2011
Evaluation
� Task: Reconstruct ontology fragments
� Datasets: � 50 hypernym ontology fragments from WordNet
� Evaluation Metrics: Precision, Recall, and F1-measure for parent-child pairs� averaged by 50 Leave-One-Out cross validation
34Ph.D. Defense, Nov 9, 2011
Comparison to State-of-the-art
35
System Precision Recall F1
Hearst 1992 0.85 0.32 0.46
Girju et al. 2003 - - -
Snow et al. 2006 0.75 0.73 0.74
Our Approach 0.82 0.79 0.82
WordNet is-aSystem Precision Recall F1
Hearst 1992 0.31 0.29 0.30
Girju et al. 2003 - - -
Snow et al. 2006 0.60 0.72 0.64
Our Approach 0.64 0.70 0.67
ODP is-a
System Precision Recall F1
Hearst 1992 - - -
Girju et al. 2003 0.75 0.25 0.38
Snow et al. 2006 0.68 0.52 0.57
Our Approach 0.69 0.55 0.61
WordNet part-of
Ph.D. Defense, Nov 9, 2011
Features vs. Relations
36
Feature Is-a Sibling Part-of Benefited Rel.
Co-occurrence 0.48 0.41 0.28 All
Pattern 0.46 0.41 0.30 All
Contextual 0.21 0.42 0.12 Sibling
Syntactic 0.22 0.36 0.12 Sibling
Word Length 0.16 0.16 0.16
Definition 0.12 0.18 0.10
All 0.82 0.79 0.61 All
Best Features Co-occurrence,
Pattern
Contextual,
Co-occurrence,
Pattern,
Syntactic
Co-occurrence,
Pattern
Metric: F1. WordNet
Ph.D. Defense, Nov 9, 2011
Features vs. Abstractness
37
Feature Level 2 Level 3 Level 4 Level
5
Level 6
Co-occurrence 0.47 0.56 0.45 0.41 0.41
Pattern 0.47 0.44 0.42 0.39 0.40
Contextual 0.29 0.31 0.35 0.36 0.36
Syntactic 0.31 0.28 0.36 0.38 0.40
Word Length 0.16 0.16 0.16 0.16 0.16
Definition 0.12 0.12 0.12 0.12 0.12
Metric: F1. WordNet /is-a
Ph.D. Defense, Nov 9, 2011
Features vs. Abstractness
38
Feature Abstract Concepts Concrete Concepts
Co-occurrenceGood Good
Pattern
ContextualBad Good
Syntactic
Word LengthBad
Definition
Metric: F1. WordNet /is-a
Ph.D. Defense, Nov 9, 2011
Outline
� A general ontology learning framework
� Put human in the loop
� Efficient hierarchy similarity measure – FBS
� Study of user behaviors
39Ph.D. Defense, Nov 9, 2011
Put Human in the Loop
� Purpose: Customize the Ontology to Suit Individual Needs
� Collect guidance from human
� Guidance in a representation that can be understood by machine
40Ph.D. Defense, Nov 9, 2011
OntoCop (Ontology Construction Panel)
41
Ph.D. Defense, Nov 9, 2011
After a Few Human Edits – Interact!
42
Ph.D. Defense, Nov 9, 2011
OntoCop Makes Suggestions
43
Ph.D. Defense, Nov 9, 2011
Matrix Representation for Ontology
� Before human edits
� Before Matrix
� After human edits
� After Matrix
44
10000
01000
00110
00110
00001
10000
01100
01100
00010
00001person
leader
president
prime minister
Obama
person
leader
president
prime minister
Obama
person
leader president
prime minister Obama
person
leader
presidentprime minister
Obama
Ph.D. Defense, Nov 9, 2011
Manual Guidance
45
10000
01000
00110
00110
00001
Before Matrix After Matrix
10000
01100
01100
00010
00001
Different rows
Different columns
110
110
001
Manual Guidance
Ph.D. Defense, Nov 9, 2011
How to Incorporate Manual Guidance
� Nearest neighbors
� Find the most similar pairs (nearest neighbors) to the manual guidance, & predict accordingly
�Why not
� Conflicts among multiple guidance’s predictions
� Lost transitivity of distance
� Using our ontology learning framework !
46Ph.D. Defense, Nov 9, 2011
Manual Guidance as Training Data
47
( )
0 subject to
),(),(min||
1
||
1
2)()(1)()()(
)( )(
=
−∑∑= =
−
fW
ccfeaturesWccfeaturesd
i iG
x
G
y
i
y
i
x
Ti
y
i
x
i
xyW
Training Data
Manual Guidance
WordNet
ODP
Smoothing
Ph.D. Defense, Nov 9, 2011
Update the Ontology
� Predict Distance Scores for Unmodified Concepts
� Organize concepts in the updated ontology
�When is small (<0.5), the relation between
is true;
� The relation can be of any type
� but one relation in one ontology
48
),(),( )1()1(1)()1()1()1( ++−+++=
i
m
i
l
iTi
m
i
l
i
lm ccfeaturesWccfeaturesd
)1( +i
lmd )1()1( , ++ i
m
i
l cc
Ph.D. Defense, Nov 9, 2011
User Study
� Task: Build personal ontologies from a set of given concepts and documents
� 20 Datasets
� 10 NAICS (North America Industry Classification System)� Information, health care, administrative services, professional services, finance, construction, public administration, …
� 5 Web � Find a good kindergarten, buy a used car, plan a trip to DC, make a cake, and find a wedding videographer, …
� 5 Public Comments� protect polar bear, protect wolf, mercury pollution, transportation registration fee, and national organic program, …
49Ph.D. Defense, Nov 9, 2011
User Study
� A within-subject study for 24 grad & undergrad students
� Procedure:� Start with a tool training
� Everyone did both manual and interactive ontology construction for the testing tasks
� Questionnaire: dataset difficulty, system learning ability, editquality, compare manual vs. interactive, etc
� 12 participants repeated the tasks after 3 weeks
50Ph.D. Defense, Nov 9, 2011
Accuracy of OntoCop’s Suggestions
51
accuracy
=# accepted suggestions
# total suggestions
Ph.D. Defense, Nov 9, 2011
� The accuracy of suggestions is high across all datasets
Accuracy of OntoCop’s Suggestions
52
Better
Ph.D. Defense, Nov 9, 2011
More Results
� Efficiency: OntoCop save 20% time (p<.001), 25% edits (p<.001) per dataset on average than manual runs
� Compare to reference ontologies: OntoCop produces ontologies more similar (0.82) to reference ontology than manual (0.74)
� Dataset difficulty:
� correlates to dataset type - NAICS>Web,Comments
� more difficult dataset � longer construction time, less confidence