Introduction to Cross-Document Coreference Amit Bagga StreamSage/Comcast [email protected].

Introduction to Cross-Document Coreference

Amit Bagga

StreamSage/Comcast

[email protected]

Outline• Motivation and Definition• Comparison with Within-Document Coreference,

WSD and other NL tasks• Methodologies for Entity Cross-Document

Coreference• Other types of Cross-Document Coreference

– Concept Cross-Document Coreference– Event Cross-Document Coreference– Cross-Media Coreference– Cross-Language, Cross-Document Coreference

• Scoring Methodologies

Motivation

• Proper names comprise approximately 10% of news text (Coates-Stephens, 1992)

• Names are often ambiguous across documents– increasingly becoming a challenge for NLP

systems as collection size and generality grow– also as systems break the “document boundary”

Definition

• Cross-Document Coreference (CDC) for entities, in broad terms, asks– how can one computationally disambiguate the

intended referent of a name• Winchester & Lee 2002

– for example, it asks, which ‘John Smith’ is meant by a particular occurrence of the string “John Smith”

Comparison with Within-Document Coreference

• Within a document– Identical or similarly named entities seldom

appear in the same context• when they do, writers distinguish them explicitly • i.e. it is usually the case that we have one referent

per discourse

– Variant form of the same name generally obey certain regularities which are predictable

• For example: Michael Jordan may be referred to by the following – Michael, Mr. Jordan, Jordan, etc.

• Across documents– Assumption that same or similar names refer to

same entity is not valid– Linguistics theories do not apply– The only way to distinguish between these

entities is to examine context

Comparison with WSD• CDC can be thought of as disambiguating the

“sense” of usage of a name• In WSD:

– Usually possible to enumerate a priori all possible senses of word

– Number of possible senses of word is small (1-10)

• In CDC:– A large corpus can contain 10s or 100s of entities with

same name which are impossible to enumerate a priori– From linguistic perspective, all entities equally

plausible

The Role of Context

• Similar to WSD, context is vital for CDC– context can be of different sizes

• window of words centered around a name, sentence containing name, group of sentences, or even whole document

– modeling context can be done in many different ways

• bag of words, set of phrases, set of entities, set of relations, etc.

• All CDC systems use context in one form or another

Bag of Words Approach– Bagga and Baldwin, 1998

– Within-document coreference system is used to identify all mentions of entity

– Sentences containing mention are extracted from each document

• “summaries” with respect to entity

– Set of summaries compared using VSM (tf*idf)

– Single-link clustering used

– Version 2 (1999) eliminates use of within document coreference system

• sentences containing any variant of name extracted

Corpus, Evaluation, and Results

– 197 articles containing “John Smith” extracted from 2 years of New York Times data

• 35 different John Smiths

– B-CUBED algorithm used– Version 2 results

• 84% F-Measure

• 90% Precision, 78% Recall

• < 1% F-Measure drop when compared to original system

Minimizing Context Matches

• Kazi and Ravin, 2000• Problem with Bagga and Baldwin, 1998

– Prohibitively expensive in terms of storage and n-to-n comparisons (specially in a large corpus)

• Use IBM’s Nominator for named entity identification and within document coreference (non-pronominal)

• CDC task is merging canonical names from different documents that refer to same entity

• Context analysis done by use of a Context Thesaurus– Given a name, returns a ranked list of terms that are related to

name in the corpus

# Docs Nominator Output

17 Bush (unspecified gender)

1 Christopher Bush (male)

1 Douglas Bush (male)

26 George Bush (male)

2 George Bush; President Bush (male)

1George W. Bush; Gov. George W. Bush; President George Bush (male)

1 Mr. Bush (male)

2 President Bush (male)

7 Vannevar Bush (unspecified gender)

E1 Christopher Bush

E2 Douglas Bush

E3 George W. Bush

E4 Vannevar Bush

M1

George Bush,

mergeable with E3

(first name and gender)

M2Mr. Bush,

mergeable with E1-E4

M3President Bush,

mergeable with E1-E4

M4-M9

Bush,

mergeable with E1-E4

• E = Exclusives – i.e. no merging possible

• M = Mergeables – i.e. compatible with some or all exclusives

• Tables are created by analyzing two lists sorted by ambiguity– PERS names

• George Walker Bush > George W. Bush > George Bush > G. Bush > Bush

– PLACE names• Albany, NY > Albany

• Merging steps– Merge identical canonical strings >= 2 words

• Merges 28 George Bush, 2 President Bush 7 Vannevar Bush articles into 3 equivalence classes

– Between mergeables and exclusives, combine if any variants share a common prefix

• Merges E3, M1 and M3 (common prefix = President)

• Reduces # of context matches from 58x58 to 7x4

Corpus, Evaluation, and Results

• Corpus – 1998 editions of New York Times• 15 name families

– For example: Berger, Black, Brown, Bush, Clinton, Gore, etc.

• B-CUBED algorithm for scoring• Without context comparisons:

– Avg Precision = 98.5%– Avg Recall = 72.85%

• No results reported when context comparisons are used (Ravin and Kazi, 1999)

3 Models of Similarity• Gooi and Allan, 2004• Methodology similar to Bagga and Baldwin

– extract 55 word snippets centered at name or its variant

• Problem with Bagga and Baldwin– sharp drop off in F-Measure around threshold

• 3 different models of similarity– Incremental Vector Space

• tf*idf, but with average link clustering

– KL divergence• snippets are represented as probability distribution of words• similarity = “distance” between two probability distributions

– Agglomerative Vector Space• tf*idf with bottom-up, complete-link clustering

Corpus• John Smith corpus (Bagga and Baldwin)• Person-x corpus

– created by querying TREC collection with queries like arts, business, sports, etc.

– BBN’s IdentiFinder used for named entity recognition

– one name (and its corresponding variants) randomly replaced with phrase Person-x

– 34,404 documents; 14,767 actual unique entities

Evaluation and Results

• B-CUBED algorithm used for scoring• Agglomerative VS best

– 88.2% F-Measure for John Smith corpus– 83% F-Measure for Person-x corpus

• When run on each sub-corpus (arts, sports, etc.) of Person-x corpus– F-Measure drops to 77%– shows that a more homogenous corpus is more difficult

• Results for Agglomerative VS degrade much more smoothly around threshold than others

Second Order Co-Occurrence• Three methods – independently published• Bagga, Baldwin, and Ramesh, 2001 - 2-pass

algorithm– First pass: as before– Second pass:

• for each chain, compute set of most frequent overlapping words in chain (signature words for chain)

• for each singleton document after pass 1, compare to each chain

– use signature words to extract additional sentences– compare enhanced summary to every summary in chain– merge if similarity > threshold– if not merged with any chain, remains singleton

• Winchester and Lee, 2001– named entity detection and conflation within

documents is done as pre-processing step

– based on Schutze’s (1998) algorithm for context-group discrimination

– 3 types of vectors are created• Term Vectors – formed for each name occurring in context of

entity of interest and its variants– stores co-occurrence stats for term across whole corpus

• Context Vectors – formed for entity of interest by summing all term vectors associated with its context

– term vectors are weighted with their idf scores before sum

• Entity Vectors – for each entity, it is centroid of set of context vectors

– entity disambiguation is done by comparing Entity Vectors using VSM with single-link clustering

Corpus, Evaluation, Results• Bagga, Baldwin, and Ramesh

– John Smith corpus, B-CUBED scoring– new F-Measure 91% (+7 from before)

• Winchester and Lee– 30 name sets; 10 each of PER, LOC, ORG– from 6000 WSJ articles– B-CUBED scoring– discovered that selective creation of 3 types of vectors

boosts performance• for example, LOC helps disambiguate other LOC• Birmingham, Alabama vs UK; John Smith associated with

Pocahontas– overall F-Measure 78.5%

• NAM – 90.3%, LOC – 79.2%, ORG – 72.5%

• Guha and Garg, 2004– mine descriptions associated with entity of interest

(sketch)• descriptions are other entities + professions that are in close

proximity

– comparing descriptions• different weights given to different descriptions given type of

entity of interest and entity-type of description– for example: location is more likely to be disambiguated by

another location than by the name of a person

– Corpus and Evaluation• 26 entities (names + places), 2-6 instances identified of each

• sent as queries to search engines, top 150 results collated and manually tagged for truth

• best F-Measure = 90.3%

Maximum Entropy Model• Fleischman and Hovy, 2004 – use ME to determine if two

concept/instance pairs are same entity– concept/instance pairs – ACL dataset (2M pairs)

• John Edwards/lawyer and John Edwards/politician– Name features: NAME-COMMON (census), NAME-FAME (ACL

dataset), WEB-FAME (Google)– Web features: based on # of Google hits with name plus headwords of

concepts used as queries– Overlap features: based on # words overlapping in context of names and

concepts– Semantic features: based on semantic relatedness of concepts (WordNet)

• for example: lawyers are more likely to become politicians– Estimated Statistics features: probabilities that a name is associated with a

particular concept (computed over entire ACL dataset)• Disambiguation using group-average agglomerative clustering• Tested on set of 31 concept/instance pairs (1875 used for training)

– 20 had a single referent– F-Measure = 93.9%– baseline (all in same chain) = 92.4%

Robust Reading Approach• Li, Morie, and Roth, 2004

– a global probabilistic view of how documents are generated and how entities are “sprinkled” into them

• Model 1 (simplest – no notion of author)– entities are present in a document with a prior probability,

independent of other entities– mentions (references) are selected according to probability

distribution P(mj|ei)– i.e. entity referenced by a mention is not dependent on other

mentions• Model 2 (more expressive)

– # of entities in doc and # of mentions follow uniform distribution– entities enter doc with a prior probability, independent of others– representative (canonical form) for each entity is selected

according to P(rj|ei) – for each representative, mentions are selected by P(mk|rj)– i.e. entity referenced by a mention depends on other mentions in

the same document

• Model 3 (least relaxation)– # of entities based on uniform distribution – but not independent of each

other– entities in doc viewed as nodes in a weighted directed graph with edges

labeled as P(ej|ei)– entities inserted in document via a random walk starting at an entity with

prior probability P(ek)– representatives and mentions follow the same probabilities as Model 2– i.e. entity referenced by a mention depends on other mentions in same

document, but also on other entities in entire corpus• Models learned using truncated EM algorithm• Evaluation

– 300 NYT articles from TREC corpus– 8000 mentions corresponding to 2000 entities (people, locations,

organizations)– compared to SOFT-TF-IDF and baseline (entities with identical writing

are same)– overall F-Measure = 89% (model 2)– baseline = 70.7% and SOFT-TF-IDF = 79.8%

• Model 3 does not perform best because– global dependencies enforces restrictions over groupings of similar

mentions– because of limited document set, estimating global dependency is

inaccurate

Using IE Features• 3 different methods published• Mann and Yarowsky, 2003

– use unsupervised learning to learn patterns from corpus that capture biographical features

• birth day, birth year, birth place and occupation

– use bottom-up centroid agglomerative clustering for disambiguation

– vectors for each document are generated by using the following

• all words (plain) or proper nouns (nnp)• most relevant words (mi and tf-idf)• basic biographical features (feat)• extended biographical features (extfeat)

Corpus, Evaluation, and Results• Mann and Yarowsky

– Pseudoname corpus• query Google with names of 8 people

– take 28 possible pairs and replace with different pseudonames

– Naturally occurring corpus• query for 4 naturally occurring polysemous names

– example: Jim Clark

• 60 articles for each name• 3-way classification (top 2 occurring people + “others”)

– Disambiguating accuracy for Pseudonames• 86.4% with nnp+feat+tf-idf

– For naturally occurring corpus• using mutual information 88% Precision and 73% Recall

• Niu, Li, and Srihari, 2004 - use 3 different categories of contextual features– set of 50 words centered around name (or alias)– other entities occurring in 50 word context of name (or

alias)– automatic extracted relationships (25 possible)

• birth day, age, affiliation, title, address, degree, etc.

– features combined using Maximum Entropy Model

• Evaluation using B-CUBED algorithm– 4 sets of 4 famous names mixed together using

pseudonames • 88% F-Measure achieved

– 2 naturally occurring sets • Peter Sutherland – 96% F-Measure• John Smith – 85% F-Measure

• Dozier and Zielund, 2004– CDC for people in legal domain

• attorneys, judges, and expert witnesses

– Combine IE techniques with record linkage techniques• biographical records for attorneys and judges created manually

from Westlaw Legal Directory• biographical record for expert witnesses created through text

mining• IE techniques extract templates associated with each type from

document• record linkage part uses Bayesian network to match templates

with biographical records

– Evaluation• for docs with stereotypical syntax and full names – 98%

precision and 95% recall• Otherwise, 95% precision and 60% recall

Baseline

• Guha and Garg, 2004– established baseline when full docs were

compared using TF-IDF without considering context for 26 entities (names and places)

– 2-6 instances of each entity considered– for each instance, top 10 results evaluated– 22.5% accuracy overall

Types of CDC• Named Entities

– described earlier• Terms or Concept

– Kazi and Ravin, 2000• Events

– Bagga and Baldwin, 1999• Cross-Media and/or Multimedia Coreference

– Between text and pictures for names (Bagga and Hu, unpublished)– Between text and video for names (Satoh and Kanade, 1997) – Between video streams (using image and text) for events (Bagga,

Hu, and Zhong, 2002)• Cross-Language, Cross-Document Coreference

– parallel corpus (Harabagiu and Maiorano, 2000)– non-parallel corpus – open problem, although manual results

encouraging (Bagga and Baldwin, unpublished)

Term or Concept CDC

• Single or multi-word terms refer to concepts occurring in domain

• Multi-word terms– identified by Terminator (rule-based)

• form subset of noun phrases in document– discard those that occur only once in document

• for example: price rose where rose is mistakenly identified as noun

– discard those that are found only as proper sub-strings• for example: dimension space (part of lower dimension space)

– are seldom ambiguous and are merged across documents

Single Word Terms• Capitalized single words are most common

sources of ambiguity– for example: Wired – name of magazine and an

adjective that is first word in sentence• Within-doc categorization of single words

– If capitalized word occurs in lowercase in document – consider as regular word

– If capitalized word appears as capitalized in middle of sentence – consider as name

– If no lowercase occurrences and word appears at beginning of sentence or in title/header - consider as term

– All other single words not identified as part of name or multi-word terms – consider as lower-case term

Disambiguating Single Words Across Documents

Lower-case

Term

Uncat.

Name

enliven Enliven

bush Bush

wired Wired

Unambiguous cases – no merging

Upper-case Term

Lower-case term

Uncat. Name

Name is variant of

Finds finds ---- ----

Loss loss ---- ----

Allied ---- Allied ----

Microsoft ---- MicrosoftMicrosoft Corp.

N.Y. ---- ---- New YorkAmbiguous cases –merge if only nameor only lower-case term found in corpus

• Single occurrences of single capitalized terms can be merged with occurrences of corresponding names if names occur more than once in at least one document

• No evaluation was performed

Upper-case Term

Uncat. Name # Docs# occurrences within doc

Find Find 2 1

Please Please 8 1

Met Met 5 2-3

Sun Sun 12 1-3

Apple Apple 203 1-46

Event CDC• Bagga and Baldwin, 1999

– similar approach to entity-based CDC• Two events are coreferent iff the players, time, and

location are the same• Event CDC system extracts as “summaries”

sentences which contain:– main event verb (for example: resign)– nominalization of main verb (for example: resignation)– synonyms (for example: quit)

• Summaries are clustered using single-link clustering and VSM similarity

Evaluation and Results• Articles chosen for 3 events: resignations, elections, and

espionage– 2 years of New York Times data

• B-CUBED algorithm used for scoring

Event # docs F-Measure Precision RecallF-Measure

(2-pass algorithm)

resignations 219 84 95 75 84

elections 135 43 50 37 45

espionage 184 76 79 74 81

Analysis• Events are harder than entities:

– no within-document coreference– no explicit references– are at time spread over the entire document

• Analysis of Elections event– elections are temporal in nature

• disambiguating phrases largely use temporal references (for example – upcoming fall elections, elections last year, next elections, etc)

• exposes weakness of using a bag of words approach– presence of sub-events

• US General election consists of both Presidential elections and Congressional elections

– “players” are the same due to high rate of incumbency– descriptions of events are very similar

• issues in every election are similar (inflation, unemployment, economy)

Cross-Media Coreference – Between Text and Video (Names)

• Satoh and Kanade, 1997

• Association of face and name in video– given unknown face, infer name or,– given name, guess faces which are likely to

have that name

• Use closed caption transcripts and video images for correlation

• Face extraction: neural-network based face detector to locate faces in images

• Name candidate extraction: use Oxford Text Archive dictionary (appx 70k words)– Word is considered to be a proper noun if

• annotated as one in dictionary• not found in dictionary

• Face similarity: eigenvector based method to compute distance between two faces

• Face and name co-occurrence: use co-occurrence factor– captures how well name and face co-occur in

time

Corpus, Evaluation, and Results

• No large scale evaluation done

• Problem with technique: false positives– specially for famous

people– Clinton mentioned by

news anchor repeatedly– name gets associated

with news anchor

Between Text and Pictures (Names)

• Bagga and Hu, unpublished (2004)• Algorithm

– Use text and image based features to identify coreference

– Tested on web pages• Text narrowed by extracting sentences containing name

variants of entity

• Image features computed by analyzing distribution of colors in L*a*b perceptual color space

– Across URLs, first compute text similarity (VSM) and image similarity (L*a*b) and then combine

Preliminary Results

Maps related to Captain John Smith’s explorations

Portraits of Captain John Smith

Captain John Smith as portrayed in the movie Pocahontas

Cross-Media Coreference

• Goal: identify and track “important” news events in broadcast news video

• Observations:– “important” stories of the day are repeated

within/across stations– common footage scenes can be used as

representative clips for these stories

Scene 1 Scene 2 Scene 3 Scene 6Scene 5Scene 4

Story 1 Story 2

images

sound

images

sound

images

sound

Scene 7

Story seg. 1 Story seg. 2 Story seg. 3Commercial Segment 1

News

ClosedCaption

ClosedCaption

ClosedCaption

Structure of Broadcast News

Methodology

• For each video source, use closed caption text:– to identify segment boundaries (>> signs indicate

speaker change)– identify and eliminate commercial segments (based

upon text-tiling method)– cluster story segments into stories

• Use complete link, hierarchical clustering to identify overlapping stories between programs– identify common footage scenes between each pair of

overlapping stories

Common Footage Detection

Overlapping Story

Scenes from video source 1

Visualsimilarity

Textsimilarity

Combined-Media

clustering

key frames

key frames

Common Footages

Scenes from video source 2

text

text

Examples – Found by SystemNews conferenceOn Iraqi bombing

CBS 4257

CBS 2829 CBS 3873 NBC 3885 NBC 5061

CBS 13833 NBC 16317

CBS 38805 NBC 20805

Flood rescue -> rescue school bus

US submarine->US submarine incident

CBS 4125 NBC 7377

Topic: US/Iraq->US bombingof Iraq.

More Examples

Same stories and similar key-frame images, but not reallyidentical footage.

CBS 2253 NBC 4173

CBS 2001 NBC 3177

Night at Baghdad->night bombingat Iraq.

Iraqi map

CBS 5193 NBC 30021

UN cars->UN inspectors leaving IraqFound by algorithm, butmissed by human subjects

CBS 501 CBS 13305 NBC 16977

US submarine incident.Missed because weak textlink and image intensitychange.

Missed by system

False positive:

Death of Dale Earnhardt

Results• System achieves on average 71% recall, 37%

precision– 4 test sets– each set consisted of 2 thirty minute news programs

from CBS and NBC (same day)

• Majority of false positives occur due to presence of studio scenes

• If studio scenes are eliminated from results (when stories are the same)– precision increases to 87%

Cross-Language CDC:Parallel Corpus

• Harabagiu and Maiorano, 2000 • Use parallel corpus English and Romanian

– Romanian obtained by manually translating MUC-6 and MUC-7 corpora

• Within-document coreference system run within each language

• Parallelism used to improve coreference in each language by using features/coreference chain information from the other– English precision increases from 84% to 87% while

preserving recall– Romanian precision increases from 72% to 76% while

preserving recall

Cross-Language CDC:Non-Parallel Corpus

• Bagga and Baldwin (unpublished)• Algorithm evaluated manually on a small set of documents

in English and Korean– for each document, extract sentences containing mentions of entity

(name variants only) – “summary”– translate each summary from non-English language to English

using a bi-lingual dictionary (word for word translation, without regard for sense)

– Compare “approximate translations” with English summaries using VSM

• Initial results were promising with limited decline in F-Measure

• Identification of transliterated names is a major problem

Cross-Language CDC:Arabic Non-Parallel Corpus

• Sayeed, et al., 2009• Based on Bagga and Baldwin, 1998

– Use BBN’s Serif for computing Within-Document Coreference chains– for each document, extract windows of 50 words around mentions of entity

– “summary”– One variation of the system tries to address the name transliteration problem

by • a) translating the longest names in each document into English• b) correlating which ones are “similar” in English, and• c) attempting to find xdoc coreference between these discovered pairs

– A baseline system identified xdoc coreference when longest matching names were exact matches

• Tested on 412 document set from ACE 2008 corpus– Baseline B-Cubed F-Measure = 40.6 (best F-Measure for task = 69)– F-Measure for System without name translation = 40.6– F-Measure for system with name translation = 41.3

Evaluation Methodologies

• MUC-6/7 algorithm – Vilain, et al., 1996– originally developed for within-document coreference

• B-CUBED– Bagga and Baldwin, 1998

• Clustering– Treat CDC as a clustering problem

• ACE – Automatic Content Extraction Program– developed for Entity Detection and Tracking (EDT)

task (currently, used for within-document EDT only)

The MUC Scorer: Example

Truth:Response A:

Response B:

MUC Scoring Algorithm

• Precision Error is determined by asking:– How many links must be added to truth (key) to

have the same equivalence classes as the response?

• For recall error, reverse the roles above.

Problem: All Errors are Equal• For response A:

– Precision = 9/10

– Recall = 9/9

• For response B:– Precision = 9/10

– Recall = 9/9

• Unintuitive results in the extreme cases– N = # of entities– m = # of chains (truth)– All entities in same

chain:

– P => 1, if N >> m

1 - N

m - N P

An Intuition for Scoring Differently

Truth:Response A:

Response B:

A Mistake!

A Bigger Mistake!!

B-CUBED Algorithm: An entity based approach

• For each entity, i:

i

ii

element containingchain output in elements of #element containingchain output in elementscorrect of #Precision

i

ii

element containingchain truth in the elements of #element containingchain output in elementscorrect of #Recall

N

1Precision *

N

1 Precision Final i

Example: PrecisionResponse A: Response B:

%)76(21

16 5*7

52*7

25*5

5 * 12

1

%)58(

12

7 5*10

52*2

25*10

5 * 12

1

Recall for both responses is 100%

ACE Scoring Algorithm

• Types of errors: miss and false alarm• Score is calculated as a function of “cost”• Cost depends on

– entity type• person, organization, geo-political entity, location, and facility

– entity level• name, nominal reference, and pronominal reference

• used for evaluation purposes only

• No published CDC evaluation using this algorithm

Type/

LevelPER ORG GPE LOC FAC

NAM 1 0.5 0.25 0.1 0.05

NOM 0.2 0.1 0.05 0.02 0.01

PRO 0.04 0.02 0.01 0.004 0.002

The cost of a single miss or false alarm

NREF = total number of reference entities in source, SDenominator is normalization factor

= cost when no entities are output

CEDT(S) =

sum over type, tsum over level, l {CMiss(t, l)*NMiss(t, l) + CFA(t, l)*NFA(t, l)}

sum over type, tsum over level, l {CMiss(t, l)*NRef(t, l)}

Applications• IR, EDT, and TDT• Name Matching Problem (Patman and Thomson, 2003)

– When are different name strings potential references to the same entity? (Qaddafi, Gadafi, Gaddafi, Kaddafi, Qaddafy, etc.)

• Cross-Document IE and Information Fusion– increases chances of a pattern match – information may be more explicit in one or more articles– the set of articles may contain more information than any one

• Multi-Document Summarization– 2002 DUC evaluation – earthquakes– systems had difficulty distinguishing between earthquakes

• Question Answering– When was Kennedy born? – which Kennedy is being referred to?

• Link Analysis– linking entities is a first step towards identifying more complex

relationships across documents

Conclusions• CDC is a feasible task

– context (text/images/video) around entity/event provides enough information to disambiguate

• Entity-based CDC – many different methods/models– performance over different, large corpora is consistently in mid

80s• Other types of CDC

– simple models/methods have been tried – plenty of opportunity to explore more sophisticated contextual

models• Evaluation Methodologies

– several different ones exist; no consensus on best one• Applications

– time is ripe for integrating entity-based CDC in higher level applications