Identity Resolution in Email Collections Tamer Elsayed and Douglas W. Oard CLIP Colloquium, UMD, Feb 2009 Department of Computer Science, UMIACS, and iSchool.

Identity Resolution in Email CollectionsIdentity Resolution in Email Collections

Tamer Elsayed and Douglas W. Oard

CLIP Colloquium, UMD, Feb 2009CLIP Colloquium, UMD, Feb 2009

Department of Computer Science, UMIACS, and iSchool

Identity Resolution in Email Collections

Real Problem

National ArchivesNational Archives

Clinton Clinton White HouseWhite House Tobacco Tobacco

PolicyPolicy

search search requestrequest

hired 25 hired 25 personspersons

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~32 million

emails

200,000

80,000

for 6 months …

Date: Wed Dec 20 08:57:00 EST 2000From: Kay Mann <kay.mann@enron.com>To: Suzanne Adams <suzanne.adams@enron.com>Subject: Re: GE Conference Call has be rescheduled

Did Sheila want Scott to participate? Looks like the

call will be too late for him.

Sheila

Identity Resolution in Email

WHO?WHO?WHO?WHO?

Enron Collection

-----Original Message-----From: SStack@reliant.com@ENRONSent: Monday, July 30, 2001 2:24 PMTo: Sager, Elizabeth; Murphy, Harlan; jcrespo@hess.com; wfhenze@jonesday.comCc: ntillett@reliant.comSubject: Shhhh.... it's a SURPRISE !

Message-ID: <1494.1584620.JavaMail.evans@thyme>Date: Mon, 30 Jul 2001 12:40:48 -0700 (PDT)From: elizabeth.sager@enron.comTo: sstack@reliant.comSubject: RE: Shhhh.... it's a SURPRISE !X-From: Sager, Elizabeth </O=ENRON/OU=NA/CN=RECIPIENTS/CN=ESAGER>X-To: 'SStack@reliant.com@ENRON'

Hope all is well.Count me in for the group present.See ya next week if not earlier

Please call me (713) 207-5233

Elizabeth Sager713-853-6349

Hi Shari

Thanks!

55 Sheila’s !!55 Sheila’s !!weisman

pardoglover

richjones

breedenhuckaby

tweedmcintyrechadwick

birminghamkahanekforakertasmanfisherpetitt

DomboRobbinschang

jarnotkirby

knudsenboehringer

lutzgloverwollamjortnerneylon

whangernagel

gravesmclaughlin

venvillerappazzo

millerswatekhollis

maynesnacey

ferrarinidey

macleodhowarddarlingwatsonperlickadvanihesterkennerlewis

waltonwhitmanberggrenosowski

Rank Rank CandidatesCandidates

Generative Model

1. Choose “personperson” c to mention

2. Choose appropriate “contextcontext” X to mention c

p(X | c)

3. Choose a “mentionmention” l

p(l | X, c)““sheila”sheila”

GEGEconferenceconference

callcall

3-Step Solution(1) Identity(1) Identity ModelingModeling

Posterior DistributionPosterior Distribution

(3) Mention Resolution(3) Mention Resolution

(2) Context Reconstruction(2) Context Reconstruction

Outline

Introduction and Approach Overview Computational Model of Identity Context Reconstruction Mention Resolution Evaluation on Existing Collections Scalable MapReduce Implementation New Test Collection Conclusion and Future Work

“Easy References” of Identity

-----Original Message-----From: SStack@reliant.com@ENRONSent: Monday, July 30, 2001 2:24 PMTo: Sager, Elizabeth; Murphy, Harlan; jcrespo@hess.com; wfhenze@jonesday.comCc: ntillett@reliant.comSubject: Shhhh.... it's a SURPRISE !

Message-ID: <1494.1584620.JavaMail.evans@thyme>Date: Mon, 30 Jul 2001 12:40:48 -0700 (PDT)From: elizabeth.sager@enron.comTo: sstack@reliant.comSubject: RE: Shhhh.... it's a SURPRISE !X-From: Sager, Elizabeth </O=ENRON/OU=NA/CN=RECIPIENTS/CN=ESAGER>X-To: 'SStack@reliant.com@ENRON'

Hope all is well.Count me in for the group present.See ya next week if not earlier

Please call me (713) 207-5233

Elizabeth Sager713-853-6349

Hi Shari

Thanks!

Email Email StandardsStandards

Email-Client Email-Client BehaviorBehavior

User User RegularitiesRegularities

Representational Model of Identity

77,240 “non-trivial” models

sheila.glover@enron.com

14 (Quoted Headers)

sheila glover

932 (Main Headers)

sheila

19 (Salutation)

216 (Signature)

sg19 (Signature)

sheila glover

1170 (User Name)

Representational ModelRepresentational Model

Computational Model of Identity

identity

observed mention

name type

ctfreq

ctfreqctp

),()|(

),,(),|(

cassocl

ctmfreq

ctmfreqctmp

ctpctmpcmp )|(),|()|(

cassoc

cassoccp

Identity Models

Candidates

Likelihood: p Likelihood: p ( “sheila” | ( “sheila” | cc))

Outline

Introduction and Approach Overview Computational Model of Identity Context ReconstructionContext Reconstruction Mention Resolution Evaluation on Existing Collections Scalable MapReduce Implementation New Test Collection Conclusion and Future Work

Contextual Space

LocalLocalContextContext

Conversational Conversational ContextContext

Topical ContextTopical Context

Topical Context

Date: Fri Dec 15 05:33:00 EST 2000From: david.oxley@enron.comTo: vince j kaminski <vince.kaminski@enron.com>Cc: sheila walton <sheila.walton@enron.com>Subject: Re: Grant Masson

Great news. Lets get this moving along. Sheila, can you work out GE letter?

Vince, I am in London Monday/Tuesday, back Weds late. I'll ask Sheila to fix this for you and if you need me call me on my cell phone.

sheila.walton@enron.com

Did Sheila want Scott to participate? Looks like the call will be too late for

Sheila call

Sheila

Contextual Space

Social ContextSocial Context

Social Context

Date: Tue, 19 Dec 2000 07:07:00 -0800 (PST)From: rebecca.walker@enron.comTo: kay.mann@enron.com Subject: ESA Option Execution

KayCan you initial the ESA assignment and assumption agreement or should I ask Sheila Tweed to do it? I believe she is currently en route from Portland.Thanks, Rebecca

Sheila Tweed

kay.mann@enron.com

Formally

A context of an email is a probability probability distributiondistribution over emails

Probability estimated based on type of context

Contextual Space is a linear combination of 4 contexts

))(|( ikj exep

Context Expansion

topical

people

content

social

conversationallocal

Temporal similarity affects social and topical similarity

Temporal Similarity

Decay over time Gaussian and Linear functions Time difference / Rank

Social Similarity

Two sets of participants (email adresses) Binary, Overlap, Jacaard, Both

Temporal Effect

Temporal Sim Pure Social Sim

Social SimNormalize

Social Context

Topical Similarity

Standard IR Similarity: BM25

Email as a DOCUMENT? Subject Body (+Subject) Root of thread Concatenated path to root

Combined similarly with temporal similarity

reply / forward

Contextual Space (emails)

Social ContextSocial Context

Contextual Space (mentions)

“Sheila”

social

conversational

social

topical

social

topical

“Sheila Tweed”

“sheila”

“jsheila@enron.com”

“sg”

“Sheila Walton”

“Sheila”

Outline

Introduction and Approach Overview Computational Model of Identity Context Reconstruction Mention ResolutionMention Resolution Evaluation on Existing Collections Scalable MapReduce Implementation New Test Collection Conclusion and Future Work

Mention Resolution

Candidates

Likelihood: p Likelihood: p ( “sheila” | ( “sheila” | cc))

Goal: estimate p(c|m, X(m)) and rank accordingly

Sheila

[1] Context-Free Resolution

“Sheila”

social

conversational

social

topical

social

topical

“Sheila Tweed”

“sheila”

“sg”

“Sheila Walton”

“Sheila”

Context-FreeResolution

[2] Contextual Resolution

“Sheila”

social

social social

topical

“Sheila Tweed”

“sheila”

“sg”

“Sheila Walton”

“Sheila”

Outline

Introduction and Approach Overview Computational Model of Identity Context Reconstruction Mention Resolution Evaluation on Existing CollectionsEvaluation on Existing Collections Scalable MapReduce Implementation New Test Collection Conclusion

Test Collections

Collection Emails Identities Mention Candidates

Queries Min. Avg. Max.

Sager 1,628 627 51 1 4 11

Shapiro 974 855 49 1 8 21

Enron-subset 54,018 27,340 78 1 152 489

Enron-all 248,451 123,783 78 3 518 1785

Shapiro

Enron-subsetEnron-all

Evaluation Measures

Commonly used in “known-item” retrieval

Success @1 (i.e., Precision @1) One-best

MRR (Mean Reciprocal Rank) Inverse of the harmonic mean of the ranks of true

answer ri

i irnMRR

Comparison w/Literature

MRRMRR Success @ 1Success @ 1

ContextContext Lit.Lit. ContextContext Lit.Lit.

CollectionCollection ExpansionExpansion BestBest ExpansionExpansion BestBest

Sager 0.911 0.889 0.863 0.804

Shapiro 0.913 0.879 0.878 0.779

Enron-subset 0.91 - 0.846 (0.82)

Enron-all 0.89 - 0.821 -

ContextContextExpansionExpansion

Lit.Lit.BestBest

ContextContextExpansionExpansion

Lit.Lit.BestBest

Earlier expansion approach, reported in ACL 2008

Improvedexpansion

0.870.92

Limitations

Resolving single mentions

All mention-queries are sampled from Enron to Enron emails

All mention-queries refer to Enron Employee Small for train/test split

Scalable Implementation for Resolving All Mentions

New Test Collection

Outline

Introduction and Approach Overview Computational Model of Identity Context Reconstruction Mention Resolution Evaluation on Existing Collections Scalable MapReduce ImplementationScalable MapReduce Implementation New Test Collection Conclusion

Scalable Implementation

Two Bottlenecks:1. Context expansion of ALL emails

For each email: ranked list of “Similar” emails

2. Resolution of ALL mentionsResolution of one mention depends on resolution of all other mentions in context

Context Expansion of ALL Emails

Goal: For each email: ranked list of “Similar” emails Need for BOTH social and topical contexts Efficient implementation

Abstract Problem:Computing Pairwise Similarity

Trivial Solution

load each vector o(N) times load each term o(dft2) times

scalable and efficient solutionfor large collections

Better Solution

Load weights for each term once Each term contributes o(dft2) partial scores

Each term contributes only if appears in

MapReduce Framework

mapmap

reducereduce

output

ShufflingShuffling

group values group values by: [by: [keyskeys]]

(a) Map(a) Map (b) Shuffle(b) Shuffle (c) Reduce(c) Reduce

handles low-level details transparentlytransparently

(k2, [v2])(k1, v1)

[(k3, v3)][k2, v2]

reducereduce

Decomposition

Load weights for each term once Each term contributes o(dft2) partial scores

Each term contributes only if appears in

mapmap

Expansion Using MapReduce

Using generic pairwise-similarity for both topical and social expansion

~~~~~~~~~~~~

~~~~~~~~~~~--

~~~~~~~~~~~~

doc rep.

time window

rankcut-off

df-cut

topical : body/root/pat

hsocial : participants

temporal sim model

contextsim model

contextgraph

Context Mention-Graph

“Sheila”

social

conversational

social

topical

social

topical

“Sheila Tweed”

“sheila”

“sg”

“Sheila Walton”

“Sheila”

mapmap

reducereduce

Resolution System Using MapReduce

EmailsThreads Identity Models

Social Expansion

Conv.Expansion

Topical Expansion

LocalExpansion

Local Graph Social GraphTopical GraphConv. Graph

Mention Recognition and

Prior Computation

Prior Resolution

Posterior Resolution

Social Resolution

Conv.Resolution

Topical Resolution

LocalResolution

Merging Context Resolutions

PriorPriorPrior

PreprocessingPreprocessing

Outline

Introduction and Approach Overview Computational Model of Identity Context Reconstruction Mention Resolution Evaluation on Existing Collections Scalable MapReduce Implementation New Test CollectionNew Test Collection Conclusion

New Test Collection

Random Sample from CMU-Enron collection “Annotation + Search” interface available Total annotators: 3 Annotation time: ~50 hours. Not only resolutions

Time, difficulty, confidence, evidence, and comments

Total mention-queries : 584 80% resolvable, 82% of them to Enron domain Overall inter-annotator agreement: ~81 %

Mention-Query Selection

Distribution of Names Based on Resolution

Enron-Resolvable390 (66%)

Non-Enron-Resolvable

80 (14%)

Unresolvable114 (20%)

Probably- Enron 24 (4%)

Probably-Non-Enron

62 (11%)

Unknown28 (5%)

Distribution Based on Difficulty

Enron-Resolvable Non-Enron-Resolvable

Hard Moderately Hard Easy

Distribution Based on Confidence

Unresolvable

Not Confident Somewhat Confident Very Confident

Distribution of Time Spent

R2 = 0.98

R2 = 0.95

R2 = 0.88

0 1 2 3 4 5 6 7 8 9 10 11 12

Time (minutes)

Enron-Resolvable

Non-Enron-Resolvable

Unresolvable

Non-Enron

Unresolvable

Evaluation again …

0.690.65

0.760.71

0.800.76

Enron Non-Enron Enron All

topical social combination

Old Collection New Collection

Pairwise Agreement

50 190

16/16 (100%)

2/4 (50%)

4/7 (57%)

12/16 (75%)

1/5 (20%)

2/2 (100%)

35/38 (92%)

2/2 (100%)

6/10 (60%)

24/27 (89%)

5/12 (42%)

11/11 (100%)

Enron-resolvable

Non-enron-resolvable

Unresolvable

Individual Annotator Agreement

37/40 2/224/27

Enron-Resolvable Non-enron-Resolvable

Unresolvable

a1 a2 a3

Overall Agreement

0.810.77

Enron-Resolvable

Non-enron-Resolvable

ResolvableOverall

Unresolvable

Agreement Based on Difficulty

Easy Non-Easy

Agreement Based on Confidence

Unresolvable

Very-Confident Not-Very-Confident

Conclusion and Future Work

Identity Resolution by non-participants is feasible And automatic systems for that can be built ~90-75% accurate

Proposed generative probabilistic model Context Expansion using temporal similarity Scalable Implementation using “Pairwise Sim with MapReduce”

Developed largest test collection for the task 80% resolvable, 82% of them to Enron employees

Effectiveness scales well to large collections

Efficiency Results Evaluation using double-assessments Iterative approach for “joint resolution”

Thank You!

Related Work

Diehl et al. (SIAM, 2006) Developed Enron-subset collection Temporal traffic models Candidates must have communicated with sender

Minkov et al. (SIGIR, 2006) Developed Sager and Shapiro collections Graphical framework Large collections?

Identity Resolution in Email Collections Tamer Elsayed and Douglas W. Oard CLIP Colloquium, UMD, Feb 2009 Department of Computer Science, UMIACS, and iSchool.

sheila identity resolution

elizabeth x

easy references of identity

context reconstruction

liza elizabeth sager

ischool slide

elizabeth murphy

c px c mention

Documents

Dr. Abdullah M. Elsayed

iSchool Drexel - Student Organization

CoCollage UW iSchool 20090515

Financial Cyberinfrastructure - umiacs - University of...

CV Mohamed Elsayed Rev.091216

SJSU iSchool Information Session: January 2015

NYC iSchool By-Laws

iSchool E-Club

MLE EM MAP - UMIACS

Elsayed Elsayed Hafez (IYSV) Egyptian Isolate , Ahmed A. …...

SJSU iSchool Information Session

El Avión con Fusion Douglas W. Oard iSchool and UMIACS...

Elsayed Amer CV

What Question Answering can Learn from Trivia Nerds ·...

Presentation Elsayed Elasy

Mahmoud Amin ElSayed