Search Results Need to be Diverse Mark Sanderson University of Sheffield.

Search Results Need to be Diverse

Mark SandersonUniversity of Sheffield

How to have fun while running an evaluation campaignMark SandersonUniversity of Sheffield

Aim

• Tell you about our test collection work in Sheffield

• How we’ve been having fun building test collections

04/18/23

3

Organising this is hard

• TREC•Donna, Ellen

• CLEF•Carol

• NTCIR•Noriko

• Make sure you enjoy it

04/18/23

4

ImageCLEF

• Cross language image retrieval

• Running for 6 years

• Photo

• Medical

• And other tasks

• Imageclef.org

04/18/23

5

How do we do it?

• Organise and conduct research

• imageCLEFPhoto 2008• Study diversity in search results

•Diversity?

04/18/23

6

SIGIR

04/18/23

7

ACL

04/18/23

8

Mark Sanderson

04/18/23

9

Cranfield model

04/18/23

10

Operational search engine

• Ambiguous queries•What is correct interpretation?

• Don’t know• Serve as diverse a range as possible

04/18/23

11

Diversity is studied

• Carbonell, J. and Goldstein, J. (1998) The use of MMR, diversity-based reranking for reordering documents and producing summaries. In ACM SIGIR, 335-336.

• Zhai, C. (2002) Risk Minimization and Language Modeling in Text Retrieval, PhD thesis, Carnegie Mellon University.

• Chen, H. and Karger, D. R. (2006) Less is more: probabilistic models for retrieving fewer relevant documents. In ACM SIGIR, 429-436.

Cluster hypothesis

• “closely associated documents tend to be relevant to the same requests”• Van Rijsbergen (1979)

04/18/23

13

Most test collections

• Focussed topic

• Relevance judgments•Who says what is relevant?

• (almost always) one person

•Consideration of interpretations• Little or none

• Gap between test and operation

Few test collections

• Hersh, W. R. and Over, P. (1999) Trec-8 interactive track report. TREC-8

• Over P. (1997) TREC-5 Interactive Track Report. TREC-5, 29-56

• Clarke, C. L., Kolla, M., Cormack, G. V., Vechtomova, O., Ashkan, A., Büttcher, S., and MacKinnon, I. (2008) Novelty and diversity in information retrieval evaluation. In ACM SIGIR.

04/18/23

15

Study diversity

• What sorts of diversity is there?• Ambiguous query words

• How often is it a feature of search?•How often are queries ambiguous?

• How can we add it into test collections?

04/18/23

16

Extent of diversity?

• “Ambiguous queries: test collections need more sense”, SIGIR 2008

• How do you define ambiguity?•Wikipedia

•WordNet

04/18/23

17

Disambiguation page

04/18/23

18

Wikipedia stats

• enwiki-20071018-pages-articles.xml• (12.7Gb)

• Disambiguation pages easy to spot• “_(disambiguation)” in title

Chicago

• “{{disambig}}” templateGeorge_bush

Conventional source

• Downloaded WordNet v3.0

• 88K words

04/18/23

20

Query logs

Log Unique queries (all)

Most frequent (fr)

Year(s) gathered

Web 1,000,000 8,719 2006

PA 507,914 14,541 2006-7

Fraction of ambiguous

1 2 3Name Wi WN WN+Wi

Web freq 7.6% 4.0% 10.0%

all 2.5% 0.8% 3.0%PA freq 10.5% 6.4% 14.7%

all 2.1% 0.8% 2.7%

Conclusions

• Ambiguity is a problem

• Ambiguity is present in query logs•Not just Web search

• Ambiguity present?•Need for IR systems to produce diverse

results

04/18/23

23

Test collections

• Don’t test for diversity

• Do search systems deal with it?

04/18/23

24

ImageCLEFPhoto

• Build a test collection• Encourage the study of diversity

• Study how others deal with diversity

•Have some fun

04/18/23

25

Collection

• IAPR TC-12

• 20,000 travel photographs

• Text captions

• 60 existing topics•Used in two previous studies

• 39 used for diversity study

04/18/23

26

Diversity needs in topic

• “Images of typical Australian animals”

04/18/23

27

Types of diversity

• 22 geographical• “Churches in Brazil”

• 17 other• “Australian animals”

04/18/23

28

Relevance judgments

• Clustered existing qrels

• Multiple assessors•Good level of agreement on clusters

04/18/23

29

Evaluation

• Precision at 20• P(20)

• Fraction of relevant in top 20

• Cluster recall at 20•CR(20)

• Fraction of different clusters in top 20

04/18/23

30

Track was popular

• 24 groups

• 200 runs in total

04/18/23

31

Submitted runs

04/18/23

32

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.0 0.1 0.2 0.3 0.4 0.5 0.6

CR20

P20

Compare with past years

• Same 39 topics used in 2006, 2007• But without clustering

• Compare cluster recall on past runs• Based on identical P(20)

• Cluster recall increased• Substantially

• Significantly

04/18/23

33

Meta-analysis

• This was fun

• We experimented on participants outputs

• Not by design• Lucky accident

04/18/23

34

Not first to think of this

• Buckley and Voorhees• SIGIR 2000, 2002

• Use submitted runs to generate new research

04/18/23

35

Conduct user experiment

• Do users prefer diversity?

• Experiment• Build a system to do this

• Show users• your system• Baseline system

•Measure users

04/18/23

36

Why bother…

• …when others have done the work for you

• Pair up randomly sampled runs•High CR(20)

• Low CR(20)

• Show to users

04/18/23

37

Animals swimming

04/18/23

38

Numbers

• 25 topics

• 31 users

• 775 result pairs compared

04/18/23

39

User preferences

• 54.6% more diversified;

• 19.7% less diversified;

• 17.4% both were equal;

• 8.3% preferred neither.

04/18/23

40

Conclusions

• Diversity appears to be important

• System don’t do diversity by default

• Users prefer diverse results

• Test collections don’t support diversity• But can be adapted

04/18/23

41

and

• Organising evaluation campaigns is rewarding• And can generate novel research

04/18/23

42