Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics

Musings at the Crossroads ofMusings at the Crossroads ofDigital Libraries, Information Retrieval, Digital Libraries, Information Retrieval,

and Scientometricsand Scientometrics

http://bit.ly/rguCabanac2012http://bit.ly/rguCabanac2012

Guillaume Cabanacguillaume.cabanac@univ-tlse3.fr

March 28th, 2012

Outline of these Musings

Musings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

Digital LibrariesDigital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity

Information RetrievalInformation Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators

ScientometricsScientometrics Recommendation based on topics and social clues Landscape of research in Information Systems

The submission-date bias in peer-reviewed conferences

Guillaume Cabanac

Question DL-1

How to transpose paper-based annotations into digital documents?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Collective annotation: Perspectives for information retrieval improvement.” RIAO’07 : Proceedings of the 8th conference on Information Retrieval and its Applications, pages 529–548. CID, may 2007.

Characteristics of paper annotation Secular activity: older than 4 centuries Numerous applicative contexts: theology, science, literature … Personal use: “active reading” (Adler & van Doren, 1972)

Collective use: review process, opinion exchange …

From Individual Paper-based Annotation …

US students

(Marshall, 1998)

Annotated bible

(Lortsch, 1910)

Fermat’s last theorem

(Kleiner, 2000)

Annotations from Blake, Keats…

(Jackson, 2001)

Les Misérables

Victor Hugo

1630 1790 1830 1881 1998

Guillaume Cabanac

… to Collective Digital Annotations

author

reader13%

1993 2005

ComMentor … iMarkup … Yawas … Amaya …

> 20 annotation systems(Cabanac et al., 2005)

Web servers (Ovsiannikov et al., 1999)

Annotation server

a discussion thread

Hard to share ‘lost’

hardcopy

Guillaume Cabanac

W3C Annotea / Amaya (Kahan et al., 2002)

Digital Document Annotation: Examples

a reader’s comment

discussionthread

Arakne, featuring “fluid annotations” (Bouvin et al., 2002)

Guillaume Cabanac

Collective Annotations Reviewed 64 systems designed during 1989–2008

Collective Annotation Objective data

Owner, creation date Anchoring point within the document. Granularity: all doc, words…

Subjective information Comments, various marks: stars, underlined text… Annotation types: support/refutation, question… Visibility: public, private, group…

Purpose-oriented annotation categoriesAnnotation remark

Annotation reminder

Annotation argumentation

Personal Annotation Space

Guillaume Cabanac

Question DL-2

How to measure the social validity ofa statement according to the

argumentative discussion it sparked off?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Social validation of collective annotations : Definition and experiment.” Journal of the American Society for Information Science and Technology, 61(2):271–287, feb. 2010, Wiley. DOI:10.1002/asi.21255

Scalability issue

Which annotationsshould I read?

Social validation = degree of consensus of the group

Social Validation

Social Validation of Argumentative DebatesMusings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

Social Validation of Argumentative Debates

BeforeAnnotation magma

AfterFiltered display

Informing readers about how validated each annotation is

Guillaume Cabanac

Overview

Two proposed algorithms Empirical Recursive Scoring Algorithm (Cabanac et al., 2005)

Bipolar Argumentation Framework Extension based on Artificial Intelligence research works (Cayrol & Lagasquie-Schiex, 2005)

Social Validation Algorithms

validity

0socially neutral

– 1 socially refuted

1socially confirmed

case 1case 2case 3 case 4

Guillaume Cabanac

Example

Computing the social validity of a debated annotation

Social Validation AlgorithmMusings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

Validation with a User-study

Design

Corpus: 13 discussion threads= 222 annotations + answers

Task of a participant Label opinion type Infer overall opinion

Volunteer subjects

Aim: social validation vs human perception of consensus

Guillaume Cabanac

Q1 Do people agree when labeling opinions? Kappa coefficient (Fleiss, 1971; Fleiss et al., 2003)

Inter-rater agreement among n > 2 raters

Weak agreement, with variability subjective task

Experimenting the Social Validation of Debates

Debate Id

Fair to good

Guillaume Cabanac

agreement

Q2 How well SV approximates HP? HP = Human Perception of consensus SV = Social Validation algorithm

1. Test whether PH and VS are different (p < 0.05) Student’s paired t-test: (p = 0,20) > ( = 0,05)

2. Correlate HP et SV Pearson’s coefficient of correlation r

r(HP, SV) = 0.48 shows a weak correlation

Experimenting the Social Validation of Debates

HP – SV

Density y = p(HP – SV)

example: HP = SV for 24 % of all cases

Guillaume Cabanac

Question DL-3

How to harness a quiescent capital present in any community:

its documents?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Organization of digital resources as an original facet for exploring the quiescent information capital of a community.” International Journal on Digital Libraries, 11(4):239–261, dec. 2010, Springer. DOI:10.1007/s00799-011-0076-6

Personal Documents Filtered, validated, organized information…

… relevant to activities in the organization

Paradox: profitable, but under-exploited Reason 1 – folders and files are private

Reason 2 – manual sharing

Reason 3 – automated sharing

Consequences People resort to resources available outside of the community Weak ROI why would we have to look outside when it’s already there?

Documents as a Quiescent WealthMusings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

Mapping the documents of the community SOM [Kohonen, 2001] Umap [Triviumsoft] TreeMap [Fekete & Plaisant, 2001]…

Limitations Find the documents with same topicssame topics as D Find documents that colleagues useuse with D

concept of usage: grouping documentsgrouping documents ⇆ keeping stuff in commonkeeping stuff in common

How to Benefit from Documents in a Community?Musings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

Organization-based similarities inter-folder

inter-document

inter-user

Guillaume CabanacHow to Benefit from Documents in a Community?

Purpose: Offering a global view of … people and their documents

Based on document contents Based on document usage/organization

Requirement: non-intrusiveness and confidentiality

OperationalOperational needs Find documents

With related materials With complementary materials

Seeking people ⇆ seeking documents

ManagerialManagerial needs Visualize the global/individual activity Work position required documents

How to Help People to Discover/Find/Use Documents?

community

Guillaume Cabanac

4 views = {documents, people} {group, unit}

1. Group of documents Main topics Usage groups

2. A single document Who to liaise with? What to read?

3. Group of people Community of interest Community of use

4. A single people Interests Similar users (potential help)

Proposed System: Static AspectMusings at the Crossroads of DL, IR, and SCIM

Guillaume Cabanac

Question IR-1

Is document tie-breaking affecting the evaluation of

Information Retrieval systems?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac, Gilles Hubert, Mohand Boughanem, Claude Chrisment. “Tie-breaking Bias : Effect of an Uncontrolled Parameter on Information Retrieval Evaluation.” M. Agosti, N. Ferro, C. Peters, M. de Rijke, and A. F. Smeaton (Eds.) CLEF’10 : Proceedings of the 1st Conference on Multilingual and Multimodal Information Access Evaluation, volume 6360 de LNCS, pages 112–123. Springer, sep. 2010. DOI:10.1007/978-3-642-15998-5_13

Measuring the Effectiveness of IR systems User-centered vs. System-focused [Spärck Jones & Willett,

Evaluation campaigns 1958 Cranfield, UK 1992 TREC (Text Retrieval Conference), USA 1999 NTCIR (NII Test Collection for IR Systems), Japan 2001 CLEF (Cross-Language Evaluation Forum), Europe …

“Cranfield” methodology Task Test collection

Corpus Topics Qrels

Measures : MAP, P@X ... using trec_eval

[Voorhees, 2007]

Guillaume Cabanac

Runs are Reordered Prior to Their EvaluationQrels = qid, iter, docno, rel Run = qid, iter, docno, rank, sim,

run_id

Reordering by trec_evalqid asc, sim desc, docno desc

Effectiveness measure = f (intrinsic_quality, )MAP, P@X, MRR…

Guillaume Cabanac

Consequences of Run Reordering Measures of effectiveness for an IRS s

RR(s,t) 1/rank of the 1st relevant document, for topic t

P(s,t,d) precision at document d, for topic t

AP(s,t) average precision for topic t

MAP(s) mean average precision

Tie-breaking bias

Is the Wall Street Journal collection more relevant than Associated Press?

Problem 1comparing 2 systemsAP(s1, t) vs. AP(s2, t)

Problem 2 comparing 2 topicsAP(s, t1) vs. AP(s, t2)

ChrisChris

EllenEllen

Sensitive to document

Guillaume Cabanac

What we Learnt: Beware of Tie-breaking for AP Poor effect on MAP, larger effect on AP

Measure bounds APRealistic APConventionnal APOptimistic

Failure analysis for the ranking process Error bar = element of chance potential for improvement

padre1, adhoc’94

Guillaume Cabanac

Question IR-2

How to retrieve documents matching keywords and

spatiotemporal constraints?

IRIRDLDL

SCIMSCIM

Damien Palacio, Guillaume Cabanac, Christian Sallaberry, Gilles Hubert. “On the evaluation of geographic information retrieval systems: Evaluation framework and case study.” International Journal on Digital Libraries, 11(2):91–109, june 2010, Springer. DOI:10.1007/s00799-011-0070-z

Geographic Information Retrieval Query = “Road trip around Aberdeen summer 1982”

Search engines Topic term {road, trip, Aberdeen, summer}

spatial {AberdeenCity, AberdeenCounty…} Geographic temporal [21-JUN-1982 .. 22-SEP-1982]

term {road, trip, Aberdeen, summer}

1/6 queries = geographic queries Excite (Sanderson et al., 2004) AOL (Gan et al., 2008) Yahoo! (Jones et al., 2008)

Current issue worth studying

Guillaume Cabanac

The Internals of a Geographic IR System 3 dimensions to process

Topical, spatial, temporal

1 index per dimension Topic bag of words, stemming, weighting, comparing with VSM… Spatial spatial entity detection, spatial relation resolution… Temporal temporal entity detection…

Query processing with sequential filtering e.g., priority to theme, then filtering according to other dimensions

Issue: effectiveness of GIRSs vs state-of-the-art IRSs?

Hypothesis: GIRSs better than state-of-the-art IRSs

Guillaume Cabanac

Case Study: the PIV GIR System Indexing: one index per dimension

Topical = Terrier IRS Spatial = tiling Temporal = tiling

Retrieval Identification of the 3 dimensions in the query Routing towards each index Combination of results with CombMNZ [Fox & Shaw, 1993; Lee 1997]

Guillaume Cabanac

Case Study: the PIV GIR System Principle of CombMNZ and Borda Count

Guillaume Cabanac

Case Study: the PIV GIR System Gain in effectiveness

Guillaume Cabanac

Question IR-3

Do operators in search queries improve the effectiveness of search results?

IRIRDLDL

SCIMSCIM

Gilles Hubert, Guillaume Cabanac, Christian Sallaberry, Damien Palacio. “Query Operators Shown Beneficial for Improving Search Results.” S. Gradmann, F. Borri, C. Meghini, H. Schuldt (Eds.) TPDL’11 : Proceedings of the 1st International Conference on Theory and Practice of Digital Libraries, volume 6966 de LNCS, pages 118–129. Springer, sep. 2011. DOI:10.1007/978-3-642-24469-8_14.

Various Operators Quotation marks, Must appear (+), boosting operator (^),

Boolean operators, proximity operators…

Information need

“I’m looking for research projects funded in the DL domain”

Regular query Query with operators

Search Engines Offer Query Operators

Guillaume Cabanac

Our Research Questions

Guillaume Cabanac

Our Methodology in a Nutshell

Regular query V1: Query variant with operators

V4VN. . .

Guillaume Cabanac

Effectiveness of Query Operators TREC-7 per Topic Analysis: Boxplots

‘+’ and ‘^’

Guillaume Cabanac

Effectiveness of Query Operators Per Topic Analysis: Box plot

AP of TREC’s regular query

Query variant highest AP

32Topics

Query variant lowest AP

Guillaume Cabanac

Effectiveness of Query Operators TREC-7 Per Topic Analysis

‘+’ and ‘^’

MAP = 0.1554

MAP ┬ = 0.2099+35.1%

Guillaume Cabanac

Question SCIM-1

How to recommend researchers according to their research topics

and social clues?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac. “Accuracy of inter-researcher similarity measures based on topical and social clues.” Scientometrics, 87(3):597–620, june 2011, Springer. DOI:10.1007/s11192-011-0358-1

Recommendation of Literature (McNee et al., 2006)

Collaborative filtering Principle: mining the preferencespreferences of researchers

those who liked this paper also liked…

Snowball effect / fad Innovation? Relevance of theme?

Cognitive filtering Principle: mining the contentscontents of articles

profile of resources (researcher, articles) citation graph

Hybrid approach

Guillaume Cabanac

Foundations: Similarity Measures Under Study

Model Coauthors graph authors auteurs Venues graph authors conferences / journals

Social similarities Inverse degree of separation length of the shortest path Strength of the tie number of shortest paths Shared conferences number of shared conference editions

Thematic similarity Cosine on Vector Space Model di = (wi

1, … , win)

built on titles (doc / researcher)

Guillaume Cabanac

Computing Similarities with Social Clues Task of literature review

Requirement topical relevance Preference social proximity (meetings, project…)

re-rank topical results with social clues

Combination with CombMNZ (Fox & Shaw, 1993)

Final result: list of recommended researchers

CombMNZDegree of separation

Strength of ties

Shared conferences

Social list

Topical list

CombMNZ TS listTS list

Guillaume Cabanac

Evaluation Design Comparison of recommendations and researchers’ perception

Q1 : Effectiveness of topical (only) recommendations? Q2 : Gain due to integrating social clues?

IR experiments: Cranfield paradigm (TREC…) Does the search engine retrieve relevant documents?

Doc relevant?

assessor

relevance judgments{0, 1} binary[0, N] gradual

trec_eval

Effectiveness measuresMean Average PrecisionNormalized Discounted Cumulative Gain

topic S1 S2

1 0.5687 0.6521

… … …

50 0.7124 0.7512

avg 0.6421 0.7215

improvement +12.3 % significativity p < 0.05 (paired t-test)

search engine xinput

corpus

Guillaume Cabanac

Evaluating Recommendations

doc relevant ?

assessor

relevance judgments{0, 1} binary[0, N] gradual

trec_eval

Effectiveness measures Mean Average PrecisionNormalized Discounted Cumulative Gain

topic S1 S21 0.5687 0.6521

… … …

50 0.7124 0.7512

avg 0.6421 0.7215

improvement +12.3 % significativity p < 0.05 (paired t-test)

search engine xinput

corpus

name of a researcher

researcher

« With whom would you like to chat for improving your research? »

recommender system

topical topical + social

#subjects

Top 25

Guillaume Cabanac

Experiment Features

Data dblp.xml (713 MB = 1.3M publications for 811,787 researchers) Subjects 90 researchers-contacts contacted by mail

74 researchers began to fill the questionnaire. 71 completed it

Interface for assessing recommendations

Guillaume Cabanac

Experiments: Profile of the Participants Experience of the 71 subjects Mdn = 13 years

Productivity of the 71 subjects Mdn = 15 publications

Guillaume Cabanac

Seniority

Number of publications

Empirical Validation of our Hypothesis Strong baseline effective approach based on VSM

+8.49 % = significant improvement (p < 0.05 ; n = 70)

of topical recommendations by social clues

global < 15 publis >= 15 publis < 13 ans >= 13 ans

Thématique Thématique + Social

productivity experience

+8,49 %+8,49 % +10,39 %+10,39 % +7,03 %+7,03 % +6,50 %+6,50 % +10,22 %+10,22 %

Guillaume Cabanac

Topical Topical + social

yearsyears

Guillaume Cabanac

Question SCIM-2

What is the landscape of research in Information Systems from the perspective of gatekeepers?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac. “Shaping the landscape of research in Information Systems from the perspective of editorial boards : A scientometric study of 77 leading journals.” Journal of the American Society for Information Science and Technology, 63, to appear in 2012, Wiley. DOI:10.1002/asi.22609

Landscape of Research in Information Systems The gatekeepers of science

Guillaume Cabanac

Landscape of Research in Information Systems The 77 core peer-reviewed IS journals in the WoS

Guillaume Cabanac

Landscape of Research in Information Systems Exploratory data analysis

Guillaume Cabanac

Landscape of Research in Information Systems Exploratory data analysis

Guillaume Cabanac

Landscape of Research in Information Systems Topical map of the IS field

Guillaume Cabanac

Landscape of Research in Information Systems Most influential

gatekeepers

Guillaume Cabanac

Landscape of Research in Information Systems Number of gatekeepers per country

Guillaume Cabanac

Landscape of Research in Information Systems Geographic and gender diversity

Guillaume Cabanac

Question SCIM-3

What if submission date influenced the acceptance of conference papers?

IRIRDLDL

SCIMSCIM

Guillaume Cabanac. “What if submission date influenced the acceptance of conference papers?” Submitted to the Journal of the American Society for Information Science and Technology, Wiley.

Conferences Affected by a Submission-Date bias? Peer-review

Guillaume Cabanac

The Submission-Date bias Dataset from the ConfMaster conference management system

Guillaume Cabanac

The Submission-Date bias Influence of submission date on bids

Guillaume Cabanac

The Submission-Date bias Influence of submission date on average marks

Guillaume Cabanac

Conclusion

Guillaume Cabanac

Thank you

http://www.irit.fr/~Guillaume.Cabanachttp://www.irit.fr/~Guillaume.Cabanac

Twitter: @tafanorTwitter: @tafanor

Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics

Education

SYNCOPATED MUSINGS

History and Institutionalisation of Scientometrics Berlin

Scientometrics class

Millennial Musings

paper in Scientometrics Journal, 2010

Musings of Thad

brain Scientometrics

Lever Delayed Musings

Philosophical Musings

Chapter Five Scientometrics / Bibliometrics...

Scrum Musings

Scientometrics with Information Retrieval and Psychology

Madras Musings

International Society for Scientometrics and Informetrics -....

international panorama of scientometrics...Scientometrics...

MUSEUM MUSINGS