Time-aware Evaluation of Cumulative Citation Recommendation Systems Krisztian Balog University of Stavanger SIGIR 2013 workshop on Time-aware Information Access (#TAIA2013) | Dublin, Ireland, Aug 2013 Laura Dietz, Jeffrey Dalton CIIR, University of Massachusetts, Amherst
22
Embed
Time-aware Evaluation of Cumulative Citation Recommendation Systems
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Time-aware Evaluation of Cumulative Citation Recommendation Systems
Krisztian Balog University of Stavanger
SIGIR 2013 workshop on Time-aware Information Access (#TAIA2013) | Dublin, Ireland, Aug 2013
Laura Dietz, Jeffrey DaltonCIIR, University of Massachusetts, Amherst
- Filter a time-ordered corpus for documents that are highly relevant to a predefined set of entities
- For each entity, provide a ranked list of documents based on their “citation-worthiness”
Evaluation metrics are set-based (using a confidence cut-off)
Aims- Develop a time-aware evaluation paradigm for
streaming collections- Capture how retrieval effectiveness changes over time- Deal with ground truth of bursty nature- Accommodate various underlying user models
- Test the ideas on CCR
Overview
time1. Slicing time
2. Measuring slice relevance
3. Aggregating slice relevance.87
.65
Slice importance
Overview
time
.87
.65
Slice importance
1. Slicing time
Slicing time- Simplifying assumptions
- Slices are non-overlapping- Unconcerned about slices that don’t contain any
relevant documents
(A) Uniform slicing- Slices of equal length
(B) Non-uniform slicing- Slices of varying length
#relevant
time
(A)(B)
ti
Overview
time
.87
.65
Slice importance
2. Measuring slice relevance
Measuring slice relevance- Ranked list of documents within a given slice
- Evaluation metric
- Standard IR metrics- MAP, R-Prec, NDCG
d =< d1, . . . , dn >
m(di, q)
Overview
time
.87
.65
Slice importance
3. Aggregating slice relevance
Aggregating slice relevance- Probabilistic formulation to estimate the
likelihood of relevance
P (r = 1|d, q,m) =X
i2I
P (r = 1|di, q, i)P (i|q)
Slice-based relevance
Slice importance
⇡ m(di, q)
Slice importance- Uniform slicing
- All slices are equally important
- Non-uniform slicing- Bursty periods (i.e., slices with more relevant
documents) are more important
P (i|q) =1I
P (i|q) =#R(i, q)Pi2I #R(i, q)
Experiments- Official TREC 2012 KBA CCR runs
- 8 systems, best run for each system
- Only uniform time slicing- Binary relevance
ResultsAtemporal vs. temporal ranking (MAP, weekly slicing)