/
.
.: 974
()
2015
..
.
- - ( ) - , . - , . , - , , -. -
, . , , - . , , . , , - /, . , , , ., -
, -. Bayes .
i
ABSTRACT
Information retrieval constitutes an important scientific area
of the com-puter science, that focuses on the extraction of amounts
of unstructuredinformation (usually text from documents) from large
collections (corpora,etc.) according to a special information need
of a user. Over the last years,one major task of information
retrieval research is the evaluation of the re-trieval process. As
a result, a vast amount of evaluation metrics and usermodels have
been developed, trying to best model users behaviour duringthe
search.
In this thesis we propose a new evaluation metric which aims at
the bestevaluation of search process from the perspective of users
behaviour. A con-ventional approach when estimating the relevance
of a document is by usingrelevance judgements from assessors that
are responsible to assess whether adocument is relevant according
to a specific query. However, relevance judge-ments do not always
reflect the opinion of every user, rather from a smallproportion
only. Our evaluation metric introduces a novel factor of
relevance,document popularity which can be seen as users vote for a
document. Thus,by employing a linear combination of relevance
judgements and popularity,we achieve a better explanation of users
behaviour.
Additionally, we present a novel click user model which by the
best mod-elling of users navigational behaviour, aims at the best
estimation of the rel-evance of a document. This particular user
model, is based on the dynamicBayesian networks theory and employs
the notion of popularity in order tobetter estimate actual document
relevance, rather perceived relevance, thatmost other models
do.
ii
-, . , , . .
. , ., ,
.
iii
1 11.1 . . . . . . . . . . . . . . 61.2 . . . . . . . . . . . .
. . . . . . . . . . . . . . 61.3 . . . . . . . . . . . . . . . 71.4
. . . . . . . . . . . . . . . . . 81.5 . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . 81.6 . . . . . . . . . . . . . . . .
. . . . . . . 11
1.6.1 . . . . . . . . . . . . . . . . . . 111.6.2 . . . . . . .
. . . 13
1.7 . . . . . . . . . . . . . . . . . . . . . 14
2 162.1 . . . . . 17
2.1.1 - TREC . . . 192.2 . . . . . . . . . . . . . . . . . . . .
. . . 22
2.2.1 . . . . . . . . . . . . . . . . . . 232.2.2 . . . . . . .
. . . . . . . . . . 26
2.3 (DCG) . . . . . . . . . 292.3.1 Rank Biased Precision . . .
. . . . . . . . . . . . . . . 322.3.2 Bpref . . . . . . . . . . . .
. . . . . . . . . . . . . . . . 342.3.3 Expected Reciprocal Rank .
. . . . . . . . . . . . . . . 36
2.4 . . . . . . . . . . . . . 392.5 . . . . . . . . . . . . . .
. . . 412.6 . . . . . . . . . . . . . . 42
2.6.1 Crowdsourcing . . . . . . . . . . . . . . . . . . . . . .
432.6.2 / . . . . . . . . . . . . . . . . . . . . . . . 44
3 453.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. 453.2 Reciprocal Rank using Document Popularity . . . . . . . . .
. 47
iv
3.2.1 . . . . . . . . . . . . . . . . . . . . . 473.2.2 . . . .
. . . . . . . . . . . . . . 483.2.3 . . . . . . . . . . . . . . . .
. . 49
3.3 . . . . . . . . . . . . . . . . . . . . . . 513.3.1 . . . .
. . . . . . . . . . 51
3.4 . . . . . . . . . . . . . . . . . . . 543.4.1 . . . . 56
4 594.1 . . . . . . . . . . . . . . . . . . . . . . . . . .
61
4.1.1 COEC . . . . . . . . . . . . . . . . . . . . 614.1.2 . . .
. . . . . . . . . . . . . . . . 624.1.3 . . . . . . . . . . . . . .
. . . . . 62
4.2 cascade . . . . . . . . . . . . . . . . . . . . . . . .
624.3 UBM . . . . . . . . . . . . . . . . . . . . . . . . . 634.4
DCM . . . . . . . . . . . . . . . . . . . . . . . . . 654.5 DBN . .
. . . . . . . . . . . . . . . . . . . . . . . 68
5 - 735.1 . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . 735.2 . . . . . . . . . . . . . . . . . . . . . . . . 755.3 . .
. . . . . . . . . . . . . . . . . . . . 77
5.3.1 . . . . . . . . . . . . . . 775.3.2 . . . . . . . . . . .
. . . . . 78
6 806.1 . . . . . . . . . . . . 826.2 . . . . . . . . . . . . .
. . . . . . . 83
Bayes 85.1 . . . . . . . . . . . . . . . . . . . . . . . . . . .
86.2 . . . . . . . . . . . . . . . . . . . . . . 90
v
1.1 Yahoo!, http://www.yahoo.com, 1995. . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . 4
1.2 . . . . . . . . . . . . . . . . 51.3 XML . . . . . . . . . .
. . . . . . . . . . . 71.4 .
, - back-end , - - . . . . . . . . . . . . . . . . . . . . . . .
. . . . . 10
1.5 Google 1998. . . . . . . . . . . . . 121.6 . . . . . . . . .
. . 131.7 snippet . . . . . 14
2.1 Cranfield. , . , - ., , stopwords , - ( [CMK66]). . . .
18
2.2 TREC TREC 7 1998. Ellen Voorhees, . . . . . . . . . . . . .
. . . . . 22
2.3 TREC . . . . . . . . . . . . . . 232.4 I.
R - F . D . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . 24
2.5 2.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. 25
vi
2.6 . - 10. 6, 4 . . . . . . . . . . . . . . . . . . . 27
2.7 RBP . [08]. . . . . . . . . 322.8 RBP TREC-5 50 -
. p , 10 . - 100 p. [08]. . . . . . . . . . . . 34
2.9 - . [BV04]. . . . . . . 36
2.10 . ERR . 1, 0. [CMZG09]. . . . . . . . . . . . . . . . .
39
2.11 Kendalls - 61 TREC-5 . . . . . . . . . . . . . . . . . .
40
2.12 /. . . . . . . . . . . . . . . . . 44
3.1 , TREC Web Tracks. . . . . . . . . . . . . . . . . . . . . .
. . 52
3.2 . . . . . . . . . . . 57
3.3 41 . . . . . . 58
3.4 83 .. . . . . 58
4.1 DCM . . . . . . . . . . . . . . . . . . . . . . . . . 674.2
-
. . . . . . . . . . 684.3 DBN . . . . . . . . . . . . . . . . .
. 704.4 CTR
1 () KL- (). . . . . . . . . . . . . . . . . . . . . . 71
4.5 NDCG DBN . . . . . . . . 72
vii
5.1 . - - . . . . . . . . . . . . . . . . . . . . . . . 76
5.2 Log-Likelihood . DBN . . . . . . 79
.1 HMM DBN , 3 . 87.2 HMM (
) . - : P (X1 = i) = (i), P (Xt = j|Xt1 = i) =A(i, j), P (Yt =
j|Xt = i) = B(i, j). . . . . . . . . . . . . . 88
.3 HMM Gaussians . . . . . . . . . 89.4 Auto-regressive HMM. . .
. . . . . . . . . . . . . . . . . . . . 89
viii
2.1 - . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
3.1 4 . . . . . . . . . . 493.2 . 553.3
. . . . . . . . . . . . . . 56
ix
1
, , , . , . , -
, Taube [Tau54], , -. , . Cranfield , Cleverdon - .
.., . - Cleverdon 18.000 1.200 - . , , -.
, , , - 1952 Cleverdon , -
1
. -
, . - . (relevance judgments ) , - . 1990, -
, . , , . , - , , . ,
, . , , - - . , , , , . , -
, . , - . , , , , , .,
, . ,
2
, - . , [KT03] - . -
, - [ABDR06]. . [CZTR08], [DP08] - ., -
, , . , - , . (click-logs) .
, , . , - -, . ( 1.1)
Boolean , , -. , - -. , Salton[Sal68] , -, .
-
3
1.1: Yahoo!, http://www.yahoo.com, 1995.
. ( ) , [MRS08] ( 1.2). , -
. - , . , , ,
4
1.2:
.
- , , , , . - 1990. (). - , - ., ,
. -, , . , - , - . , , . ,
5
- . . -
, , - , - . , - .
1.1
. , , , . , -
. , , - , . , , -, . , - .
1.2
. - , . ; -. , . - . , , .
6
, . , - , ., -
. - . , , , , .. - , . XML - ( 1.3).
1.3: XML .
1.3
, . , - . ( ), . , - , -. - . ,
7
, , - . , , Q1 Q2.
1.4
. . (hypertext) -, 1990. - ( ), , , , HTML . , - , . - , .
1.5
.html , - , . , - http://ceid.upatras.gr/contact.html , . URL
(Univer-sal Resource Locator) HTTP webserver . browsers , - URL .
HTML ,
8
. web . , . - , . -, , .
, - . :
Altavista, Excite Infoseek.
Yahoo!.
- - . . , . , - - , , - . , - . , - . , . , -
, - . . - . ,
9
- . () - . , . , . , . -
, . 1995, Altavista - 30 . . -, , . - , 1.4. URL ? . , , 1995,
Al-tavista .
1.4: . , back-end , .
10
1.6
, - . , . , - . , - - . , - ( ) , . - - 2 3, , Boolean , .
Google ( 1.5). , Google , , . . Google . , , user interface .
Google , , , ( 1.6).
1.6.1
:
,
,
11
1.5: Google 1998.
.
, (.. Facebook ), - , - . , . , , , . , - , . , .
. . , - .
12
1.6: .
1.6.2
- . , . . . , URL . 1.7.
. - . - , - (examination model ). . - , . , -
.
13
1.7: snippet .
cascade . , . - . , .
1.7
, 2 - . Cranfield , - . , . 3 -
, relevance judg-ments . TREC - click-through - .
14
4 -, , . - , . 5, -
. Bayesian , . - , ., 6
, .
15
2
. 1990, , . - , . - -, . - , , . , -
, 1990, - . 2.1 - . Cranfield , - , - , TREC. 2.2 , , nDCG ,
Expected Reciprocal Rank .. ,
16
2.3, click-through .
2.1 -
, - [BYRN99]. , 1952, Cranfield -, Cleverdon . - Mortimer Taube
, Uniterm. , -. , - , Uniterm . Cleverdon ,
, Cranfield-1 , , Uniterm . Cleverdon 18.000 1.200 . , , - . ,
Cleverdon -
. , Cranfield-2 , , -, ( 2.1). - . Cranfield-2 ,
17
, - . , , Cleverdon . , , -, .
2.1: Cranfield. , . , . , , stopwords , ( [CMK66]).
- , , , qrels .
18
, . qrels , - , . , ,
, - , . - , Cranfield Cranfield [Voo02], [VH05],[BV04].
, Cranfield -, . Cranfield - . , , , . - -, ., - . , Cranfield ,
.
2.1.1 - TREC
Cranfiled-2 , - . (reference collections ). - D, , I rj , - . ,
rj = 0, , rj = 1, -
19
. , , 0 1. , 5 , . 5 0 4, 5 - : , , , , [Kek05]. -
, - ., -, - . , ( - ), . , - , . ,
. - . , - . k , - . TREC. 1992 -
, , - . (NIST) , - , TREC . TREC , , -. TREC TIPSTER 750.000 , .
- 2GB
20
, [Har93]. TREC ,
, . , . - ( 2.2) , . , TREC -. . TREC ,
( ), , , . TREC [AMWZ09]. - , TREC . -, . , . , TREC-9 -
, -. - ad-hoc TREC . - ( 2.3). TREC .
, . ad-hoc - . , .
21
2.2: TREC TREC 7 1998. Ellen Voorhees, .
. 2005, 117 TREC,
7 , , -, spam. , -. TREC, NTCIR, 1999, [KKK+99], (CLEF) 2000, -
-, XML (INEX), 2002 [GK02].
2.2
, -
22
2.3: TREC .
, .
2.1. - .[BYRN99].
- . - 0 1
.rel = rd1 , rd2 , rd3 , ..., rdn,
rdi {0, 1} , , .
2.2.1
- . D I. - D, F . F
23
R - I, 2.2 :
Precision =|R F||F|
(2.1)
Recall =|R F||R|
(2.2)
- . - Cleverdon , , , . , , , . - - -, .
2.4: I. R - F . D .
- - , -. 2.1
24
2.1: - - .
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1 1 1 0.6 0.6 0.6
0.6 0.6 0.44 0.44 0.44
. 2.5 2.1. - . , .
2.5: 2.1.
, - . , -, ., ., Nq , . :
P (rj) =
Nqi=1
Pi(rj)
Nq(2.3)
25
P (rj) rj Pi(rj) rj i- .
- . - . , - . , .
2.2.2
, , - . , . .
n , , . , , 5, 10 20 . Cranfield-2 , -
. , . , - [GJG04], . , precision at 5 (P@5), precision
at 10 (P@10), precision at 20 (P@20) - .
26
Average Precision (AP) . - , . 0. -. , :
AP@k(r) =1
R
kd=1
rd
di=1
ri
d(2.4)
k , , rd . , . 2.6.
2.6: . - 10. 6, 4 .
R- - R-. - R- , R . R- :
RPrec(r) =1
R
Ri=1
ri (2.5)
27
R-Precision -. R-, R - .
- -, van Rijsbergen [Rij79], - - . - :
E(j) = 1 1 + b2
b2
rj++ 1
P (j)
(2.6)
r(i) i- P (i) i- . b (b 0) , -. b - b - . -, b 1, . , b 1, .
F - - F -. F - , . -, F -, - 1 . F - , . F - :
F (j) =2
1r(j)
+ 1P (j)
(2.7)
28
2.3 (DCG)
. - . - Cleverdon , , , . , , , . - -, . - . , .
, - . [JK02] - . , - . , - . , ,
, k , . , , G
, . ,
G = {3, 1, 2, 3, 2, 0, 3, 1, 0, 0, 2, ...}.
G - i, 1 i. CG,
29
:
CG[i] =
{G[1], if i = 1CG[i 1] +G[i], otherwise. (2.8)
, G, :
CG = {3, 4, 6, 9, 11, 11, 14, 15, 15, 15, 17, ...}.
, [Kek05] , . , . . , , , . :
DCG[i] =
{CG[1], if i < bDCG[i 1] +G[i]/ logb i, if i b.
(2.9)
b . , 2. , . -
. , - 100% . , DCG . DCG DCG.
. , . , -
:IG = {3, 3, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1, 0, 0, 0, ...}.
30
, - (ICG) (IDCG) . -
DCG, - , . [Kek05] nDCG. - . nDCG , -
DCG IDCG . [BYRN99], Nq , - ( overlineDCG) :
DCG[i] =1
Nq
Nqj=1
DCGj[i] (2.10)
:
IDCG[i] =1
Nq
Nqj=1
IDCGj[i] (2.11)
nDCG qi DCG qi IDCG . :
nDCG[i] =DCG[i]
IDCG[i](2.12)
, , , nDCG . -, nDCG . ,
31
, - , .
2.3.1 Rank Biased Precision
- , . . , - . , . - , 4. Moffat Zobel [MZ08]
, . , - . , -
. , , p 1 p. , , , p, p2, 1p . 2.7 BRP .
2.7: RBP . [08].
, q,
32
R = r1, r2, ..., rd i -, :
di=1
ri pi1
i .
,
, i=1
i pi1
(1 p) = 11p ,
. RBP , 0.0 1.0, :
RBP = (1 p) di=1
ri pi1. (2.13)
RBP , . - 1, 2, 6, 11 17. p 0.50, RBP 2 11, 2.13 0.5000 0.0010 .
, p = 0.80, 0.8000 0.1074. , p
. p 1.0, - -. , p , . , 10 p = 0.5, . , RBP
p. p .
33
p, , . , p , RBP . 2.8 RBP TREC-5 p 10. : 10, p 0.7 , 100, p
0.95 .
2.8: RBP TREC-5 50 -. p , 10 . 100 p. [08].
2.3.2 Bpref
- . , . -
34
, . , - . , , . Buckley - Voorhees [BV04]
bpref , , , . , dj dk, . bpref . - , , , ., R , bpref
:
bpref =1
R
r
1 |n ranked higher than r|R
(2.14)
r n R - . bpref ,
, . bpref bpref 10, 10 . , bpref 10 :
bpref =1
R
r
1 |n ranked higher than r|10 +R
(2.15)
n 10 +R . 2.9 -
, 10, R- bpref 10, - . - . TREC-8 . ,
35
, bpref 10 .
2.9: - . [BV04].
2.3.3 Expected Reciprocal Rank
2.2.3, - NDCG . , . , - NDCG , ., nDCG,
. , ., [CZTR08], [CZ09] i, . , Chappelle et al. [CZ09] -
,
36
. ExpectedReciprocal Rank (ERR), - . Cascade user model -. ,
Cascade user model - , i, , Ri. , . . Cascade user model (ERR)
:
ERR =nr=1
1
rP (user stops at position r).
, (r) (reciprocal rank) , , (1) = 1 (r) 0 R ., cascade , R :
P (user stops at position r) =r1i=1
(1Ri)Rr. (2.16)
- . , Ri , - 2.16 . , ERR :
ERR =nr=1
1
r
r1i=1
(1Ri)Rr, (2.17)
n . - 1 ERR . O(n2) .
37
Algorithm 1 Algorithm to compute ERR metric in linear time
Require: Relevance grades gi, 1 i n, and a mapping function R.p
1, ERR 0.for r=1 to n do
R R(gr)ERR ERR + p R/rp p (1R)
end forreturn ERR
, . , ., ERR . , , ., -
. , . , , . Radlinski et al. [RKJ08], -
, ERR, ERR . 2.10 ,
ERR , UCTR,Min,Max and Mean Reciprocal Ranks, Search Success,
Precision at LowestRank . - ERR ERR , .
38
2.10: . ERR . 1, 0. [CMZG09].
2.4
, - , - . , - , . , . - , . .
Kendalls
Kendalls , - [YAR08]. n, C ( ) A - ( ). , Kendalls
39
:
=C D
N(N 1)/2(2.18)
N(N 1)/2 . Kendalls , - , 1 . , Kendalls , - . , Kendalls 1, - ,
Kendalls -1, , Kendalls 0. 2.11 -
Kendalls . , 61 TREC-5 . 1.0 , .
2.11: Kendalls - 61TREC-5 .
Pearson
2.2.6 , . ,
40
- . [HH07], - , . - 2.3., , -
Pearson . Pearson , X Y , [1, 1], -1 , 0 1 . Pearson :
rxy =
ni=1
(xi x)(yi y)ni=1
(xi x)
ni=1
(yi y)(2.19)
, n , xi yi x y .
2.5
. , , , . , - , . , - . ,
, . - , ,
41
[JFM97], [LF07]. - . , Fox et al. [FKM+05] , - , . - .
2.6
2.1.1 ., , . . - . , . , - .
. - ( ) -. Joachims [Joa03], - , . , - , . -
, (click-through-rate (CTR)) . , ,
42
, - ., .,
. , . . , - , . , -
[CJRY12] . - r , . , - , .
2.6.1 Crowdsourcing
, , . , -, Yahoo!, Google, Microsoft .., crowd-
sourcing , . - . crowdsourcing ( - ) , . .
43
Amazon Ama-zon Mechanical Turk (AMT), crowd-sourcing - , . , , .
- - , .
2.12: /.
2.6.2 /
/ - , . crowdsourcing , . / - , , . , -, . . 2.12 /. .
44
3
? AIAI 2014 : Xenophon Evangelopou-los, Christos Makris, and
Yannis Plegas. Reciprocal rank using web pagepopularity. AIAI,
Rhodes 2014
3.1
, , . Cranfield [Cyr91] .
- . , - - , , , NDCG Jarvelin - Kekalainen [Kek05], ERR[CMZG09]
[YSCR10] .,
45
. - , . - - 5 . , -
, , -, . , - , click-through [CJRY12]. , - [RDR07a]., [HH07],
[SZ05]. click-through -
, [HH07]., Carterette Allan [CA05], Sanderson - Joho
[SJ04] . . Buckley - Voorhees [BV04] , - bpref , - . , Sakai ,
[Sak07]. -
. [CZTR08]. , (position models) cascade [CZ09]. , . cascade , .
- NDCG RBP [MZ08], ERR cascade .
46
- . cascade , ( ) . 3.2 - . 3.3 , . ,
. , . Di, , web traffic . - click-metrics . , .
3.2 Reciprocal Rank using Document Popularity
3.2.1
- . . , . - , , cascade [CZTR08], Bayes [CZ09] . cascade -
. , , . , . Ri i - -
.
47
-. Craswell et al. [CZTR08], Ri . , . , , . , (daily pageviews)
, .
, . Ri , i, i, .
3.2.2
, - . Cho et al. [CRA05] - V (p,t) p - t. , :
3.1. (P (p,t)) ( -) d, (pv) t.
, , . , - -. . , 3.1,
,
48
3.1: 4
WebSite Daily Page views Popularity Gradehttp://google.com
584.640.000 4http://wikipedia.com 30.451.680
3http://ceid.upatras.gr 11.228 1http://sample-site.wordpress.com 11
0
- . , pv u, :
pu =
ln pvu
5
(3.1)
(3.1) - . , pv - ( ) 500.000.000 ( Google ) . - , . , 5 0 4, .
(3.1) 0
4 : , , , , . 3.1 .
3.2.3
- , . , cascade - . . gi i pi
49
i, 3.2.2. , gi pi.
Ri = R(gi, pi), (3.2) f -
, . f :
f(r) =2r 12rmax
, (3.3)
r =g + p
2, r {0, ..., rmax}. (3.4)
(3.4) - . - . - . , , . , -
. - , ( ) - . , . , (g = 1), (p = 4), , . ,
. , , , 1 0. , (1) = 1 (R) 0 r +. (r) = 1/r :
RRP =nr=1
1
r
r1i=1
(1Ri)Rr. (3.5)
50
3.3
- . , - . -, [CMZG09],[CSdR13] . , [RKJ08] . , , click-through
.
3.3.1
, click-through . - , crowdsourcing . - Indri? , TREC Web Tracks
2008-2012. , .
TREC Web Tracks 2008-2012 ClueWeb09, . - 1.000 , 25 (5 ) . 2009.
, ClueWeb09 ,
?http://www.lemurproject.org/. The lemur project. University of
Massachusetts andCarnegie Mellon University.
51
http://www.lemurproject.org/
50.000.000 . , (topics ) MSN Microsoft Research.
3.1: , TREC Web Tracks.
200 TREC Web Tracks , .html . , .aspx C # , , 20 ( Indri ) -. ,
, URL ( 3.3). .
Indri Indri [SMTC04]
52
- . Indri - :
, -
-.
, - . , , , , Indri - . . Indri ,
. , - . , ,, , . , In-
dri . ., , Indri
, , . Indri . Indri , click-through
. - TRECWeb Tracks (0 4). ,
53
, 3.2. , (0 4) .
- . TREC Web Tracks , - -. , 167 .
3.4
- - . , - ( ) . - :
Normalized Discounted Cumulative Gain
Average Precision
Expected Reciprocal Rank
Reciprocal Rank using Document Popularity
:
Precision at Lowest Rank
Max, Min and Mean Reciprocal Ranks of the clicks
UCTR
3.2 ( - ) . Pearson , .
54
3.2: -.
PLC MeanRR MinRR MaxRR UCTRnDCG 0.498 0.497 0.503 0.445 -0.024AP
0.402 0.417 0.395 0.396 -0.004ERR 0.528 0.512 0.517 0.459 0.064RRP
0.559 0.554 0.588 0.472 0.041
3.2, - . , RRP - nDCG cascade ERR. -, ERR, RRP , Mean, Max, Min
Reciprocal Rank PLC 3.2. , ERR RRP reciprocalrank 1/r. ,
UCTR . UCTR , ., -
, . , r - gi pi ( 3.4). , - . , - , . :
r = 0.7 gi + 0.3 pi, (3.6)
. : , , . 3.3 -
55
3.3: - .
PLC MeanRR MinRR MaxRR UCTRnDCG 0.498 0.497 0.503 0.445 -0.024AP
0.402 0.417 0.395 0.396 -0.004ERR 0.528 0.512 0.517 0.459 0.064RRP
0.578 0.562 0.588 0.490 0.083
(3.7) . , RRP , 3.2. , - .
3.4.1
- . , - . , . Buckley - Voorhees[BV04] ( 2), , Sakai , [Sak07].
, -
, - . , -, . ., 167 ,
56
, 41 167 - 83 167 . . 3.2, 3.3 3.4 . 3.3, , - . , , 3.4 ( ), - .
, - .
3.2: .
57
3.3: 41 .
3.4: 83 ..
58
4
. , -, - . , - , . ,
, - . . , - , . , - ., -
, , - . , , . - , .
, - , ,
59
[ABDR06] .. 3. click-through -, . (CTR ) - , .
. - - , . , , -, . , - url , , . . . ,
, - , -, . - . , -
, , . - . , , cascade . , - , . - 4.1 , 4.2 cascade 4.3, 4.4 4.5
- . User Browsing Model , Dependent Click Model Dynamic Bayesian
Network Click Model.
60
4.1
, , - [DP08], [RDR07b]. , , , . , - d p [CZTR08]:
P (C = 1|d, p) = adbp,
ad d bp - p, . , . , ad , , - . , b1 = 1, , ad (CTR) 1. , , COEC
, .
4.1.1 COEC
, bp CTR N p [ZJ07] . ci p i, COEC ( ) D :
ad =
Nn=1
ci
Nn=1
bpi
(4.1)
- . ,
61
, COEC , .
4.1.2
- ad. , bp, ad :
ad = argmaxn
Ni=1
ci log(abpi) + (1 ci) log(1 abpi). (4.2)
bp - ad bp. , , , - , . ad > 1. , ad , 1 Expectation
Maximization .
4.1.3
. , . :
P (C = 1|d, p) = 11 + exp(ad bp)
. (4.3)
4.2 cascade
, - , -. , cascade [CZTR08] - . , cascade p . , .
62
Ri, , . , , r:
r1i=1
(1Ri)Rr (4.4)
i -. cascade 1[CMZG09]. , cascade .
Algorithm 2 Cascade model algorithm
Require: R1, R2, ..., Ri.i = 1User examines position i.if
random(0,1) Ri then User is satisfied with document in position
iand stops.else i = i+ 1; go to 2.end if
4.3 UBM
, - [DP08]. , , . cascade
, . cascade , User Browsing Model (UBM)
63
. , , . cascade , . , , . -
a e . [CZTR08] - , , . , .
ds , p. , - . d (a) q P (a|d, q), - Bernoulli :
P (a|d, q) = aad,q(1 ad,g)1a (4.5)
ad,q - d q. , :
P (e|p, ds) = ep,ds(1 p,ds)1e (4.6)
p,ds ds p. P (c, a, e, d, q, dr, p). - :
P (c, a, e|d, q, ds, p) = P (c|a, e)P (e|ds, p)P (a|d, q)= P
(c|a, e)ep,ds(1 p,ds)1eaad,q(1 ad,g)1a
(4.7)
P (c|a, e) - .
64
, . , (CTR) . , (c, d, q, ds) . , c = 1 , , (a = 1) (e = 1). ,
4.7 :
P (c = 1|d, q, ds, p) = ad,qp,dsP (c = 0|d, q, ds, p) = 1
ad,qp,ds
a . cascade .,
. , - . UBM . - . m,
:
P (e|p, dis,m) = ep,ds,m(1 p,ds,m)1e (4.8)
Expectation Maximization a, m.
4.4 DCM
- , cascade, , . , -, ,
65
. - , . , Guo et al. [GLK+09]
. , - , . , :
1, - .
.
:edi,i = 1, cdi,i = rdi . (4.9)
. , IndependentClick Model (ICM). , . , C1, C2, ..., CM d1, d2,
..., dM , :
LICM =Mi=1
(Ci log rdi + (1 Ci) log (1 rdi)). (4.10)
Ci Bernoulli . , - , rd - 4.10. rd
rd = number of clicks on d/measured relevance of d.
, -
66
. , - , i. 4.9
, . :
cdi,i = edi,irdi (4.11)
edi+1,i+1 = icdi,i + (edi,i cdi,i) (4.12)
[GLW09] , Dependent Click Model. 4.1. , - . :
LICM l
i=1
(Ci log rdi + (1 Ci) log (1 rdi)) +l1i=1
(Ci log i + log (1 i)),
(4.13)
4.1: DCM .
-.
67
ICM . :
rd =number of clicks on d
measured relevance of d at position l.
i = number of query sessions when last click occurs at i
number of query sessions when position i is clicked.
, ICM DCM - ( 4.2). (8% - ), DCM ICM . .
4.2: - .
4.5 DBN
([CZ09], [GLK+09]) ,
68
. , -, , . , 3
1 2 , 3 . , 1 2 , [08] 3 . Chapelle et al. [CZ09] -
, , . Ci i ( ). , Ei, i, Ai, - i Si i. , -
, . Ai i, . , Si i, , , . , , -
, i . , cascade , . , . , 1 . , cascade . , i, .
69
4.3: DBN .
, 4.3.
Ai = 1, Ei = 1 C = 1
P (Ai = 1) = ad
P (Si = 1|Ci = 1) = sdCi = 0 Si = 0
Si = 1 Ei+1 = 0
P (Ei+1 = 1|Ei = 1, Si = 0) =
Ei = 0 Ei+1 = 0
4.14, DBN ad sd, - . , . ad sd ExpectationMaximization , .
70
, . . -, CTR 1, - , , , ad. 4.4 KL- .
4.4: CTR 1 () KL- ().
, - . , sd , - NDCG. . 4.5 NDCG - . DBN , .
71
4.5: NDCG DBN .
72
5
? SPIRE : Xenophon Evangelopoulos andChristos Makris, Modeling
Clicks using Document Popularity. SPIRE, Lon-don 2015.
5.1
, - , - . - , , , , [RKJ08]. ,
-, , -. , .
73
, , . , ,
. , . , - , , , , [KT03]. , -
- . [GLW09], [GLK+09] . Dupret et al. [DP08] UBM , -, , ds -.
-
DBN [CZ09], - . - -. cascade , DBN - . , , . , -
. 5.2 5.3 - .
74
5.2
, - , . , . , -
. . 5.1. [Mur02].
10 . 5.1 2 , 8 . :
Ei: i
Ci: i
Pi: i
Ri: i
-. , Ci, . (rd) , -. , -
, . cascade , . , - . , . ,
75
5.1: . - - .
. [EMP14] , () [0, 4], , pi . (pi) -: 0, 1, 2, 3 4, 0.5, 0.6,
0.7, 0.8 0.9. ,
, -. , , , 1 , . - , . : - . pi . -
76
:
=Ni=1
piN, (5.1)
. :
P (Ci = 1|Ei = 0) = 0 (5.2)
P (Ei+1 = 1|Ei = 0) = 0 (5.3)
P (Ri = 1|Ci = 1) = rd (5.4)
P (Ei+1 = 1|Ei = 1, Ri = 0, Pi = 1) = (5.5)
P (Ei+1 = 1|Ei = 1, Ri = 1, Pi = 0) = 1 (5.6)
ru, - Expectation Maximization .
5.3
- state-of-the-art. , , DBN -.
5.3.1
. , 10 . , crowdsourcing TREC Web Tracks , [EMP14].
-. , ,
77
-. , - , -. MATLAB R2009b, -
BNT ?.
5.3.2
- (Log-Likelihood). LL , . - LL . 5.2 DBN
LL. . , DBN . 110 -3.21, DBN -3.29. , 50 , LL -3.11, DBN -3.13.
, state-of-the-art.
?https://code.google .com/p/BNT/ Bayes Toolbox for Matlab. By
Kevin Murphy,1997 - -2002
78
https://code.google.com/p/bnt/
5.2: Log-Likelihood . DBN .
79
6
- . , - . , , - , - , ( ) . ,
- , . . - , , , . . , . , -
, , . ,
80
. , , , . , , .
, - . , - , , . , , . ,
:
.
- , -. , - . , , - . , , . , , - . , , - , .
81
. Bayes . . , .
6.1
, . , , . . , , . -
, - : , , . - . , -. , , . -
, . , - , micro-blogging. -, , .
82
, . - , , click-through , - . , - , , , , .. ,
. - , AOL 2008. - , , . , - - .
6.2
. , . , , - . , , , . , , crowdsourcing -. , , . 5,
. ,
83
, .,
, . .
84
Bayes
(DBN) [DK89], [Mur02] - Bayes , - - . Z1, Z2, ..., , Zt = (Ut,
Xt, Yt) , - . , t . , , . DBN (B1, B), B1
Bayesian P (Z1) B Bayes (2TBN) P (Zt|Zt1) :
P (Zt|Zt1) =Ni=1
P (Zit |Pa(Zit))
Zit i- t, Pa(Zit)
Zit . 2TBN , 2TBN (CPD) , P (Zit |Pa(Zit)) t > 1. ,
Pa(Zit),
. , .
85
Zit1 Zit ,
. , DAG., -
. , CPD -
, . , . , - , . DBN 2TBN
T . :
P (Z1:T ) =Tt=1
Ni=1
P (Zti|Pa(Zti))
Hidden Markov Models (HMMs) Baysian . DBN HMM DBN -, Xt1, . . .
, QtNh , , . , HMM , Xt.
.1
HMM DBN , .1. -
. , - , -. .1 : Xt+1 Xt1|Xt ( Markov ) Yt Yt|Xt, t 6= t. Bayes ,
-
(CPD) . .1, P (X1),P (Xt|Xt1) P (Yt|Xt). (CPD) P (X1) - , ,, P
(X1 = i) = (i), 0 (i) 1
i (i) = 1.
(CPD) P (Xt|Xt1) ,
86
.1: HMM DBN , 3 .
, P (Xt = j|Xt1 = i) = A(i, j) 1. (CPD) P (Yt|Xt) . Yt - , - ,
:P (Yt = j|Xt = i) = B(i, j). Yt , Gaussian Gaussians. , -
P (X1), P (X2|X1) P (Y1|X1). CPDs . . .2. HMM DBN . , ,
P (Yt|Xt = i) Gaus-sians i. Gaussians CPD , .3. CPDs Y M :
P (Yt|Xt = i,Mt = m) = N (yt;i,m,i,m)P (Mt = m|Xt = i) =
C(i,m)
, CPD -
87
.2: HMM ( ) . : P (X1 =i) = (i), P (Xt = j|Xt1 = i) = A(i, j), P
(Yt = j|Xt = i) = B(i, j).
, ,
P (Yt|Xt = i,Mt = m) = N (Ayt;i,m,i,m)
( t) [KA96]. , . - , ( ) ( [KA96] ). HMM Yt Yt|Xt, ,
, .4. - HMM (ARHMM). , [Rab89]. - HMM DBN , - .
88
.3: HMM Gaussians .
.4: Auto-regressive HMM.
ARHMM Xt, Yt1 Yt. . Y , CPD Y .
Y , CPD Y
P (Yt = yt|Xt = i, Yt1 = yt1) = N (yt;Wiyt1 + i,i)
Wi , Xt i. HMM -, Gaussian HMM , Markov [Ham90].
89
.2
- . , - . . -
. HMM , , , CPD P (Xt|Xt1). , - [Bra99] .
. -, . - , . , Expec-tation Maximization .
Online offline . offline , - ( online ). online - . , online -
offline .
. (MLE) CPD . , . - D = {D1, . . . , DM} , :
L = logMm=1
Pr(Dm|G) =ni=1
Mm=1
logP (Xi|Pa(Xi), Dm)
90
Pa(Xi) Xi. . ( - : .)
CPD Dm(Xi, Pa(Xi)). ( CPD , :
mDm(Xi, Pa(Xi)).)
- , . -
, - . Expectation Maximization (EM) . Jensen
[CT91] , . Jensen , f ,
f
(j
jyj
)j
jf(yj)
j j = 1. , f f , f . , Jensen :
L =m
logh
P(H = h, Vm)
=m
logh
q(h|Vm)P(H = h, Vm)
q(h|Vm)
m
h
q(h|Vm) logP(H = h, Vm)
q(h|Vm)
=m
h
q(h|Vm) logP(H = h, Vm)m
h
q(h|Vm) log q(h|Vm)
91
q
h q(h|Vm) = 1 0 q(h|Vm) 1, . q
q(h|Vm) = P(h|Vm) E (expectation) , . , [NH98]. -
.
lc()q =m
h
q(h|Vm) logP(H = h, Vm)
M (maximization) . , , q . q(h|Vm) = P(h|Vm), EM ,
:
Q(|) =m
h
P (h|Vm, ) logP (h, Vm|)
Dempster et al. [DLR77]
Q(|) > Q(|) P (D|) > P (D|), - , - . q(h|Vm) = P(h|Vm) , -
. , . CPDs ,
:
Q(|) =ijk
E[Nijk] log ijk
ENijk =
m P (Xi = k, Pa(Xi) = j|Dm, ), , :=arg max Q(
|),
ijk =ENijkk ENijk
92
[ABDR06] Eugene Agichtein, Eric Brill, Susan Dumais, and Robert
Ragno.Learning user interaction models for predicting web search
re-sult preferences. In Proceedings of the 29th Annual
InternationalACM SIGIR Conference on Research and Development in
Infor-mation Retrieval, SIGIR 06, pages 310, New York, NY,
USA,2006. ACM.
[AMWZ09] Timothy G. Armstrong, Alistair Moffat, William Webber,
andJustin Zobel. Improvements that dont add up: Ad-hoc
retrievalresults since 1998. In Proceedings of the 18th ACM
Conferenceon Information and Knowledge Management, CIKM 09,
pages601610, New York, NY, USA, 2009. ACM.
[Bra99] M. Brand. Structure learning in conditional probability
modelsvia an entropic prior and parameter extinction. Neural
Compu-tation, 11:11551182, 1999.
[BV04] Chris Buckley and Ellen M. Voorhees. Retrieval evaluation
withincomplete information. In Proceedings of the 27th Annual
Inter-national ACM SIGIR Conference on Research and Developmentin
Information Retrieval, SIGIR 04, pages 2532, New York,NY, USA,
2004. ACM.
[BYRN99] Ricardo A. Baeza-Yates and Berthier Ribeiro-Neto.
Modern In-formation Retrieval. Addison-Wesley Longman Publishing
Co.,Inc., Boston, MA, USA, 1999.
[CA05] B. Carterette and J. Allan. : Incremental test
collections. In:Proceedings of the 14th ACM International
Conference on In-formation and Knowledge Management, CIKM 2005,
2005.
[CJRY12] Olivier Chapelle, Thorsten Joachims, Filip Radlinski,
andYisong Yue. Large-scale validation and analysis of
interleaved
93
search evaluation. ACM Trans. Inf. Syst., 30(1):6:16:41,
March2012.
[CMK66] Cyril Cleverdon, Jack Mills, and Michael Keen. Aslib
cranfieldresearch project: factors determining the performance of
index-ing systems, 1966.
[CMZG09] Olivier Chapelle, Donald Metlzer, Ya Zhang, and
PierreGrinspan. Expected reciprocal rank for graded relevance.
InProceedings of the 18th ACM Conference on Information
andKnowledge Management, CIKM 09, pages 621630, New York,NY, USA,
2009. ACM.
[CRA05] J. Cho, S. Roy, and R. E. Adams. : Page quality: In
search ofa unbiased web ranking. In: Proceedings of the 2005 ACM
SIG-MOD International Conference on Management of Data, SIG-MOD
2005, 2005.
[CSdR13] A. Chuklin, P. Serdyukov, and M. de Rijke. : Click
model-basedinformation retrieval metric. In: Proceedings of the
36th Inter-national ACM SIGIR Conference on Research and
Developmentin Information Retrieval, SIGIR 20013, 2013.
[CT91] T. M. Cover and J. A. Thomas. Elements of Information
Theory.John Wiley, 1991.
[Cyr91] W. Cyril. Cleverdon. The significance of the cranfield
test on in-dex languages. In: Proceedings of the 14th Annual
Inter-nationalACM SIGIR Conference on Research and Development in
Infor-mation Retrieval, SIGIR 1991, 1991.
[CZ09] Olivier Chapelle and Ya Zhang. A dynamic bayesian
networkclick model for web search ranking. In Proceedings of the
18thInternational Conference on World Wide Web, WWW 09, pages110,
New York, NY, USA, 2009. ACM.
[CZTR08] Nick Craswell, Onno Zoeter, Michael Taylor, and Bill
Ramsey.An experimental comparison of click position-bias models.
InProceedings of the 2008 International Conference on Web Searchand
Data Mining, WSDM 08, pages 8794, New York, NY, USA,2008. ACM.
[DK89] T. Dean and K. Kanazawa. A model for reasoning about
persis-tence and causation. Artificial Intelligence, 93(12):127,
1989.
94
[DLR77] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum
like-lihood from incomplete data via the EM algorithm. J. of
theRoyal Statistical Society, Series B, 34:138, 1977.
[DP08] Georges E. Dupret and Benjamin Piwowarski. A user
brows-ing model to predict search engine click data from past
obser-vations. In Proceedings of the 31st Annual International
ACMSIGIR Conference on Research and Development in
InformationRetrieval, SIGIR 08, pages 331338, New York, NY, USA,
2008.ACM.
[EMP14] Xenophon Evangelopoulos, Christos Makris, and Yannis
Plegas.Reciprocal rank using web page popularity. In Lazaros
Iliadis,Ilias Maglogiannis, Harris Papadopoulos, Spyros Sioutas,
andChristos Makris, editors, Artificial Intelligence Applications
andInnovations, volume 437 of IFIP Advances in Information
andCommunication Technology, pages 116125. Springer Berlin
Hei-delberg, 2014.
[FKM+05] Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan
Dumais,and Thomas White. Evaluating implicit measures to improveweb
search. ACM Trans. Inf. Syst., 23(2):147168, April 2005.
[GJG04] Laura A. Granka, Thorsten Joachims, and Geri Gay.
Eye-tracking analysis of user behavior in www search. In
Proceedingsof the 27th Annual International ACM SIGIR Conference
onResearch and Development in Information Retrieval, SIGIR 04,pages
478479, New York, NY, USA, 2004. ACM.
[GK02] Norbert Govert and Gabriella Kazai. Overview of the
initiativefor the evaluation of xml retrieval (inex) 2002. In IN:
PROC. OFTHE FIRST WORKSHOP OF THE INITIATIVE FOR THEEVALUATION OF
XML RETRIEVAL (INEX), DAGSTUHL,2002, pages 117, 2002.
[GLK+09] Fan Guo, Chao Liu, Anitha Kannan, Tom Minka, Michael
Tay-lor, Yi-Min Wang, and Christos Faloutsos. Click chain model
inweb search. In Proceedings of the 18th International Conferenceon
World Wide Web, WWW 09, pages 1120, New York, NY,USA, 2009.
ACM.
[GLW09] Fan Guo, Chao Liu, and Yi Min Wang. Efficient
multiple-clickmodels in web search. In Proceedings of the Second
ACM Inter-
95
national Conference on Web Search and Data Mining, WSDM09, pages
124131, New York, NY, USA, 2009. ACM.
[Ham90] J. Hamilton. Analysis of time series subject to changes
in regime.J. Econometrics, 45:3970, 1990.
[Har93] Donna Harman. Overview of the first trec conference. In
Proceed-ings of the 16th Annual International ACM SIGIR
Conferenceon Research and Development in Information Retrieval,
SIGIR93, pages 3647, New York, NY, USA, 1993. ACM.
[HH07] S. B. Huffman and M. Hochster. : How well does result
rele-vance predict session satisfaction? In: Proceedings of the
30thAnnual International ACM SIGIR Conference on Research
andDevelopment in Information Retrieval SIGIR 2007, 2007.
[JFM97] Thorsten Joachims, Dayne Freitag, and Tom Mitchell.
Web-watcher: A tour guide for the world wide web. In INPROCEEDINGS
OF THE FIFTEENTH INTERNATIONALJOINT CONFERENCE ON ARTIFICIAL
INTELLIGENCE,pages 770775. Morgan Kaufmann, 1997.
[JK02] Kalervo Jarvelin and Jaana Kekalainen. Cumulated
gain-basedevaluation of ir techniques. ACM Trans. Inf. Syst.,
20(4):422446, October 2002.
[Joa03] Thorsten Joachims. Evaluating retrieval performance
usingclickthrough data, 2003.
[KA96] N. Kumar and A. G. Andreou. A generalization of linear
dis-criminant analysis in maximum likelihood framework. In Proc.of
the Joint Statistical Meeting, Statistical Computing
section,1996.
[Kek05] Jaana Kekalainen. Binary and graded relevance in ir
evaluations-comparison of the effects on ranking of ir systems.
Inf. Process.Manage., 41(5):10191033, September 2005.
[KKK+99] Noriko Kando Kazuko, Noriko K, Kazuko Kuriyama,
ToshihikoNozue, Koji Eguchi, Hiroyuki Kato, Soichiro Hidaka, and
JunAdachi. The ntcir workshop : the first evaluation workshop
onjapanese text retrieval and cross-lingual information
retrieval,1999.
96
[KT03] Diane Kelly and Jaime Teevan. Implicit feedback for
inferringuser preference: A bibliography. SIGIR Forum,
37(2):1828,September 2003.
[LF07] Yiqun Liu and Yupeng Fu. Automatic search engine
performanceevaluation with click-through data analysis. In In
Proceedings ofthe 16th international conference on World Wide Web
(WWW,2007.
[MRS08] Christopher D. Manning, Prabhakar Raghavan, and
HinrichSchutze. Introduction to Information Retrieval. Cambridge
Uni-versity Press, New York, NY, USA, 2008.
[Mur02] Kevin Patrick Murphy. Dynamic bayesian networks:
Represen-tation, inference and learning, 2002.
[MZ08] Alistair Moffat and Justin Zobel. Rank-biased precision
formeasurement of retrieval effectiveness. ACM Trans. Inf.
Syst.,27(1):2:12:27, December 2008.
[NH98] R. M. Neal and G. E. Hinton. A new view of the EM
algorithmthat justifies incremental and other variants. In M.
Jordan, edi-tor, Learning in Graphical Models. MIT Press, 1998.
[Rab89] L. R. Rabiner. A tutorial on Hidden Markov Models and
selectedapplications in speech recognition. Proc. of the IEEE,
77(2):257286, 1989.
[RDR07a] M. Richardson, E. Dominowska, and R. Ragno. :
Predictingclicks: Estimating the click-through rate for new ads.
In: Pro-ceedings of the 16th International Conference on World
WideWeb, WWW 2007, 2007.
[RDR07b] Matthew Richardson, Ewa Dominowska, and Robert
Ragno.Predicting clicks: Estimating the click-through rate for new
ads.In Proceedings of the 16th International Conference on
WorldWide Web, WWW 07, pages 521530, New York, NY, USA,2007.
ACM.
[Rij79] C. J. Van Rijsbergen. Information Retrieval.
Butterworth-Heinemann, Newton, MA, USA, 2nd edition, 1979.
[RKJ08] Filip Radlinski, Madhu Kurup, and Thorsten Joachims.
Howdoes clickthrough data reflect retrieval quality? In Proceedings
of
97
the 17th ACM Conference on Information and Knowledge
Man-agement, CIKM 08, pages 4352, New York, NY, USA, 2008.ACM.
[Sak07] T. Sakai. : Alternatives to bpref. In: Proceedings of
the 30thAnnual International ACM SIGIR Conference on Research
andDevelopment in Information Retrieval, SIGIR 2007, 2007.
[Sal68] Gerard. Salton. Automatic Information Organization and
Re-trieval. McGraw Hill Text, 1968.
[SJ04] M. Sanderson and H. Joho. : Forming test collections with
nosystem pooling. In: Proceedings of the 27th annual
internationalACM Conference on Research and Development in
InformationRetrieval, SIGIR 2004, 2004.
[SMTC04] T. Strohman, D. Metzler, H. Turtle, and W. B. Croft.
Indri: Alanguage model-based search engine for complex queries.
Pro-ceedings of the International Conference on Intelligence
Analy-sis, 2004.
[SZ05] M. Sanderson and J. Zobel. : Information retrieval system
eval-uation: Effort, sensitivity, and reliability. In: Proceedings
of the28th Annual International ACM SIGIR Conference on
Re-searchand Development in Information Retrieval, SIGIR 2005,
2005.
[Tau54] M. et al. Taube. The uniterm coordinate indexing of
reports. InThe Technical Report. New York, Reinhold, 1954.
[VH05] Ellen M. Voorhees and Donna K. Harman. TREC:
Experimentand Evaluation in Information Retrieval (Digital
Libraries andElectronic Publishing). The MIT Press, 2005.
[Voo02] Ellen M. Voorhees. The philosophy of information
retrievalevaluation. In Revised Papers from the Second Workshop
ofthe Cross-Language Evaluation Forum on Evaluation of
Cross-Language Information Retrieval Systems, CLEF 01, pages
355370, London, UK, UK, 2002. Springer-Verlag.
[YAR08] Emine Yilmaz, Javed A. Aslam, and Stephen Robertson. A
newrank correlation coefficient for information retrieval. In
Proceed-ings of the 31st Annual International ACM SIGIR
Conferenceon Research and Development in Information Retrieval,
SIGIR08, pages 587594, New York, NY, USA, 2008. ACM.
98
[YSCR10] E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson.
: Ex-pected browsing utility for web search evaluation. In:
Proceedingsof the 19th ACM International Conference of Information
andknowledge management, CIKM 2010, 2010.
[ZJ07] V. Zhang and R. Jones. Comparing click logs and editorial
labelsfor training query rewriting. in query log analysis: Social
andtechnological challenges. In Workshop of the 2007
InternationalWorld Wide Web Conference, 2007.
99
- TREC
(DCG)Rank Biased PrecisionBprefExpected Reciprocal Rank
Crowdsourcing A/B
o Reciprocal Rank using Document Popularity
COEC
cascade UBM DCM DBN
Bayes