Click Models for Web Search - Lecture 3clickmodels.weebly.com/uploads/5/2/2/5/52257029/russir2016-click... · EvaluationData and toolsResults Click Models for Web Search Lecture 3

Evaluation Data and tools Results

Click Models for Web SearchLecture 3

Aleksandr Chuklin§,¶ Ilya Markov§ Maarten de Rijke§

[email protected] [email protected] [email protected]

§University of Amsterdam¶Google Research Europe

AC–IM–MdR Click Models for Web Search 1

Evaluation Data and tools Results

Course overview

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

AC–IM–MdR Click Models for Web Search 2

Evaluation Data and tools Results

This lecture

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

AC–IM–MdR Click Models for Web Search 3

Evaluation Data and tools Results

What do click models give us?

General:

Understanding of user behavior

Specific:

Conditional click probabilities

Full click probabilities

Attractiveness and satisfactoriness for query-document pairs

AC–IM–MdR Click Models for Web Search 4

Evaluation Data and tools Results

Lecture outline

1 EvaluationLikelihoodPerplexityRanking evaluation

2 Data and tools

3 Results

AC–IM–MdR Click Models for Web Search 5

Evaluation Data and tools Results

Evaluation summary

Click model’s output Evaluation

Conditional click probabilities Log-likelihoodFull click probabilities PerplexityParameter values Ranking evaluation

AC–IM–MdR Click Models for Web Search 6

Evaluation Data and tools Results

Lecture outline

1 EvaluationLikelihoodPerplexityRanking evaluation

AC–IM–MdR Click Models for Web Search 7

Evaluation Data and tools Results

Likelihood

Likelihood measures how well a click model estimatesconditional click probabilities given observed clicks.

LL(M) =1

|S|∑s∈S

logPM

(C1 = c

(s)1 , . . . ,Cn = c

(s)n

)Cr – binary random variable denoting a click at rank r

c(s)r – observed click at rank r in a search session s

P(Cr = c

(s)r

)– probability of observing c

(s)r in session s

P(C1 = c

(s)1 , . . . ,Cn = c

(s)n

)– probability of observing

sequence c(s)1 , . . . , c

(s)n in session s

AC–IM–MdR Click Models for Web Search 8

Evaluation Data and tools Results

Likelihood

PM

(C1 = c

(s)1 , . . . ,Cn = c

(s)n

)= PM

(C1 = c

(s)1

)· PM

(C2 = c

(s)2 , . . . ,Cn = c

(s)n | C1 = c

(s)1

)= PM

(C1 = c

(s)1

)· PM

(C2 = c

(s)2 | C1 = c

(s)1

)· PM

(C3 = c

(s)3 , . . . ,Cn = c

(s)n | C1 = c

(s)1 ,C2 = c

(s)2

)=

n∏r=1

PM

(Cr = c

(s)r | C<r = c

(s)<r

)

AC–IM–MdR Click Models for Web Search 9

Evaluation Data and tools Results

Likelihood: summary

LL(M) =1

|S|∑s∈S

logPM

(C1 = c

(s)1 , . . . ,Cn = c

(s)n

)LL(M) =

1

|S|∑s∈S

n∑r=1

logPM

(Cr = c

(s)r | C<r = c

(s)<r

)

Likelihood measures how well a click model estimatesconditional click probabilities given observed clicks.

LL(M) ∈ [−∞..0]

AC–IM–MdR Click Models for Web Search 10

Evaluation Data and tools Results

Lecture outline

1 EvaluationLikelihoodPerplexityRanking evaluation

AC–IM–MdR Click Models for Web Search 11

Evaluation Data and tools Results

Perplexity

Perplexity measures how well a click model estimatesfull click probabilities (i.e., when clicks are not observed).

pr (M) = 2− 1

|S|∑

s∈S

(log2 PM(C

(s)r =c

(s)r ))

pr (M) ∈ [1..2]

AC–IM–MdR Click Models for Web Search 12

Evaluation Data and tools Results

Lecture outline

1 EvaluationLikelihoodPerplexityRanking evaluation

AC–IM–MdR Click Models for Web Search 13

Evaluation Data and tools Results

Ranking evaluation

R̂el i Reli

αu1q 4

αu2q 2

αu3q 1

αu4q 4

αu5q 2

DCG =n∑

i=1

2Reli − 1

log2(i + 1)

AC–IM–MdR Click Models for Web Search 14

Evaluation Data and tools Results

Evaluation summary

Click model’s output Evaluation

Conditional click probabilities Log-likelihoodFull click probabilities PerplexityParameter values Ranking evaluation

AC–IM–MdR Click Models for Web Search 15

Evaluation Data and tools Results

Lecture outline

1 Evaluation

2 Data and tools

3 Results

AC–IM–MdR Click Models for Web Search 16

Evaluation Data and tools Results

Datasets

AOL2006: raw queries and clicked documents (no SERPs)

MSN2006: contains only clicked documents (no SERPs)

Workshop on Web Search Click Data (WSCD)

WSCD2012: predict document relevanceWSCD2013: detect search engine switchWSCD2014: search personalization

SogouQ

Tsinghua University: eye fixation

AC–IM–MdR Click Models for Web Search 17

Evaluation Data and tools Results

Dataset statistics

Dataset Queries URLs Users Sessions

AOL 2006 10,154,742 1,632,788 657,426 21,011,340MSN 2006 8,831,280 4,975,897 – 7,470,915SogouQ 2012 8,939,569 15,095,269 9,739,704 25,530,711WSCD 2012 30,717,251 117,093,258 – 146,278,823WSCD 2013 10,139,547 49,029,185 956,536 17,784,583WSCD 2014 21,073,569 70,348,426 5,736,333 65,172,853

AC–IM–MdR Click Models for Web Search 18

Evaluation Data and tools Results

Software

Click model packages

clickmodels project by Aleksandr ChuklinPyClick by Ilya Markov et al.

Infer.NET

General-purpose languages

OctaveMatlab

AC–IM–MdR Click Models for Web Search 19

Evaluation Data and tools Results

Lecture outline

1 Evaluation

2 Data and tools

3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation

AC–IM–MdR Click Models for Web Search 20

Evaluation Data and tools Results

Experimental setup

Data

first 1M query sessions from WSCD 2012 dataset75% for training, 25% for testingrepeat 15 times, each time with next 1M sessions

PyClick

50 iterations for EM

AC–IM–MdR Click Models for Web Search 21

Evaluation Data and tools Results

Studied click models

CTR models: counting clicks

Position-based model (PBM): examination and attractiveness

Cascade model (CM): previous examinations and clicks matter

Dynamic Bayesian network model (DBN): satisfactoriness

User browsing model (UBM): rank of previous click

AC–IM–MdR Click Models for Web Search 22

Evaluation Data and tools Results

Lecture outline

3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation

AC–IM–MdR Click Models for Web Search 23

Evaluation Data and tools Results

Log-likelihood

RCM RCTR DCTR PBM CM UBM SDCM CCM DBN SDBN0.40

0.35

0.30

0.25

0.20

0.15

0.10

0.05

0.00

Log-l

ikelih

ood

Cascade model: LL = −∞Complex models (DBN, UBM) win over simple onesMany examination parameters win over few: UBM > PBMSatisfaction parameters help: DBN > PBM

AC–IM–MdR Click Models for Web Search 24

Evaluation Data and tools Results

Lecture outline

3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation

AC–IM–MdR Click Models for Web Search 25

Evaluation Data and tools Results

Perplexity

RCM RCTR DCTR PBM CM UBM SDCM CCM DBN SDBN1.0

1.1

1.2

1.3

1.4

1.5

Perp

lexit

y

Complex models win over simple ones

Most complex models have similar perplexity

AC–IM–MdR Click Models for Web Search 26

Evaluation Data and tools Results

Perplexity by rank

1 2 3 4 5 6 7 8 9 100.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

GCTRRCTR

DCTRPBM

CMUBM

DCMCCM

DBNSDBN

Picture taken from A. Grotov, A. Chuklin, I. Markov, L. Stout, F. Xumara, and M. de Rijke. A comparative studyof click models for web search. In CLEF. Springer, September 2015.

AC–IM–MdR Click Models for Web Search 27

Evaluation Data and tools Results

Lecture outline

3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation

AC–IM–MdR Click Models for Web Search 28

Evaluation Data and tools Results

Training time

RCM RCTR DCTR PBM CM UBM SDCM CCM DBN SDBN0

500

1000

1500

2000

2500

3000

3500

4000

Tim

e (

sec)

MLE is much faster than EM

PBM and UBM are fast enough compared to DBN

AC–IM–MdR Click Models for Web Search 29

Evaluation Data and tools Results

Lecture outline

3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation

AC–IM–MdR Click Models for Web Search 30

Evaluation Data and tools Results

Experimental setup

Full WSCD 2012 dataset

146,278,823 query sessions30,717,251 unique queries117,093,258 unique URLs41,275 relevance labels (for 4,991 queries)

50% for training, 50% for testing

PyClick

AC–IM–MdR Click Models for Web Search 31

Evaluation Data and tools Results

Log-likelihood and perplexity

Click model Perplexity Log-likelihood

DBN 1.3510 −0.2824DCM 1.3627 −0.3613CCM 1.3692 −0.3560UBM 1.3431 −0.2646

UBM is the best in terms of predicting user click behavior

UBM has the largest number of examination parameters (55)

AC–IM–MdR Click Models for Web Search 32

Evaluation Data and tools Results

Ranking evaluation

NDCG

Click model @1 @3 @5 @10

DBN 0.717 0.725 0.764 0.833DCM 0.736 0.746 0.780 0.844CCM 0.741 0.752 0.785 0.846UBM 0.724 0.737 0.773 0.838

CCM is the best in terms of ranking

Not covered in this corse (but covered in the book)

AC–IM–MdR Click Models for Web Search 33

Evaluation Data and tools Results

Lecture 3 summary

Click model’s output Evaluation Best model

Conditional click probabilities Log-likelihood UBMFull click probabilities Perplexity UBMParameter values Ranking evaluation CCM

Training time MLE-based

AC–IM–MdR Click Models for Web Search 34

Evaluation Data and tools Results

Course overview

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

AC–IM–MdR Click Models for Web Search 35

Evaluation Data and tools Results

Up next

Practical Session 1

AC–IM–MdR Click Models for Web Search 36

Evaluation Data and tools Results

Acknowledgments

All content represents the opinion of the authors which is not necessarily shared orendorsed by their respective employers and/or sponsors.

AC–IM–MdR Click Models for Web Search 37