Click Models for Web Search - Lecture 1 · Lecture 1 Lecture 2 Lecture 4 Practical 2 Lecture 5 Lecture 3 Lecture 2 Practical 1 AC{IM{MdR Click Models for Web Search 10. ... # shown

Post on 04-Jun-2018

226 Views

Category:

Documents

0 Downloads

Preview:

Click to see full reader

Transcript

Introduction Basic click models Click probabilities

Click Models for Web SearchLecture 1

Aleksandr Chuklin§,¶ Ilya Markov§ Maarten de Rijke§

a.chuklin@uva.nl i.markov@uva.nl derijke@uva.nl

§University of Amsterdam¶Google Research Europe

AC–IM–MdR Click Models for Web Search 1

Introduction Basic click models Click probabilities

Aleksandr Chuklin

Currently at Google Zurich

Previously at Yandex Moscow

Research interests: user experience evaluation and modelling

Participated at RuSSIR 2009 in Petrozavodsk and RuSSIR2011 in Saint Petersburg

AC–IM–MdR Click Models for Web Search 2

Introduction Basic click models Click probabilities

Ilya Markov

Postdoctoral researcher at the University of Amsterdam

PhD at the University of Lugano

Research interests: heterogeneous search environments

Distributed IR, federated search, aggregated searchUser behavior, user-oriented evaluation

Teach MSc courses on IR and Web Search

AC–IM–MdR Click Models for Web Search 3

Introduction Basic click models Click probabilities

Long-term relations with RuSSIR

RuSSIR 2007, student

RuSSIR 2010, lecturer on Distributed IR (with Fabio Crestani)

RuSSIR 2011, member of organizing committee

RuSSIR 2015, chair of program committee

RuSSIR 2016, lecturer

AC–IM–MdR Click Models for Web Search 4

Introduction Basic click models Click probabilities

Course on Information Retrieval in St. Petersburg

http://compsciclub.ru/courses/

information-retrieval/2016-autumn/

AC–IM–MdR Click Models for Web Search 5

Introduction Basic click models Click probabilities

Maarten de Rijke

Currently at the University of Amsterdam

Ongoing collaborations with Bloomberg Labs, Google,Microsoft Research, Yandex Moscow

Research interests: semantic search, online learning to rank

Always looking for strong new PhD students

AC–IM–MdR Click Models for Web Search 6

Introduction Basic click models Click probabilities

The book

http://clickmodels.weebly.com/the-book.html

AC–IM–MdR Click Models for Web Search 7

Introduction Basic click models Click probabilities

Other course materials

clickmodels.weebly.com/russir-2016-course.html

Demos and practical sessions:

clickmodels.weebly.com/russir-2016-setup.html

github.com/markovi/PyClick

AC–IM–MdR Click Models for Web Search 8

Introduction Basic click models Click probabilities

Course content

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

AC–IM–MdR Click Models for Web Search 9

Introduction Basic click models Click probabilities

Lectures

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

Lecture 1 Lecture 2

Lecture 4Practical 2 Lecture 5

Lecture 3Practical 1Lecture 2

AC–IM–MdR Click Models for Web Search 10

Introduction Basic click models Click probabilities

Course overview

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

AC–IM–MdR Click Models for Web Search 11

Introduction Basic click models Click probabilities

This lecture

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

AC–IM–MdR Click Models for Web Search 12

Introduction Basic click models Click probabilities

Lecture outline

1 Introduction

2 Basic click models

3 Click probabilities

AC–IM–MdR Click Models for Web Search 13

Introduction Basic click models Click probabilities

Web search

AC–IM–MdR Click Models for Web Search 14

Introduction Basic click models Click probabilities

Why clicks?

AC–IM–MdR Click Models for Web Search 15

Introduction Basic click models Click probabilities

Why clicks?

Reflect user interests

Help to improve search

Help to evaluate search

Ongoing and future research: other user search interactions

mouse movementsscrollingtouch gestures

AC–IM–MdR Click Models for Web Search 16

Introduction Basic click models Click probabilities

What can we do with clicks?

AC–IM–MdR Click Models for Web Search 17

Introduction Basic click models Click probabilities

What can we do with clicks?

countclick-through rate (CTR)

Global CTR = # clicks# shown docs

Rank-based CTR = # clicks at rank r# shown docs at rank r

Query-document CTR = # u is clicked for q# u is shown for q

Some notation: u – URL (or document), q – query

AC–IM–MdR Click Models for Web Search 18

Introduction Basic click models Click probabilities

Why click models?

AC–IM–MdR Click Models for Web Search 19

Introduction Basic click models Click probabilities

Why click models?

Scientific modelling is a scientific activity, the aim of which is tomake a particular part or feature of the world easier to understand,define, quantify, visualize, or simulate by referencing it to existingand usually commonly accepted knowledge.

Wikipedia, Scientific modelling

AC–IM–MdR Click Models for Web Search 20

Introduction Basic click models Click probabilities

Why click models?

Click models make user clicks in web searcheasier to understand, define, quantify, visualize, or simulate

using (mostly) probabilistic graphical models.

AC–IM–MdR Click Models for Web Search 21

Introduction Basic click models Click probabilities

Click log

Yandex Relevance Prediction Challengehttp://imat-relpred.yandex.ru/en

AC–IM–MdR Click Models for Web Search 22

Introduction Basic click models Click probabilities

Why do we need click models?

Understand users

Simulate users

Approximate document relevance

Evaluate search

AC–IM–MdR Click Models for Web Search 23

Introduction Basic click models Click probabilities

Lecture outline

1 Introduction

2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model

3 Click probabilities

AC–IM–MdR Click Models for Web Search 24

Introduction Basic click models Click probabilities

Lecture outline

2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model

AC–IM–MdR Click Models for Web Search 25

Introduction Basic click models Click probabilities

Random click model

Pclick

Pclick

Pclick

Pclick

Pclick

AC–IM–MdR Click Models for Web Search 26

Introduction Basic click models Click probabilities

Random click model

Terminology

Cu – binary random variable denoting a click on document uDocument u is clicked: Cu = 1Document u is not clicked: Cu = 0P(Cu = 1) – probability of click on document uP(Cu = 0) = 1− P(Cu = 1)

Random click model (RCM)

Any document can be clicked with the same (fixed) probability

P(Cu = 1) = const = ρ

AC–IM–MdR Click Models for Web Search 27

Introduction Basic click models Click probabilities

Random click model

P(Cu1 = 1) = ρ

P(Cu2 = 1) = ρ

P(Cu3 = 1) = ρ

P(Cu4 = 1) = ρ

P(Cu5 = 1) = ρ

ρ =# clicks

# shown docs= Global CTR

AC–IM–MdR Click Models for Web Search 28

Introduction Basic click models Click probabilities

Lecture outline

2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model

AC–IM–MdR Click Models for Web Search 29

Introduction Basic click models Click probabilities

Rank-based CTR model

P(Cu1 = 1) = ρ1

P(Cu2 = 1) = ρ2

P(Cu3 = 1) = ρ3

P(Cu4 = 1) = ρ4

P(Cu5 = 1) = ρ5

P(Cur = 1) = ρr =# clicks at rank r

# shown docs at rank r

AC–IM–MdR Click Models for Web Search 30

Introduction Basic click models Click probabilities

Query-document CTR model

P(Cu1 = 1) = ρu1q

P(Cu2 = 1) = ρu2q

P(Cu3 = 1) = ρu3q

P(Cu4 = 1) = ρu4q

P(Cu5 = 1) = ρu5q

P(Cu = 1) = ρuq =# u is clicked for q

# u is shown for q

AC–IM–MdR Click Models for Web Search 31

Introduction Basic click models Click probabilities

CTR models: summary

Random click model (global CTR):

P(Cu = 1) = ρ

Rank-based CTR:

P(Cur = 1) = ρr

Query-document CTR:

P(Cu = 1) = ρuq

AC–IM–MdR Click Models for Web Search 32

Introduction Basic click models Click probabilities

CTR models: demo

Demo

AC–IM–MdR Click Models for Web Search 33

Introduction Basic click models Click probabilities

Lecture outline

2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model

AC–IM–MdR Click Models for Web Search 34

Introduction Basic click models Click probabilities

Position-based model

Pread(1) , Pclick(u1q)

Pread(2), Pclick(u2q)

Pread(3) , Pclick(u3q)

Pread(4) , Pclick(u4q)

Pread(5) , Pclick(u5q)

AC–IM–MdR Click Models for Web Search 35

Introduction Basic click models Click probabilities

Position-based model: examination

TerminologyExamination = reading a snippet

Er – binary random variable denoting examinationof a snippet at rank r

Snippet at rank r is examined: Er = 1

Snippet at rank r is not examined: Er = 0

P(Er = 1) – probability of examination of rank r

P(Er = 0) = 1− P(Er = 1)

Position-based model (PBM)Examination depends on rank

P(Er = 1) = γr

AC–IM–MdR Click Models for Web Search 36

Introduction Basic click models Click probabilities

Position-based model

γ1 , Pclick(u1q)

γ2 , Pclick(u2q)

γ3 , Pclick(u3q)

γ4 , Pclick(u4q)

γ5 , Pclick(u5q)

AC–IM–MdR Click Models for Web Search 37

Introduction Basic click models Click probabilities

Position-based model: attractiveness

TerminologyAttractiveness = a user wants to click on a documentafter examining (reading) its snippet

Au – binary random variable showing whether document uis attractive to a user, given query q

Document u is attractive: Au = 1

Document u is not attractive: Au = 0

P(Au = 1) – probability of attractiveness of document u

P(Au = 0) = 1− P(Au = 1)

Position-based model (PBM)Attractiveness depends on a query-document pair

P(Auq = 1) = αuq

AC–IM–MdR Click Models for Web Search 38

Introduction Basic click models Click probabilities

Position-based model

γ1 , αu1q

γ2 , αu2q

γ3 , αu3q

γ4 , αu4q

γ5 , αu5q

AC–IM–MdR Click Models for Web Search 39

Introduction Basic click models Click probabilities

Position-based model: summary

P(Eru = 1) = γru

P(Au = 1) = αuq

P(Cu = 1) = P(Eru = 1) · P(Au = 1)

AC–IM–MdR Click Models for Web Search 40

Introduction Basic click models Click probabilities

Position-based model: probabilistic graphical model

document u

Eu

Cu

Au

↵uq�ru

AC–IM–MdR Click Models for Web Search 41

Introduction Basic click models Click probabilities

Position-based model: exercises

P(Eru = 1) = γru

P(Au = 1) = αuq

P(Cu = 1) = P(Eru = 1) · P(Au = 1)

Eru = 0⇒ Cu = 0

Au = 0⇒ Cu = 0

Eru = 1⇒ (Cu = 1 ⇐⇒ Au = 1)

Au = 1⇒ (Cu = 1 ⇐⇒ Eru = 1)

AC–IM–MdR Click Models for Web Search 42

Introduction Basic click models Click probabilities

Position-based model: demo

Demo

AC–IM–MdR Click Models for Web Search 43

Introduction Basic click models Click probabilities

Lecture outline

2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model

AC–IM–MdR Click Models for Web Search 44

Introduction Basic click models Click probabilities

Position-based model

P(Eru = 1) = γru

P(Au = 1) = αuq

P(Cu = 1) = P(Eru = 1) · P(Au = 1)

AC–IM–MdR Click Models for Web Search 45

Introduction Basic click models Click probabilities

Cascade model

1 Start from the first document

2 Examine documents one by one

3 If click, then stop

4 Otherwise, continue

AC–IM–MdR Click Models for Web Search 46

Introduction Basic click models Click probabilities

Cascade model

Er = 1 and Aur = 1⇔ Cr = 1

P(Aur = 1) = αurq

P(E1 = 1)︸ ︷︷ ︸start from first

= 1

P(Er = 1 | Er−1 = 0)︸ ︷︷ ︸examine one by one

= 0

P(Er = 1 | Cr−1 = 1)︸ ︷︷ ︸if click, then stop

= 0

P(Er = 1 | Er−1 = 1,Cr−1 = 0)︸ ︷︷ ︸otherwise, continue

= 1

AC–IM–MdR Click Models for Web Search 47

Introduction Basic click models Click probabilities

Cascade model: probabilistic graphical model

document urdocument ur�1

Er�1

Cr�1

Ar�1

Er

Cr

Ar

......

↵ur�1q ↵urq

AC–IM–MdR Click Models for Web Search 48

Introduction Basic click models Click probabilities

Click models so far

CTR models

+ count clicks (simple and fast)– do not distinguish examination and attractiveness

Position-based model (PBM) −→ User browsing model+ examination and attractiveness– examination of a document at rank r does not depend

on examinations and clicks above r

Cascade model (CM) −→ Dynamic Bayesian network+ cascade dependency of examination at r

on examinations and clicks above r– only one click is allowed

AC–IM–MdR Click Models for Web Search 49

Introduction Basic click models Click probabilities

Lecture outline

2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model

AC–IM–MdR Click Models for Web Search 50

Introduction Basic click models Click probabilities

Dynamic Bayesian network model

Cascade model −→ Dynamic Bayesian network

AC–IM–MdR Click Models for Web Search 51

Introduction Basic click models Click probabilities

Dynamic Bayesian network model

1 Start from the first document

2 Examine documents one by one

3 If click, read actual documentand can be satisfied

4 If satisfied, stop

5 Otherwise, continue with fixedprobability

AC–IM–MdR Click Models for Web Search 52

Introduction Basic click models Click probabilities

Dynamic Bayesian network model: satisfaction

TerminologySatisfaction = a user reads the clicked documentand satisfies his/her information needSu – binary random variable showing whether document uis satisfactory for query qP(Su = 1) – probability of satisfactoriness of document u,P(Su = 0) = 1− P(Su = 1)

Dynamic Bayesian network model (DBN)If a user is satisfied, he/she stopsOtherwise, continues with fixed probability

P(Er = 1 | Sr−1 = 1)︸ ︷︷ ︸if satisfied, stop

= 0

P(Er = 1 | Er−1 = 1, Sr−1 = 0)︸ ︷︷ ︸otherwise, continue

= γ

AC–IM–MdR Click Models for Web Search 53

Introduction Basic click models Click probabilities

Dynamic Bayesian network model: summary

Er = 1 and Aur = 1⇔ Cr = 1

P(Aur = 1) = αurq

P(E1 = 1) = 1

P(Er = 1 | Er−1 = 0) = 0

P(Sr = 1 | Cr = 1)︸ ︷︷ ︸if click, can be satisfied

= σurq

P(Er = 1 | Sr−1 = 1)︸ ︷︷ ︸if satisfied, stop

= 0

P(Er = 1 | Er−1 = 1, Sr−1 = 0)︸ ︷︷ ︸otherwise, continue

= γ

AC–IM–MdR Click Models for Web Search 54

Introduction Basic click models Click probabilities

Dynamic Bayesian network: probabilistic graphical model

document urdocument ur�1

Er�1

Cr�1

Ar�1

Er

Cr

Ar

......

↵ur�1q ↵urq

Sr�1 Sr

�ur�1q �urq

AC–IM–MdR Click Models for Web Search 55

Introduction Basic click models Click probabilities

Dynamic Bayesian network model: demo

Demo

AC–IM–MdR Click Models for Web Search 56

Introduction Basic click models Click probabilities

Lecture outline

2 Basic click modelsRandom click modelCTR modelsPosition-based modelCascade modelDynamic Bayesian network modelUser browsing model

AC–IM–MdR Click Models for Web Search 57

Introduction Basic click models Click probabilities

User browsing model

Position-based model −→ User browsing model

AC–IM–MdR Click Models for Web Search 58

Introduction Basic click models Click probabilities

Position-based model

γ1 , αu1q

γ2 , αu2q

γ3 , αu3q

γ4 , αu4q

γ5 , αu5q

P(Er = 1 | Cr ′ = 1,Cr ′+1 = 0, . . . ,Cr−1 = 0) = γrr ′

AC–IM–MdR Click Models for Web Search 59

Introduction Basic click models Click probabilities

User browsing model

γ10 , αu1q

γ21 , αu2q

γ31 , αu3q

γ41 , αu4q

γ54 , αu5q

P(Er = 1 | Cr ′ = 1,Cr ′+1 = 0, . . . ,Cr−1 = 0) = γrr ′

AC–IM–MdR Click Models for Web Search 60

Introduction Basic click models Click probabilities

User browsing model: summary

P(Cu = 1) = P(Eru = 1) · P(Au = 1)

P(Au = 1) = αuq

P(Er = 1 |Cr ′ = 1,

Cr ′+1 = 0, . . . ,Cr−1 = 0) = γrr ′

AC–IM–MdR Click Models for Web Search 61

Introduction Basic click models Click probabilities

User browsing model: probabilistic graphical model

document ur

Er

Cr

Ar

...

↵urq

�rr0

AC–IM–MdR Click Models for Web Search 62

Introduction Basic click models Click probabilities

Basic click models summary

CTR models: counting clicks

Position-based model (PBM): examination and attractiveness

Cascade model (CM): previous examinations and clicks matter

Dynamic Bayesian network model (DBN): satisfactoriness

User browsing model (UBM): rank of previous click

AC–IM–MdR Click Models for Web Search 63

Introduction Basic click models Click probabilities

Probability theory

Partitioned probability: A = A1 ∪ A2, A1 ∩ A2 = ∅

P(A) = P(A1,A2) = P(A1) + P(A2)

Bayes’ rule

P(A | B) · P(B) = P(B | A) · P(A)

B causes A: B → A

P(B) = P(B | A) · P(A)

AC–IM–MdR Click Models for Web Search 64

Introduction Basic click models Click probabilities

Probability theory (cont’d)

B → A, A = A1 ∪ A2, A1 ∩ A2 = ∅

P(B) = P(B | A) · P(A)

= P(B | A1,A2) · P(A1,A2)

= P(B | A1,A2) · (P(A1) + P(A2))

= P(B | A1,A2) · P(A1) + P(B | A1,A2) · P(A2)

= P(B | A1) · P(A1) + P(B | A2) · P(A2)

P(B) = P(B | A1) · P(A1) + P(B | A2) · P(A2)

AC–IM–MdR Click Models for Web Search 65

Introduction Basic click models Click probabilities

Lecture outline

1 Introduction

2 Basic click models

3 Click probabilities

AC–IM–MdR Click Models for Web Search 66

Introduction Basic click models Click probabilities

Click probabilities

Full probability – probabilitythat a user clickson a document at rank r

P(Cr = 1)

Conditional probability –probability that a user clickson a document at rank rgiven previous clicks

P(Cr = 1 | C1, . . . ,Cr−1)

AC–IM–MdR Click Models for Web Search 67

Introduction Basic click models Click probabilities

Dependency between examination and clicks

document u

Eu

Cu

Au

↵uq�ru

AC–IM–MdR Click Models for Web Search 68

Introduction Basic click models Click probabilities

Full click probability

P(Cr = 1) = +P(Cr = 1 | Er = 1) · P(Er = 1)

P(Cr = 1 | Er = 0) · P(Er = 0)

= P(Aur = 1) · P(Er = 1) + 0

= αurqεr

AC–IM–MdR Click Models for Web Search 69

Introduction Basic click models Click probabilities

Cascade models: dependency between examinations

document urdocument ur�1

Er�1

Cr�1

Ar�1

Er

Cr

Ar

......

↵ur�1q ↵urq

AC–IM–MdR Click Models for Web Search 70

Introduction Basic click models Click probabilities

Full click probability

P(Cr = 1) = P(Aur = 1) · P(Er = 1) = αurqεr

εr+1 = P(Er+1 = 1)

= +P(Er = 1) · P(Er+1 = 1 | Er = 1)

P(Er = 0) · P(Er+1 = 1 | Er = 0)

= εr · P(Er+1 = 1 | Er = 1) + 0

= εr ·

(+P(Er+1 = 1 | Er = 1,Cr = 1) · P(Cr = 1 | Er = 1)

P(Er+1 = 1 | Er = 1,Cr = 0) · P(Cr = 0 | Er = 1)

)

AC–IM–MdR Click Models for Web Search 71

Introduction Basic click models Click probabilities

Full click probability: Dynamic Bayesian network model

Dynamic Bayesian network model: satisfactoriness

document urdocument ur�1

Er�1

Cr�1

Ar�1

Er

Cr

Ar

......

↵ur�1q ↵urq

Sr�1 Sr

�ur�1q �urq

P(Cr+1 = 1) = αur+1qεr ·

(+P(Er+1 = 1 | Er = 1,Cr = 1) · P(Cr = 1 | Er = 1)

P(Er+1 = 1 | Er = 1,Cr = 0) · P(Cr = 0 | Er = 1)

)

P(Cr+1 = 1) = αur+1qεr ·

(+

(1− σurq)γ · αurq

γ · (1− αurq)

)AC–IM–MdR Click Models for Web Search 72

Introduction Basic click models Click probabilities

Conditional click probability

P(Cr = 1 | C1, . . . ,Cr−1) = P(Cr = 1 | C<r )

= +P(Cr = 1 | Er = 1,C<r ) · P(Er = 1 | C<r )

P(Cr = 1 | Er = 0,C<r ) · P(Er = 0 | C<r )

= P(Aur = 1) · P(Er = 1 | C<r ) + 0

= αurqεr

εr+1 = +

P(Er+1 = 1 | Er = 1,Cr = 1) · c(s)r

P(Er+1 = 1 | Er = 1,Cr = 0) · εr (1− αurq)

1− αurqεr· (1− c(s)

r )

c(s)r – a click on rank r in query session s

AC–IM–MdR Click Models for Web Search 73

Introduction Basic click models Click probabilities

Click probabilities summary

Full probability

P(Cr+1 = 1) =

αur+1qεr ·

(+P(Er+1 = 1 | Er = 1,Cr = 1) · P(Cr = 1 | Er = 1)

P(Er+1 = 1 | Er = 1,Cr = 0) · P(Cr = 0 | Er = 1)

)

Conditional probability

P(Cr+1 = 1 | C1, . . . ,Cr )

= αur+1q ·

+

P(Er+1 = 1 | Er = 1,Cr = 1) · c(s)r

P(Er+1 = 1 | Er = 1,Cr = 0) · εr (1− αurq)

1− αurqεr· (1− c(s)

r )

AC–IM–MdR Click Models for Web Search 74

Introduction Basic click models Click probabilities

Lecture 1 summary

CTR models: counting clicks

Position-based model (PBM): examination and attractiveness

Cascade model (CM): previous examinations and clicks matter

Dynamic Bayesian network model (DBN): satisfactoriness

User browsing model (UBM): rank of previous click

AC–IM–MdR Click Models for Web Search 75

Introduction Basic click models Click probabilities

Lecture 1 summary

What do click models give us?

General

Understanding of user behavior

Specific

Conditional click probabilitiesFull click probabilitiesAttractiveness and satisfactoriness for query-document pairs

AC–IM–MdR Click Models for Web Search 76

Introduction Basic click models Click probabilities

Course overview

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

AC–IM–MdR Click Models for Web Search 77

Introduction Basic click models Click probabilities

Next lecture

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research

AC–IM–MdR Click Models for Web Search 78

Introduction Basic click models Click probabilities

Acknowledgments

All content represents the opinion of the authors which is not necessarily shared orendorsed by their respective employers and/or sponsors.

AC–IM–MdR Click Models for Web Search 79

top related