Evaluating Implicit Measures to Improve the Search Experience SIGIR 2003 Steve Fox
Mar 26, 2015
Evaluating Implicit Measures to Improve the
Search Experience
SIGIR 2003
Steve Fox
Outline
Background Approach Data Analysis Value-Add Contributions Result-Level Findings Session-Level Findings
Background
Interested in implicit measures to improve user’s search experience What the user wants What satisfies them Significant implicit measures
Needed to prove it! Two goals:
Test association between implicit measures and user satisfaction
Understand what implicit measures were useful within this association
Approach
Architecture Internet Explorer add-in Client-Server Configured for MSN Search and Google
Deployment Internal MS employees (n = 146) – work environment Implicit measures and explicit feedback SQL Server back-end
Approach, cont’d
Data Analysis Bayesian modeling at result and session
level Trained on 80% and tested on 20% Three levels of SAT – VSAT, PSAT & DSAT Implicit measures:
Result-Level Session-Level
Diff Secs, Duration Secs Averages of result-level measures (Dwell Time and Position)
Scrolled, ScrollCnt, AvgSecsBetweenScroll, TotalScrollTime, MaxScroll
Query count
TimeToFirstClick, TimeToFirstScroll Results set count
Page, Page Position, Absolute Position Results visited
Visits End action
Exit Type
ImageCnt, PageSize, ScriptCnt
Added to Favorites, Printed
Data Analysis, cont’d
Result-Level Findings
1. Dwell time, clickthrough and exit type strongest predictors of SAT
2. Printing and Adding to Favorites highly predictive of SAT when present
3. Combined measures predict SAT better than clickthrough
Result Level Findings, cont’d
Only clickthrough
Combined measures
Combined measures with confidence of > 0.5 (80-20
train/test split)
Session-Level Findings
Four findings:1. Strong predictor of session-level SAT was
result-level SAT
2. Dwell time strong predictor of SAT
3. Combination of (slightly different) implicit measures could predict SAT better than clickthrough
4. Some gene sequences predict SAT (preliminary and descriptive)
Session Level Findings, cont’d
Common patterns in gene analysis, e.g. SqLrZ Session starts (S) Submit a query (q) Result list returned (L) Click a result (r) Exit on result (Z)
PatternFrequenc
y%VSA
T%PSA
T%DSA
TAvg. VSAT Dwell Time
Avg. PSAT Dwell Time
Avg. DSAT Dwell Time
SqLrZ 117 75 15 9 64 6 11
Value-Add Contributions
Deployed in the work setting Collected data in context of web search
Rich user behavior data stream Annotated data stream with explicit judgment
Used new methodology to analyze the data ‘Gene analysis’ to analyze usage patterns Mapped usage patterns to SAT
Question(s)