Large scale-ctr-prediction lessons-learned-florian-hartl

Florian [email protected]

Large Scale CTR PredictionLessons Learned

Yelp’s MissionConnecting people with great

local businesses.

92M 3272%108M

Yelp StatsAs of Q2 2016

CTR Prediction

CTR: Click-Through RatepCTR: predicted CTR

QuestionHow likely is the user to click on the ad?

WhyProxy for relevance

5.5%

0.8%

9.2%

?

Logistic Regression with

thousands of features,

trained and tested on

millions of samples.

Current pCTR Model

Kuvasz

pCTR Model History

(CC) from Flickr: "Wednesday Freedom 11"by Parker Knight

(CC) from Flickr: "Icelandig sheepdog"by Thomas Quine(CC) from Flickr: by Craige Moore

FrenchBrittany

Icelandic Sheepdog

Jindo Kuvasz

Lessons Learned(CC) from Flickr: "WEL" by luckyno3

user feedbackservice

onlineoffline

data model

logs

(CC) from Flickr: "The huge crossing" by Miroslav Petrasko

Infrastructure

(CC) from Flickr: "KOGI and WEL" by luckyno3


onlineoffline

data model

logs


logs

Log at source of online prediction→ Prevents downstream modifications of data

Logging


onlineoffline

data model

logs

data

logsprediction verification

Assert validity of logged data

Verification

model


onlineoffline

data model


data model


fastscalable

Make offline training iterations fast & scalable

Automation is key→ end-to-end pipeline→ automated visualizations

Tools: mrjob, Spark

Iterations

Offline Training at Yelp

merge logs sampling feature extraction

model training evaluation

mrjobAWS EMR

daily scheduled pipelinekicked off manually

mrjobAWS EMR

Spark

mrjobAWS EMR

mrjobAWS EMR

mrjobAWS EMR

new features

(CC) from Flickr: "Cloud" by Jason Pratt

Lessons Learned

InfrastructureLog at source of online predictionVerify predictionsMake offline iterations fast & scalable

Model Comprehension

(CC) from Flickr: "Bella" by Maureen Lee


onlineoffline

data model


fastscalable

Focus on a single metric(but don't trust it blindly)

Evaluation

data model

prediction verification

evaluation

fastscalable

Our Metric

Focus on a single metric(but don't trust it blindly)

Create helpful visualizations

Tools: Zeppelin

Evaluation

data model

prediction verification

evaluation

fastscalable

Visualizations...

feature 1feature 2feature 3

...

feature contribution

Feature contributionssd(feature) * coef

Feature value vs. CTR count

feature value

CTR


onlineoffline

data model


evaluation

fastscalable

logs

Beware of biased training data→ offline != online→ pCTR threshold

Thresholds


pCTR Threshold

CTR pCTR

Model 1Good

CTR pCTR

Model 2Bad

CTR pCTR

Model 3Good

pCTR Threshold

time

training data

Model 1 Model 2 Model 3 Model 4Idea:Frequent retraining

Better:Deliberate sampling of bad ads

CTR pCTR

Online Evaluation

CTR pCTR

Model 1Good

CTR pCTR

Model 2Bad

CTR pCTR

Model 3Good

Online Evaluation

CTR pCTR

Model 1Good

CTR pCTR

Model 2Bad

CTR pCTR

Model 3Good

Combined Rescoring

new modelcurrent model

online

offline

Combined Rescoring

new modelcurrent model

online

offline

evaluation

Lessons Learned


Model ComprehensionEvaluate, evaluate, evaluateBe aware of threshold effects


onlineoffline

data model


evaluation

fastscalable


onlineoffline

data model


evaluation

fastscalable

simplicity

simplicity

rule-based approach

simple models

Occam's razor

appropriate metric

documentation

"Simple Made Easy"


onlineoffline

data model


evaluation

fastscalablewell documented


simplicity


onlineoffline

data model


evaluation



simplicity

Lessons Learned

Above all, keep it simple.


Model ComprehensionEvaluate, evaluate, evaluateBe aware of threshold effects

@YelpEngineering

engineeringblog.yelp.com

github.com/yelp

yelp.com/careers

Large scale-ctr-prediction lessons-learned-florian-hartl

Data & Analytics