Modeling and Decision Optimization in Real-time Bidding Display Advertising 实时竞价广告中的建模与决策优化 Kan Ren Apex Data and Knowledge Mangement Lab Shanghai Jiao Tong University Aug. 2018 Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display Advertising Aug. 2018 1 / 98
106
Embed
Modeling and Decision Optimization in Real-time Bidding ... · Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Modeling and Decision Optimizationin Real-time Bidding Display Advertising
实时竞价广告中的建模与决策优化
Kan Ren
Apex Data and Knowledge Mangement LabShanghai Jiao Tong University
Aug. 2018
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 1 / 98
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 16 / 98
Research Problems User Response Prediction
Related Work of User Response PredictionRegression Model
Logistic Regression (LR)K.-c Lee et al. Estimating Conversion Rate in Display Advertising from Past Performance Data. KDD 2012
Tree-based ModelX. He et al. Practical Lessons from Predicting Clicks on Ads at Facebook. ADKDD 2014
Factorization MachinesA.K. Menon et al. Response prediction using collaborative filtering with hierarchies and side-information. KDD2011
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 17 / 98
Research Problems User Response Prediction
Related WorkOther Variants
Bayesian Probit RegressionWeb-scale Bayesian Click-through Rate Prediction for Sponsored Search Advertising in Microsoft’s Bing SearchEngine, T. Graepel et al. ICML 2010
Factorization Machine with FTRLA.-P. Ta. Factorization Machines with Follow-The-Regularized-Leader for CTR prediction in DisplayAdvertising. Big Data 2015
Deep Neural NetworksQ. Liu et al. A Convolutional Click Prediction Model. CIKM 2015.W. Zhang et al. Deep Learning over Multi-field Categorical Data – A Case Study on User Response Prediction.ECIR 2016.Y. Qu, H. Cai, K. Ren. Product-based Neural Networks for User Response Prediction. ICDM 2016.G. Zhou et al. Deep Interest Network for Click-Through Rate Prediction. KDD 2018
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 18 / 98
Research Problems Objective Function
Related Work (cont.)Objective Function
Squared Error
LSE =1
2(y − y)2, y ∈ {0, 1}, y ∈ [0, 1]
Cross Entropy
LCE = −y log y − (1− y) log(1− y), y ∈ {0, 1}, y ∈ [0, 1]
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 19 / 98
Research Problems Evaluation Measurements
Related Work (cont.)Evaluation Measurements
Area under ROC Curve (AUC)
Relative Information Gain (Cross Entropy)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 20 / 98
Research Problems Evaluation Measurements
Related Work (cont.)Traditional Bidding Function
Truthful Bidding Function
b(x) = Vaction · f (x),
where f is the utility estimation function, such as pCTR.
Linear Bidding Function
b(x) = φ · Vaction · f (x) = b0 · f (x).C. Perlich et al. Bid optimizing and inventory scoring in targeted online advertising. KDD 2012.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 21 / 98
Research Problems Problem Setup
Problem Setup
Motivation
To make the user response prediction more precisely accurate consideringthe context.
Rethinking about CTR estimation
Why do we regard the CTR estimation as a classification task?
What is the optimization objective for the advertiser? The accuracyof pCTR?
Why is the bid price linear or positive correlated to the pCTR?
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 22 / 98
Research Problems Problem Setup
Problem Setup
Motivation
To make the user response prediction more precisely accurate consideringthe context.
Rethinking about CTR estimation
Why do we regard the CTR estimation as a classification task?
What is the optimization objective for the advertiser? The accuracyof pCTR?
Why is the bid price linear or positive correlated to the pCTR?
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 22 / 98
Research Problems Problem Setup
ProblemsIsolation Optimization and Joint Optimization
The related works only take classification error as the loss.They do not consider the subsequent usage of the prediction model.
Our Solution
Embed the user response prediction model into the whole procedureof the bidding. And take the overall profit as our learning objectivefunction, to maximize the gains of the advertiser.
K. Ren et al. User Response Learning for Directly Optimizing Campaign Performance in Display Advertising.CIKM, 2016.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 23 / 98
Research Problems Optimization for Campaign Performance
Notations and descriptions
Notation Description
y The true label of user response.x The bid request represented by its features.θ The parameter of CTR estimation function.
fθ(x) the CTR estimation function to learn.b(fθ(x)) The bid price determined by the estimated CTR,
b for short.Rθ(·) The utility function.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 24 / 98
Research Problems Optimization for Campaign Performance
Market Modeling
Market Price
The second highest price proposed during an RTB auction (2nd price).
Market Price Distribution (p.d.f.)
pz(z), z ∈ N .
Winning Probability when Bidding at price b (c.d.f.)
w(b) =
∫ b
0pz(z)dz . (1)
Expected Cost under 2nd Price Auction (if winning)
c(b) =
∫ b0 zpz(z)dz
w(b)=
∫ b0 zpz(z)dz∫ b0 pz(z)dz
. (2)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 25 / 98
Research Problems Optimization for Campaign Performance
Market Modeling
Market Price
The second highest price proposed during an RTB auction (2nd price).
Market Price Distribution (p.d.f.)
pz(z), z ∈ N .
Winning Probability when Bidding at price b (c.d.f.)
w(b) =
∫ b
0pz(z)dz . (1)
Expected Cost under 2nd Price Auction (if winning)
c(b) =
∫ b0 zpz(z)dz
w(b)=
∫ b0 zpz(z)dz∫ b0 pz(z)dz
. (2)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 25 / 98
Research Problems Optimization for Campaign Performance
Market Modeling
Market Price
The second highest price proposed during an RTB auction (2nd price).
Market Price Distribution (p.d.f.)
pz(z), z ∈ N .
Winning Probability when Bidding at price b (c.d.f.)
w(b) =
∫ b
0pz(z)dz . (1)
Expected Cost under 2nd Price Auction (if winning)
c(b) =
∫ b0 zpz(z)dz
w(b)=
∫ b0 zpz(z)dz∫ b0 pz(z)dz
. (2)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 25 / 98
Research Problems Optimization for Campaign Performance
Objective Function
Objective Function
θ∗ = arg maxθ
∫xRθ(x , y ; b, v , c ,w)px(x)dx . (3)
Rθ(·) is the Utility Function.
Constant click value v limits the max bid.
We will propose two variants of Rθ(·).
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 26 / 98
Research Problems Optimization for Campaign Performance
Profit = [gain - (expected cost)] × winning probability
gain = click value × click indicator .
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 27 / 98
Research Problems Optimization for Campaign Performance
Objective of EU
The overall expected direct profit of all the auctions can be calculated byreplacing the winning probability funcion w(b(·)) and the expected costfunction c(b(·)) into EU objective function as∑
(x ,y)∈D
REUθ (x , y) =
∑(x ,y)∈D
[vy − c(b(fθ(x)))] · w(b(fθ(x)))
=∑
(x ,y)∈D
[vy −
∫ b(fθ(x))0 z · pz(z)dz∫ b(fθ(x))
0 pz(z)dz
]·∫ b(fθ(x))
0pz(z)dz
=∑
(x ,y)∈D
∫ b(fθ(x))
0(vy − z) · pz(z)dz . (5)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 28 / 98
Research Problems Optimization for Campaign Performance
Optimal Parameter
Taking Eq. (5) into Eq. (3) with a regularization term turns our learningproblem into convex optimization:
θEU = arg minθ−
∑(x ,y)∈D
REUθ (x , y) +
λ
2‖θ‖2
2 (6)
= arg minθ
∑x
∫ b(fθ(x))
0(z − vy) · pz(z)dz +
λ
2θTθ.
where the optimal value of θ is obtained by taking a gradient descentalgorithm.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 29 / 98
Research Problems Optimization for Campaign Performance
Gradient of EU
The gradient of REUθ (x , y) with regard to θ is calculated as
∂REUθ (x , y)
∂θ= (
bid error︷ ︸︸ ︷b(fθ(x))− vy) ·
market sensitivity︷ ︸︸ ︷pz(b(fθ(x))) ·
∂b(fθ(x))
∂fθ(x)
∂fθ(x)
∂θ+ λθ.
(7)
and we update for each data instance as θ ← θ − η ∂REUθ (x ,y)∂θ by above
chain rule. (SGD)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 30 / 98
Research Problems Optimization for Campaign Performance
Illustration of EU Update
0 50 100 150 200 250 300Bid Price
300
200
100
0
100
200
300
Bid
Err
or
Negative Response
Positive Response
0 50 100 150 200 250 300Bid Price
0.000
0.002
0.004
0.006
0.008
0.010
0.012
0.014
0.016
0.018
Mark
et
Sensi
tivit
y
5
4
3
2
1
0
1
Bid
Err
or ×
Mark
et
Sensi
tivit
y
Market Sensitivity
Negative Response
Positive Response
Figure: The illustration of the impact from the bid and market price of ExpectedUtility (EU); click value v = 300.
Gradient of EU
∂REUθ (x , y)
∂θ= (
bid error︷ ︸︸ ︷b(fθ(x))− vy) ·
market sensitivity︷ ︸︸ ︷pz(b(fθ(x))) ·∂b(fθ(x))
∂fθ(x)
∂fθ(x)
∂θ+ λθ.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 31 / 98
Research Problems Realization of Model
Question
Gradient of EU
∂REUθ (x , y)
∂θ= (
bid error︷ ︸︸ ︷b(fθ(x))− vy) ·
market sensitivity︷ ︸︸ ︷pz(b(fθ(x))) ·∂b(fθ(x))
∂fθ(x)
∂fθ(x)
∂θ+ λθ.
Gradient of RR
∂RRRθ (x , y)
∂θ=( bid error︷ ︸︸ ︷− vy
b(fθ(x))+
v(1− y)
v − b(fθ(x))
)·
market sensitivity︷ ︸︸ ︷pz(b(fθ(x)))
·∂b(fθ(x))
∂fθ(x)
∂fθ(x)
∂θ+ λθ.
fθ(x)? b(fθ(x))? pz(z)?
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 32 / 98
Research Problems Realization of Model
Question
Gradient of EU
∂REUθ (x , y)
∂θ= (
bid error︷ ︸︸ ︷b(fθ(x))− vy) ·
market sensitivity︷ ︸︸ ︷pz(b(fθ(x))) ·∂b(fθ(x))
∂fθ(x)
∂fθ(x)
∂θ+ λθ.
Gradient of RR
∂RRRθ (x , y)
∂θ=( bid error︷ ︸︸ ︷− vy
b(fθ(x))+
v(1− y)
v − b(fθ(x))
)·
market sensitivity︷ ︸︸ ︷pz(b(fθ(x)))
·∂b(fθ(x))
∂fθ(x)
∂fθ(x)
∂θ+ λθ.
fθ(x)? b(fθ(x))? pz(z)?
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 32 / 98
Research Problems Realization of Model
Question
Gradient of EU
∂REUθ (x , y)
∂θ= (
bid error︷ ︸︸ ︷b(fθ(x))− vy) ·
market sensitivity︷ ︸︸ ︷pz(b(fθ(x))) ·∂b(fθ(x))
∂fθ(x)
∂fθ(x)
∂θ+ λθ.
Gradient of RR
∂RRRθ (x , y)
∂θ=( bid error︷ ︸︸ ︷− vy
b(fθ(x))+
v(1− y)
v − b(fθ(x))
)·
market sensitivity︷ ︸︸ ︷pz(b(fθ(x)))
·∂b(fθ(x))
∂fθ(x)
∂fθ(x)
∂θ+ λθ.
fθ(x)? b(fθ(x))? pz(z)?
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 32 / 98
Research Problems Realization of Model
Realization of Model
Response Prediction Model fθ(x)
fθ(x) ≡ σ(θTx) =1
1 + e−θT x. (8)
Linear Bidding Strategy
b(fθ(x)) ≡ φ · v · fθ(x), (9)
where φ is the scaling parameter.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 33 / 98
Research Problems Realization of Model
Linear Gradient
Linear Gradient of EU
∂REUθ (x , y)
∂θ=φv2(σ(θTx)− y) · pz(b(fθ(x))) · (10)
σ(θTx)(1− σ(θTx))x + λθ.
Linear Gradient of RR
∂RRRθ (x , y)
∂θ=φv
(− y
σ(θTx)+
1− y
1− σ(θTx)
)· (11)
pz(b(fθ(x))) · σ(θTx)(1− σ(θTx))x + λθ.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 34 / 98
Adopting Truthful Bidding and uniform market price distribution,EU & RR have totally degenerated to SE & CE!
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 37 / 98
Research Problems Links to Previous Work
Summary of the Discussion
Table: The comparison of the model gradients (without regularization). LR:logistic regression, TB: truthful bidding, LB: linear bidding, UM: uniform marketprice distribution. LR and LR+TB+UM are equivalent (LR+TB reduces to thebaseline LR when assuming the uniform market price distribution).
2 Research ProblemsUser Response PredictionBidding Strategy OptimizationReinforcement Learning for AdvertisingConversion AttributionBid Landscape Forecasting
3 Related Literatures
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 50 / 98
Research Problems Bidding Strategy Optimization
Decision Optimization: the Bidding Function
Problem Definition
Propose the optimal bidding function b(x) to maximize the overall gains(clicks, conversions or profits), under the constraints of budget B.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 51 / 98
Research Problems Bidding Strategy Optimization
Related Work (cont.)Traditional Bidding Function
Truthful Bidding Function
b(x) = Vaction · f (x),
where f is the utility estimation function, such as pCTR.
Linear Bidding Function
b(x) = φ · Vaction · f (x) = b0 · f (x).C. Perlich et al. Bid optimizing and inventory scoring in targeted online advertising. KDD 2012.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 52 / 98
Research Problems Bidding Strategy Optimization
Related Work (cont.)Non-linear Bidding Function
Optimal Real-time Bidding Strategy
b()ORTB = arg maxb()
∫x
clicks dx
subject to
∫x
expected costs dx ≤ B .
⇒
b()ORTB = arg maxb()
∫xf (x)w(b(f (x)))px(x)dx ,
s.t.
∫xb(f (x))w(b(f (x)))px(x)dx ≤ B.
W. Zhang et al. Optimal Real-Time Bidding for Display Advertising. KDD 2014.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 53 / 98
Research Problems Bidding Strategy Optimization
Related Work (cont.)Non-linear Bidding Function
θ is the pCTR function, w(·) is the winning probability estimationfunction, b(·) is the bidding funciton.
W. Zhang et al. Optimal Real-Time Bidding for Display Advertising. KDD 2014.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 54 / 98
Research Problems Bidding Strategy Optimization
Problems of the Related Work
Naive assumption for the bidding function.
Zhang’s paper only considers first-price auction, which is notappropriate in practice.
Our Solution
Unified learning objective of the overall profits for utility estimation, costestimation and bidding strategy optimization.
K. Ren et al. Bidding Machine: Learning to Bid for Directly Optimizing Profits in Display Advertising. TKDE 2018
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 55 / 98
Research Problems Bidding Strategy Optimization
Bidding Machine - Joint Optimization Framework
(x , y): feature and label
v : value of click (constant)
fθ: utility (CTR) estimation function
b: bidding function (strategy)
wφ: winning probability
c : expected cost
The expected profit formulation is
R(b,θ,φ) =
∫x
[vy − c]wφ · px(x)dx
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 56 / 98
Research Problems Bidding Strategy Optimization
Bidding Machine - Joint Optimization Framework
wφ(b|x) =
∫ b
0pz(z |x ;φ)dz ,
c(b) =
∫ b0 zpz(z)dz∫ b0 pz(z)dz
,
R(b,θ,φ) =
∫x
[vy − c(b(fθ(x)))]wφ(b(fθ(x)))px(x)dx
=∑
(x ,y)∈D
[vy − c(b(fθ(x)))]wφ(b(fθ(x))).
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 57 / 98
Research Problems Bidding Strategy Optimization
Derivation Result 1Some interesting findings
Table: The comparison of the model gradients (without regularization). LR:logistic regression, TB: truthful bidding, LB: linear bidding, UM: uniform marketprice distribution. LR and LR+TB+UM are equivalent (LR+TB reduces to thebaseline LR when assuming the uniform market price distribution).
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 58 / 98
Research Problems Bidding Strategy Optimization
Derivation Result 2Optimal Bidding Strategy for Profit Optimization under 2nd Price Auction withoutBudget Constraint
R(b,θ,φ) =
∫x
[vy − c(b(fθ(x)))]wφ(b(fθ(x)))px(x)dx
=∑
(x ,y)∈D
[vy − c(b(fθ(x)))]wφ(b(fθ(x))).
Theorem
We can theoretically prove that the optimal bidding function is the truthfulbidding for profit maximization under second-price auction.
b(x) = v · f (x) ,
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 59 / 98
Research Problems Bidding Strategy Optimization
Derivation Result 3Optimal Bidding Strategy for Profit Optimization under 2nd Price Auction
Theorem
The optimal bidding function under a symmetric game of repeatedauctions with budget constraints is linear to the estimated utility.
maxb()
T
∫r[u(r)− c(b(τ))]wb(b(τ))pr (r)dr ,
s.t. T
∫rc(b(τ))wb(b(τ))pr (r)dr = B ,
(22)
here we assume that the bidding is based on a signal τ related with theCTR r = f (x).Here v is the click value of the advertiser.We derive in the paper that
b(r) =vr
λ+ 1. (23)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 60 / 98
Research Problems Bidding Strategy Optimization
Derivation Result 4What if all the advertisers adopt the same bidding strategy?
Theorem
The bid price is monotonously increasing w.r.t. the number of theparticipating advertiser bidders, and the tragedy of the commons willoccur in the market.
b(r) =vr
λ+ 1⇒ Br
T∫r
∫ r0 t(n − 1)Fr (t)n−2 pr (t) dt pr (r)dr
. (24)
The profit of the platform will increase :)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 61 / 98
Research Problems Bidding Strategy Optimization
Derivation Result 4What if all the advertisers adopt the same bidding strategy?
Theorem
The bid price is monotonously increasing w.r.t. the number of theparticipating advertiser bidders, and the tragedy of the commons willoccur in the market.
b(r) =vr
λ+ 1⇒ Br
T∫r
∫ r0 t(n − 1)Fr (t)n−2 pr (t) dt pr (r)dr
. (24)
The profit of the platform will increase :)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 61 / 98
Research Problems Bidding Strategy Optimization
Derivation Result 4What if all the advertisers adopt the same bidding strategy?
Theorem
The bid price is monotonously increasing w.r.t. the number of theparticipating advertiser bidders, and the tragedy of the commons willoccur in the market.
b(r) =vr
λ+ 1⇒ Br
T∫r
∫ r0 t(n − 1)Fr (t)n−2 pr (t) dt pr (r)dr
. (24)
The profit of the platform will increase :)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 61 / 98
Research Problems Bidding Strategy Optimization
Bidding Machine - Joint Optimization Framework
User
Other Advertisers
0. Ad Request
Ad Exchanger
1. BidRequest
3. WinNotice
2. BidResponse
4. Ad Creative
5. UserResponse
z: Market Price
x: RequestFeature
Training Flow
LearningPerforming
Data Flow
Bidding Machine y: true user response
Bid Strategy
User Response Prediction
Bid Landscape Forecasting
Bid Optim
ization
MarketModeling
Utilit
y Es
timat
ion
yzx
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 62 / 98
Research Problems Bidding Strategy Optimization
Bidding MachineExp. Results
1 2 3 4 5 6 7 8 9Training Round
0.6
0.7
0.8
0.9
1.0
Valu
e
10
20
30
40
50
60
AN
LP V
alu
e
Learning of Bidding Machine (iPinYou 3476)
AUCProfit(£3:3e10)
ANLP
1 2 3 4 5 6 7 8 9Training Round
0.75
0.80
0.85
0.90
0.95
1.00
Valu
e
10
20
30
40
50
60
AN
LP V
alu
e
Learning of Bidding Machine (iPinYou 3358)
AUCProfit(£2:6e10)
ANLP
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 63 / 98
Research Problems Bidding Strategy Optimization
Bidding MachineOffline Results
0.0 0.1 0.2 0.3 0.4 0.5Budgets
1.2
1.4
1.6
1.8
2.0
Pro
fit
1e7 Profit with Budgets
CELINORTBPRUDBM(FULL)
0.0 0.1 0.2 0.3 0.4 0.5Budgets
5
10
15
20
25
RO
I
ROI with Budgets
CELINORTBPRUDBM(FULL)
0.0 0.1 0.2 0.3 0.4 0.5Budgets
20
40
60
80
100
eC
PC
eCPC with Budgets
CELINORTBPRUDBM(FULL)
0.0 0.1 0.2 0.3 0.4 0.5Budgets
0.002
0.004
0.006
0.008
0.010
0.012
0.014
0.016
0.018
CTR
CTR with Budgets
CELINORTBPRUDBM(FULL)
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 64 / 98
Research Problems Bidding Strategy Optimization
Bidding MachineOffline Results
Table: Campaign profit for Single CTR estimation and Binary Optimization withmarket modeling.
1458 2259 2261 2821 2997 3358 3386 3427 3476 Average
2 Research ProblemsUser Response PredictionBidding Strategy OptimizationReinforcement Learning for AdvertisingConversion AttributionBid Landscape Forecasting
3 Related Literatures
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 68 / 98
Research Problems Reinforcement Learning for Advertising
Bidding as Sequential Decision Making
Relationship between RTB & RL
Real-time Bidding is a sequence of decision making.
The goal is to maximize the cumulative rewards (clicks, etc.) of theadvertiser (bidder).
The constraint is the total budget of the advertiser.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 69 / 98
Research Problems Reinforcement Learning for Advertising
RL for Bidding
Bidding AgentEnvironment[s] left volume t
[s] bid request x
[a] bid a
[r] auction win, cost δ[r] user click r
[s] left budget b
[s] state [a] action [r] reward
Figure: Real-time Bidding as reinforcement learning.
MDP Formulation
state: (t, b, x) with remaining auction num. t, remaining budget b and thereceived auction feature x .
action: bid price.
reward: predicted CTR (as model-baed RL).
P = Pr((t − 1, b − δ, x t−1)|(t, b, x t), δ) which is the winning probability,where delta is the bid price.
1) H. Cai, K. Ren, et al. Real-Time Bidding by Reinforcement Learning in Display Advertising. WSDM 20172) Y. Song, K. Ren, et al. Volume Ranking and Sequential Selection Programmatic Display Advertising. CIKM 2017
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 70 / 98
Research Problems Reinforcement Learning for Advertising
Learned Value Function
b (×10 3)
2040
6080
100t (×10
2 )
010
2030
4050
D(t, b
)(×
10−5
)
0123456789
D(t, b)
b (×10 3)
2040
6080
100t (×10
2 )
010
2030
4050
V(t, b
)
0.00.20.40.60.81.01.21.41.61.8
V(t, b)
Figure: The learned value function over states.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 71 / 98
Research Problems Reinforcement Learning for Advertising
Online Results
Lin RLB0.0
0.5
1.0
1.5
2.0
2.5
3.0Bids (×106)
Lin RLB0.00.51.01.52.02.53.03.54.0
Impressions (×105)
Lin RLB0
50100150200250300350400
Total Clicks
Lin RLB0.000.020.040.060.080.100.120.14
CTR (%)
Lin RLB0.00.20.40.60.81.01.2
CPM (CNY)
Lin RLB0.00.20.40.60.81.01.2
eCPC (CNY)
(a) The overall results.
0 50 100 150 200 250
Episodes
0
50
100
150
200
250
300
350
400
Tota
lClic
ks
LinRLB
0 50 100 150 200 250
Episodes
0
50
100
150
200
250
300
350
Cos
t(C
NY
)
LinRLB
(b) Performance over time.
Figure: The online results on VLion ad platform.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 72 / 98
2 Research ProblemsUser Response PredictionBidding Strategy OptimizationReinforcement Learning for AdvertisingConversion AttributionBid Landscape Forecasting
3 Related Literatures
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 73 / 98
Research Problems Conversion Attribution
Conversion AttributionProblem Definition
Two view of the problem
Horizontal View: Given a sequence of user activities leading to a conversion,assign the attribution credits to each touch point for the (negative) contributionon the final conversion.Vertical View: Calculate the conversion attribution over difference channels orsubcampaigns.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 74 / 98
Research Problems Conversion Attribution
Problem Challenge: Multi-touch Conversion Attribution
Cons of the traditional methods
Prediction upon single point: ignore the sequential data patterns in modeltraining.
Rule-based method: heuristically assign the conversion credits on to themultiple touches.
Search Social Website
No Conversion
User 1
Search Website Search
No Conversion
User 3
SocialSearch Search Website
Conversion
User 2
Impression
Click
Our SolutionUse recurrent neural network to model the sequential user activities.
Assign “attention” to the touch points to model the conversion attributions.
Simultaneously model impression-level and click-level patterns for conversion estimation.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 75 / 98
Research Problems Conversion Attribution
Dual-attention Mechanism for Conversion Attribution
x1
x2
xm -1
x
i
m i
1
2
m -1
z
i
m i
h1
h2
h0
m -1ih
m ih
s0
s1
s2
m -1is
m isxm i
ci2v cc2v
Encoder Decoderr
y
Ai2v Ac2v
z
z
z
g
g
g
g
fe
fe
fe
fe
fd
fd
f d
fd
i
ImpressionFeatures
PredictedClickRates
Dual-Attention
Predicted Conversion Rate for ith sequence
m ih
m ih
m ih
m ih
K. Ren et al. Learning Conversion Attribution with Dual-attention Mechanism for Online Advertising. CIKM, 2018.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 76 / 98
Research Problems Conversion Attribution
Attention Implementation
x
hxm i
ah
Softmax
Ee
j
j
j
j
(1, .., j, .., m )i
j
1h1 a h
c
m i m i
Attention
fhj-1
a
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 77 / 98
Research Problems Conversion Attribution
Visualization of the AttributionHorizontal Sequence Level
0 2 4 6 8touch point index
0.00
0.05
0.10
0.15
0.20
0.25
0.30
atttr
ibut
ion
Attribution on sequence length = 10AHAMTAARNNDARNN
0 1 2 3 4touch point index
0.0
0.1
0.2
0.3
0.4
0.5
atttr
ibut
ion
Attribution on sequence length = 5AHAMTAARNNDARNN
Figure: Touch point level attribution statistics (Miaozhen).
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 78 / 98
Research Problems Conversion Attribution
Visualization of the AttributionVertical Channel Level
Figure: Attribution of different channels on Miaozhen.Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 79 / 98
Research Problems Conversion Attribution
Visualization of the Attribution PreferencesClick-level v.s. Impression-level
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9λ
0.000
0.005
0.010
0.015
0.020
0.025
0.030
ratio
The distribution of λ
Figure: The distribution of λ over Criteo dataset.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 80 / 98
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 91 / 98
Research Problems Bid Landscape Forecasting
Related Work: Censorship Handling with Mixture Model
zi = [Pr(zi < bi )βW + (1− Pr(zi < bi ))βL]T xi
= βTmixxi ,
Pr(zi < bi ) = p(x) .
(26)
W. Wu et al. Predicting Winning Price in Real Time Bidding with Censored Data. KDD 2015
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 92 / 98
Research Problems Bid Landscape Forecasting
Related WorkDeepHit Model for Survival Analysis
C. Lee et al. DeepHit: A Deep Learning Approach to Survival Analysis with Competing Risks. AAAI 2018
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 93 / 98
Research Problems Bid Landscape Forecasting
Problems in the Related Work
Heuristic assumption on the distribution.
Naive ensorship handling
Mixture model: combines probability and cumulative probabilityin a simple way.Tree-based model: uses counting-based statistics for censorshiphandling.
DeepHit model: Sparse gradient signals, without consideration ofsequential patterns along time.
Our Solution (Under Review)
Using deep recurrent neural network to model the event rate at eachtimestep (price).
Using maximum partial likelihood for censorship handling.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 94 / 98
Research Problems Bid Landscape Forecasting
Deep Survival Analysis
We utilize recurrent neural network to model sequential patterns inthe time series space.
We also adopt partial likelihood for censorship handling.
The model achieves state-of-the-art performance.
It can also inspire the survival analysis in other fields such as clinicalresearch.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 95 / 98
Related Literatures
Related Literatures
Ren K, Fang Y, Zhang W, et al. Learning Multi-touch Conversion Attribution withDual-attention Mechanisms for Online Advertising[C]//Proceedings of the 27th ACMInternational on Conference on Information and Knowledge Management.
Ren K, Zhang W, Chang K, et al. Bidding Machine: Learning to Bid for DirectlyOptimizing Profits in Display Advertising[J]. IEEE Transactions on Knowledge and DataEngineering, 2018.
Ren K, Zhang W, Rong Y, et al. User response learning for directly optimizing campaignperformance in display advertising[C]//Proceedings of the 25th ACM International onConference on Information and Knowledge Management. ACM, 2016: 679-688.
Song Y, Ren K, Cai H, et al. Volume Ranking and Sequential Selection in ProgrammaticDisplay Advertising[J]. 2017.
Qu Y, Cai H, Ren K, et al. Product-based neural networks for user response prediction[J].2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, 2016, pp.1149-1154.
Wang Y, Ren K, Zhang W, et al. Functional bid landscape forecasting for displayadvertising[C]//Joint European Conference on Machine Learning and KnowledgeDiscovery in Databases. Springer International Publishing, 2016: 115-131.
Cai H, Ren K, Zhang W, et al. Real-Time Bidding by Reinforcement Learning in DisplayAdvertising[C]//10th ACM International Conference on Web Search and Data Mining(WSDM), 2017.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 96 / 98
Related Literatures
Other and Working Paper
Ren K, et al. Deep Survival Analysis. Working paper.
Lantao Y*, Xuejian W* (Equal Contribution), Ren K, et al. A Dynamic Attention DeepModel for Article Recommendation by Learning Human Editors’ Demonstration[C]//Proceedings of the 23th ACM SIGKDD International Conference on Knowledge Discoveryand Data Mining. ACM, 2017.
Zhou Z, Cai H, Rong S, Song Y, Ren K, Zhang W, Wang J, Yu Y. ActivationMaximization Generative Adversarial Nets[J]. 2018.
Zhu C, Ren K, Liu X, et al. A Graph Traversal Based Approach to AnswerNon-Aggregation Questions Over DBpedia[C]//Joint International Semantic TechnologyConference. Springer, Cham, 2015: 219-234.
Kan Ren (Shanghai Jiao Tong University) Modeling and Decision Optimization in Real-time Bidding Display AdvertisingAug. 2018 97 / 98