Big Data Analytics: Optimization and Randomization Tianbao Yang † , Qihang Lin , Rong Jin * ‡ Tutorial@SIGKDD 2015 Sydney, Australia † Department of Computer Science, The University of Iowa, IA, USA Department of Management Sciences, The University of Iowa, IA, USA * Department of Computer Science and Engineering, Michigan State University, MI, USA ‡ Institute of Data Science and Technologies at Alibaba Group, Seattle, USA August 10, 2015 Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 1 / 234
315
Embed
Big Data Analytics: Optimization and Randomizationhomepage.cs.uiowa.edu/~tyng/kdd15-tutorial.pdf · Big Data Analytics: Optimization and Randomization Tianbao Yang†, Qihang Lin\,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Big Data Analytics: Optimizationand Randomization
Tianbao Yang†, Qihang Lin\, Rong Jin∗‡
Tutorial@SIGKDD 2015Sydney, Australia
†Department of Computer Science, The University of Iowa, IA, USA\Department of Management Sciences, The University of Iowa, IA, USA
∗Department of Computer Science and Engineering, Michigan State University, MI, USA‡Institute of Data Science and Technologies at Alibaba Group, Seattle, USA
August 10, 2015
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 1 / 234
URL
http://www.cs.uiowa.edu/˜tyng/kdd15-tutorial.pdf
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 2 / 234
NoThis tutorial is not an exhaustive literature surveyIt is not a survey on different machine learning/data miningalgorithms
YesIt is about how to efficiently solve machine learning/data mining(formulated as optimization) problems for big data
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 3 / 234
Outline
Part I: BasicsPart II: OptimizationPart III: Randomization
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 4 / 234
Big Data Analytics: Optimization and Randomization
Part I: Basics
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 5 / 234
Basics Introduction
Outline
1 BasicsIntroductionNotations and Definitions
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 6 / 234
Basics Introduction
Three Steps for Machine Learning
Model Optimization
20 40 60 80 1000
0.05
0.1
0.15
0.2
0.25
0.3
iterations
dist
ance
to o
ptim
al o
bjec
tive
0.5T
1/T2
1/T
Data
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 7 / 234
Basics Introduction
Big Data Challenge
Big Data
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 8 / 234
Basics Introduction
Big Data Challenge
Big Model
60 million parameters
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 9 / 234
Basics Introduction
Learning as Optimization
Ridge Regression Problem:
minw∈Rd
1n
n∑i=1
(yi −w>xi )2 +
λ
2 ‖w‖22
xi ∈ Rd : d-dimensional feature vectoryi ∈ R: target variablew ∈ Rd : model parametersn: number of data points
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 10 / 234
Basics Introduction
Learning as Optimization
Ridge Regression Problem:
minw∈Rd
1n
n∑i=1
(yi −w>xi )2
︸ ︷︷ ︸Empirical Loss
+λ
2 ‖w‖22
xi ∈ Rd : d-dimensional feature vectoryi ∈ R: target variablew ∈ Rd : model parametersn: number of data points
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 11 / 234
Basics Introduction
Learning as Optimization
Ridge Regression Problem:
minw∈Rd
1n
n∑i=1
(yi −w>xi )2 +
λ
2 ‖w‖22︸ ︷︷ ︸
Regularization
xi ∈ Rd : d-dimensional feature vectoryi ∈ R: target variablew ∈ Rd : model parametersn: number of data points
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 12 / 234
Basics Introduction
Learning as Optimization
Classification Problems:
minw∈Rd
1n
n∑i=1
`(yiw>xi ) +λ
2 ‖w‖22
yi ∈ +1,−1: labelLoss function `(z): z = yw>x
1. SVMs: (squared) hinge loss `(z) = max(0, 1− z)p, where p = 1, 2
2. Logistic Regression: `(z) = log(1 + exp(−z))
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 13 / 234
Basics Introduction
Learning as Optimization
Feature Selection:
minw∈Rd
1n
n∑i=1
`(w>xi , yi ) + λ‖w‖1
`1 regularization ‖w‖1 =∑d
i=1 |wi |λ controls sparsity level
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 14 / 234
Basics Introduction
Learning as Optimization
Feature Selection using Elastic Net:
minw∈Rd
1n
n∑i=1
`(w>xi , yi )+λ(‖w‖1 + γ‖w‖2
2
)
Elastic net regularizer, more robust than `1 regularizer
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 15 / 234
Basics Introduction
Learning as Optimization
Multi-class/Multi-task Learning:
minW
1n
n∑i=1
`(Wxi , yi ) + λr(W)
W ∈ RK×d
r(W) = ‖W‖2F =
∑Kk=1
∑dj=1 W 2
kj : Frobenius Normr(W) = ‖W‖∗ =
∑i σi : Nuclear Norm (sum of singular values)
r(W) = ‖W‖1,∞ =∑d
j=1 ‖W:j‖∞: `1,∞mixed norm
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 16 / 234
Basics Introduction
Learning as Optimization
Regularized Empirical Loss Minimization
minw∈Rd
1n
n∑i=1
`(w>xi , yi ) + R(w)
Both ` and R are convex functionsExtensions to Matrix Cases are possible (sometimes straightforward)Extensions to Kernel methods can be combined with randomizedapproachesExtensions to Non-convex (e.g., deep learning) are in progress
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 17 / 234
Basics Introduction
Data Matrices and Machine Learning
The Instance-feature Matrix: X ∈ Rn×d
X =
x>1x>2···
x>n
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 18 / 234
Basics Introduction
Data Matrices and Machine Learning
The output vector: y =
y1y2···
yn
∈ Rn×1
continuous yi ∈ R: regression (e.g., house price)discrete, e.g., yi ∈ 1, 2, 3: classification (e.g., species of iris)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 19 / 234
Basics Introduction
Data Matrices and Machine LearningThe Instance-Instance Matrix: K ∈ Rn×n
Similarity MatrixKernel Matrix
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 20 / 234
Basics Introduction
Data Matrices and Machine LearningSome machine learning tasks are formulated on the kernel matrix
ClusteringKernel Methods
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 21 / 234
Basics Introduction
Data Matrices and Machine Learning
The Feature-Feature Matrix: C ∈ Rd×d
Covariance MatrixDistance Metric Matrix
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 22 / 234
Basics Introduction
Data Matrices and Machine Learning
Some machine learning tasks requires the covariance matrixPrincipal Component AnalysisTop-k Singular Value (Eigen-Value) Decomposition of the CovarianceMatrix
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 23 / 234
Basics Introduction
Why Learning from Big Data is Challenging?
High per-iteration cost
High memory cost
High communication cost
Large iteration complexity
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 24 / 234
Basics Notations and Definitions
Outline
1 BasicsIntroductionNotations and Definitions
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 25 / 234
Basics Notations and Definitions
Norms
Vector x ∈ Rd
Euclidean vector norm: ‖x‖2 =√
x>x =√∑d
i=1 x2i
`p-norm of a vector: ‖x‖p =(∑d
i=1 |xi |p)1/p
where p ≥ 1
1 `2 norm ‖x‖2 =√∑d
i=1 x2i
2 `1 norm ‖x‖1 =∑d
i=1 |xi |3 `∞ norm ‖x‖∞ = maxi |xi |
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 26 / 234
Basics Notations and Definitions
Norms
Vector x ∈ Rd
Euclidean vector norm: ‖x‖2 =√
x>x =√∑d
i=1 x2i
`p-norm of a vector: ‖x‖p =(∑d
i=1 |xi |p)1/p
where p ≥ 1
1 `2 norm ‖x‖2 =√∑d
i=1 x2i
2 `1 norm ‖x‖1 =∑d
i=1 |xi |3 `∞ norm ‖x‖∞ = maxi |xi |
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 26 / 234
Basics Notations and Definitions
Norms
Vector x ∈ Rd
Euclidean vector norm: ‖x‖2 =√
x>x =√∑d
i=1 x2i
`p-norm of a vector: ‖x‖p =(∑d
i=1 |xi |p)1/p
where p ≥ 1
1 `2 norm ‖x‖2 =√∑d
i=1 x2i
2 `1 norm ‖x‖1 =∑d
i=1 |xi |3 `∞ norm ‖x‖∞ = maxi |xi |
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 26 / 234
Basics Notations and Definitions
Matrix Factorization
Matrix X ∈ Rn×d
Singular Value Decomposition X = UΣV>1 U ∈ Rn×r : orthonormal columns (U>U = I): span column space2 Σ ∈ Rr×r : diagonal matrix Σii = σi > 0, σ1 ≥ σ2 . . . ≥ σr3 V ∈ Rd×r : orthonormal columns (V>V = I): span row space4 r ≤ min(n, d): max value such that σr > 0: rank of X5 UkΣkV>k : top-k approximation
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 45 / 234
Optimization (Sub)Gradient Methods
Gradient Method in Machine Learning
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
Suppose `(z) is smoothFull gradient: ∇F (w) = 1
n∑n
i=1∇`(w>xi , yi ) + λwPer-iteration cost: O(nd)
Gradient Descent
wt = wt−1 − γt∇F (wt−1)
step size
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 46 / 234
Optimization (Sub)Gradient Methods
Gradient Method in Machine Learning
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
Suppose `(z) is smoothFull gradient: ∇F (w) = 1
n∑n
i=1∇`(w>xi , yi ) + λwPer-iteration cost: O(nd)
Gradient Descent
wt = wt−1 − γt∇F (wt−1)
step size
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 46 / 234
Optimization (Sub)Gradient Methods
Gradient Method in Machine Learning
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
If λ = 0: R(w) is non-strongly convexIteration complexity O( 1
ε )
If λ > 0: R(w) is λ-strongly convexIteration complexity O( 1
λ log( 1ε ))
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 47 / 234
Optimization (Sub)Gradient Methods
Accelerated Gradient Method
Accelerated Gradient Descent
wt = vt−1 − γt∇F (vt−1)
vt = wt + ηt(wt −wt−1)
MomentumStep
wt is the output and vt is an auxiliary sequence.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 48 / 234
Optimization (Sub)Gradient Methods
Accelerated Gradient Method
Accelerated Gradient Descent
wt = vt−1 − γt∇F (vt−1)
vt = wt + ηt(wt −wt−1)
MomentumStep
wt is the output and vt is an auxiliary sequence.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 48 / 234
Optimization (Sub)Gradient Methods
Accelerated Gradient Method
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
If λ = 0: R(w) is non-strongly convexIteration complexity O( 1√
ε), better than O( 1
ε )
If λ > 0: R(w) is λ-strongly convexIteration complexity O( 1√
λlog( 1
ε )), better than O( 1λ log( 1
ε )) for smallλ
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 49 / 234
Optimization (Sub)Gradient Methods
Deal with `1 regularizer
Consider a more general case
minw∈Rd
F (w) =1n
n∑i=1
`(w>xi , yi ) + R ′(w) + τ‖w‖1︸ ︷︷ ︸R(w)
R(w) = R ′(w) + τ‖w‖1
R ′(w): λ-strongly convex and smooth
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 50 / 234
Optimization (Sub)Gradient Methods
Deal with `1 regularizer
Consider a more general case
minw∈Rd
F (w) =1n
n∑i=1
`(w>xi , yi ) + R ′(w)︸ ︷︷ ︸F ′(w)
+τ‖w‖1
R(w) = R ′(w) + τ‖w‖1
R ′(w): λ-strongly convex and smooth
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 51 / 234
Optimization (Sub)Gradient Methods
Deal with `1 regularizer
Accelerated Gradient Descent
wt = arg minw∈Rd
∇F ′(vt−1)>w +
12γt‖w− vt−1‖2
2 + τ‖w‖1
vt = wt + ηt(wt −wt−1)
Proximalmapping
Proximal mapping has close-form solution: Soft-thresholdingIteration complexity and runtime remain unchanged.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 52 / 234
Optimization (Sub)Gradient Methods
Deal with `1 regularizer
Accelerated Gradient Descent
wt = arg minw∈Rd
∇F ′(vt−1)>w +
12γt‖w− vt−1‖2
2 + τ‖w‖1
vt = wt + ηt(wt −wt−1)
Proximalmapping
Proximal mapping has close-form solution: Soft-thresholdingIteration complexity and runtime remain unchanged.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 52 / 234
Optimization (Sub)Gradient Methods
Sub-Gradient Method in Machine Learning
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
Suppose `(z) is non-smoothFull sub-gradient: ∂F (w) = 1
n∑n
i=1 ∂`(w>xi , yi ) + λw
Sub-Gradient Descent
wt = wt−1 − γt∂F (wt−1)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 53 / 234
Optimization (Sub)Gradient Methods
Sub-Gradient Method in Machine Learning
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
Suppose `(z) is non-smoothFull sub-gradient: ∂F (w) = 1
n∑n
i=1 ∂`(w>xi , yi ) + λw
Sub-Gradient Descent
wt = wt−1 − γt∂F (wt−1)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 53 / 234
Optimization (Sub)Gradient Methods
Sub-Gradient Method
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
If λ = 0: R(w) is non-strongly convexIteration complexity O( 1
ε2 )
If λ > 0: R(w) is λ-strongly convexIteration complexity O( 1
λε)
No efficient acceleration scheme in general
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 54 / 234
Optimization (Sub)Gradient Methods
Problem Classes and Iteration Complexity
minw∈Rd
1n
n∑i=1
`(w>xi , yi ) + R(w)
Iteration complexity`(z) ≡ `(z , y)
Non-smooth Smooth
R(w)Non-strongly convex O
(1ε2
)O(
1√ε
)λ-strongly convex O
(1λε
)O(
1√λ
log(
1ε
))Per-iteration cost: O(nd), too high if n or d are large.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 55 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Outline
2 Optimization(Sub)Gradient MethodsStochastic Optimization Algorithms for Big Data
Stochastic OptimizationDistributed Optimization
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 56 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Stochastic First-Order Method by Data Sampling
Stochastic Gradient Descent (SGD)
Stochastic Variance Reduced Gradient (SVRG)
Stochastic Average Gradient Algorithm (SAGA)
Stochastic Dual Coordinate Ascent (SDCA)
Accelerated Proximal Coordinate Gradient (APCG)
Assumption: ‖xi‖ ≤ 1 for any i
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 57 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Basic SGD (Nemirovski & Yudin (1978))
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
Full sub-gradient: ∂F (w) = 1n∑n
i=1 ∂`(w>xi , yi ) + λw
Randomly sample i ∈ 1, . . . , nStochastic sub-gradient: ∂`(wT xi , yi ) + λw
Ei [∂`(wT xi , yi ) + λw] = ∂F (w)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 58 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Basic SGD (Nemirovski & Yudin (1978))
Applicable in all settings!
minw∈Rd
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
sample: it ∈ 1, . . . , n
update: wt = wt−1 − γt(∂`(wT
t−1xit , yit ) + λwt−1)
output: wT =1T
T∑t=1
wt
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 59 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Basic SGD (Nemirovski & Yudin (1978))
Applicable in all settings!
minw∈Rd
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
sample: it ∈ 1, . . . , n
update: wt = wt−1 − γt(∂`(wT
t−1xit , yit ) + λwt−1)
output: wT =1T
T∑t=1
wt
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 59 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Basic SGD (Nemirovski & Yudin (1978))
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22
If λ = 0: R(w) is non-strongly convexIteration complexity O( 1
ε2 )
If λ > 0: R(w) is λ-strongly convexIteration complexity O( 1
λε)
Exactly the same as sub-gradient descent!
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 60 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Total Runtime
Per-iteration cost: O(d)
Much lower than full gradient methode.g. hinge loss (SVM)
stochastic gradient: ∂`(w>xit , yit ) =
−yit xit , 1− yit w>xit > 0
0, otherwise
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 61 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Total Runtime
minw∈Rd
1n
n∑i=1
`(w>xi , yi ) + R(w)
Iteration complexity`(z) ≡ `(z , y)
Non-smooth Smooth
R(w)Non-strongly convex O
(1ε2
)O(
1ε2
)λ-strongly convex O
(1λε
)O(
1λε
)For SGD, only strongly convexity helps but the smoothness does notmake any difference!
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 62 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Full Gradient V.S. Stochastic Gradient
Full gradient method needs fewer iterationsStochastic gradient method has lower cost per iteration
For small ε, use full gradientSatisfied with large ε, use stochastic gradient
Full gradient can be acceleratedStochastic gradient cannot
Full gradient’s iterations complexity depends on smoothness andstrong convexityStochastic gradient’s iteration complexity only depends on strongconvexity
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 63 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 71 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SAGA: efficient update of averaged gradient
Gt and Gt−1 only differs in gi for i = itBefore we update gi , we update
Gt =1n
n∑i=1
gi = Gt−1 −1n git +
1n(∇`(w>t−1xit , yit ) + λwt−1
)computation cost: O(d)
To implemente SAGA, we have to store and update all: g1, g2, . . . , gn
Require extra memory space O(nd)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 72 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SAGA: efficient update of averaged gradient
Gt and Gt−1 only differs in gi for i = itBefore we update gi , we update
Gt =1n
n∑i=1
gi = Gt−1 −1n git +
1n(∇`(w>t−1xit , yit ) + λwt−1
)computation cost: O(d)
To implemente SAGA, we have to store and update all: g1, g2, . . . , gn
Require extra memory space O(nd)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 72 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SAGA (Defazio et al. (2014))
Per-iteration cost: O(d)
Iteration complexity`(z) ≡ `(z , y)
Non-smooth Smooth
R(w)Non-strongly convex N.A. O
(nε
)λ-strongly convex N.A. O
((n + 1
λ
)log(
1ε
))Total Runtime (strongly convex): O
(d(
n + 1λ
)log(
1ε
)). Same as
SVRG!Use proximal mapping for `1 regularizer
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 73 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SAGA (Defazio et al. (2014))
Per-iteration cost: O(d)
Iteration complexity`(z) ≡ `(z , y)
Non-smooth Smooth
R(w)Non-strongly convex N.A. O
(nε
)λ-strongly convex N.A. O
((n + 1
λ
)log(
1ε
))Total Runtime (strongly convex): O
(d(
n + 1λ
)log(
1ε
)). Same as
SVRG!Use proximal mapping for `1 regularizer
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 73 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Compare the Runtime of SGD and SVRG/SAGA
Smooth but non-strongly convex:SGD: O
( dε2
)SAGA: O
( dnε
)Smooth and strongly convex:
SGD: O( dλε
)SVRG/SAGA: O
(d(n + 1
λ
)log( 1ε
))For small ε, use SVRG/SAGASatisfied with large ε, use SGD
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 74 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Conjugate Duality
Define `i (z) ≡ `(z , yi )
Conjugate function: `∗i (α)⇐⇒ `i (z)
`i (z) = maxα∈R
[αz − `∗(α)] , `∗i (α) = maxz∈R
[αz − `(z)]
E.g. hinge loss: `i (z) = max(0, 1− yi z)
`∗i (α) =
αyi if − 1 ≤ αyi ≤ 0+∞ otherwise
E.g. square hinge loss: `i (z) = max(0, 1− yi z)2
`∗i (α) =
α2
4 + αyi if αyi ≤ 0+∞ otherwise
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 75 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Conjugate Duality
Define `i (z) ≡ `(z , yi )
Conjugate function: `∗i (α)⇐⇒ `i (z)
`i (z) = maxα∈R
[αz − `∗(α)] , `∗i (α) = maxz∈R
[αz − `(z)]
E.g. hinge loss: `i (z) = max(0, 1− yi z)
`∗i (α) =
αyi if − 1 ≤ αyi ≤ 0+∞ otherwise
E.g. square hinge loss: `i (z) = max(0, 1− yi z)2
`∗i (α) =
α2
4 + αyi if αyi ≤ 0+∞ otherwise
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 75 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SDCA (Shalev-Shwartz & Zhang (2013))
Stochastic Dual Coordinate Ascent (liblinear (Hsieh et al., 2008))Applicable when R(w) is λ-strongly convexSmoothness is not requiredFrom Primal problem to Dual problem:
minw
1n
n∑i=1
`(w>xi︸ ︷︷ ︸z
, yi ) +λ
2 ‖w‖22
= minw
1n
n∑i=1
maxαi∈R
[αi (w>xi )− `∗i (αi )
]+λ
2 ‖w‖22
= maxα∈Rn
1n
n∑i=1−`∗i (αi )−
λ
2
∥∥∥∥∥ 1λn
n∑i=1
αixi
∥∥∥∥∥2
2
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 76 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SDCA (Shalev-Shwartz & Zhang (2013))
Stochastic Dual Coordinate Ascent (liblinear (Hsieh et al., 2008))Applicable when R(w) is λ-strongly convexSmoothness is not requiredFrom Primal problem to Dual problem:
minw
1n
n∑i=1
`(w>xi︸ ︷︷ ︸z
, yi ) +λ
2 ‖w‖22
= minw
1n
n∑i=1
maxαi∈R
[αi (w>xi )− `∗i (αi )
]+λ
2 ‖w‖22
= maxα∈Rn
1n
n∑i=1−`∗i (αi )−
λ
2
∥∥∥∥∥ 1λn
n∑i=1
αixi
∥∥∥∥∥2
2
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 76 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SDCA (Shalev-Shwartz & Zhang (2013))
Solve Dual Problem:
maxα∈Rn
1n
n∑i=1−`∗i (αi )−
λ
2
∥∥∥∥∥ 1λn
n∑i=1
αixi
∥∥∥∥∥2
2
Sample it ∈ 1, . . . , n. Optimize αit while fixing others
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 77 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SDCA (Shalev-Shwartz & Zhang (2013))
Maintain a primal solution: wt = 1λn∑n
i=1 αti xi
Change variable αi −→ ∆αi
max∆α∈Rn
1n
n∑i=1−`∗i (αt
i + ∆αi )−λ
2
∥∥∥∥∥ 1λn
( n∑i=1
αti xi +
n∑i=1
∆αixi
)∥∥∥∥∥2
2
⇐⇒ max∆α∈Rn
1n
n∑i=1−`∗i (αt
i + ∆αi )−λ
2
∥∥∥∥∥wt +1λn
n∑i=1
∆αixi
∥∥∥∥∥2
2
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 78 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SDCA (Shalev-Shwartz & Zhang (2013))
Dual Coordinate Updates
∆αit = max∆αit∈Rn
−1n `∗it (−αt
it −∆αit )− λ
2
∥∥∥∥wt +1λn ∆αit xit
∥∥∥∥2
2
αt+1it = αt
it + ∆αit
wt+1 = wt +1λn ∆αit xi
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 79 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SDCA updates
Close-form solution for ∆αi : hinge loss, squared hinge loss, absoluteloss and square loss (Shalev-Shwartz & Zhang (2013))e.g. square loss
∆αi =yi −w>t xi − αt
i1 + ‖xi‖2
2/(λn)
Per-iteration cost: O(d)
Approximate solution: logistic loss (Shalev-Shwartz & Zhang (2013))
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 80 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SDCA
Iteration complexity`(z) ≡ `(z , y)
Non-smooth Smooth
R(w)Non-strongly convex N.A. N.A.λ-strongly convex O
(n + 1
λε
)O((
n + 1λ
)log(
1ε
))Total Runtime (smooth loss): O
(d(
n + 1λ
)log(
1ε
)). The same as
SVRG and SAGA!
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 81 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SDCA
Iteration complexity`(z) ≡ `(z , y)
Non-smooth Smooth
R(w)Non-strongly convex N.A. N.A.λ-strongly convex O
(n + 1
λε
)O((
n + 1λ
)log(
1ε
))Total Runtime (smooth loss): O
(d(
n + 1λ
)log(
1ε
)). The same as
SVRG and SAGA!
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 81 / 234
Optimization Stochastic Optimization Algorithms for Big Data
SVRG V.S. SDCA V.S. SGD
`2-regularized logistic regression with λ = 10−4
MNIST dataJohnson & Zhang (2013)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 82 / 234
Optimization Stochastic Optimization Algorithms for Big Data
APCG (Lin et al. (2014))
Recall the acceleration scheme for full gradient methodAuxiliary sequence (βt)Momentum step
Maintain a primal solution: wt = 1λn∑n
i=1 βti xi
Dual Coordinate Updates
∆βit = max∆βit∈Rn
−1n `∗it (−βt
it −∆βit )− λ
2
∥∥∥∥wt +1λn ∆βit xit
∥∥∥∥2
2
αt+1it = βt
it + ∆βit
βt+1 = αt+1 + ηt(αt+1 − αt)
Momentum Step
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 83 / 234
Optimization Stochastic Optimization Algorithms for Big Data
APCG (Lin et al. (2014))
Recall the acceleration scheme for full gradient methodAuxiliary sequence (βt)Momentum step
Maintain a primal solution: wt = 1λn∑n
i=1 βti xi
Dual Coordinate Updates
∆βit = max∆βit∈Rn
−1n `∗it (−βt
it −∆βit )− λ
2
∥∥∥∥wt +1λn ∆βit xit
∥∥∥∥2
2
αt+1it = βt
it + ∆βit
βt+1 = αt+1 + ηt(αt+1 − αt)
Momentum Step
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 83 / 234
Optimization Stochastic Optimization Algorithms for Big Data
APCG (Lin et al. (2014))
Per-iteration cost: O(d)
Iteration complexity`(z) ≡ `(z , y)
Non-smooth Smooth
R(w)Non-strongly convex N.A. N.A.λ-strongly convex O
(n +
√nλε
)O((
n +√
nλ
)log(
1ε
))Compared to SDCA, APCG has shorter runtime when λ is very small.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 84 / 234
Optimization Stochastic Optimization Algorithms for Big Data
APCG V.S. SDCA
squared hinge loss SVMreal data
datasets number of samples n number of features d sparsityrcv1 20,242 47,236 0.16%
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 85 / 234
Optimization Stochastic Optimization Algorithms for Big Data
APCG V.S. SDCALin et al. (2014)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 86 / 234
Optimization Stochastic Optimization Algorithms for Big Data
For general R(w)
Dual Problem:
maxα∈Rn
1n
n∑i=1−`∗i (αi )− R∗
(1λn
n∑i=1
αixi
)
R∗ is the conjugate of RSample it ∈ 1, . . . , n. Optimize αit while fixing othersCan be still updated in O(d) in many cases (Shalev-Shwartz & Zhang(2013))Iteration complexity and runtime of SDCA and APCG remainunchanged.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 87 / 234
Optimization Stochastic Optimization Algorithms for Big Data
APCG for primal problem
minw∈Rd
F (w) =1n
n∑i=1
`(w>xi , yi ) +λ
2 ‖w‖22 + τ‖w‖1
Suppose d >> n. Per-iteration cost O(d) is too highApply APCG to the primal instead of dual problemSample over features instead of dataPer-iteration cost becomes O(n)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 88 / 234
Optimization Stochastic Optimization Algorithms for Big Data
APCG for primal problem
minw∈Rd
F (w) =12‖Xw− y‖2
2 +λ
2 ‖w‖22 + τ‖w‖1
X = [x1, x2, · · · , xn]
Full gradient: ∇F (w) = X T (Xw− y) + λwPartial gradient: ∇i F (w) = xT
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 114 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DiSCO (Zhang & Xiao, 2015)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 115 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DiSCO (Zhang & Xiao, 2015)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 116 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DiSCO (Zhang & Xiao, 2015)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 117 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DiSCO (Zhang & Xiao, 2015)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 118 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DiSCO (Zhang & Xiao, 2015)
For high-dimensional data (e.g. d ≥ 1000), computingP−1 ×∇2F (wt)× v is time costly.Instead, use SDCA in machine 1 to solve
P−1 ×∇2F (wt)× v ≈ arg minu∈Rd
12u>Pu− u>F (wt)v
Local runtime: O( Nm + 1+µ
λ+µ)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 119 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DiSCO (Zhang & Xiao, 2015)
Suppose SDCA is the local solverLocal Runtime Per-round: O( N
m + 1+µλ+µ)
Rounds of Communication: O(√
µλ log
(1ε
))
Choice of m (how many machines to use?). (µ = O(√
mN ))
Fast machines but slow network: Use small mFast network but slow machine: Use large m
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 120 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DiSCO (Zhang & Xiao, 2015)
Suppose SDCA is the local solverLocal Runtime Per-round: O( N
m + 1+µλ+µ)
Rounds of Communication: O(√
µλ log
(1ε
))
Choice of m (how many machines to use?). (µ = O(√
mN ))
Fast machines but slow network: Use small mFast network but slow machine: Use large m
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 120 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DSVRG (Lee et al., 2015)
A distributed version of SVRG using a “round-robin” schemeAssume the user can control the distribution of data before algorithm.Applicable when fj(w) is smooth and λ-strongly convex
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 121 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DSVRG (Lee et al., 2015)
Iterate s = 1, . . . ,T − 1Let w0 = ws and compute ∇F (ws)
Each machine can only sample from its own data. However,Eit∈Sj [git (wt−1)] 6= ∇F (wt−1)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 122 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DSVRG (Lee et al., 2015)
Solution:Store a second set of data Rj in machine j , which are sampled withreplacement from x1, x2, . . . , xn before the algorithm starts.Construct the stochastic gradient git (wt−1) by sampling it ∈ Rj andremoving it from Rj after.
Eit∈Rj [git (wt−1)] = ∇F (wt−1)
When Rj = ∅, pass wt to next machine.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 123 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Full Gradient Step
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 124 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Full Gradient Step
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 125 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Stochastic Gradient Step
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 126 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Stochastic Gradient Step
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 127 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Stochastic Gradient Step
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 128 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DSVRG (Lee et al., 2015)
Suppose |Rj | = r for all j .Local Runtime Per-round: O( N
m + r)
Rounds of Communication: O( 1rλ log
(1ε
))
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 129 / 234
Optimization Stochastic Optimization Algorithms for Big Data
DSVRG (Lee et al., 2015)
Choice of m (how many machines to use?).Fast machines but slow network: Use small mFast network but slow machine: Use large m
Choice of r (how many data points to pre-sample in Rj?).The larger, the betterRequired machine memory space: |Sj |+ |Rj | = N
m + r
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 130 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Other Distributed Optimization Methods
ADMM (Boyd et al., 2011; Ozdaglar, 2015)Rounds of Communication:O(Network Graph Dependency Term × 1√
λlog( 1
ε ))
DANE (Shamir et al., 2014)Approximate Newton direction with a difference approach from DISCO
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 131 / 234
Optimization Stochastic Optimization Algorithms for Big Data
Optimization Stochastic Optimization Algorithms for Big Data
Thank You! Questions?
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 137 / 234
Randomized Dimension Reduction
Big Data Analytics: Optimization and Randomization
Part III: Randomization
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 138 / 234
Randomized Dimension Reduction
Outline
1 Basics
2 Optimization
3 Randomized Dimension Reduction
4 Randomized Algorithms
5 Concluding Remarks
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 139 / 234
Randomized Dimension Reduction
Random Sketch
Approximate a large data matrix
by a much smaller sketch
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 140 / 234
Randomized Dimension Reduction
The Framework of Randomized Algorithms
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 141 / 234
Randomized Dimension Reduction
The Framework of Randomized Algorithms
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 142 / 234
Randomized Dimension Reduction
The Framework of Randomized Algorithms
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 143 / 234
Randomized Dimension Reduction
The Framework of Randomized Algorithms
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 144 / 234
Randomized Dimension Reduction
Why randomized dimension reduction?
Efficient
Robust (e.g., dropout)
Formal Guarantees
Can explore parallel algorithms
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 145 / 234
Randomized Dimension Reduction
Randomized Dimension Reduction
Johnson-Lindenstauss (JL) transforms
Subspace embeddings
Column sampling
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 146 / 234
Randomized Dimension Reduction
JL Lemma
JL Lemma (Johnson & Lindenstrauss, 1984)For any 0 < ε, δ < 1/2, there exists a probability distribution on m × dreal matrices A such that there exists a small universal constant c > 0 andfor any fixed x ∈ Rd with a probability at least 1− δ, we have∣∣∣‖Ax‖2
2 − ‖x‖22
∣∣∣ ≤ c
√log(1/δ)
m ‖x‖22
or for m = Θ(ε−2 log(1/δ)), then with a probability at least 1− δ∣∣∣‖Ax‖22 − ‖x‖2
2
∣∣∣ ≤ ε‖x‖22
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 147 / 234
Randomized Dimension Reduction
Embedding a set of points into low dimensional space
Given a set of points x1, . . . , xn ∈ Rd , we can embed them into a lowdimensional space Ax1, . . . ,Axn ∈ Rm such thatthe pairwise distance between any two points are well preserved in the lowdimensional space
‖Axi − Axj‖22 = ‖A(xi − xj)‖2
2 ≤ (1 + ε) ‖xi − xj‖22
‖Axi − Axj‖22 = ‖A(xi − xj)‖2
2 ≥ (1− ε) ‖xi − xj‖22
In other words, in order to have all pairwise Euclidean distances preservedup to 1± ε, only m = Θ(ε−2 log(n2/δ)) dimensions are necessary
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 148 / 234
Randomized Dimension Reduction
JL transforms: Gaussian Random Projection
Gaussian Random Projection (Dasgupta & Gupta, 2003): A ∈ Rm×d
Aij ∼ N (0, 1/m)
m = Θ(ε−2 log(1/δ))
Computational cost of AX : where X ∈ Rd×n
mnd for dense matricesnnz(X )m for sparse matrices
Computational Cost is very High (could be as high as solving manyproblems)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 149 / 234
Randomized Dimension Reduction
Accelerate JL transforms: using discrete distributions
Using Discrete Distributions (Achlioptas, 2003):Pr(Aij = ± 1√
m ) = 0.5
Pr(Aij = ±√
3m ) = 1
6 , Pr(Aij = 0) = 23
Database friendlyReplace multiplications by additions and subtractions
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 150 / 234
Randomized Dimension Reduction
Accelerate JL transforms: using Hadmard transform (I)
Fast JL transform based on randomized Hadmard transform:
Motivation: Can we simply use random sampling matrix P ∈ Rm×d thatrandomly selects m coordinates out of d coordinates (scaled by
√d/m)?
Unfortunately: by Chernoff bound
|‖Px‖22 − ‖x‖2
2| ≤√
d‖x‖∞‖x‖2
√3 log(2/δ)
m ‖x‖22
Unless√
d‖x‖∞‖x‖2
≤ c, the random sampling doest not work
Remedy is given by randomized Hadmard transform
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 151 / 234
Randomized Dimension Reduction
Accelerate JL transforms: using Hadmard transform (I)
Fast JL transform based on randomized Hadmard transform:
Motivation: Can we simply use random sampling matrix P ∈ Rm×d thatrandomly selects m coordinates out of d coordinates (scaled by
√d/m)?
Unfortunately: by Chernoff bound
|‖Px‖22 − ‖x‖2
2| ≤√
d‖x‖∞‖x‖2
√3 log(2/δ)
m ‖x‖22
Unless√
d‖x‖∞‖x‖2
≤ c, the random sampling doest not work
Remedy is given by randomized Hadmard transform
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 151 / 234
Randomized Dimension Reduction
Accelerate JL transforms: using Hadmard transform (I)
Fast JL transform based on randomized Hadmard transform:
Motivation: Can we simply use random sampling matrix P ∈ Rm×d thatrandomly selects m coordinates out of d coordinates (scaled by
√d/m)?
Unfortunately: by Chernoff bound
|‖Px‖22 − ‖x‖2
2| ≤√
d‖x‖∞‖x‖2
√3 log(2/δ)
m ‖x‖22
Unless√
d‖x‖∞‖x‖2
≤ c, the random sampling doest not work
Remedy is given by randomized Hadmard transform
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 151 / 234
Randomized Dimension Reduction
Randomized Hadmard transform
Hadmard transform:H ∈ Rd×d : H =
√1d H2k
H1 = [1] , H2 =
[1 11 −1
], H2k =
[H2k−1 H2k−1
H2k−1 −H2k−1
]
‖Hx‖2 = ‖x‖2 and H is orthogonalComputational costs of Hx : d log(d)
randomized Hadmard transform: HDD ∈ Rd×d : a diagonal matrix Pr(Dii = ±1) = 0.5HD is orthogonal and ‖HDx‖2 = ‖x‖2
Key property:√
d‖HDx‖∞‖HDx‖2
≤√
log(d/δ) w.h.p 1− δ
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 152 / 234
Randomized Dimension Reduction
Randomized Hadmard transform
Hadmard transform:H ∈ Rd×d : H =
√1d H2k
H1 = [1] , H2 =
[1 11 −1
], H2k =
[H2k−1 H2k−1
H2k−1 −H2k−1
]
‖Hx‖2 = ‖x‖2 and H is orthogonalComputational costs of Hx : d log(d)
randomized Hadmard transform: HDD ∈ Rd×d : a diagonal matrix Pr(Dii = ±1) = 0.5HD is orthogonal and ‖HDx‖2 = ‖x‖2
Key property:√
d‖HDx‖∞‖HDx‖2
≤√
log(d/δ) w.h.p 1− δ
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 152 / 234
Randomized Dimension Reduction
Randomized Hadmard transform
Hadmard transform:H ∈ Rd×d : H =
√1d H2k
H1 = [1] , H2 =
[1 11 −1
], H2k =
[H2k−1 H2k−1
H2k−1 −H2k−1
]
‖Hx‖2 = ‖x‖2 and H is orthogonalComputational costs of Hx : d log(d)
randomized Hadmard transform: HDD ∈ Rd×d : a diagonal matrix Pr(Dii = ±1) = 0.5HD is orthogonal and ‖HDx‖2 = ‖x‖2
Key property:√
d‖HDx‖∞‖HDx‖2
≤√
log(d/δ) w.h.p 1− δ
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 152 / 234
Randomized Dimension Reduction
Accelerate JL transforms: using Hadmard transform (I)
Fast JL transform based on randomized Hadmard transform (Tropp, 2011):
A =
√dm PHD
yields
|‖Ax‖22 − ‖x‖2
2| ≤
√3 log(2/δ) log(d/δ)
m ‖x‖22
m = Θ(ε−2 log(1/δ) log(d/δ)) suffice for 1± εadditional factor log(d/δ) can be removedComputational cost of AX : O(nd log(m))
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 153 / 234
Randomized Dimension Reduction
Accelerate JL transforms: using a sparse matrix (I)
Random hashing (Dasgupta et al., 2010)
A = HD
where D ∈ Rd×d and H ∈ Rm×d
random hashing: h(j) : 1, . . . , d → 1, . . . ,mHij = 1 if h(j) = i : sparse matrix (each column has only one non-zeroentry)D ∈ Rd×d : a diagonal matrix Pr(Dii = ±1) = 0.5[Ax]j =
∑i :h(i)=j xi Dii
Technically speaking, random hashing does not satisfy JL lemma
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 154 / 234
Randomized Dimension Reduction
Accelerate JL transforms: using a sparse matrix (I)
Random hashing (Dasgupta et al., 2010)
A = HD
where D ∈ Rd×d and H ∈ Rm×d
random hashing: h(j) : 1, . . . , d → 1, . . . ,mHij = 1 if h(j) = i : sparse matrix (each column has only one non-zeroentry)D ∈ Rd×d : a diagonal matrix Pr(Dii = ±1) = 0.5[Ax]j =
∑i :h(i)=j xi Dii
Technically speaking, random hashing does not satisfy JL lemma
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 154 / 234
Randomized Dimension Reduction
Accelerate JL transforms: using a sparse matrix (I)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 155 / 234
Randomized Dimension Reduction
Accelerate JL transforms: using a sparse matrix (II)
Sparse JL transform based on block random hashing (Kane & Nelson,2014)
A =
1√s Q1
. . .1√s Qs
Each Qs ∈ Rv×d is an independent random hashing (HD) matrixSet v = Θ(ε−1) and s = Θ(ε−1 log(1/δ))
Computational Cost of AX : O(nnz(X )
εlog[1δ
])
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 156 / 234
Randomized Dimension Reduction
Randomized Dimension Reduction
Johnson-Lindenstauss (JL) transforms
Subspace embeddings
Column sampling
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 157 / 234
Randomized Dimension Reduction
Subspace Embeddings
Definition: a subspace embedding given some parameters0 < ε, δ < 1, k ≤ d is a distribution D over matrices A ∈ Rm×d such thatfor any fixed linear subspace W ∈ Rd with dim(W ) = k it holds that
PrA∼D
(∀x ∈W , ‖Ax‖2 ∈ (1± ε)‖x‖2) ≥ 1− δ
It impliesIf U ∈ Rd×k is orthogonal matrix (contains the orthonormal bases)
AU ∈ Rm×k is of full column rank‖AU‖2 ∈ (1± ε)(1− ε)2 ≤ ‖U>A>AU‖2 ≤ (1 + ε)2
These are key properties in the theoretical analysis of manyalgorithms (e.g., low-rank matrix approximation, randomizedleast-squares regression, randomized classification)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 158 / 234
Randomized Dimension Reduction
Subspace Embeddings
Definition: a subspace embedding given some parameters0 < ε, δ < 1, k ≤ d is a distribution D over matrices A ∈ Rm×d such thatfor any fixed linear subspace W ∈ Rd with dim(W ) = k it holds that
PrA∼D
(∀x ∈W , ‖Ax‖2 ∈ (1± ε)‖x‖2) ≥ 1− δ
It impliesIf U ∈ Rd×k is orthogonal matrix (contains the orthonormal bases)
AU ∈ Rm×k is of full column rank‖AU‖2 ∈ (1± ε)(1− ε)2 ≤ ‖U>A>AU‖2 ≤ (1 + ε)2
These are key properties in the theoretical analysis of manyalgorithms (e.g., low-rank matrix approximation, randomizedleast-squares regression, randomized classification)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 158 / 234
Randomized Dimension Reduction
Subspace Embeddings
Definition: a subspace embedding given some parameters0 < ε, δ < 1, k ≤ d is a distribution D over matrices A ∈ Rm×d such thatfor any fixed linear subspace W ∈ Rd with dim(W ) = k it holds that
PrA∼D
(∀x ∈W , ‖Ax‖2 ∈ (1± ε)‖x‖2) ≥ 1− δ
It impliesIf U ∈ Rd×k is orthogonal matrix (contains the orthonormal bases)
AU ∈ Rm×k is of full column rank‖AU‖2 ∈ (1± ε)(1− ε)2 ≤ ‖U>A>AU‖2 ≤ (1 + ε)2
These are key properties in the theoretical analysis of manyalgorithms (e.g., low-rank matrix approximation, randomizedleast-squares regression, randomized classification)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 158 / 234
Randomized Dimension Reduction
Subspace Embeddings
From a JL transform to a Subspace Embedding (Sarlos, 2006).Let A ∈ Rm×d be a JL transform. If
m = O
k log[
kδε
]ε2
Then w.h.p 1− δk , A ∈ Rm×d is a subspace embedding w.r.t ak-dimensional space in Rd
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 159 / 234
Randomized Dimension Reduction
Subspace Embeddings
Making block random hashing a Subspace Embedding (Nelson & Nguyen,2013).
A =
1√s Q1
. . .1√s Qs
Each Qs ∈ Rv×d is an independent random hashing (HD) matrixSet v = Θ(kε−1 log5(k/δ)) and s = Θ(ε−1 log3(k/δ))
w.h.p 1− δ, A ∈ Rm×d with m = Θ(
k log8(k/δ)ε2
)is a subspace
embedding w.r.t a k-dimensional space in Rd
Computational Cost of AX : O(nnz(X )
εlog3
[kδ
])
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 160 / 234
Randomized Dimension Reduction
Sparse Subspace Embedding (SSE)
Random hashing is SSE with a Constant Probability (Nelson & Nguyen,2013)
A = HD
where D ∈ Rd×d and H ∈ Rm×d
m = Ω(k2/ε2) suffice for a subspace embedding with a probability 2/3Computational Cost AX : O(nnz(X ))
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 161 / 234
Randomized Dimension Reduction
Randomized Dimensionality Reduction
Johnson-Lindenstauss (JL) transforms
Subspace embeddings
Column (Row) sampling
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 162 / 234
Randomized Dimension Reduction
Column sampling
Column subset selection (feature selection)More interpretableUniform sampling usually does not work (not a JL transform)Non-oblivious sampling (data-dependent sampling)
leverage-score sampling
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 163 / 234
Randomized Dimension Reduction
Column sampling
Column subset selection (feature selection)More interpretableUniform sampling usually does not work (not a JL transform)Non-oblivious sampling (data-dependent sampling)
leverage-score sampling
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 163 / 234
Randomized Dimension Reduction
Column sampling
Column subset selection (feature selection)More interpretableUniform sampling usually does not work (not a JL transform)Non-oblivious sampling (data-dependent sampling)
leverage-score sampling
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 163 / 234
Randomized Dimension Reduction
Leverage-score sampling (Drineas et al., 2006)
Let X ∈ Rd×n be a rank-k matrixX = UΣV>: U ∈ Rd×k , Σ ∈ Rk×k
Leverage scores ‖Ui∗‖22, i = 1, . . . , d
Let pi =‖Ui∗‖2
2∑di=1 ‖Ui∗‖2
2, i = 1, . . . , d
Let i1, . . . , im ∈ 1, . . . , d denote m indices selected by following pi
Let A ∈ Rm×d be sampling-and-rescaling matrix:
Aij =
1√mpj
if j = ij
0 otherwise
AX ∈ Rm×n is a small sketch of X
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 164 / 234
Randomized Dimension Reduction
Leverage-score sampling (Drineas et al., 2006)
Let X ∈ Rd×n be a rank-k matrixX = UΣV>: U ∈ Rd×k , Σ ∈ Rk×k
Leverage scores ‖Ui∗‖22, i = 1, . . . , d
Let pi =‖Ui∗‖2
2∑di=1 ‖Ui∗‖2
2, i = 1, . . . , d
Let i1, . . . , im ∈ 1, . . . , d denote m indices selected by following pi
Let A ∈ Rm×d be sampling-and-rescaling matrix:
Aij =
1√mpj
if j = ij
0 otherwise
AX ∈ Rm×n is a small sketch of X
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 164 / 234
Randomized Dimension Reduction
Leverage-score sampling (Drineas et al., 2006)
Let X ∈ Rd×n be a rank-k matrixX = UΣV>: U ∈ Rd×k , Σ ∈ Rk×k
Leverage scores ‖Ui∗‖22, i = 1, . . . , d
Let pi =‖Ui∗‖2
2∑di=1 ‖Ui∗‖2
2, i = 1, . . . , d
Let i1, . . . , im ∈ 1, . . . , d denote m indices selected by following pi
Let A ∈ Rm×d be sampling-and-rescaling matrix:
Aij =
1√mpj
if j = ij
0 otherwise
AX ∈ Rm×n is a small sketch of X
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 164 / 234
Randomized Dimension Reduction
Leverage-score sampling (Drineas et al., 2006)
Let X ∈ Rd×n be a rank-k matrixX = UΣV>: U ∈ Rd×k , Σ ∈ Rk×k
Leverage scores ‖Ui∗‖22, i = 1, . . . , d
Let pi =‖Ui∗‖2
2∑di=1 ‖Ui∗‖2
2, i = 1, . . . , d
Let i1, . . . , im ∈ 1, . . . , d denote m indices selected by following pi
Let A ∈ Rm×d be sampling-and-rescaling matrix:
Aij =
1√mpj
if j = ij
0 otherwise
AX ∈ Rm×n is a small sketch of X
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 164 / 234
Randomized Dimension Reduction
Leverage-score sampling (Drineas et al., 2006)
Let X ∈ Rd×n be a rank-k matrixX = UΣV>: U ∈ Rd×k , Σ ∈ Rk×k
Leverage scores ‖Ui∗‖22, i = 1, . . . , d
Let pi =‖Ui∗‖2
2∑di=1 ‖Ui∗‖2
2, i = 1, . . . , d
Let i1, . . . , im ∈ 1, . . . , d denote m indices selected by following pi
Let A ∈ Rm×d be sampling-and-rescaling matrix:
Aij =
1√mpj
if j = ij
0 otherwise
AX ∈ Rm×n is a small sketch of X
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 164 / 234
Randomized Dimension Reduction
Properties of Leverage-score sampling
When m = Θ(
kε2 log
[2kδ
]), w.h.p 1− δ,
AU ∈ Rm×k is full column rankσ2
i (AU) ≥ (1− ε) ≥ (1− ε)2
σ2i (AU) ≤ 1 + ε ≤ (1 + ε)2
Leverage-score sampling performs like a subspace embedding (only forU, the top singular vector matrix of X )Computational cost: compute top-k SVD of X , expensiveRandomized algoritms to compute approximate leverage scores
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 165 / 234
Randomized Dimension Reduction
Properties of Leverage-score sampling
When m = Θ(
kε2 log
[2kδ
]), w.h.p 1− δ,
AU ∈ Rm×k is full column rankσ2
i (AU) ≥ (1− ε) ≥ (1− ε)2
σ2i (AU) ≤ 1 + ε ≤ (1 + ε)2
Leverage-score sampling performs like a subspace embedding (only forU, the top singular vector matrix of X )Computational cost: compute top-k SVD of X , expensiveRandomized algoritms to compute approximate leverage scores
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 165 / 234
Randomized Dimension Reduction
When uniform sampling makes sense?
Coherence measureµk =
dk max
1≤i≤d‖Ui∗‖2
2
Valid when the coherence measure is small (some real data miningdatasets have small coherence measures)The Nystrom method usually uses uniform sampling (Gittens, 2011)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 166 / 234
margin is preserved: if data is linearly separable (Balcan et al., 2006) aslong as m ≥ 12
ε2 log( 6mδ )
generalization performance is preserved: if the data matrix if of lowrank and m = Ω( kploy(log(k/δε))
ε2 ) (Paul et al., 2013)How to recover an accurate model in the original high-dimensionalspace?Dual Recovery (Zhang et al., 2014) and Dual Sparse Recovery (Yanget al., 2015)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 170 / 234
margin is preserved: if data is linearly separable (Balcan et al., 2006) aslong as m ≥ 12
ε2 log( 6mδ )
generalization performance is preserved: if the data matrix if of lowrank and m = Ω( kploy(log(k/δε))
ε2 ) (Paul et al., 2013)How to recover an accurate model in the original high-dimensionalspace?Dual Recovery (Zhang et al., 2014) and Dual Sparse Recovery (Yanget al., 2015)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 170 / 234
margin is preserved: if data is linearly separable (Balcan et al., 2006) aslong as m ≥ 12
ε2 log( 6mδ )
generalization performance is preserved: if the data matrix if of lowrank and m = Ω( kploy(log(k/δε))
ε2 ) (Paul et al., 2013)How to recover an accurate model in the original high-dimensionalspace?Dual Recovery (Zhang et al., 2014) and Dual Sparse Recovery (Yanget al., 2015)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 170 / 234
margin is preserved: if data is linearly separable (Balcan et al., 2006) aslong as m ≥ 12
ε2 log( 6mδ )
generalization performance is preserved: if the data matrix if of lowrank and m = Ω( kploy(log(k/δε))
ε2 ) (Paul et al., 2013)How to recover an accurate model in the original high-dimensionalspace?Dual Recovery (Zhang et al., 2014) and Dual Sparse Recovery (Yanget al., 2015)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 170 / 234
margin is preserved: if data is linearly separable (Balcan et al., 2006) aslong as m ≥ 12
ε2 log( 6mδ )
generalization performance is preserved: if the data matrix if of lowrank and m = Ω( kploy(log(k/δε))
ε2 ) (Paul et al., 2013)How to recover an accurate model in the original high-dimensionalspace?Dual Recovery (Zhang et al., 2014) and Dual Sparse Recovery (Yanget al., 2015)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 170 / 234
For random sketch: JL transforms, sparse subspace embedding all workJL transform: m = O(k log(k/(εδ))
ε2 )
Sparse subspace embedding: m = O( k2
ε2δ )
ε relates to the approximation accuracyAnalysis of approximation error for K-means can be formulates asConstrained Low-rank Approximation (Cohen et al., 2015)
minQ>Q=I
‖X − QQ>X‖2F
where Q is orthonormal.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 182 / 234
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 188 / 234
Randomized Algorithms Randomized Kernel methods
The Nystrom method vs RFF (Yang et al., 2012)
functional approximation frameworkThe Nystrom method: data-dependent basesRFF: data independent basesIn certain cases (e.g., large eigen-gap, skewed eigen-valuedistribution): the generalization performance of the Nystrom methodis better than RFF
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 189 / 234
Randomized Algorithms Randomized Kernel methods
The Nystrom method vs RFF
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 190 / 234
Krylov subspace method (e.g. Lanczos algorithm):O(kTmult + (n + d)k2), where Tmult denotes the cost of matrix-vectorproduct.
Randomized AlgorithmsSpeed can be faster (e.g., O(nd log(k)))Output more robust (e.g. Lanczos requires sophisticatedmodifications)Can be pass efficientCan exploit parallel algorithms
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 194 / 234
Krylov subspace method (e.g. Lanczos algorithm):O(kTmult + (n + d)k2), where Tmult denotes the cost of matrix-vectorproduct.
Randomized AlgorithmsSpeed can be faster (e.g., O(nd log(k)))Output more robust (e.g. Lanczos requires sophisticatedmodifications)Can be pass efficientCan exploit parallel algorithms
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 194 / 234
Randomization perspective: reduce data size, exploring properties ofdata
randomized feature reduction (e.g., reduce the number of features)randomized instance reduction (e.g., reduce the number of instances)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 200 / 234
Concluding Remarks
How can we address big data challenge?
Optimization perspective: improve convergence rates, exploringproperties of functions
Pro: can obtain the optimal solutionCon: high computational/communication costs
Randomization perspective: reduce data size, exploring properties ofdata
Pro: fastCon: still exists recovery error
Can we combine the benefits of two techniques?
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 201 / 234
Concluding Remarks
Combine Randomization and Optimization (Yang et al.,2015)
Use randomization (Dual Spare Recovery) to obtain a good initialsolution
Initialize distributed optimization (DisDCA) to reduce cost ofcomputation/communication
Observe 1 or 2 epochs of computations (1 or 2 communications)suffice to obtain the same performance of pure optimization
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 202 / 234
Concluding Remarks
Big Data Experiments
KDDcup Data: n = 8, 407, 752, d = 29, 890, 095, 10 machines, m = 1024
0
5
10
15
Testing E
rro
r (%
)
kdd
DSRRDSRR−Rec
DSRR−DisDCA−1
DSRR−DisDCA−2DisDCA
0
50
100
150
200
250
tim
e (
s)
kdd
DSRR
DSRR−Rec
DSRR−DisDCA−1
DSRR−DisDCA−2
DisDCA
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 203 / 234
Concluding Remarks
Thank You! Questions?
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 204 / 234
Concluding Remarks
References I
Achlioptas, Dimitris. Database-friendly random projections:Johnson-Lindenstrauss with binary coins. Journal of Computer andSystem Sciences, 66(4):671 – 687, 2003.
Balcan, Maria-Florina, Blum, Avrim, and Vempala, Santosh. Kernels asfeatures: on kernels, margins, and low-dimensional mappings. MachineLearning, 65(1):79–94, 2006.
Boyd, S., Parikh, N., Chu, E., Peleato, B., and Eckstein, J. Distributedoptimization and statistical learning via the alternating directionmethods of multiplies. Foundations and Trends in Machine Learning,3(1):1–122, 2011.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 205 / 234
Concluding Remarks
References II
Cohen, Michael B., Elder, Sam, Musco, Cameron, Musco, Christopher,and Persu, Madalina. Dimensionality reduction for k-means clusteringand low rank approximation. In Proceedings of the Forty-SeventhAnnual ACM on Symposium on Theory of Computing (STOC), pp.163–172, 2015.
Dasgupta, Anirban, Kumar, Ravi, and Sarlos, Tamas. A sparse johnson:Lindenstrauss transform. In Proceedings of the 42nd ACM symposiumon Theory of computing, STOC ’10, pp. 341–350, 2010.
Dasgupta, Sanjoy and Gupta, Anupam. An elementary proof of a theoremof Johnson and Lindenstrauss. Random Structures & Algorithms, 22(1):60–65, 2003.
Defazio, Aaron, Bach, Francis, and Lacoste-Julien, Simon. Saga: A fastincremental gradient method with support for non-strongly convexcomposite objectives. In NIPS, 2014.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 206 / 234
Concluding Remarks
References III
Drineas, Petros and Mahoney, Michael W. On the nystrom method forapproximating a gram matrix for improved kernel-based learning.Journal of Machine Learning Research, 6:2005, 2005.
Drineas, Petros, Mahoney, Michael W., and Muthukrishnan, S. Samplingalgorithms for l2 regression and applications. In ACM-SIAM Symposiumon Discrete Algorithms (SODA), pp. 1127–1136, 2006.
Drineas, Petros, Mahoney, Michael W., Muthukrishnan, S., and Sarlos,Tamas. Faster least squares approximation. Numerische Mathematik,117(2):219–249, February 2011.
Gittens, Alex. The spectral norm error of the naive nystrom extension.CoRR, 2011.
Golub, Gene H. and Ye, Qiang. Inexact preconditioned conjugate gradientmethod with inner-outer iteration. SIAM J. Sci. Comput, 21:1305–1320,1997.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 207 / 234
Concluding Remarks
References IV
Halko, Nathan, Martinsson, Per Gunnar., and Tropp, Joel A. Findingstructure with randomness: Probabilistic algorithms for constructingapproximate matrix decompositions. SIAM Review, 53(2):217–288, May2011.
Hsieh, Cho-Jui, Chang, Kai-Wei, Lin, Chih-Jen, Keerthi, S. Sathiya, andSundararajan, S. A dual coordinate descent method for large-scale linearsvm. In ICML, pp. 408–415, 2008.
Johnson, Rie and Zhang, Tong. Accelerating stochastic gradient descentusing predictive variance reduction. In NIPS, pp. 315–323, 2013.
Johnson, William and Lindenstrauss, Joram. Extensions of Lipschitzmappings into a Hilbert space. In Conference in modern analysis andprobability (New Haven, Conn., 1982), volume 26, pp. 189–206. 1984.
Kane, Daniel M. and Nelson, Jelani. Sparser johnson-lindenstrausstransforms. Journal of the ACM, 61:4:1–4:23, 2014.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 208 / 234
Lin, Qihang, Lu, Zhaosong, and Xiao, Lin. An accelerated proximalcoordinate gradient method and its application to regularized empiricalrisk minimization. In NIPS, 2014.
Ma, Chenxin, Smith, Virginia, Jaggi, Martin, Jordan, Michael I., Richtarik,Peter, and Takac, Martin. Adding vs. averaging in distributedprimal-dual optimization. In ICML, 2015.
Nelson, Jelani and Nguyen, Huy L. OSNAP: faster numerical linear algebraalgorithms via sparser subspace embeddings. CoRR, abs/1211.1002,2012.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 209 / 234
Concluding Remarks
References VI
Nelson, Jelani and Nguyen, Huy L. OSNAP: faster numerical linear algebraalgorithms via sparser subspace embeddings. In 54th Annual IEEESymposium on Foundations of Computer Science (FOCS), pp. 117–126,2013.
Nemirovski, A. and Yudin, D. On cezari’s convergence of the steepestdescent method for approximating saddle point of convex-concavefunctons. Soviet Math Dkl., 19:341–362, 1978.
Nesterov, Yurii. Efficiency of coordinate descent methods on huge-scaleoptimization problems. SIAM Journal on Optimization, 22:341–362,2012.
Ozdaglar, Asu. Distributed multiagent optimization linear convergencerate of admm. Technical report, MIT, 2015.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 210 / 234
Concluding Remarks
References VII
Paul, Saurabh, Boutsidis, Christos, Magdon-Ismail, Malik, and Drineas,Petros. Random projections for support vector machines. In Proceedingsof the International Conference on Artificial Intelligence and Statistics(AISTATS), pp. 498–506, 2013.
Rahimi, Ali and Recht, Benjamin. Random features for large-scale kernelmachines. In Advances in Neural Information Processing Systems 20,pp. 1177–1184, 2008.
Recht, Benjamin. A simpler approach to matrix completion. JournalMachine Learning Research (JMLR), pp. 3413–3430, 2011.
Roux, Nicolas Le, Schmidt, Mark, and Bach, Francis. A stochasticgradient method with an exponential convergence rate forstrongly-convex optimization with finite training sets. CoRR, 2012.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 211 / 234
Concluding Remarks
References VIII
Sarlos, Tamas. Improved approximation algorithms for large matrices viarandom projections. In 47th Annual IEEE Symposium on Foundations ofComputer Science (FOCS), pp. 143–152, 2006.
Shalev-Shwartz, Shai and Zhang, Tong. Stochastic dual coordinate ascentmethods for regularized loss. Journal of Machine Learning Research, 14:567–599, 2013.
Shamir, Ohad, Srebro, Nathan, and Zhang, Tong.Communication-efficient distributed optimiztion using an approximatenewton-type method. In ICML, 2014.
Tropp, Joel A. Improved analysis of the subsampled randomized hadamardtransform. Advances in Adaptive Data Analysis, 3(1-2):115–126, 2011.
Tropp, Joel A. User-friendly tail bounds for sums of random matrices.Found. Comput. Math., 12(4):389–434, August 2012. ISSN 1615-3375.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 212 / 234
Concluding Remarks
References IX
Xiao, L. and Zhang, T. A proximal stochastic gradient method withprogressive variance reduction. SIAM Journal on Optimization, 24(4):2057–2075, 2014.
Yang, Tianbao. Trading computation for communication: Distributedstochastic dual coordinate ascent. NIPS’13, pp. –, 2013.
Yang, Tianbao, Li, Yu-Feng, Mahdavi, Mehrdad, Jin, Rong, and Zhou,Zhi-Hua. Nystrom method vs random fourier features: A theoretical andempirical comparison”. In Advances in Neural Information ProcessingSystems (NIPS), pp. 485–493, 2012.
Yang, Tianbao, Zhang, Lijun, Jin, Rong, and Zhu, Shenghuo. Theory ofdual-sparse regularized randomized reduction. In Proceedings of the32nd International Conference on Machine Learning, (ICML), pp.305–314, 2015.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 213 / 234
Concluding Remarks
References X
Zhang, Lijun, Mahdavi, Mehrdad, and Jin, Rong. Linear convergence withcondition number independent access of full gradients. In NIPS, pp.980–988. 2013.
Zhang, Lijun, Mahdavi, Mehrdad, Jin, Rong, Yang, Tianbao, and Zhu,Shenghuo. Random projections for classification: A recovery approach.IEEE Transactions on Information Theory (IEEE TIT), 60(11):7300–7316, 2014.
Zhang, Yuchen and Xiao, Lin. Communication-efficient distributedoptimization of self-concordant empirical loss. In ICML, 2015.
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 214 / 234
Appendix
Examples of Convex functions
ax + b, Ax + bx2, ‖x‖2
2exp(ax), exp(w>x)
log(1 + exp(ax)), log(1 + exp(w>x))
x log(x),∑
i xi log(xi )
‖x‖p, p ≥ 1, ‖x‖2p
maxi (xi )
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 215 / 234
Appendix
Operations that preserve convexity
Nonnegative scale: a · f (x) where a ≥ 0Sum: f (x) + g(x)
Composition with affine function f (Ax + b)
Point-wise maximum: maxi fi (x)
Examples:Least-squares regression: ‖Ax− b‖2
SVM: 1n∑n
i=1 max(0, 1− yiw>xi ) + λ2‖w‖
22
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 216 / 234
Appendix
Smooth Convex function
smooth: e.g. logistic loss f (x) = log(1 + exp(−x))
‖∇f (x)−∇f (y)‖2 ≤ L‖x − y‖2
where L > 0
smoothnessconstant
Second Order Derivative is upperbounded ‖∇2f (x)‖2 ≤ L
−5 −4 −3 −2 −1 0 1 2 3 4 5−1
0
1
2
3
4
5
6
log(1+exp(−x))
f(y)+f’(y)(x−y)
y
f(x)
Quadratic Function
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 217 / 234
Appendix
Smooth Convex function
smooth: e.g. logistic loss f (x) = log(1 + exp(−x))
‖∇f (x)−∇f (y)‖2 ≤ L‖x − y‖2
where L > 0
smoothnessconstant
Second Order Derivative is upperbounded ‖∇2f (x)‖2 ≤ L
−5 −4 −3 −2 −1 0 1 2 3 4 5−1
0
1
2
3
4
5
6
log(1+exp(−x))
f(y)+f’(y)(x−y)
y
f(x)
Quadratic Function
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 217 / 234
Appendix
Strongly Convex function
strongly convex: e.g. Euclidean norm f (x) = 12‖x‖
22
‖∇f (x)−∇f (y)‖2 ≥ λ‖x − y‖2
where λ > 0
strong convexityconstant
Second Order Derivative is lowerbounded ‖∇2f (x)‖2 ≥ λ
−1 −0.5 0 0.5 1−0.2
0
0.2
0.4
0.6
0.8
x2
gradient
smooth
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 218 / 234
Appendix
Strongly Convex function
strongly convex: e.g. Euclidean norm f (x) = 12‖x‖
22
‖∇f (x)−∇f (y)‖2 ≥ λ‖x − y‖2
where λ > 0
strong convexityconstant
Second Order Derivative is lowerbounded ‖∇2f (x)‖2 ≥ λ
−1 −0.5 0 0.5 1−0.2
0
0.2
0.4
0.6
0.8
x2
gradient
smooth
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 218 / 234
Appendix
Smooth and Strongly Convex function
smooth and strongly convex: e.g. quadratic function:f (z) = 1
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 233 / 234
Appendix
When uniform sampling makes sense?
Coherence measureµk =
dk max
1≤i≤d‖Ui∗‖2
2
When µk ≤ τ and m = Θ(
kτε2 log
[2kδ
])w.h.p 1− δ,
A formed by uniform sampling (and scaling)AU ∈ Rm×k is full column rankσ2
i (AU) ≥ (1− ε) ≥ (1− ε)2
σ2i (AU) ≤ (1 + ε) ≤ (1 + ε)2
Valid when the coherence measure is small (some real data miningdatasets have small coherence measures)The Nystrom method usually uses uniform sampling (Gittens, 2011)
Yang, Lin, Jin Tutorial for KDD’15 August 10, 2015 234 / 234