Support Vector Machines
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Support Vector Machines
Supervised Learning
quail apple apple corn cornLabeled
Data
Model Class
Consider classifiers of the form y = f(x;w)
Features1.1 -0.5 0 0 0.3 …
quail-1 0 1.2 -0.4 0.1 …
apple1.1 -0.5 0 0 0.3 … -1 0 1.2 -0.4 0.1 …
apple corn
Learning Find that works well on the training data ww
x
y
Optimization!
Linear Classifiers• A simple and effective family of classifiers
(xi, yi)
y = sign [w · x+ b]
• The training problem:
• Given a set of n training points
• Find for the “best fitting” classifier w, b
Training Linear Classifiersy = sign [w · x+ b]
• How do we find it?
• If there exists a classifier with zero training error we can find one with the __________ algorithmperceptron
Many Possible Solutions • If there exists one solution, there exist many.
• Which one should we choose?
• Intuitively: one that’s farther away from the points.
Maximum Margin Classifier• For every point denote its distance
from the hyperplane.
• Margin of a classifier: the shortest distance to the hyperplane:
• Goal: find classifier that maximizes
xi d(xi,w, b)
mini
d(xi,w, b)
mini
d(xi,w, b)
ML and Optimization• We have an optimization problem
• Namely we want to find a set of parameters that will maximize some objective function (the margin) subject to some constraints (classifying correctly)
• Need a toolbox for solving such problems
• In what follows we provide an overview
Unconstrained Optimization• Use w to denote optimization variables
• For example:
• Generally:
• Solve by:
• Find all w such that
• These stationary points are the candidates for the global minimum (and asymptotes)
minw1,w2
(w1 � 2w2)2
minw
f(w)
@f(w)
@w= 0
Constrained Minimization• Suppose we are only interested in variables that
satisfy
• The optimization problem is:
• The zero gradient point may not satisfy the constraint.
h(w) = 0
min f(w)s.t. h(w) = 0
w1
w2h(w) = 0
Directional Derivative• Given function f(w) and direction v.
• What happens if we make a small change in direction v (where |v|2=1?)
• It is
• The direction along the curve h(w)=0 has zero directional derivative.
• Thus the orthogonal is the gradientw1
w2h(w) = 0
rf(w) · v
rh(w)
w1
w2 v
w
f(w + ↵v)
↵
rf(w)
Constrained Minimization• The optimization problem is:
• Consider f along h
• Vector of movement along curve v is orthogonal to
min f(w)s.t. h(w) = 0
w1
w2h(w) = 0
rf(w) = �rh(w)
rh(w)
f(w)
w : h(w) = 0
rh(w)
• Gradient along curve
• Is zero iff
rf(w) · v
vrf(w)
Lagrange Multiplier• The optimum points should satisfy:
rf(w) = �rh(w) For some �h(w) = 0 Constraint satisfied
1.2.
• Alternative formulation. Define Lagrangian:
L(w,�) = f(w) + �h(w)
• The optimum should satisfy:rwL(w,�) = 0
r�L(w,�) = 0
1.2.
Example• What is the distance between hyperplane and point?
x̄
w · x+ b = 0
minx:w·x+b=0
0.5kx� x̄k22
• Use primal feasibility to solve for �
(x̄� �w) ·w + b = 0
L(x,�) = 0.5kx� x̄k22 + � (w · x+ b)
rx
L(x,�) = (x� x̄) + �w = 0
� =x̄ ·w + b
kwk
kx� x̄k2 =|x̄ ·w + b|
kwk
x = x̄� �w
Multiple Constraints• Solve:
min f(w)s.t. hi(w) = 0 8i = 1, . . . , p
• Introduce multiplier per constraint �1, · · · ,�p
• Lagrangian:
• Optimality conditions:rwL(w,�) = 01.r�iL(w,�) = 02.
• May be several such points. Need to check which one is the global optimum.
L(w,�,↵) = f(w) +X
i
�ihi(w)
Inequality Constraints• Solve:
• Optimality conditions:
1.
w1
w2h(w) 0min f(w)
s.t. h(w) 0rh(w)
h(w) 0 Constraint satisfied
w1
w2h(w) 0
rh(w)�rf(w)
• When we are “stuck” if the only directions that decrease f take us outside the constraints. Namely:
h(w) = 0
For some ↵ � 0rf(w) = �↵rh(w)2a.h(w) < 0• When we need:
rf(w) = 02b.
w1
w2h(w) 0
rh(w)�rf(w)
Progress possible
Stuck
Complementary Slackness
• Called the Karush Kuhn Tucker (KKT) conditions. Always necessary.
• Sufficient for convex optimization.
rf(w) = �↵rh(w)
h(w) < 0 rf(w) = 0
h(w) = 0
• Summarize as:rf(w) = �↵rh(w)
↵h(w) = 0
↵ � 0, h(w) 0
Lagrange Multipliers• Consider the general problem:
min f(w)s.t. hi(w) = 0 8i = 1, . . . , p
gi(w) 0 8i = 1, . . . ,m
• Define the Lagrangian:L(w,�,↵) = f(w) +
X
i
�ihi(w) +X
i
↵igi(w)
• Optimum must satisfy:rwL(w,�,↵) = 0
↵igi(x) = 0 8i↵i � 0, gi(w) 0, hi(w) = 0
Typically easy if someone hands us ! ↵,�
Convex Optimization• General optimization problem may have
many local minima/maxima and saddle points. w
f(w)
• Makes minimization hard (e.g., exponential in dimension).
• Convex optimization problems are a “nice” subclass. Require:
• Convex f(w),g(x)
• Linear h(x) w
f(w)
Convex Optimization• Convex function if:
• Value on line is less than linear function.
• Non-negative second derivative (or Hessian)
• Examples:f(w) = w · xf(w) = max [w · x, 0]
f(w) = wTAw A ⌫ 0
Convex Optimization• Nice things:
• No local optima
• KKT conditions are sufficient for global optimality.
• Multipliers can be solved via dual.
Convex Duality• For every convex problem, we can define a dual
problem that has the same value
• Optimization is over the Lagrange multipliers.
• Solution to dual implies solution to primal via KKT
• Dual might be easier to solve.
Convex Duality• Recall the Lagrangian:
L(w,�,↵) = f(w) +X
i
�ihi(w) +X
i
↵igi(w)
• Then:min f(w)s.t. hi(w) = 0
gi(w) 0
= min
wmax
�,↵�0L(w,�,↵) Why?
min
wmax
�,↵�0L(w,�,↵) max
�,↵�0min
wL(w,�,↵)
• Replacing min and max gives:
• In the convex case it is an equality
Convex Duality• Define:
• Dual problem:
g(�,↵) = minw
L(w,�,↵)
max
�,↵�0g(�,↵)
• Has same value as primal problem.
• The resulting are optimal. You can recover the “primal” variables w via KKT. This is often easy.
�,↵
rwL(w,�,↵) = 0
↵igi(x) = 0 8i↵i � 0, gi(w) 0, hi(w) = 0
Maximum Margin Classifier• For every point denote its distance
from the hyperplane.
• Margin of a classifier: the shortest distance to the hyperplane:
• Goal: find classifier that maximizes
xi d(xi,w, b)
mini
d(xi,w, b)
mini
d(xi,w, b)
Geometry of Linear Classifiers
y = sign [w · x+ b]
• w is the orthogonal direction to the hyperplane.
• Proof: if on hyperplane then
What'is'b?
wMx + ; = 0
;w
w x
|x ·w + b|kwk
• Distance from origin to hyperplane is |b|kwk
x1,x2 w · (x1 � x2) = 0
Max Margin Hyperplane• Find a hyperplane that maximizes the minimum distance
• Solve: maxw1
kwk mini |w · xi + b|s.t. yi (w · xi + b) � 0
• Any solution can be rescaled to and not affect the objective or constraints.
(w, b)(cw, cb)
• We can rescale such that mini
|w · xi + b| = 1
maxw kwk�1
s.t. yi (w · xi + b) � 0
mini |w · xi + b| = 1
Max Margin Hyperplanemaxw kwk�1
s.t. yi (w · xi + b) � 0
mini |w · xi + b| = 1
• Equivalently:maxw kwk�1
s.t. mini yi (w · xi + b) = 1
• We can relax to an inequality (why?): mini
yi (w · xi + b) � 1
maxw kwk�1
s.t. yi (w · xi + b) � 1
minw kwk2s.t. yi (w · xi + b) � 1
Support Vector Machines (SVM)• The SVM classifier is the solution to:
• The where this is an equality are “support vectors” xi
• It is a convex optimization problem. Called a convex quadratic program (quad. objective and linear constraints)
minw 0.5kwk2s.t. yi (w · xi + b) � 1
Factor 0.5 doesn’t affect the optimum
SVM History• Initial version by Vapnik and Chervonenkis (63)
• Non linear version by Boser, Guyon, Vapnik (92)
• Much work on generalization theory since (by Bartlett, Shawe Taylor, Mendelson, Schoelkopf, Smola, and others).
• Many variants for regression, unsupervised learning etc.
Solving SVM• The SVM classifier is the solution to:
• You can plug this into a solver and get w, b
• Lets use Lagrangian to understand solution.
minw 0.5kwk2s.t. yi (w · xi + b) � 1
w =X
i
↵iyixi
rbL(w, b,↵) = �X
i
↵iyi = 0X
i
↵iyi = 0
L(w, b,↵) = 0.5kwk2 +X
i
↵i [1� yi (w · xi + b)]
rwL(w, b,↵) = w �X
i
↵iyixi = 0
The Representer Theorem • The optimal weight is a weight
combination of the data pointsw =
X
i
↵iyixi
• This will be very important!
• When is ? (recall KKT) ↵i = 0
• Whenever w · xi + b < 1
• only when ↵i > 0w · xi + b = 1
• Optimal weight is a combination only of support vectors!
Deriving via Dual • How do we find and then b?
• Use the dual!
w =X
i
↵iyixi↵i
• We know the minimizing w. Plug into Lagrangian.
g(↵) = 0.5kX
i
↵iyixik22 �X
i
↵i
2
4yi
0
@
0
@X
j
↵jyjxj
1
A · xi + b
1
A� 1
3
5
X
i
↵iyi = 0
=X
i
↵i � 0.5X
i,j
↵i↵jyiyjxi · xj
• Constrain because otherwise g(↵) = �1
L(w, b,↵) = 0.5kwk2 +X
i
↵i [1� yi (w · xi + b)]
g(↵) = minw,b
L(w, b,↵)
The SVM Dual• The dual problem is:
max
Pi ↵i � 0.5
Pi,j ↵i↵jyiyjxi · xj
s.t. ↵i � 0,P
i ↵iyi = 0
• Number of variables and constraints is number of training points.
• Also a convex quadratic program (why?)
• Obtaining the primal w: w =X
i
↵iyixi
Finding b• Recall from KKT that support vectors ( ) satisfy:
w · xi + b = 1
↵i > 0
• Since we know w we can solve for b.
• Should give same value for all support vectors.
Non Separable Case• So far we assumed a separating
hyperplane exists.• If it doesn’t, our optimization problem
is infeasible. • For real data, we don’t want to make
this assumption. Because:
• Data may be noisy. Linear classifier may still do ok.
• May come from a non linear rule. Next class!
Non Separable Case• Ideally, we would like to find the
classifier that minimizes training error.• But:
• Turns out this is NP hard.
• How do we incorporate margin?
• Let’s start from the separable case.
Non Separable Case
• Separable case:
• Need to “relax” the constraints.
• Allow violation by , but “pay” for violation.
minw 0.5kwk2s.t. yi (w · xi + b) � 1
⇠i � 0
• C is a constant that determines how much we care about classification errors as opposed to margin.
minw 0.5kwk2 + CP
i ⇠is.t. yi (w · xi + b) � 1� ⇠i , ⇠i � 0
Dual for non separable
• Dual is:max
Pi ↵i � 0.5
Pi,j ↵i↵jyiyjxi · xj
s.t. 0 ↵i C,P
i ↵iyi = 0
• Mapping to primal is as before.
Alternative Interpretation
• Primal is: minw 0.5kwk2 + CP
i ⇠is.t. yi (w · xi + b) � 1� ⇠i , ⇠i � 0
• Can solve for to get: ⇠i ⇠i = max [0, 1� yi(w · xi + b)]
• Problem becomes:
min
wCX
i
max [0, 1� yi(w · xi + b)] + 0.5kwk22
Alternative Interpretation
• Primal is: min
wCX
i
max [0, 1� yiw · xi] + 0.5kwk22
• The function: is called the hinge loss.
max [0, 1� yiw · xi]Hinge'loss
• SVM'uses'the'hinge'loss'max(0,1+ − %#1 x# )• an'approximation'to'the'0]1'loss
%#1 x#0
1
2
3
4
5
_3 _2 _1 0 1 2 3 4
0_1hinge
yiw · xi
• Upper bounds the true classification error
• A convex upper bound!
Alternative Interpretation
• Primal is: min
wCX
i
max [0, 1� yiw · xi] + 0.5kwk22
Bound on loss Regularization• Very common design pattern.
• Other losses and regularizers can be considered.
• Logistic loss:
• L1 regularization: . Sparsity inducing.
1
ln 2ln(1 + e�yiw·xi)
kwk1 =X
i
|wi|
SVM and Generalization
• Intuitively choosing a large margin should improve generalization
• Assume true distribution and classifier are such that the margin is
• Expect generalization to behave like
• But can always increase by rescaling
• Denote R the largest norm of x
• Generalization scales with
�
���1
�
R��1
SVM and Generalization
• Assume training error is zero.
• Can be shown that generalization satisfies (up to some logarithmic factors):
�error(w) c1
m
R
2
�
2+
c2
m
log
m
�
• The VC dimension is replaced by
• Appeared in “Structural Risk Minimization over Data-Dependent Hierarchies” (98)
R2
�2
Leave one out bounds
• Another intuition: using few support vectors should lead to good generalization.
• We will show this via leave one out error.
• Denote training sample without
• Denote the hypothesis from training on S
S�i (xi, yi)
hS
R̂LOO(S) =1
m
mX
i=1
I [hS�i(xi) 6= yi]
Leave one out bounds
• LOO error is similar in spirit to generalization error. But we only train on m-1 points.
• Denote R(h) the generalization error of h
• Can show:
• LOO error and generalization error have same expected value
ESm
hR̂LOO(Sm)
i= ESm�1
⇥R(hSm�1)
⇤
R(h) = E(x,y)⇠D
I [h(x) 6= y]
Leave one out bounds for SVM
• What is the expected LOO error of SVM (separable case).
• If a non-support vector is left out, the solution will not change, and error will be zero.
• Otherwise there might be an error:R̂LOO(Sm) NSV (Sm)
m
ESm�1
⇥R(hSm�1)
⇤ 1
mESm [NSV (Sm)]• Therefore:
• Generalization related to number of SVs.
Related Documents
