Duality (I)
Lijun [email protected]://cs.nju.edu.cn/zlj
Outline
The Lagrange Dual Function The Lagrange Dual Function Lower Bound on Optimal Value The Lagrange Dual Function and
Conjugate Functions The Lagrange Dual Problem Making Dual Constraints Explicit Weak Duality Strong Duality and Slaterβs Constraint
Qualification
Optimization Problems
Standard Form
Domain is nonempty
Denote the optimal value by β
We do not assume the problem is convex
π dom π β© dom β
min π π₯ s. t. π π₯ 0, π 1, . . . , π
β π₯ 0 π 1, . . . , π 1
The Lagrangian
: the Lagrange multiplier associated with
the -th inequality constraint π : the Lagrange multiplier associated with
the -th equality constraint Vectors and : dual variables or
Lagrange multiplier vectors
The Lagrangian
πΏ π₯, π, π π π₯ π π π₯ π β π₯
The Lagrange Dual Function
When is unbounded below in , is concave π is the pointwise infimum of a family of
affine functions of π, π£ It is unconstrained
π π, π infβπ
πΏ π₯, π, π
infβπ
π π₯ π π π₯ π β π₯
Outline
The Lagrange Dual Function The Lagrange Dual Function Lower Bound on Optimal Value The Lagrange Dual Function and
Conjugate Functions The Lagrange Dual Problem Making Dual Constraints Explicit Weak Duality Strong Duality and Slaterβs Constraint
Qualification
Lower Bounds on
For any and any
Proof is a feasible point for original problem
Since
Therefore
π π, π πβ
π π₯ 0, β π₯ 0
π π π₯ π β π₯ 0
πΏ π₯, π, π π π₯ π π π₯ π β π₯ π π₯
Lower Bounds on
For any and any
Proof Hence
Note that for any feasible Discussions The lower bound is vacuous, when π π, π β It is nontrivial only when π β½ 0, π, π β dom π Dual feasible: π, π with π β½ 0, π, π β dom π
π π, π πβ
π π, π infβπ
πΏ π₯, π, π πΏ π₯, π, π π π₯
Example
A Simple Problem with Lower bound from a dual feasible point
Solid curve: objective function π Dashed curve: constraint function π Feasible set: 0.46, 0.46 (indicated by the two
dotted vertical lines)
Example
A Simple Problem with Lower bound from a dual feasible point
Optimal point and value: π₯β 0.46, πβ 1.54 Dotted curves: πΏ π₯, π for π 0.1, 0.2, β¦ , 1.0.
β’ Each has a minimum value smaller than πβ as on the feasible set (and for π 0), πΏ π₯, π π π₯
πΏ π₯, π π π₯
π π infβπ
πΏ π₯, π
Example
The dual function
Neither nor is convex, but the dual function is concave
Horizontal dashed line: β (the optimal value of the problem)
Example
Least-squares Solution of Linear Equations
No inequality constraints (linear) equality constraints
Lagrangian
Domain:
min π₯ π₯ s. t. π΄π₯ π
πΏ π₯, π π₯ π₯ π π΄π₯ π
Example
Least-squares Solution of Linear Equations
Dual Function
Optimality condition
π π inf πΏ π₯, π inf π₯ π₯ π π΄π₯ π
π» πΏ π₯, π 2π₯ π΄ π 0 β π₯ 1/2 π΄ π
min π₯ π₯ s. t. π΄π₯ π
Example
Least-squares Solution of Linear Equations
Dual Function
Concave Function Lower Bound Property
β π π πΏ 1/2 π΄ π, π 1/4 π π΄π΄ π π π
1/4 π π΄π΄ π π π inf π₯ π₯ π΄π₯ π
min π₯ π₯ s. t. π΄π₯ π
Example
Standard Form LP
Inequality constraints: Lagrangian
Dual Function
min π π₯ s. t. π΄π₯ π
π₯ β½ 0
πΏ π₯, π, π π π₯ β π π₯ π π΄π₯ π
π π π π΄ π π π₯
π π, π inf πΏ π₯, π, π π π inf π π΄ π π π₯
Example
Standard Form LP
Inequality constraints: Dual Function
The lower bound is nontrivial only when and satisfy and
min π π₯ s. t. π΄π₯ π
π₯ β½ 0
π π, π π π π΄ π π π 0,β otherwise.
Outline
The Lagrange Dual Function The Lagrange Dual Function Lower Bound on Optimal Value The Lagrange Dual Function and
Conjugate Functions The Lagrange Dual Problem Making Dual Constraints Explicit Weak Duality Strong Duality and Slaterβs Constraint
Qualification
Conjugate Function Its conjugate function is
dom πβ π¦|πβ π¦ β πβ is always convex
πβ π¦ supβ
π¦ π₯ π π₯
The Lagrange Dual Function and Conjugate Functions
A Simple Example
Lagrangian
Dual Function
min π π₯ s. t. π₯ 0
πΏ π₯, π π π₯ π π₯
π π inf π π₯ π π₯
sup π π₯ π π₯ πβ π
The Lagrange Dual Function and Conjugate Functions
A More General Example
Lagrangian
Dual Function
min π π₯ s. t. π΄π₯ βΌ π
πΆπ₯ π
πΏ π₯, π, π π π₯ π π΄π₯ π π πΆπ₯ π
π π, π inf π π₯ π π΄π₯ π π πΆπ₯ π
π π π π inf π π₯ π΄ π πΆ π π₯
π π π π πβ π΄ π πΆ π
The Lagrange Dual Function and Conjugate Functions
A More General Example
Lagrangian
Dual Function
β
min π π₯ s. t. π΄π₯ βΌ π
πΆπ₯ π
π π, π π π π π πβ π΄ π πΆ π
πΏ π₯, π, π π π₯ π π΄π₯ π π πΆπ₯ π
Example
Equality Constrained Norm Minimization
Conjugate of
The Dual Function
min π₯ s. t. π΄π₯ π
πβ π¦ 0 π¦ β 1,β otherwise.
π π π π πβ π΄ π π π π΄ π β 1,β otherwise.
Example
Entropy Maximization
Conjugate of
The Dual Function
min π π₯ π₯ log π₯
s. t. π΄π₯ βΌ π π π₯ 1
πβ π¦ π
π π, π π π π πβ π΄ π π£π= π π π β π
π π π π β π
Outline
The Lagrange Dual Function The Lagrange Dual Function Lower Bound on Optimal Value The Lagrange Dual Function and
Conjugate Functions The Lagrange Dual Problem Making Dual Constraints Explicit Weak Duality Strong Duality and Slaterβs Constraint
Qualification
The Lagrange Dual Problem
For any and any
What is the best lower bound? Lagrange Dual Problem
Primal Problem
π π, π πβ
max π π, πs. t. π β½ 0
min π π₯ s. t. π π₯ 0, π 1, . . . , π
β π₯ 0 π 1, . . . , π 1
The Lagrange Dual Problem
For any and any
What is the best lower bound? Lagrange Dual Problem
Dual feasible: with Dual optimal or optimal Lagrange
multipliers: β β
A convex optimization problem
π π, π πβ
max π π, πs. t. π β½ 0
Outline
The Lagrange Dual Function The Lagrange Dual Function Lower Bound on Optimal Value The Lagrange Dual Function and
Conjugate Functions The Lagrange Dual Problem Making Dual Constraints Explicit Weak Duality Strong Duality and Slaterβs Constraint
Qualification
Making Dual Constraints Explicit
Motivation The may have
dimension Identify the equality constraints that are
βhiddenβ or βimplicitβ in Standard Form LP
Dual Function
dom π π, π π π, π β
π π, π π π π΄ π π π 0,β otherwise.
min π π₯ s. t. π΄π₯ π
π₯ β½ 0
Example
Lagrange Dual of Standard Form LP Lagrange Dual Problem
An Equivalent Problem
Make equality constraints explicit
max π π, π π π π΄ π π π 0,β otherwise.
s. t. π β½ 0
max π π s. t. π΄ π π π 0
π β½ 0
Example
Lagrange Dual of Standard Form LP Lagrange Dual Problem
Another Equivalent Problem
An LP in inequality form
max π π, π π π π΄ π π π 0,β otherwise.
s. t. π β½ 0
max π π s. t. π΄ π π β½ 0
Standard Form LP Inequality Form LPLagrange Dual
Example
Lagrange Dual of Inequality Form LP Inequality form LP (Primal Problem)
Lagrangian
Lagrange dual function
min π π₯ s. t. π΄π₯ βΌ π
πΏ π₯, π π π₯ π π΄π₯ π π π π΄ π π π₯
π π inf πΏ π₯, π π π inf π΄ π π π₯
π π π΄ π π 0,β otherwise.
Example
Lagrange Dual of Inequality Form LP Inequality form LP (Primal Problem)
Lagrange Dual Problem
An Equivalent Problem
min π π₯ s. t. π΄π₯ βΌ π
max π π π π π΄ π π 0,β otherwise.
s. t. π β½ 0
max π π s. t. π΄ π π 0
π β½ 0
Example
Lagrange Dual of Inequality Form LP Inequality form LP (Primal Problem)
An Equivalent Problem
An LP in standard form
min π π₯ s. t. π΄π₯ βΌ π
max π π s. t. π΄ π π 0
π β½ 0
Standard Form LPInequality Form LPLagrange Dual
Outline
The Lagrange Dual Function The Lagrange Dual Function Lower Bound on Optimal Value The Lagrange Dual Function and
Conjugate Functions The Lagrange Dual Problem Making Dual Constraints Explicit Weak Duality Strong Duality and Slaterβs Constraint
Qualification
Weak Duality
For any and any
What is the best lower bound? Lagrange Dual Problem
Optimal value β
Weak Duality
π π, π πβ
max π π, πs. t. π β½ 0
πβ πβ
Does not rely on convexity!
Weak Duality
Weak Duality
If the primal problem is unbounded below, i.e., β , we must have β
, i.e., the Lagrange dual problem is infeasible
If β , we must have β
Optimal duality gap
Nonegativeπβ πβ
πβ πβ
Outline
The Lagrange Dual Function The Lagrange Dual Function Lower Bound on Optimal Value The Lagrange Dual Function and
Conjugate Functions The Lagrange Dual Problem Making Dual Constraints Explicit Weak Duality Strong Duality and Slaterβs Constraint
Qualification
Strong Duality
Strong Duality
The optimal duality gap is zero The best bound that can be obtained
from the Lagrange dual function is tight In general, does not hold
Usually hold for convex optimization
are convex
β β
min π π₯ s. t. π π₯ 0, π 1, . . . , π
π΄π₯ π
Slaterβs Constraint Qualification
Constraint Qualifications Sufficient conditions for strong duality
Slaterβs condition such that
Such a point is called strictly feasible If Slaterβs condition holds and the
problem is convex Strong duality holds Dual optimal value is attained when β
π π₯ 0, π 1, β¦ , π, π΄π₯ π
Slaterβs Constraint Qualification
Constraint Qualifications Sufficient conditions for strong duality
Slaterβs condition (weaker form) If the first constraint functions are affine such that
When constraints are all linear equalities and inequalities, and is open Reduce to feasibility
π π₯ 0, π 1, β¦ , π π π₯ 0, π π 1, β¦ , ππ΄π₯ π
Example
Least-squares Solution of Linear Equations
Dual Problem
Slaterβs condition The primal problem is feasible, i.e.,
Strong duality always holds Even when
min π₯ π₯ s. t. π΄π₯ π
max 1/4 Ξ½ π΄π΄ π π π
Example
Lagrange dual of LP
Strong duality holds for any LP If the primal problem is feasible or the
dual problem is feasible
Strong duality may fail If both the primal and dual problems are
infeasible
Standard Form LP Inequality Form LPLagrange Dual
Example
QCQP (Primal Problem)
π β π and π β π , π 1, β¦ , π
Dual Problem
π π π β π π , π π π β π π π π π β π π
Slaterβs condition βπ₯ , 1/2 π₯ π π₯ π π₯ π 0, π 1, β¦ , π
min 1/2 π₯ π π₯ π π₯ π s. t. 1/2 π₯ π π₯ π π₯ π 0, π 1, β¦ , π
max 1/2 π π π π π π π πs. t. π β½ 0
Example
A Nonconvex Quadratic Problem (Primal Problem)
Lagrangian
Dual Function
min π₯ π΄π₯ 2π π₯s. t. π₯ π₯ 1
πΏ π₯, π π₯ π΄π₯ 2π π₯ π π₯ π₯ 1 π₯ π΄ ππΌ π₯ 2π π₯ π
π π π π΄ ππΌ π π π΄ ππΌ β½ 0, π β β π΄ ππΌβ otherwise
Example
A Nonconvex Quadratic Problem (Primal Problem)
Dual Problem
A convex optimization problem
min π₯ π΄π₯ 2π π₯s. t. π₯ π₯ 1
max π π΄ ππΌ π π s. t. π΄ ππΌ β½ 0, π β β π΄ ππΌ
Example
A Nonconvex Quadratic Problem (Primal Problem)
Dual Problem
A convex optimization problem and : eigenvalues and corresponding
(orthonormal) eigenvectors of
min π₯ π΄π₯ 2π π₯s. t. π₯ π₯ 1
max β π π / π π πs. t. π π π΄
Strong duality holds
Example
A Nonconvex Quadratic Problem (Primal Problem)
Dual Problem
A convex optimization problem and : eigenvalues and corresponding
(orthonormal) eigenvectors of
min π₯ π΄π₯ 2π π₯s. t. π₯ π₯ 1
max β π π / π π πs. t. π π π΄
Strong duality holds for anyoptimization problem withquadratic objective and onequadratic inequality constraint,provided Slaterβs conditionholds
Summary
The Lagrange Dual Function The Lagrange Dual Function Lower Bound on Optimal Value The Lagrange Dual Function and
Conjugate Functions The Lagrange Dual Problem Making Dual Constraints Explicit Weak Duality Strong Duality and Slaterβs Constraint
Qualification