Linear Conic Programming Theory and Applications 1 Shu-Cherng Fang Department of Industrial and Systems Engineering Graduate Program in Operations Research North Carolina State University 1 Fang, S. C., and W. Xing. “Linear Conic Programming” (2013). Conic Programming 1 / 25
251
Embed
Linear Conic Programming Theory andLinear Conic Programming Theory and Applications1 Shu-Cherng Fang Department of Industrial and Systems Engineering Graduate Program in Operations
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Linear Conic Programming Theory andApplications1
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
1Fang, S. C., and W. Xing. “Linear Conic Programming” (2013).
Conic Programming 1 / 25
Content• Part I : Introduction• Part II: Preliminary and Convex Cone Structure• Part III: Duality Theory of Linear Conic Programming• Part IV: Interior Point Methods and Solution Software• Part V: Modelling and Applications• Part VI: Recent Research• Part VII: Practical LCoP
Conic Programming 2 / 25
LCoP Part I – Introduction
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 3 / 25
Introduction
Content• Linear Conic Programs• Applications• Duality Theory and Algorithms• References
Conic Programming 4 / 25
Linear Conic Program
Min C •Xs.t. A •X = B
X ∈ K(LCoP)
where K is a closed, convex cone; A, B and C are in the space ofinterests with • being an appropriate linear operator.
Conic Programming 5 / 25
K = Rn+ (First Orthant)
When K = Rn+ = {x ∈ Rn|xi ≥ 0, i = 1, ..., n}, LCoP becomes LP.
Min cTxs.t. Ax = b
x ∈ Rn+
(LP)
where A ∈ Rm×n, b ∈ Rm and c ∈ Rn.
Equivalently,
Min cTxs.t. aTi x = bi, i = 1, . . . ,m
x ≥Rn+
0(LP)
where ai ∈ Rn, bi ∈ R and c ∈ Rn.
Conic Programming 6 / 25
K = Rn+
Figure:R1+ Figure:R2
+ Figure:R3+
Conic Programming 7 / 25
K = Ln (Lorentz Cone/Second OrderCone)
When K = Ln = {x ∈ Rn|√x21 + · · ·+ x2n−1 ≤ xn}, LCoP becomes
SOCP.Min cTxs.t. Ax = b
x ∈ Ln(SOCP)
where A ∈ Rm×n, b ∈ Rm and c ∈ Rn.
Equivalently,
Min cTxs.t. aTi x = bi, i = 1, . . . ,m
x ≥Ln 0(SOCP)
where ai ∈ Rn, bi ∈ R and c ∈ Rn.
Conic Programming 8 / 25
K = Ln
Figure: L2 Figure: L3
Conic Programming 9 / 25
K = Sn+ (Positive Semidefinite Cone)
When K = Sn+ = {X ∈ Rn×n|X = XT � 0}, LCoP becomes SDP.
Min C •Xs.t. Ai •X = bi, i = 1, ...,m
X ∈ Sn+(SDP)
where C,A1, ..., Am are given n× n symmetric matrices and b1, ..., bm aregiven scalars, and
M •X =∑i,j
MijXij = tr(MTX).
Equivalently,
Min C •Xs.t. Ai •X = bi, i = 1, ...,m
X � 0(SDP)
Conic Programming 10 / 25
K = Sn+
S2+ =
{(x, y, z) ∈ R3|
[x yy z
]� 0.
}⇐⇒ x ≥ 0, z ≥ 0, xz ≥ y2.
Conic Programming 11 / 25
Application of SOCP – I
Torricelli Point ProblemThe problem is proposed by Pierre de Fermat in 17th century. Giventhree points a, b and c on the R2 plane, find the point in the plane thatminimizes the total distance to the three given points. The solutionmethod was found by Torricelli, hence know as Torricelli point.
SOCP Formulation
Min t1 + t2 + t3
s.t.
[x− at1
]∈ L3,
[x− bt2
]∈ L3,
[x− ct3
]∈ L3
Question:C =? A =? X =? · · ·
Conic Programming 12 / 25
Application of SOCP – II
Robust Portfolio DesignAssume returns r are known within an ellipsoid
E = {r = r + κΣ1/2u : ‖u‖2 ≤ 1}.
where r is the expected return, Σ is the empirical covariance matrix,0 < κ < 1 is a given constant.
robust counterpart: (optimize the worst case)
maxω
minr∈E{rTω : eTω = 1, ω ≥ 0}.
Conic Programming 13 / 25
SOCP Formulation
Notice thatminr∈E
rTω
= min‖u‖2≤1
{rTω + κuT Σ1/2ω}
= rTω − κ‖Σ1/2ω‖2
Robust portfolio problem is an SOCP
Max rTω − κ‖Σ1/2ω‖2s.t. eTω = 1, ω ≥ 0
⇐⇒
Max ts.t. eTω = 1, ω ≥ 0[
κΣ1/2ωrTω − t
]∈ Ln+1
Question:C =? A =? X =? · · ·
Conic Programming 14 / 25
Application of SDP – I
Correlation Matrix Verification
Consider three random variables A, B and C. By definition, theircorrelation coefficients ρAB , ρAC and ρBC are valid if and only if 1 ρAB ρAC
ρAB 1 ρBC
ρAC ρBC 1
� 0
Suppose we know from some prior knowledge (e.g. empirical results ofexperiments) that −0.2 ≤ ρAB ≤ −0.1 and 0.4 ≤ ρBC ≤ 0.5. What are thesmallest and largest values that ρAC can take?
Conic Programming 15 / 25
SDP Formulation
The problem can be formulated as the following problem:
Min/Max ρAC
s.t. −0.2 ≤ ρAB ≤ −0.1
0.4 ≤ ρBC ≤ 0.5
ρAA = ρBB = ρCC = 1 ρAA ρAB ρAC
ρAB ρBB ρBC
ρAC ρBC ρCC
� 0
Conic Programming 16 / 25
SDP Formulation
In order to formulate the problem as in standard form, we handle theinequality constraints by augmenting the variable matrix and introducingslack variables, for example
The popularity of SOCP is also due to the fact that it is a generalized formof convex QCQP (Quadratically Constrained Quadratic Programming).Specifically, consider the following QCQP:
Min xTA0x+ 2bT0 x+ c0s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m
where A0 � 0, Ai � 0 for i = 1, . . . ,m.Note that
t ≥n∑
i=1
x2i ⇐⇒
∣∣∣∣∣∣∣∣∣
∣∣∣∣∣∣∣∣∣
x1...xn
(t− 1)/2
∣∣∣∣∣∣∣∣∣
∣∣∣∣∣∣∣∣∣2
≤ t+ 1
2⇐⇒
x1...xn
(t− 1)/2(t+ 1)/2
∈ Ln+2
Conic Programming 19 / 25
Other Applications - QCQP =⇒ SOCP
Therefore, for each i = 1, . . . ,m
xTAix+ 2bTi x+ ci ≤ 0⇐⇒
A1/2i x
−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ln+2
QCQP can be equivalently written as
Min u
s.t.
A1/20 x
−1/2− bT0 x+ u/2− c0/21/2− bT0 x+ u/2− c0/2
∈ Ln+2
A1/2i x
−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ln+2, i = 1, . . . ,m.
Conic Programming 20 / 25
Other Applications - SOCP =⇒ SDP
Ln+1 can be easily embedded into Sn+1+ by observing the fact that[
xt
]∈ Ln+1 ⇐⇒
[t xT
x tIn
]∈ Sn+1
+
Based on this, we will focus on the theorems and algorithms for SDP. Butthis does not mean that SOCP is useless or we should transform SOCPto SDP in any case.
Conic Programming 21 / 25
Duality Theory of LP
Theorem (Weak Duality Theorem of LP)If x is primal feasible and y is dual feasible, then cTx ≥ bT y.
Theorem (Strong Duality Theorem of LP)
• If either LP or LD has a finite optimal solution, then so does the otherand they achieve the same optimal objective value.
• If either LP or LD has an unbounded objective value, then the otherhas no feasible solution.
How about the duality theorems of LCoP?
Conic Programming 22 / 25
Algorithms for LP
Simplex Method for LP
• Starting from one vertex• Check whether current vertex is optimal or not. If yes, stop.
Otherwise, go to the next step.• Move to a neighboring vertex, go to the previous step.
The complexity of simplex method is not polynomial.
Books• Bertsekas D.P., Nedic A. and Ozdaglar A.E., Convex Analysis and
Optimization, Athena Scientific: Belmont, MA USA 2003• Boyd S. and Vandenberghe L., Convex Optimization, Cambridge University
Press: Cambridge, UK 2004• Fang S. and Puthenpura S., Linear Optimization and Extensions: Theory
and Algorithms, Prentice-Hall Inc.: Englewood Cliffs, NJ USA 1993• Nemirovski A., Lectures on Modern Convex Optimization: Analysis,
Algorithms, and Engineering Applications, Society for Industrial and AppliedMathematics: Philadelphia, PA USA 2001
• Renegar J., A Mathematical View of Interior-point Methods in ConvexOptimization, Society for Industrial and Applied Mathematics: Philadelphia,PA USA 2001
• Handbook of Semidefinite Programming: Theory, Algorithms, andApplications, edited by Wolkowicz H., Saigal R. and Vandenberghe L.,Kluwer Academic Publisher: Norwell, MA USA 2000
Conic Programming 24 / 25
References
• Rockafellar R.T., Convex Analysis, Princeton University Press:Princeton, NJ USA 1970
• Wright. S., Primal-Dual Interior-Point Methods, Society for Industrialand Applied Mathematics: Philadelphia, PA USA 1997
Others
• Ye Y., Linear Conic Programming, lecture notes online: http://www.stanford.edu/class/msande314/sdpmain.pdf
LCoP Part II – Preliminaries and Convex ConeStructures
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 1 / 39
Preliminaries and Convex Cone Structures
Content• Vectors, Matrices, and Spaces• Inner Products and Norms• Open, Closed, Interior, and Boundary Sets• Functions• Linear Systems• Convex Sets and Functions
Conic Programming 2 / 39
Vectors, Matrices and Spaces
• Real numbers: R, R+, R++
• Euclidean space: Rn
• First orthant: Rn+• n-dimensional (column) vector:
• Bounded: a set X ⊂ Rn is bounded if there exist an r > 0 such that
‖x‖ < r,∀x ∈ X
Conic Programming 8 / 39
Functions
• Continuous: f : X ⊂ Rn is continuous at x0 if( i ) x0 ∈ X(i i) limx→x0 f(x) = f(x0)
• Continuous function: f ∈ C0(X ) means f is continuous at all pointsin X ⊂ Rn.
• Gradient: For f : X ⊂ Rn → R
∇f(x) = [∂f(x)
∂x1, · · · , ∂f(x)
∂xn]1×n
• Hessian: For f : X ⊂ Rn → R
F (x) = [∂2f(x)
∂xi∂xj]n×n
• Continuously differentiable function: f ∈ Cp(X ) (p = 1, 2, · · · ) meansf is p-th continuously differentiable over X ⊂ Rn.
Conic Programming 9 / 39
Functions
Theorem (Taylor theorem)Let X be open, f ∈ Cp(X ), x1, x2 ∈ X , x1 6= x2 and
x(θ) = θx1 + (1− θ)x2 ∈ X , ∀ 0 ≤ θ ≤ 1.
Then ∃ x = θx1 + (1− θ)x2 ∈ X , 0 < θ < 1, s.t.
f(x2) = f(x1) +
p−1∑k=1
1
k!dkf(x1;x2 − x1) +
1
p!dpf(x;x2 − x1)
where dkf(x;h) is the k-th order differential of function f along h.
Conic Programming 10 / 39
Functions: Big O and Small o
Let g(·) be a real-valued function on R.
• g(x) = O(m(x))
∃ c ≥ 0 such that ∣∣∣∣ g(x)
m(x)
∣∣∣∣ ≤ c as x→ 0 (or +∞)
• g(x) = o(m(x)) ∣∣∣∣ g(x)
m(x)
∣∣∣∣ = 0 as x→ 0 (or +∞)
Conic Programming 11 / 39
Functions
Taylor theorem in small o formulation:• p = 1
f(x+ h) = f(x) +∇f(x)h+ o(‖h‖)• p = 2
f(x+ h) = f(x) +∇f(x)h+1
2hTF (x)h+ o(‖h‖2)
Conic Programming 12 / 39
Linear Systems
Given x1, · · · , xm ∈ Rn• Linear combination:
m∑i=1
λixi,
where λi ∈ R, i = 1, . . . ,m.• Linearly independent
m∑i=1
λixi = 0⇒ λ1 = · · · = λm = 0
• Affine combination: a linear combination withm∑i=1
λi = 1
• Affinely independent: if x2 − x1, · · · , xm − x1 are linearlyindependent.
Conic Programming 13 / 39
Linear Systems
• Convex combination: a linear combination withm∑i=1
λi = 1 and λi ≥ 0, i = 1, . . . ,m
• Hyperplane:
X = {x ∈ Rn|aTx =
n∑i=1
aixi = b}
• Affine space: affine combination of any two points in the space is stillin the space. (An intersection of finitely many hyperplanes.)
• Linear subspace: an affine space containing the origin.
We can always transform an affine space Y ⊂ Rn into a linearsubspace X ⊂ Rn by choosing x0 ∈ Y such that
X = {x− x0|x ∈ Y}
Conic Programming 14 / 39
Linear Systems
• Half space:
X = {x ∈ Rn|aTx =
n∑i=1
aixi ≤ b}
• Polyhedron: an intersection of finitely many half spaces.
• Polytope: a bounded polyhedron
• Dimension of a linear subspace: the maximum number of linearlyindependent vectors in the subspace.
• Dimension of an affine space: the dimension of the transformedlinear subspace.
• Dimension of a polyhedron: the dimension of the smallest affinespace containing it.
Conic Programming 15 / 39
Linear Systems
• Linear equations
a1 • x = b1a2 • x = b2· · · · · · · · ·
am • x = bm
⇒ Ax = b,
where a1, · · · , am and x are all in Rn.
A1 •X = b1A2 •X = b2· · · · · · · · ·
Am •X = bm
⇒ AX = b,
where A1, · · · , Am and X are all in Sn.• For convenience, A∗y =
∑mi=1 yiAi.
Conic Programming 16 / 39
Convex Sets and Properties
• A set X ⊂ Rn is convex if for any x1 ∈ X and x2 ∈ X , we haveλx1 + (1− λ)x2 ∈ X , for all 0 ≤ λ ≤ 1.
• Convex hull: the smallest convex set containing a given set
conv(X ) = {x ∈ Rn|x =∑mi=1 λiy
i for some m ∈ N+,λi ≥ 0,
∑mi=1 λi = 1, and yi ∈ X , i = 1, . . . ,m}
• Dimension of a convex set: the dimension of the smallest affinespace containing it.
• Relative interior of a convex set X ⊂ Rn: suppose H is the smallestaffine space containing X ,
ri(X ) = {x ∈ Rn|∃ open set Y ⊆ Rn such that x ∈ Y ∩H ⊂ X}
• Supporting hyperplane H = {x ∈ Rn|aTx = b} of a convex set X :
aT y ≥ b,∀ y ∈ X and X ∩H 6= ∅.
Conic Programming 17 / 39
Convex Functions and Properties
• Epigraph of a function f : X ⊂ Rn → R
epif = {(x, λ) ∈ Rn+1|λ ≥ f(x), x ∈ X}
• Closed function: if epif is a closed set.
• Convex function: if epif is a convex set.
• Concave function: if −f is a convex function.
• Convex hull function conv(f) of a function f : X ⊂ Rn → R is afunction on X such that epi(conv(f)) = conv(epi(f)).
Lemmaf : X ⊂ Rn → R is a convex function if and only if for any x1, x2 ∈ X and0 ≤ λ ≤ 1, we have
f(λx1 + (1− λ)x2) ≤ λf(x1) + (1− λ)f(x2).
Conic Programming 18 / 39
Convex Functions and Properties
• Subgradient d ∈ Rn of a convex function f : X ⊂ Rn at x ∈ X :
if for any y ∈ X ,f(y) ≥ f(x) + dT (y − x)
• The set {(y, λ) ∈ Rn+1|λ− dT y = f(x)− dTx} is a supportinghyperplane of epif at x.
• Subdifferential of a convex function f : X ⊂ Rn at x ∈ X :
∂f(x) = {d ∈ Rn|d is a subgradient of f at x}
Conic Programming 19 / 39
Convex Functions and Properties
Figure: (x, f(x))↔ (m, b) or (y, h(y))
Conic Programming 20 / 39
Convex Functions and Properties
• Conjugate (transform) of f : X ⊂ Rn → R:
h(y) = supx∈X{y • x− f(x)}
with h being defined on Y = {y ∈ Rn|h(y) < +∞}.
Lemmah : Y is a closed, convex function.
Lemma (Fenchel’s inequality)Given f : X and its conjugate h : Y, then
x • y ≤ f(x) + h(y), ∀ x ∈ X and y ∈ Y.
Moreover,x • y = f(x) + h(y) ⇐⇒ y ∈ ∂f(x)
Conic Programming 21 / 39
Conjugate Functions and Properties
Let f : X ⊂ Rn → R be a function with its conjugate transform h : Y.• For α ∈ R, the conjugate of f + α is h− α.• For a ∈ Rn, the conjugate of f(x) = f(x) + x • a on X ish(y) = h(y − a), ∀ y ∈ Y.
• For a ∈ Rn, the conjugate of f(x) = f(x− a) on X ish(y) = h(y) + y • a, ∀ y ∈ Y.
• For λ > 0, the conjugate of f1(x) = λf(x) on X is h1(y) = λh( yλ ),∀ y ∈ λY.
• For λ > 0, the conjugate of f2(x) = f(xλ ) on λX is h2(y) = h(λy),∀ y ∈ Y/λ.
TheoremAssume that f1 : X and f2 : X have the same convex hull function. Thenthey have the same conjugate transform h : Y when it exists.
Conic Programming 22 / 39
Conjugate Functions and Properties
We know the dual problem of LD is LP again. When will the conjugatetransform of h : Y become f : X?
Proper functionA convex function f is proper if its epigraph is non-empty and contains novertical lines, i.e. if f(x) < +∞ for at least one x and f(x) > −∞ forevery x.
TheoremLet f : X ⊂ Rn → R be a proper closed convex function with conjugatetransform h : Y. Then the conjugate transform of h : Y is f : X . Moreover,y ∈ ∂f(x) if and only if x ∈ ∂h(y). In this case,
x • y = f(x) + h(y) ⇐⇒ y ∈ ∂f(x) or x ∈ ∂h(y)
Conic Programming 23 / 39
Convex Cone Structure
Content• Convex Cones and Properties• Partial Order and Ordered Vector Space• Some Examples
Conic Programming 24 / 39
Convex Cones and Properties
• A set K ⊂ Rn is a cone if
∀x ∈ K and λ > 0⇒ λx ∈ K;
• A cone K ⊂ Rn is pointed if
K ∩ −K = {0};
• A cone K ⊂ Rn is solid ifintK 6= ∅;
• A cone K ⊂ Rn is proper if it is pointed, solid, closed and convex.
Conic Programming 25 / 39
Convex Cones and Properties
• Conic combination: a linear combination∑mi=1 λix
i with λi ≥ 0,xi ∈ Rn for all i = 1, . . . ,m.
• The conic hull of a set X ⊂ Rn is
cone(X ) = {x ∈ Rn|x =∑mi=1 λix
i, for some m ∈ N+
and xi ∈ X , λi ≥ 0, i = 1, . . . ,m.}
• The dual cone K∗ ⊂ Rn of a cone K ⊂ Rn is
K∗ = {y ∈ Rn|y • x ≥ 0,∀ x ∈ K}
K∗ is a closed, convex cone.• If K∗ = K, then K is a self-dual cone.
Conic Programming 26 / 39
Convex Cones and Properties
K, K1, K2 are convex cones in Rn.• (K∗)∗ = cl(K).• K1 ⊆ K2 ⇒ K∗2 ⊆ K∗1 .• K1 ∩K2, K1 ∪K2, K1 +K2, K1 ×K2 are all cones.• (K1 +K2)∗ = K∗1 ∩K∗2 .• K1, K2 closed⇒ K1 +K2 and K1 ×K2 are closed.• ri(K1 +K2) = ri(K1) + ri(K2).• ri(K1 ×K2) = ri(K1)× ri(K2).• The supporting hyperplane of K always contains the origin• If K is solid, then K∗ is pointed.• If K is pointed, then K∗ is solid.• If K is proper, then K∗ is proper.
Examples: Cones of NonnegativeQuadratic Functions — Nonhomogeneous
• Nonnegative quadratic functions over F ⊂ Rn
f(x) = xTAx+ 2bTx+ c ≥ 0,∀x ∈ F
f ⇔[c bT
b A
]• DF = {
[c bT
b A
]∈ Sn+1|
[1x
]T [c bT
b A
] [1x
]≥ 0,∀x ∈ F} is a closed,
convex cone.
• D∗F = cl(cone{[
1 xT
x xxT
]|x ∈ F})
• (D∗F )∗ = DF and (DF )∗ = D∗F
Conic Programming 38 / 39
Examples: Cones of NonnegativeQuadratic Functions — Nonhomogeneous
• Examples:• F = Rn
DF = D∗F = Sn+1+
• F = Rn+
DF = Cn+1 and D∗F = C∗n+1
Conic Programming 39 / 39
LCoP Part III – Duality Theory of Linear ConicProgramming
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 1 / 28
Duality Theory of Linear ConicProgramming
Content• Definition of LCoP and LCoD• Conjugate Duality Theory• Deriving LCoD from LCoP• Conic Duality Theorems for LCoP• Duality Theorems of LP, SOCP and SDP
Conic Programming 2 / 28
Linear Conic Programs
Recall thatMin C •Xs.t. A •X = B
X ∈ K(LCoP)
where K is a closed, convex cone, A, C and B are in the space ofinterests with • being an appropriate linear operator.
Note that when K = Rn+ or Ln, X is a vector; when K = Sn+, X is an
n× n matrix.
Conic Programming 3 / 28
Linear Conic Programs
Min c • xs.t. ai • x = bi, i = 1, . . . ,m
x ∈ K(LCoP)
where K is a closed, convex cone, such as Rn+, Ln, Sn+.
Max bT ys.t.
∑mi=1 yia
i + s = cs ∈ K∗, y ∈ Rm
(LCoD)
where K∗ is the dual cone of K.
Conic Programming 4 / 28
Conjugate Duality Theory
Conjugate Program
inf f(x)s.t. x ∈ X ∩K
(CP)
where f : X ⊂ Rn → R and K is a cone in Rn.
Conjugate Dual
inf h(y)s.t. y ∈ Y ∩K∗
(CD)
where h : Y is the conjugate transform of f : X and K∗ is the dual coneof K.• feas(*) denotes the feasible domain of problem (*)• opt(*) denotes the optimal solution set of problem (*)• v(*) denotes the optimal value of problem (*)
Conic Programming 5 / 28
Conjugate Duality Theory
Theorem (Conjugate duality theorem/KKT duality theorem)If x ∈ feas(CP) and y ∈ feas(CD), then
0 ≤ x • y ≤ f(x) + h(y)
with the equality holding if and only if
x • y = 0 and y ∈ ∂f(x),
in which casex ∈ opt(CP) and y ∈ opt(CD).
ProofThe inequality follows from Fenchel’s inequality and the definition of dualcone. The rest follows easily.
Conic Programming 6 / 28
Conjugate Duality Theory
Theorem (Weak duality theorem)If both CP and CD are feasible, then
(i) v(CP) is finite and
v(CP) + h(y) ≥ 0,∀ y ∈ feas(CD);
(ii) v(CD) is finite and
v(CP) + v(CD) ≥ 0.
ProofThis theorem follows from the previous KKT duality theorem.
Conic Programming 7 / 28
Conjugate Duality Theory
Theorem (Fenchel’s theorem/Strong duality theorem)Suppose that f : X and K are closed and convex. If v(CD) is finite andone of the following conditions holds:
(i) ri(K∗) ∩ ri(Y) 6= ∅,(ii) both K∗ and Y are polyhedrons,
thenv(CP) + v(CD) = 0 and opt(CP) 6= ∅.
Similarly, if v(CP) is finite and one of the following conditions holds:(i) ri(K) ∩ ri(X ) 6= ∅,(ii) both K and X are polyhedrons,
thenv(CP) + v(CD) = 0 and opt(CD) 6= ∅.
Proof: See Rockafellar’s book “Convex Analysis” Section 31.
Conic Programming 8 / 28
Deriving LCoD from LCoP
LCoP
Min c • xs.t. ai • x = bi, i = 1, . . . ,m
x ∈ K(LCoP)
Deriving LCoD in the framework of conjugate program.
Conic Programming 9 / 28
Deriving LCoD from LCoP
LCoP as CP
Variables: uT = (u0, u1, . . . , um) ∈ Rm+1;
f(u) = u0;
X = {u ∈ Rm+1|ui = bi, i = 1, . . . ,m};
K0 = {u ∈ Rm+1|u0 = c • x, ui = ai • x, x ∈ K, i = 1, . . . ,m}.
inf f(u)s.t. u ∈ X ∩K0
Conic Programming 10 / 28
Deriving LCoD from LCoP
Corresponding CD
Variables: vT = (v0, v1, . . . , vm) ∈ Rm+1;
h(v) = supu∈X{u • v − f(u)} < +∞
= supu0∈R{(v0 − 1)u0 +
m∑i=1
bivi}
Hence
h(v) =∑m
i=1 bivi;
Y = {v ∈ Rm+1|v0 = 1};
Conic Programming 11 / 28
Deriving LCoD from LCoP
Corresponding CD
Moreover,
K∗0 = {v ∈ Rm+1|v • u ≥ 0,∀u ∈ K0}
= {v ∈ Rm+1|(v0c +∑m
i=1 viai) • x ≥ 0,∀x ∈ K}
= {v ∈ Rm+1|v0c +∑m
i=1 viai ∈ K∗}.
Hence
Y ∩K∗0 = {v ∈ Rm+1|c +
m∑i=1
viai = s, s ∈ K∗}.
inf∑m
i=1 bivis.t. c +
∑mi=1 via
i = ss ∈ K∗
Conic Programming 12 / 28
Deriving LCoD from LCoP
CD to LCoDDefine variables: y = −(v1, . . . , vm)T , we have
Max bT ys.t.
∑mi=1 yia
i + s = cs ∈ K∗, y ∈ Rm
(LCoD)
Therefore, the duality theorems of conjugate programs may apply toLCoP.
Conic Programming 13 / 28
Conic Duality Theorems for LCoP
Theorem (Weak duality theorem)If both LCoP and LCoD are feasible, then
c • x ≥ bT y,∀x ∈ feas(LCoP) and (y, s) ∈ feas(LCoD).
Theorem (Strong duality theorem)
(i) If feas(LCoP) ∩ int(K) 6= ∅ and v(LCoP) is finite, then there exists(y∗, s∗) ∈ feas(LCoD) such that bT y∗ = v(LCoP).
(ii) If feas(LCoD) ∩ int(K∗) 6= ∅ and v(LCoD) is finite, then there existsx∗ ∈ feas(LCoP) such that c • x = v(LCoD).
Proof: See Aharon Ben-Tal and Arkadi Nemirovski’s book “Lectures onmodern convex optimization” Chapter 2.
Conic Programming 14 / 28
Conic Duality Theorems for LCoP
Theorem (KKT duality theorem)If feas(LCoP) and feas(LCoD) are both nonempty andfeas(LCoP) ∩ int(K) 6= ∅, then x∗ is optimal for LCoP if and only if thefollowing conditions hold:
(i) x∗ ∈ feas(LCoP);
(ii) There exists (y∗, s∗) ∈ feas(LCoD);
(iii) c • x∗ = bT y∗ (or equivalently x∗ • s∗ = c • x∗ − bT y∗ = 0).Proof: =⇒ follows from strong duality theorem.
⇐= is obvious.
Conic Programming 15 / 28
Linear Program (LP)
Min cTxs.t. Ax = b
x ≥Rn+
0(LP)
Max bT ys.t. AT y + s = c
s ≥Rn+
0(LD)
Conic Programming 16 / 28
Linear Program (LP)
Theorem (LP duality theorem)
(i) If either LP or LD is unbounded, then the other one is infeasible.
(ii) If either v(LP) or v(LD) is finite, then there exist x∗ ∈ feas(LP) and(y∗, s∗) ∈ feas(LD) such that v(LP) = cTx∗ = bT y∗ = v(LD).
(iii) If LP is feasible and v(LP) is finite, then x∗ is optimal for LP if andonly if the following conditions hold:
(a) Ax∗ = b, x∗ ≥Rn+
0;
(b) there exists (y∗, s∗) satisfying AT y∗ + s∗ = c, s ≥Rn+
0;
(c) (x∗)T s∗ = cTx∗ − bT y∗ = 0.
Conic Programming 17 / 28
Second Order Cone Program (SOCP)
Min cTxs.t. Ax = b
x ≥K 0(SOCP)
where K = Ln1 × · · · × Lnr = {x ∈ Rn|n1 + · · ·+ nr = n, (x1, ..., xn1)T ∈
Ln1 , ..., (xn−nr+1, ..., xn)T ∈ Lnr}.
Max bT ys.t. AT y + s = c
s ≥K 0(SOCD)
Conic Programming 18 / 28
Second Order Cone Program (SOCP)
Theorem (SOCP duality theorem)
(i) If either SOCP or SOCD is unbounded, then the other one isinfeasible.
(ii) If there exists a feasible solution x such that x ∈ int(K), andv(SOCP) is finite, then there exist (y∗, s∗) ∈ feas(SOCD) such thatv(SOCP) = bT y∗ = v(SOCD).
(iii) If there exists a feasible solution (y, s) such that s ∈ int(K), andv(SOCD) is finite, then there exist x∗ ∈ feas(SOCP) such thatv(SOCP) = cTx∗ = v(SOCD).
Conic Programming 19 / 28
Second Order Cone Program (SOCP)
Theorem (SOCP duality theorem)
(iv) If both SOCP and SOCD are feasible, and there exists a feasiblesolution x such that x ∈ int(K), then x∗ is optimal for SOCP if andonly if the following conditions hold:
(a) Ax∗ = b, x∗ ≥K 0;
(b) there exists (y∗, s∗) satisfying AT y∗ + s∗ = c, s∗ ≥K 0;
(c) (x∗)T s∗ = cTx∗ − bT y∗ = 0.
Conic Programming 20 / 28
Difference between LP and SOCP (interior feasible solution):
Min −x2
s.t. x1 − x3 = 0x ∈ L3
Max 0 · y
s.t.
0−10
− y
10−1
=
−y−1y
∈ L3
v(SOCP ) = 0 but SOCD is infeasible.
Figure: Feasible domain is a ray x1 = x3 in hyperplane x2 = 0. No feasibleinterior point.
Conic Programming 21 / 28
Second Order Cone Program (SOCP)
Finite nonzero duality gap:
Min −x2
s.t. x1 + x3 − x4 + x5 = 0x2 + x4 = 1x ∈ L3 × L2
Max y2
s.t.
y1 +s1 = 0y2 +s2 = −1
y1 +s3 = 0−y1 +y2 +s4 = 0y1 +s5 = 0
s ∈ L3 × L2
x∗ =
−101
× [11
]y∗ =
[−1−1
]s∗ =
101
× [01
]
v(SOCP) = 0 6= −1 = v(SOCD)
Conic Programming 22 / 28
Second Order Cone Program (SOCP)
Zero duality gap with non-attainable value:
Min x1
s.t. −x2 −x3 = 0x2 = −1
x ∈ L3
Max −y2s.t. s1 = 1
−y1 +y2 +s2 = 0−y1 +s3 = 0
s ∈ L3
x∗ =
0−11
v(SOCD) = 0 but not attainable.
Conic Programming 23 / 28
Semidefinite Program (SDP)
Min C •Xs.t. AX = b
X � 0(SDP)
Max bT ys.t. A∗y + S = C
S � 0(SDD)
Note:
A∗y =
m∑i=1
yiAi
Conic Programming 24 / 28
Semidefinite Program (SDP)
Theorem (SDP duality theorem)
(i) If either SDP or SDD is unbounded, then the other one is infeasible.
(ii) If there exists a feasible solution X such that X � 0, and v(SDP) isfinite, then there exist (y∗, S∗) ∈ feas(SDD) such thatv(SDP) = bT y∗ = v(SDD).
(iii) If there exists a feasible solution (y, S) such that S � 0, and v(SDD)is finite, then there exist X∗ ∈ feas(SDP) such thatv(SDP) = C •X∗ = v(SDD).
Conic Programming 25 / 28
Semidefinite Program (SDP)
Theorem (SDP duality theorem)
(iv) If both SDP and SDD are feasible, and there exists a feasiblesolution X such that X � 0, then X∗ is optimal for SDP if and only ifthe following conditions hold:
v(SDP ) = 0 but is not attainable. y∗ = 0 and S∗ =
[1 00 0
].
Conic Programming 27 / 28
Semidefinite Program (SDP)
Finite nonzero duality gap:
C =
0 0 00 0 10 1 0
, A1 =
0 0 00 1 00 0 0
, A2 =
1 0 00 0 −10 −1 0
, b =
[01
]
X∗ =
1 0 00 0 00 0 0
, y∗ =
[0−1
], S∗ =
1 0 00 0 00 0 0
v(SDP ) = 0 6= −1 = v(SDD)
Conic Programming 28 / 28
LCoP Part IV – Interior Point Methods andSolution Software
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 1 / 39
Interior Point Methods and SolutionSoftware
Content• Interior Points and Primal-Dual Model• Barrier Functions and Optimal Systems• Central Path and Newton Methods• Path Following Method• CVX Solution Software
Conic Programming 2 / 39
Interior Point Methods
• Interior point approach• Start from an interior point solution.• If the current solution is not good enough, then move to another interior
point solution.• Stop at an interior point solution whose objective value is close to the
optimum (within an ε gap).• Advantages:
• Polynomial time complexity (comparing to the simplex method for LP)• Excellent computational performance in practice (comparing to the
ellipsoid method)• Three types: primal; dual; primal-dual
Conic Programming 3 / 39
Primal-dual Model
• Primal-dual type of LP
Min sTxs.t. Ax = b
AT y + s = cx ≥Rn
+0, s ≥Rn
+0
(LPD)
• Primal-dual type of SDP
Min S •Xs.t. AX = b
A∗y + S = CX � 0, S � 0
(SDPD)
• Note:AX = [A1 •X, · · · , Am •X]T
and A∗y =∑mi=1 yiAi
Conic Programming 4 / 39
Interior Points
feas+(LP) = {x ∈ Rn|Ax = b, x >Rn+
0}feas+(LD) = {(y, s) ∈ Rm ×Rn|AT y + s = c, s >Rn
Linear transformation: Given an invertible matrix L ∈ Rn×n, letA = (A1, . . . , Am), Ai = LTAiL for i = 1, . . . ,m.X0 = L−1X0L−T , S0 = LTS0L, C = LTCL.
A4X = 0
A∗dy + 4S = 0
4XS0 + X04S = γµ0I − X0S0
X0 +4X � 0, S0 +4S � 0
• L = (X0)1/2: X0 = I ⇒ X0 +4X � 0, ∀‖4X‖F < 1 (Primal)• L = (S0)−1/2: S0 = I ⇒ S0 +4S � 0, ∀‖4S‖F < 1 (Dual)• LLT = (S0)−1/2[(S0)1/2X0(S0)1/2]1/2(S0)−1/2:V 0 = X0 = S0 (Primal-dual)
• Short Step Algorithm• Long Step Algorithm• Predictor-Corrector Algorithm• Largest Step Algorithm
Reference: Handbook of Semidefinite Programming: Theory, Algorithms,and Applications, edited by Wolkowicz H., Saigal R. and VandenbergheL., Kluwer Academic Publisher: Norwell, MA USA 2000
Conic Programming 35 / 39
CVX Solution Software - Example 1
Conic Programming 36 / 39
CVX Solution Software - Example 2
Conic Programming 37 / 39
CVX Solution Software - Example 3
Conic Programming 38 / 39
CVX Solution Software - Example 4
Conic Programming 39 / 39
LCoP Part V – Modeling and Applications
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
Conic Programming 1 / 34
Modeling and Applications
Content• Weber Problem• Matrix Optimization• Approximating Solutions of Linear Equations• Minimum of a Univariate Polynomial of Degree 2n
• Stochastic Queue Location Problem• Conic Reformulations of QCQP and Extensions• Robust Optimization
Conic Programming 2 / 34
Weber Problem
In 1909, the German economist Alfred Weber introduced the problem offinding a best location for the warehouse of a company, such that the totaltransportation cost to serve the customers is minimum. Suppose thatthere are m customers needing to be served. Let the location ofcustomer i be ai ∈ R2, i = 1, . . . ,m. Suppose that customer may havedifferent demands, to be translated as weight ωi for customer i,i = 1, . . . ,m. Denote the desired location of the warehouse to be x.Then, the optimization problem becomes
Minm∑i=1
ωiti
s.t.
[x− aiti
]∈ L3, i = 1, . . . ,m
Conic Programming 3 / 34
Matrix Optimization
Given A0, A1, . . . , Am, determine if there is y ∈ Rm such that
Consider the problem of finding the minimum of a univariate polynomialof degree 2n:
Min x2n + a1x2n−1 + · · ·+ a2n−1x+ a2n
s.t. x ∈ R
This problem is equivalent to
Max ts.t. x2n + a1x
2n−1 + · · ·+ a2n−1x+ a2n − t ≥ 0 for all x ∈ R
Conic Programming 7 / 34
Minimum of a Univariate Polynomial
It is well known that a univariate polynomial is nonnegative over the realdomain if and only if it can be written as sum of squares (SOS), which isequivalent to saying that there must be a positive semidefinite matrixX ∈ Sn+1
BackgroundSuppose there are m potential customers to serve in the region.Customers’ demands are random, and once a customer calls for service,then the server in the service center will need to go to the customer toprovide the required service. In case the server is occupied, then thecustomer will have to wait. The goal is to find a good location for theservice center in order to minimize the expected waiting time of service.
Conic Programming 9 / 34
Stochastic Queue Location Problem
Assumptions and notationsSuppose that the service calls from the customer are identicallyindependent distributed, and the demand process follows the Poissondistribution with overall arrival rate λ, and the probability that any servicecall is from customer i is assumed to be pi for i = 1, . . . ,m. The queueingprinciple is First Come First Service, and there is only one server in theservice center. This model can be regarded as M/G/1 queue as inQueueing theory, and the expected service time, including waiting timeand traveling, can be explicitly computed. To this end, denote the velocityof the server to be v, and the location of customer i is ai, i = 1, . . . ,m,and the location of the service center to be x.
Conic Programming 10 / 34
Stochastic Queue Location Problem
Problem formulationThe expected waiting time for customer i is given by
ωi(x) =
(2λ/v2)m∑j=1
pj‖x− aj‖22
1− (2λ/v)m∑j=1
pj‖x− aj‖22+
1
v‖x− ai‖2,
where the first term in the expected term is the expected waiting time forthe server to be free and the second the term is the waiting time for theserver to travel after his departure at the service center.
Convex QCQP: A0, . . . , Am � 0Assume rank(Ai) = di, then Ai = BTi Bi, Bi ∈ Rdi×n, i = 0, . . . ,m.
xTAix+ 2bTi x+ ci ≤ 0 ⇔
Bix−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ldi+2
Reformulating Convex QCQP as SOCP:
min u
s.t.
B0x−1/2− bT0 x+ u/2− ci/21/2− bT0 x+ u/2− ci/2
∈ Ld0+2
Bix−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ldi+2 i = 1, . . . ,m
Conic Programming 14 / 34
SDP Relaxation for QCQP
Recall that
xTAix+ 2bTi x+ ci =
[1x
]T [ci bTibi Ai
] [1x
]=
[ci bTibi Ai
]•[
1 xT
x xxT
]SDP Relaxation
min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0, i = 1, . . . ,m
Y11 = 1Y � 0
Conic Programming 15 / 34
SDP Relaxation for HQCQP
Recall thatxTAix = Ai • xxT
SDP Relaxation
min A0 •Xs.t. Ai •X ≤ −ci, i = 1, . . . ,m
X � 0
Conic Programming 16 / 34
Exact Solutions from SDP Relaxation
TheoremWhen m = 1, if the optimal solution of the SDP relaxation for QCQPexists, then there exists a rank one optimal solution Y ∗ of the SDP
relaxation for QCQP. Furthermore, let Y ∗ =
[1 (x∗)T
x∗ x∗(x∗)T
], then x∗ is
optimal for QCQP.
Conic Programming 17 / 34
Exact Solutions from SDP Relaxation
LemmaLet G be a symmetric matrix and X be a positive semidefinite matrix withrank d. Suppose that G •X ≤ (=)0. Then there exists a rank-onedecomposition X =
∑di=1 xix
Ti such that xTi Gxi ≤ (=)0, i = 1, . . . , d.
TheoremWhen m = 2, suppose there exists x such that xTAix+ 2bTi x+ ci < 0,i = 1, 2, and there exists y1 ≥ 0, y2 ≥ 0 such that[c0 bT0b0 A0
]+ y1
[c1 bT1b1 A1
]+ y2
[c2 bT2b2 A2
]� 0. If, for the optimal solution Y ∗,
either[c1 bT1b1 A1
]• Y ∗ < 0 or
[c2 bT2b2 A2
]• Y ∗ < 0, then there is no gap
between QCQP and its SDP relaxation and at least one optimal solutioncan be obtained from the rank-one decomposition of Y ∗.
Conic Programming 18 / 34
Exact Solutions from SDP Relaxation
Theorem(HQCQP)When m = 2, suppose there exists x, such that xTAix+ ci < 0, i = 1, 2,and there exists y1 ≥ 0 and y2 ≥ 0 such that A0 + y1A1 + y2A2 � 0. Thenthere is no gap between HQCQP and its SDP relaxation and at least oneoptimal solution can be obtained from the rank-one decomposition of X∗.
Conic Programming 19 / 34
One Extension of QCQP
Ext-QCQP
min xTA0x+ 2bT0 x+ c0
s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m,(x21, . . . , x
2n)T ∈ X
(Ext-QCQP)
where X is a closed convex set.SDP Relaxation:
min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0, i = 1, . . . ,m
(Y22, Y33, . . . , Yn+1n+1)T ∈ XY11 = 1Y � 0
Conic Programming 20 / 34
Exact Solutions from SDP Relaxation
TheoremSuppose there is σ ∈ {−1, 1}n+1, such that all the off-diagonal elements
of Λσ
[ci bTibi Ai
]Λσ, 0 ≤ i ≤ m, are nonpositive. If Y ∗ is an optimal solution
for the SDP relaxation for Ext-QCQP, then
x∗ = σ1(σ2√Y ∗22, σ3
√Y ∗33, . . . , σn+1
√Yn+1n+1)T
is optimal for Ext-QCQP.
Conic Programming 21 / 34
More Results on Ext-QCQP
A Special Case of Ext-QCQP
min xTAx
s.t. (x21, . . . , x2n)T ∈ X
where X is a closed convex set.SDP Relaxation:
min A •Xs.t. (X11, X22, . . . , Xnn)T ∈ X
X � 0
Conic Programming 22 / 34
Approximation by Randomized Algorithm
• Suppose Z � 0, Zii = 1, i = 1, . . . , n. If ξ ∼ N(0, Z) andσ ∈ {−1, 1}n with
σi =
{1 ξi ≥ 0−1 ξi < 0
then E(σσT ) = 2π arcsinZ.
• Suppose Z � 0, Zii = 1, i = 1, . . . , n. Then arcsinZ � Z � 0.• 2
π arcsin t ≤ 1− α+ αt, ∀t ∈ [−1, 1] with α = 0.87856 . . ..
Conic Programming 23 / 34
Approximation by Randomized Algorithm
Given X � 0, let Z = D+XD+ + D, where D, D+, D are diagonalmatrices defined by
Dii =√Xii, D+
ii =
{(√Xii)
−1 Xii 6= 00 Xii = 0
, Dii =
{0 Xii 6= 01 Xii = 0
• X = DZD
• E(A • (DσσTD)) = A • (DE(σσT )D) = A • ( 2πD arcsin(Z)D).
Conic Programming 24 / 34
Approximation by Randomized Algorithm
TheoremSuppose A � 0, and X∗ is the minimizer of SDP relaxation. Let Z, D,D+, and D be define from X∗, and ξ ∼ N(0, Z) with corresponding σ.Then
E(A • (DσσTD)) ≤ 2
πmin{xTAx|(x21, . . . , x2n)T ∈ X}
Proof:E(A • (DσσTD))
= A • ( 2πD arcsin(Z)D)
≤ A • ( 2πDZD)
= 2πA •X
∗
≤ 2π min{xTAx|(x21, . . . , x2n)T ∈ X}
Conic Programming 25 / 34
Approximation by Randomized Algorithm
TheoremAssume that A � 0, Aij ≥ 0 for all i 6= j, and X∗ is the minimizer of SDPrelaxation. Let Z, D, D+ and D be defined from X∗ and ξ ∼ N(0, Z) withcorresponding σ. Then
Convex QCQP: A0, . . . , Am � 0Assume rank(Ai) = di, then Ai = BTi Bi, Bi ∈ Rdi×n, i = 0, . . . ,m.
xTAix+ 2bTi x+ ci ≤ 0 ⇔
Bix−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ldi+2
Reformulating Convex QCQP as SOCP:
Min u
s.t.
B0x−1/2− bT0 x+ u/2− c0/21/2− bT0 x+ u/2− c0/2
∈ Ld0+2
Bix−1/2− bTi x− ci/21/2− bTi x− ci/2
∈ Ldi+2 i = 1, . . . ,m
Conic Programming 4 / 46
SDP Relaxation for QCQP
Recall that
xTAix+ 2bTi x+ ci =
[1x
]T [ci bTibi Ai
] [1x
]=
[ci bTibi Ai
]•[
1 xT
x xxT
]
Let Y =
[1x
] [1x
]T=
[1 xT
x xxT
]∈ Sn+1
+ , we have SDP Relaxation
Min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0, i = 1, . . . ,m
Y11 = 1
Y � 0, rank(Y ) = 1
Note: No convexity is assumed.
Conic Programming 5 / 46
SDP Relaxation for HQCQP
Recall thatxTAix = Ai • xxT
Let X = xxT ∈ Sn+, we have SDP Relaxation
Min A0 •Xs.t. Ai •X ≤ −ci, i = 1, . . . ,m
X � 0, rank(X) = 1
Conic Programming 6 / 46
Exact Solutions from SDP Relaxation
TheoremWhen m = 1, if the optimal solution of the SDP relaxation for QCQPexists, then there exists a rank one optimal solution Y ∗ of the SDP
relaxation for QCQP, and, furthermore, let Y ∗ =
[1 (x∗)T
x∗ x∗(x∗)T
], then x∗
is optimal for QCQP.
Conic Programming 7 / 46
Exact Solutions from SDP Relaxation
LemmaLet G be a symmetric matrix and X be a positive semidefinite matrix withrank d. Suppose that G •X ≤ (=)0. Then there exists a rank-onedecomposition X =
∑di=1 xix
Ti such that xTi Gxi ≤ (=)0, i = 1, . . . , d.
TheoremWhen m = 2, assume there exists x such that xTAix+ 2bTi x+ ci < 0,i = 1, 2, and there exists y1 ≥ 0, y2 ≥ 0 such that[c0 bT0b0 A0
]+ y1
[c1 bT1b1 A1
]+ y2
[c2 bT2b2 A2
]� 0. If, for the optimal solution Y ∗,
either[c1 bT1b1 A1
]• Y ∗ < 0 or
[c2 bT2b2 A2
]• Y ∗ < 0, then there is no gap
between QCQP and its SDP relaxation, and at least one optimal solutioncan be obtained from the rank-one decomposition of Y ∗.
Conic Programming 8 / 46
Exact Solutions from SDP Relaxation
Theorem(HQCQP)When m = 2, suppose there exists x, such that xTAix+ ci < 0, i = 1, 2,and there exists y1 ≥ 0 and y2 ≥ 0 such that A0 + y1A1 + y2A2 � 0. Thenthere is no gap between HQCQP and its SDP relaxation, and at least oneoptimal solution can be obtained from the rank-one decomposition of X∗.
Conic Programming 9 / 46
One Extension of QCQP
Ext-QCQPMin xTA0x+ 2bT0 x+ c0
s.t. xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m(x2
1, . . . , x2n)T ∈ X
(Ext-QCQP)
where X is a closed convex set.SDP Relaxation:
Min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0, i = 1, . . . ,m
(Y22, Y33, . . . , Y(n+1),(n+1))T ∈ X
Y11 = 1
Y � 0, rank(Y ) = 1
Conic Programming 10 / 46
Exact Solutions from SDP Relaxation
TheoremSuppose there is σ ∈ {−1, 1}n+1, such that all the off-diagonal elements
of Λσ
[ci bTibi Ai
]Λσ, 0 ≤ i ≤ m, are nonpositive. Here
(Λσ)ij =
{σi i = j0 i 6= j
If Y ∗ is an optimal solution for the SDP relaxation for Ext-QCQP, then
x∗ = σ1(σ2
√Y ∗22, σ3
√Y ∗33, . . . , σn+1
√Y(n+1),(n+1))
T
is optimal for Ext-QCQP.
Conic Programming 11 / 46
More Results on Ext-QCQP
A Special Case of Ext-QCQP
Min xTAx
s.t. (x21, . . . , x
2n)T ∈ X
where X is a closed convex set.SDP Relaxation:
Min A •Xs.t. (X11, X22, . . . , Xnn)T ∈ X
X � 0
The previous theorem is satisfied automatically. Hence the optimalsolution for this special case of Ext-QCQP can be obtained by its SDPrelaxation.
Conic Programming 12 / 46
Recent Results on QCQP
Content• Introduction• KKT solution and Conic Relaxation for QCQP• Adaptive Approximation to QCQP
Conic Programming 13 / 46
Introduction
QCQP:
Min f(x) = xTA0x+ 2bT0 x+ c0
s.t. gi(x) = xTAix+ 2bTi x+ ci ≤ 0, i = 1, . . . ,m,
Ai ∈ Sn, bi ∈ Rn, ci ∈ R, i = 0, 1, . . . ,m.Feasible domain:
feas(QCQP) = {x ∈ Rn|gi(x) ≤ 0, i = 1, . . . ,m}
Conic Programming 14 / 46
Introduction
Previously we have learnt:• Equivalent SOCP for convex QCQP• SDP relaxation for QCQP• m = 1: optimal rank one solution from SDP relaxation• m = 2: optimal solution from non-binding solution of SDP relaxation• m = 2: optimal solution from SDP relaxation for HQCQP• Optimal solution for Ext-QCQP with all nonpositive off-diagonal
coefficients
Conic Programming 15 / 46
Recall
• Nonnegative quadratic functions over F
f(x) = xTAx+ 2bTx+ c ≥ 0,∀x ∈ F
f ⇔[c bT
b A
]• DF = {
[c bT
b A
]∈ Sn+1|
[1x
]T [c bT
b A
] [1x
]≥ 0,∀x ∈ F} is a closed,
convex cone, containing Sn+1+ .
• D∗F = cl cone{[
1 xT
x xxT
]|x ∈ F
}• (D∗F )∗ = DF and (DF )∗ = D∗F• F = Rn =⇒ DF = Sn+1
+ , D∗F = Sn+1+
• F = Rn+ =⇒ DF = Cn+1, D∗F = C∗n+1
Conic Programming 16 / 46
Question
How to use the cone of nonnegative quadratic functions to solve QCQP?
• Relation to KKT solution• Adaptively approximation
Conic Programming 17 / 46
KKT solution and Conic Relaxation forQCQP
Ideas:
Conic Programming 18 / 46
Constraint Qualification: LICQ
Active setGiven a feasible solution x, the active set I(x) is defined as
I(x) = {1 ≤ i ≤ m | gi(x) = 0}.
Linear independence constraint qualification (LICQ)Given a feasible solution x, if the vectors ∇gi(x), i ∈ I(x), are linearlyindependent, then the LICQ condition holds at x.
Conic Programming 19 / 46
Lagrangian Function
Define
A(λ) = A0 +m∑i=1
λiAi, b(λ) = b0 +m∑i=1
λibi, c(λ) = c0 +m∑i=1
λici.
Lagrangian function of QCQP
L(x, λ) = f(x) +m∑i=1
λigi(x)
= xTA(λ)x+ 2bT (λ)x+ c(λ)
where λ ∈ Rm+ .
Conic Programming 20 / 46
Karush-Kuhn-Tucker Conditions
Given a feasible solution x∗, if it satisfies the LICQ and is a localminimum, then there exists a Lagrangian multiplier vector λ∗ such that(x∗, λ∗) is a KKT solution satisfying
∇xL(x∗, λ∗) = 2A(λ∗)x∗ + 2b(λ∗) = 0,
λ∗i gi(x∗) = λ∗i ((x
∗)TAix∗ + 2bTi x
∗ + ci) = 0,
gi(x∗) = (x∗)TAix
∗ + 2bTi x∗ + ci ≤ 0, λ∗i ≥ 0, i = 1, . . . ,m.
Conic Programming 21 / 46
“Copositive Cone” w.r.t. the FeasibleDomain (Cone of Nonnegative QuadraticFunctions)
Define
Dn+1 =
{U ∈ Sn+1
∣∣∣∣ U • [ 1 xT
x xxT
]≥ 0,∀x ∈ feas(QCQP)
}D∗n+1 = cl cone
{X =
[1x
] [1x
]T ∣∣∣∣ x ∈ feas(QCQP)
}
Conic Programming 22 / 46
Sufficient Conditions
Lemma (Positive semidefiniteness condition).Let (x∗, λ∗) be a KKT solution. If A(λ∗) ∈ Sn+, then x∗ is a globally optimalsolution of QCQP.
Theorem (Copositiveness condition)Let (x∗, λ∗) be a KKT solution. If M(x∗, λ∗) ∈ Dn+1, then x∗ is an optimalsolution of the QCQP problem, where
M(x∗, λ∗) =
[c(λ∗)− f(x∗) bT (λ∗)
b(λ∗) A(λ∗)
].
Conic Programming 23 / 46
Recall
Conic Programming 24 / 46
An Equivalent Conic ProgrammingProblem of QCQP
Min
[c0 bT0b0 A0
]• Y
s.t. Y11 = 1,Y ∈ D∗n+1.
(CoP)
TheoremCoP and QCQP have the same optimal objective value.
ProofSee Sturm and Zhang’s paper “On Cones of Nonnegative QuadraticFunctions”.
Conic Programming 25 / 46
Adding Valid Inequality Constraints
CoP1:
Min
[c0 bT0b0 A0
]• Y
s.t.
[ci bTibi Ai
]• Y ≤ 0,
i = 1, . . . ,m,
Y11 = 1,Y ∈ D∗n+1.
CoD1:
Max σ
s.t.
[c(λ)− σ bT (λ)b(λ) A(λ)
]∈ Dn+1,
λ ≥Rm+
0.
TheoremIf QCQP is bounded below, then the optimal objective values of QCQP,CoP1 and CoD1 are equal.
Conic Programming 26 / 46
Finding the Lagrangian Multiplier Vector λ∗
AssumptionThere exists a KKT solution (x∗, λ∗) satisfying the LICQ condition andcopositiveness condition.
LemmaUnder the copositiveness condition assumption, (σ∗, λ∗) = (f(x∗), λ∗) isan optimal solution of CoD1. Furthermore, if the LICQ assumption holds,then for any (σ, λ) being an optimal solution of the CoD1 problem, wehave λ ≤Rn
+λ∗.
Conic Programming 27 / 46
Finding the Lagrangian Multiplier Vector λ∗
TheoremLet (x∗, λ∗) be a KKT solution. If it satisfies the LICQ condition and thecopositiveness condition, then λ∗ is the unique optimal solution of themaximization problem CoD2
Max∑mi=1 λi
s.t.
[c(λ)− σ∗ bT (λ)b(λ) A(λ)
]∈ Dn+1,
λ ≥Rn+
0,
(CoD2)
where σ∗ is the optimal objective value obtained by solving CoD1.Furthermore, if A(λ∗) is invertible, then
x∗ = −A−1(λ∗)b(λ∗).
Conic Programming 28 / 46
Relaxing the Conic Constraint
Choose a computable cone Kn+1 such that Sn+1+ ⊂ Kn+1 ⊂ Dn+1.
CoD1’:Max σ
s.t.
[c(λ)− σ bT (λ)b(λ) A(λ)
]∈ Kn+1,
λ ≥Rn+
0.
CoD2’:Max
∑mi=1 λi
s.t.
[c(λ)− σ∗ bT (λ)b(λ) A(λ)
]∈ Kn+1,
λ ≥Rn+
0.
Conic Programming 29 / 46
Algorithm (QCQP)
STEP 1: Given a QCQP problem, solve the corresponding CoD1’ problem andget the optimal value σ∗. If failed, then stop and the problem cannot besolved by the current scheme.
STEP 2: Solve CoD2’ to get the optimal λ∗.STEP 3: Compute x∗ = −A+(λ∗)b(λ∗).STEP 4: If (x∗, λ∗) is a KKT solution, then return x∗ as a global optimal solution
of QCQP with the objective value f(x∗) = σ∗. Otherwise, return σ∗ asa lower bound of QCQP.
Here A+ is the Moore-Penrose generalized inverse for a given squarematrix A ∈ Rn×n.
Conic Programming 30 / 46
Example
Min f(x) = xTA0x+ 2bT0 x
s.t. g1(x) = xTA1x+ 2bT1 x+ c1 ≤ 0,
g2(x) = xTA2x+ 2bT2 x+ c2 ≤ 0,
where
A0 =
−2 10 210 4 12 1 −7
, b0 =
−12−656
,A1 =
2 0 00 −2 00 0 8
, b1 =
0−2−64
, c1 = 512,
A2 =
2 0 00 2 00 0 2
, b2 =
−20−16
, c2 = 128.
Conic Programming 31 / 46
Example
• The feasible domain is a subset of R3+.
• If we choose K4 = S4+ (SDP relaxation), then the optimal value of the relaxed problem
is 222.88.• Instead, choose K4 = S4
+ +N4.
• x∗ = (0, 0, 8)T with f(x∗) = 224 and λ∗ = (1, 2)T .
M(x∗, λ∗) =
320 −16 −8 −40−16 4 10 2−8 10 6 1−40 2 1 5
=
320 −16 −8 −40−16 4 0 2−8 0 6 1−40 2 1 5
+
0 0 0 00 0 10 00 10 0 00 0 0 0
∈ S4+ +N4,
A0 + λ∗1A1 + λ
∗2A2 =
4 10 210 6 12 1 5
/∈ S3+.
Conic Programming 32 / 46
Summary
• Equivalent conic programming problems.
• The copositiveness condition which ensures a KKT solution beingoptimal.
• The maximal property of the Lagrangian multiplier vector under theLICQ condition and the copositiveness condition.
• Conic programming relaxation.• A way to design polynomial time algorithm to solve a class of QCQP.
Conic Programming 33 / 46
Adaptive Approximation to QCQP
Consider a box constrained quadratic program
Min xTA0x+ 2bT0 x+ c0
s.t. x ∈ G = [0, 1]n(BQP)
This problem is in general NP-hard.
Is there any polynomial time approximation?
Find an ellipsoid cover F1 ∪ · · · ∪ Fk ⊃ G such that the problem
Min xTA0x+ 2bT0 x+ c0
s.t. x ∈ F1 ∪ · · · ∪ Fk(BQPR)
satisfies opt(BQP) ≤ opt(BQPR) + ε.
Conic Programming 34 / 46
Conic Reformulation
Let H =
[c0 bT0b0 A0
], BQP becomes
Min H • Y
s.t. Y11 = 1
Y ∈ D∗G
(BCoP)
and BQPR becomes
Min H • Y
s.t. Y11 = 1
Y ∈ D∗F1∪···∪Fk
(BCoPR)
Max σ
s.t. σ
[1 00 0
]+ S = H
S ∈ DF1∪···∪Fk
Conic Programming 35 / 46
Conic Reformulation
Lemma
DF1∪···∪Fk= DF1 ∩ · · · ∩ DFk
and
D∗F1∪···∪Fk= D∗F1
+ · · ·+D∗Fk
Rewrite the relaxation BCoPR as
Min H • Y
s.t. Y11 = 1
Y = Y1 + · · · , YkYi ∈ D∗Fi
, i = 1, . . . , k
(BCRP)
What is DF1∪···∪Fkfor the dual problem of BCRP?
Conic Programming 36 / 46
Conic Reformulation: Dual Problem
Lemma (Linear Matrix Inequalities (LMI))Suppose each ellipsoid Fi = {x ∈ Rn|xTAix+ 2bTi x+ ci 6 0} with someAi � 0 and Int(Fi) 6= ∅. Then, M ∈ DF1∪···∪Fk
if and only if the followingsystem is feasible:
M + λi
[ci bTibi Ai
]� 0, λi > 0 for i = 1, 2, ..., k.
Dual of BCoRP:Max σ
σ
[1 00 0
]+ S = H
S + λi
[ci bTibi Ai
]� 0, λi ≥ 0,
i = 1, . . . , k.
(BCRD)
Conic Programming 37 / 46
Optimality Condition
(Y ∗, Y ∗1 , ..., Y∗k ) is feasible to BCRP,
(σ∗, λ∗, S∗) is feasible to BCRD,
S∗ · Y ∗i = 0 and λ∗i
[bi cTici Qi
]• Y ∗i = 0, i = 1, 2, ...k.
Conic Programming 38 / 46
Some Properties
LemmaS∗ is on the boundary of DF1∪···∪Fk
.
Lemma (Rank-one decomposition)Y ∗ can be decomposed in the form of
Y ∗ =k∑i=1
ni∑j=1
µij
[1xij
] [1xij
]T
with xij ∈ Fi, µij > 0 and∑ki=1
∑ni
j=1 µij = Y ∗11 = 1.Moreover, [
1xij
]TS∗[
1xij
]= 0,∀xij
Conic Programming 39 / 46
Improving The Relaxation
TheoremIf any xij decomposed from Y ∗ is feasible for BQP, then it is an optimalsolution of BQP.
However, in general, they are not feasible.
Let x = xij = arg mini,j
[1xij
]TH
[1xij
]with corresponding Fi.
Notice that for Fi, the feasible part is Fi ∩ G. If we could cover Fi ∩ G bysmaller sets, whose union does not contain x, then the solution of thenew cover may be improved.
Conic Programming 40 / 46
Improving The Relaxation
For simplicity, the ellipsoids have to be constructed carefully.
DefinitionLet T = {x ∈ Rn|ui 6 xi 6 vi} be a rectangular set in Rn, define aquadratic function
gT (x) =
n∑i=1
(2xi − vi − ui)2
(vi − ui)2− n for x ∈ Rn,
and an ellipsoidal set FT = {x ∈ Rn|gT (x) 6 0}. Then we say FT is thecorresponding ellipsoid of T .
Conic Programming 41 / 46
Algorithm
STEP 1: Set T = G and Nmax. Set k = 1 and let the cover be F1 = FT .STEP 2: Solve BCRP and BCRD defined by the cover F1 ∪ · · · ∪ Fk to obtain Y ∗
and (σ∗, λ∗, S∗). Set lk = σ∗.STEP 3: Decompose Y ∗ to obtain x and Fi.STEP 4: If x ∈ G, then x is optimal. Stop and return.
Otherwise, set T be the rectangle covered by Fi.Find i∗ = arg maxi{max(xi − 1,−xi)}.Partition T into T1 = {x ∈ T |xi∗ ≤ ui∗+vi∗
2} and
T2 = {x ∈ T |xi∗ ≥ ui∗+vi∗2}.
Drop Fk from the cover, and add FT1 and FT2 into the cover.Set k = k + 1.
STEP 5: If k < Nmax, then go to STEP 2. Otherwise return max{l1, ..., lk} as alower bound for BQP.
Conic Programming 42 / 46
Convergence
TheoremLet l∗k = maxi=1,...,k{li} be the largest lower bound up to the kth iteration,then limk→+∞ l∗k = opt(BQP ).
• Developed an adaptive relaxation for BQP.• It applies to convex constrained QCQP too.• It may be applied to general QCQP, but needs well-designed covers.• Other techniques such as RLT can be involved in the relaxation.
Conic Programming 46 / 46
LCoP Part VII – Practical LCoP
Shu-Cherng Fang
Department of Industrial and Systems EngineeringGraduate Program in Operations Research
North Carolina State University
December 2017
Conic Programming 1 / 40
Practical LCoP
Content• Convex Cones and Properties• Different Forms of LCoP• SOCP Formulation• SDP Formulation
Conic Programming 2 / 40
Convex Cones and Properties
K, K1, K2 are convex cones in Rn.• (K∗)∗ = cl(K).• K1 ⊆ K2 ⇒ K∗2 ⊆ K∗1 .• K1 ∩K2, K1 ∪K2, K1 +K2, K1 ×K2 are all cones.• (K1 +K2)∗ = K∗1 ∩K∗2 .• K1, K2 closed⇒ K1 +K2 and K1 ×K2 are closed.• ri(K1 +K2) = ri(K1) + ri(K2).• ri(K1 ×K2) = ri(K1)× ri(K2).• The supporting hyperplane of K always contains the origin• If K is solid, then K∗ is pointed.• If K is pointed, then K∗ is solid.• If K is proper, then K∗ is proper.
Conic Programming 3 / 40
Convex Cones and Properties
K, K1, K2 are convex cones in Rn.• Rn+ ⊆ Rn, Ln ⊆ Rn, Sn+ ⊆ Rn×n are pointed, closed, convex and