A fresh CP look at MINLP –
new formulations and relaxations
Immanuel Bomze, Universitat Wien
COST MINLP Workshop@IMUS Seville, 31 March 2015
Overview
1. Ternary fractional QPs
Overview
1. Ternary fractional QPs
2. Everything is quadratic !
Overview
1. Ternary fractional QPs
2. Everything is quadratic !
3. ... and therefore linearly copositive
Overview
1. Ternary fractional QPs
2. Everything is quadratic !
3. ... and therefore linearly copositive
4. New bounds and ...
Overview
1. Ternary fractional QPs
2. Everything is quadratic !
3. ... and therefore linearly copositive
4. New bounds and ...
5. ... new approximation hierarchies
Ternary decisions
Do you like Copositive Optimization ?
−
{ ∈ }
{−
Ternary decisions
Do you like Copositive Optimization ?
Yes !
−
{ ∈ }
{−
Ternary decisions
Do you like Copositive Optimization ?
Yes ! +1,
−
{ ∈ }
{−
Ternary decisions
Do you like Copositive Optimization ?
Yes ! +1,
No ! −1,
{ ∈ }
{−
Ternary decisions
Do you like Copositive Optimization ?
Yes ! +1,
No ! −1,
don’t know/tell 0.
{ ∈ }
{− }
Ternary decisions
Do you like Copositive Optimization ?
Yes ! +1,
No ! −1,
don’t know/tell 0.
Ternary optimization:
min {f(x) : x ∈ Tn}
with T = {−1,0,1}.
Ternary decisions
Do you like Copositive Optimization ?
Yes ! +1,
No ! −1,
don’t know/tell 0.
Ternary optimization:
min {f(x) : x ∈ Tn}
with T = {−1,0,1}.
What for ? Eg., graph tri-partitioning problems as in PageRank
for Folksonomy in social media/Semantic Web [Hotho et al.’06].
Predicting ternary decisions – TFQP
Given dissimilarities Dij > 0 (D adjacency of aversion graph):coherence within three groups/separation between them:
small x>Dx =∑i,jDijxixj
Predicting ternary decisions – TFQP
Given dissimilarities Dij > 0 (D adjacency of aversion graph):coherence within three groups/separation between them:
small x>Dx =∑i,jDijxixj ? ... too simple, ignores outside option.
Predicting ternary decisions – TFQP
Given dissimilarities Dij > 0 (D adjacency of aversion graph):coherence within three groups/separation between them:
small x>Dx =∑i,jDijxixj ? ... too simple, ignores outside option.
Better: relative to density, so
min
(x>Dx)/∑i
x2i : x ∈ Tn \ {o}
which is APX-hard [Bhaskara et al.’12].
Predicting ternary decisions – TFQP
Given dissimilarities Dij > 0 (D adjacency of aversion graph):coherence within three groups/separation between them:
small x>Dx =∑i,jDijxixj ? ... too simple, ignores outside option.
Better: relative to density, so
min
(x>Dx)/∑i
x2i : x ∈ Tn \ {o}
which is APX-hard [Bhaskara et al.’12]. More general
z∗T� := min
{f(x)
g(x): x ∈ Tn \ {o}
}for quadratic f , g.
Predicting ternary decisions – TFQP
Given dissimilarities Dij > 0 (D adjacency of aversion graph):coherence within three groups/separation between them:
small x>Dx =∑i,jDijxixj ? ... too simple, ignores outside option.
Better: relative to density, so
min
(x>Dx)/∑i
x2i : x ∈ Tn \ {o}
which is APX-hard [Bhaskara et al.’12]. More general
z∗T� := min
{f(x)
g(x): x ∈ Tn \ {o}
}for quadratic f , g. Easier variant if g(x) 6= 0 for all x ∈ Tn:
z∗T := min
{f(x)
g(x): x ∈ Tn
}.
Ternary fractional quadratic problems (TFQPs) ...
... are special cases of (Mixed-)Binary constrained fractional QPs
Theorem [Amaral/B.’15]: Both z∗T� and z∗T are in fact MBFQP
min
{f(v)
g(v): v ∈ Rd+ , Cv = c , vi ∈ {0,1} for all i ∈ B
},
with d = 3n and Burer’s key condition satisfied.
∈ ⇐⇒ − { }
Ternary fractional quadratic problems (TFQPs) ...
... are special cases of (Mixed-)Binary constrained fractional QPs
Theorem [Amaral/B.’15]: Both z∗T� and z∗T are in fact MBFQP
min
{f(v)
g(v): v ∈ Rd+ , Cv = c , vi ∈ {0,1} for all i ∈ B
},
with d = 3n and Burer’s key condition satisfied.
Proof. x ∈ T ⇐⇒ x = y − z with {y, z} ⊆ {0,1} and y + z ≤ 1.
Ternary fractional quadratic problems (TFQPs) ...
... are special cases of (Mixed-)Binary constrained fractional QPs
Theorem [Amaral/B.’15]: Both z∗T� and z∗T are in fact MBFQP
min
{f(v)
g(v): v ∈ Rd+ , Cv = c , vi ∈ {0,1} for all i ∈ B
},
with d = 3n and Burer’s key condition satisfied.
Proof. x ∈ T ⇐⇒ x = y − z with {y, z} ⊆ {0,1} and y + z ≤ 1.
... latter reminds on Densest Subgraph problem (poly-time),
but also includes APX-hard Max-Cut, k-Densest Subgraph ...
Ternary fractional quadratic problems (TFQPs) ...
... are special cases of (Mixed-)Binary constrained fractional QPs
Theorem [Amaral/B.’15]: Both z∗T� and z∗T are in fact MBFQP
min
{f(v)
g(v): v ∈ Rd+ , Cv = c , vi ∈ {0,1} for all i ∈ B
},
with d = 3n and Burer’s key condition satisfied.
Proof. x ∈ T ⇐⇒ x = y − z with {y, z} ⊆ {0,1} and y + z ≤ 1.
... latter reminds on Densest Subgraph problem (poly-time),
but also includes APX-hard Max-Cut, k-Densest Subgraph ...
Application: e.g., gene annotation graphs [Saha et al.’10] !
Burer’s key condition ? Copositive formulation !
... key condition: linear constraints force binary var.s into [0,1].
∈C{ • • • − ∈ B}
•
Burer’s key condition ? Copositive formulation !
... key condition: linear constraints force binary var.s into [0,1].
Used in [Burer’09];
∈C{ • • • − ∈ B}
•
Burer’s key condition ? Copositive formulation !
... key condition: linear constraints force binary var.s into [0,1].
Used in [Burer’09]; also here, for
Theorem [Amaral/B.’15]: Any Mixed-integer FQP with linear
constraints is, under key condition, equivalent to the linear COP
z∗cop = minY∈Cd+1
{A • Y : B • Y = 1 , Cc • Y = 0, Y0i − Yii = 0 , all i ∈ B} ,
•
Burer’s key condition ? Copositive formulation !
... key condition: linear constraints force binary var.s into [0,1].
Used in [Burer’09]; also here, for
Theorem [Amaral/B.’15]: Any Mixed-integer FQP with linear
constraints is, under key condition, equivalent to the linear COP
z∗cop = minY∈Cd+1
{A • Y : B • Y = 1 , Cc • Y = 0, Y0i − Yii = 0 , all i ∈ B} ,
with A coming from f , B from g, Cc from constraints Cv = c,
duality A • Y =∑i,j AijYij and
Y ∈ Cd+1 = conv{yy> : y ∈ Rd+1
+
}
> >
Burer’s key condition ? Copositive formulation !
... key condition: linear constraints force binary var.s into [0,1].
Used in [Burer’09]; also here, for
Theorem [Amaral/B.’15]: Any Mixed-integer FQP with linear
constraints is, under key condition, equivalent to the linear COP
z∗cop = minY∈Cd+1
{A • Y : B • Y = 1 , Cc • Y = 0, Y0i − Yii = 0 , all i ∈ B} ,
with A coming from f , B from g, Cc from constraints Cv = c,
duality A • Y =∑i,j AijYij and
Y ∈ Cd+1 = conv{yy> : y ∈ Rd+1
+
}the completely positive matrix cone.
Reason: lifting A • (yy>) = y>Ay, homogenization [Shor ’87].
General form of conic linear optimization
Let K be a convex cone of symmetric d× d matrices X.
min {C • X : Ai • X = bi , i∈ [1:m] , X ∈ K}
General form of conic linear optimization
Let K be a convex cone of symmetric d× d matrices X.
min {C • X : Ai • X = bi , i∈ [1:m] , X ∈ K} , barrier ??
General form of conic linear optimization
Let K be a convex cone of symmetric d× d matrices X.
min {C • X : Ai • X = bi , i∈ [1:m] , X ∈ K} , barrier ??
Familiar cases:
K = N ={X = X> : X≥O
}= N ∗ . . . LP, barrier:−
∑i,jlogXij ,
K
General form of conic linear optimization
Let K be a convex cone of symmetric d× d matrices X.
min {C • X : Ai • X = bi , i∈ [1:m] , X ∈ K} , barrier ??
Familiar cases:
K = N ={X = X> : X≥O
}= N ∗ . . . LP, barrier:−
∑i,jlogXij ,
and
K = P ={X = X> : X�O
}= P∗ . . .SDP, barrier:−
∑ilogλi(X) .
General form of conic linear optimization
Let K be a convex cone of symmetric d× d matrices X.
min {C • X : Ai • X = bi , i∈ [1:m] , X ∈ K} , barrier ??
Familiar cases:
K = N ={X = X> : X≥O
}= N ∗ . . . LP, barrier:−
∑i,jlogXij ,
and
K = P ={X = X> : X�O
}= P∗ . . .SDP, barrier:−
∑ilogλi(X) .
In above cases, the dual cone of K,
K∗ ={S = S> : S • X ≥ 0 for all X ∈ K
}coincides with K (self-duality)
General form of conic linear optimization
Let K be a convex cone of symmetric d× d matrices X.
min {C • X : Ai • X = bi , i∈ [1:m] , X ∈ K} , barrier ??
Familiar cases:
K = N ={X = X> : X≥O
}= N ∗ . . . LP, barrier:−
∑i,jlogXij ,
and
K = P ={X = X> : X�O
}= P∗ . . .SDP, barrier:−
∑ilogλi(X) .
In above cases, the dual cone of K,
K∗ ={S = S> : S • X ≥ 0 for all X ∈ K
}coincides with K (self-duality), but in general K∗ differs from K.
Copositive optimization (COP), duality
A very special matrix cone:
K = C = conv{xx> : x ∈ Rn, x ≥ o
},
the cone of completely positive matrices,
K∗ C∗ ≥
}
Copositive optimization (COP), duality
A very special matrix cone:
K = C = conv{xx> : x ∈ Rn, x ≥ o
},
the cone of completely positive matrices, with its dual cone
K∗ = C∗ ={S = S> is copositive; means: x>Sx ≥ 0 if x ≥ o
}6= C .
C ⊂ P ∩N ⊂ P N ⊂ C∗ ≥
∗ { • • ∈ ∈ C}
∗∈
∗ ∗
Copositive optimization (COP), duality
A very special matrix cone:
K = C = conv{xx> : x ∈ Rn, x ≥ o
},
the cone of completely positive matrices, with its dual cone
K∗ = C∗ ={S = S> is copositive; means: x>Sx ≥ 0 if x ≥ o
}6= C .
Well known relations:
C ⊂ P ∩N P N ⊂ C∗
∗ { • • ∈ ∈ C}
∗∈
∗ ∗
Copositive optimization (COP), duality
A very special matrix cone:
K = C = conv{xx> : x ∈ Rn, x ≥ o
},
the cone of completely positive matrices, with its dual cone
K∗ = C∗ ={S = S> is copositive; means: x>Sx ≥ 0 if x ≥ o
}6= C .
Well known relations:
C ⊂ P ∩N ⊂ P +N
∗ { • • ∈ ∈ C}
∗
∗ ∗
Copositive optimization (COP), duality
A very special matrix cone:
K = C = conv{xx> : x ∈ Rn, x ≥ o
},
the cone of completely positive matrices, with its dual cone
K∗ = C∗ ={S = S> is copositive; means: x>Sx ≥ 0 if x ≥ o
}6= C .
Well known relations:
C ⊂ P ∩N ⊂ P +N ⊂ C∗
∗ { • • ∈ ∈ C}
∗
∗ ∗
Copositive optimization (COP), duality
A very special matrix cone:
K = C = conv{xx> : x ∈ Rn, x ≥ o
},
the cone of completely positive matrices, with its dual cone
K∗ = C∗ ={S = S> is copositive; means: x>Sx ≥ 0 if x ≥ o
}6= C .
Well known relations:
C ⊂ P ∩N ⊂ P +N ⊂ C∗ . . . strict for n ≥ 5 .
∗ { • • ∈ ∈ C}
∗
∗ ∗
Copositive optimization (COP), duality
A very special matrix cone:
K = C = conv{xx> : x ∈ Rn, x ≥ o
},
the cone of completely positive matrices, with its dual cone
K∗ = C∗ ={S = S> is copositive; means: x>Sx ≥ 0 if x ≥ o
}6= C .
Well known relations:
C ⊂ P ∩N ⊂ P +N ⊂ C∗ . . . strict for n ≥ 5 .
Primal-dual pair in (COP):
p∗ = inf {C • X : Ai • X = bi , i∈ [1:m] , X ∈ C}and
d∗ = supy∈Rm{b>y : C−
∑i yiAi ∈ C∗
}.
Copositive optimization (COP), duality
A very special matrix cone:
K = C = conv{xx> : x ∈ Rn, x ≥ o
},
the cone of completely positive matrices, with its dual cone
K∗ = C∗ ={S = S> is copositive; means: x>Sx ≥ 0 if x ≥ o
}6= C .
Well known relations:
C ⊂ P ∩N ⊂ P +N ⊂ C∗ . . . strict for n ≥ 5 .
Primal-dual pair in (COP):
p∗ = inf {C • X : Ai • X = bi , i∈ [1:m] , X ∈ C}and
d∗ = supy∈Rm{b>y : C−
∑i yiAi ∈ C∗
}.
Usual weak (d∗ ≤ p∗) and strong (d∗ = p∗) duality results hold.
Semidefinite cone P
Why duality in (COP) ? And how ?
General rule: any dually feasible solution y ∈ Rm gives lower bound
b>y ≤ d∗ ≤ p∗
if S(y) = C−∑i yiAi ∈ C∗.
≤ ∗ ≤ ∗
−
∈ C∗
Why duality in (COP) ? And how ?
General rule: any dually feasible solution y ∈ Rm gives lower bound
b>y ≤ d∗ ≤ p∗
if S(y) = C−∑i yiAi ∈ C∗. For MBFQP
yB ≤ d∗ ≤ z∗cop = z∗
if
S(y) = A− yBB + yCCc +∑i∈B
yi
0 1 01 −1 00 0 0
∈ C∗ .
Why duality in (COP) ? And how ?
General rule: any dually feasible solution y ∈ Rm gives lower bound
b>y ≤ d∗ ≤ p∗
if S(y) = C−∑i yiAi ∈ C∗. For MBFQP
yB ≤ d∗ ≤ z∗cop = z∗
if
S(y) = A− yBB + yCCc +∑i∈B
yi
0 1 01 −1 00 0 0
∈ C∗ .Bad news: checking S ∈ C∗ is NP-hard [Dickinson/Gijben ’14];
Why duality in (COP) ? And how ?
General rule: any dually feasible solution y ∈ Rm gives lower bound
b>y ≤ d∗ ≤ p∗
if S(y) = C−∑i yiAi ∈ C∗. For MBFQP
yB ≤ d∗ ≤ z∗cop = z∗
if
S(y) = A− yBB + yCCc +∑i∈B
yi
0 1 01 −1 00 0 0
∈ C∗ .Bad news: checking S ∈ C∗ is NP-hard [Dickinson/Gijben ’14];
good news: checking S ∈ P +N ⊂ C∗ is poly-time.
Why duality in (COP) ? And how ?
General rule: any dually feasible solution y ∈ Rm gives lower bound
b>y ≤ d∗ ≤ p∗
if S(y) = C−∑i yiAi ∈ C∗. For MBFQP
yB ≤ d∗ ≤ z∗cop = z∗
if
S(y) = A− yBB + yCCc +∑i∈B
yi
0 1 01 −1 00 0 0
∈ C∗ .Bad news: checking S ∈ C∗ is NP-hard [Dickinson/Gijben ’14];
good news: checking S ∈ P +N ⊂ C∗ is poly-time. Even better:
resulting bound beats Lagrangian dual bounds (S ∈ P) ...
From linear to quadratic constraints – some thoughts
Everything is linear
From linear to quadratic constraints – some thoughts
Everything is linear – really ?
Nono – everything is nonlinear –
From linear to quadratic constraints – some thoughts
Everything is linear – really ?
Nono – everything is nonlinear – c’mon
Ahh ... everything is quadratic !
From linear to quadratic constraints – some thoughts
Everything is linear – really ?
Nono – everything is nonlinear – c’mon
Ahh ... everything is quadratic ! – wait ...
Binarity: x(1− x) = 0, quadratic, ok; also x ≥ 0 ⇔ x = t2
From linear to quadratic constraints – some thoughts
Everything is linear – really ?
Nono – everything is nonlinear – c’mon
Ahh ... everything is quadratic ! – wait ...
Binarity: x(1− x) = 0, quadratic, ok; also x ≥ 0 ⇔ x = t2
Monomial: x3 = x1x2, build∏ixmii recursively, ok.
From linear to quadratic constraints – some thoughts
Everything is linear – really ?
Nono – everything is nonlinear – c’mon
Ahh ... everything is quadratic ! – wait ...
Binarity: x(1− x) = 0, quadratic, ok; also x ≥ 0 ⇔ x = t2
Monomial: x3 = x1x2, build∏ixmii recursively, ok.
So polynomial optimization is quadratic
From linear to quadratic constraints – some thoughts
Everything is linear – really ?
Nono – everything is nonlinear – c’mon
Ahh ... everything is quadratic ! – wait ...
Binarity: x(1− x) = 0, quadratic, ok; also x ≥ 0 ⇔ x = t2
Monomial: x3 = x1x2, build∏ixmii recursively, ok.
So polynomial optimization is quadratic, and hence everything.
Then lift quadratics to linear objective, so everything is linearcopositive ;-)
From linear to quadratic constraints – some thoughts
Everything is linear – really ?
Nono – everything is nonlinear – c’mon
Ahh ... everything is quadratic ! – wait ...
Binarity: x(1− x) = 0, quadratic, ok; also x ≥ 0 ⇔ x = t2
Monomial: x3 = x1x2, build∏ixmii recursively, ok.
So polynomial optimization is quadratic, and hence everything.
Then lift quadratics to linear objective, so everything is linearcopositive ;-)
Time to get serious: [Pena/Vera/Zuluaga ’15] show indeed that
any polynomial optimization problem is linear copositive.
Quadratically/linearly constrained QPs
Consider m+ 1 (possibly nonconvex) quadratic functions
qi(x) = x>Qix− 2b>i x + ci , i∈ [0:m] ,
and the problem to minimize q0 over two different sets:
F := {x ∈ Rn : qi(x) ≤ 0 , all i∈ [1:m]} ,
∗∈ ∈
Quadratically/linearly constrained QPs
Consider m+ 1 (possibly nonconvex) quadratic functions
qi(x) = x>Qix− 2b>i x + ci , i∈ [0:m] ,
and the problem to minimize q0 over two different sets:
F := {x ∈ Rn : qi(x) ≤ 0 , all i∈ [1:m]}
Quadratically/linearly constrained QPs
Consider m+ 1 (possibly nonconvex) quadratic functions
qi(x) = x>Qix− 2b>i x + ci , i∈ [0:m] ,
and the problem to minimize q0 over two different sets:
F := {x ∈ Rn : qi(x) ≤ 0 , all i∈ [1:m]} , or
F+ :={x ∈ Rn+ : qi(x) ≤ 0 , all i∈ [1:m]
}.
∗∈
∈‖
Quadratically/linearly constrained QPs
Consider m+ 1 (possibly nonconvex) quadratic functions
qi(x) = x>Qix− 2b>i x + ci , i∈ [0:m] ,
and the problem to minimize q0 over two different sets:
F := {x ∈ Rn : qi(x) ≤ 0 , all i∈ [1:m]} , or
F+ :={x ∈ Rn+ : qi(x) ≤ 0 , all i∈ [1:m]
}.
So consider two QCQPs
z∗ := infx∈F
q0(x) and z∗+ := infx∈F+
q0(x) .
Quadratically/linearly constrained QPs
Consider m+ 1 (possibly nonconvex) quadratic functions
qi(x) = x>Qix− 2b>i x + ci , i∈ [0:m] ,
and the problem to minimize q0 over two different sets:
F := {x ∈ Rn : qi(x) ≤ 0 , all i∈ [1:m]} , or
F+ :={x ∈ Rn+ : qi(x) ≤ 0 , all i∈ [1:m]
}.
So consider two QCQPs
z∗ := infx∈F
q0(x) and z∗+ := infx∈F+
q0(x) .
Other linear constraints: slack var.s, Ax = a and x ∈ Rn+ can be
treated similarly by additional qm+1(x) = ‖Ax− a‖2 ≤ 0 .
Quadratically/linearly constrained QPs
Consider m+ 1 (possibly nonconvex) quadratic functions
qi(x) = x>Qix− 2b>i x + ci , i∈ [0:m] ,
and the problem to minimize q0 over two different sets:
F := {x ∈ Rn : qi(x) ≤ 0 , all i∈ [1:m]} , or
F+ :={x ∈ Rn+ : qi(x) ≤ 0 , all i∈ [1:m]
}.
So consider two QCQPs
z∗ := infx∈F
q0(x) and z∗+ := infx∈F+
q0(x) .
Other linear constraints: slack var.s, Ax = a and x ∈ Rn+ can be
treated similarly by additional qm+1(x) = ‖Ax− a‖2 ≤ 0 .
Important: explicit sign constraints xj ≥ 0 in F+.
(Full) Lagrangian relaxation ...
... for z∗+:
L(x; u, v) = x>Hux− 2d>u x− 2v>x + c>u ,
(Full) Lagrangian relaxation ...
... for z∗+:
L(x; u, v) = x>Hux− 2d>u x− 2v>x + c>u ,
∇xL(x; u, v) = 2[Hux− du − v]
(Full) Lagrangian relaxation ...
... for z∗+:
L(x; u, v) = x>Hux− 2d>u x− 2v>x + c>u ,
∇xL(x; u, v) = 2[Hux− du − v] and
D2xL(x; u, v) = 2Hu for all (x; u, v) ∈ Rn × Rm+ × Rn+ ,
with Hessian Hu = Q0 +m∑i=1
uiQi and du = b0 +m∑i=1
uibi.
(Full) Lagrangian relaxation ...
... for z∗+:
L(x; u, v) = x>Hux− 2d>u x− 2v>x + c>u ,
∇xL(x; u, v) = 2[Hux− du − v] and
D2xL(x; u, v) = 2Hu for all (x; u, v) ∈ Rn × Rm+ × Rn+ ,
with Hessian Hu = Q0 +m∑i=1
uiQi and du = b0 +m∑i=1
uibi.
... for z∗ similar, just drop v:
L0(x; u) = x>Hux− 2d>u x + c>u = L(x; u, o) ,
(Full) Lagrangian relaxation ...
... for z∗+:
L(x; u, v) = x>Hux− 2d>u x− 2v>x + c>u ,
∇xL(x; u, v) = 2[Hux− du − v] and
D2xL(x; u, v) = 2Hu for all (x; u, v) ∈ Rn × Rm+ × Rn+ ,
with Hessian Hu = Q0 +m∑i=1
uiQi and du = b0 +m∑i=1
uibi.
... for z∗ similar, just drop v:
L0(x; u) = x>Hux− 2d>u x + c>u = L(x; u, o) ,
∇xL0(x; u) = 2[Hux− du] and
D2xL0(x; u) = 2Hu for all (x; u) ∈ Rn × Rm+ .
(Semi-)Lagrangian dual functions and problems
... for z∗, no explicit sign constraints:
Θ0(u) := inf {L0(x; u) : x ∈ Rn} and z∗dual := sup{
Θ0(u) : u ∈ Rm+}
(Semi-)Lagrangian dual functions and problems
... for z∗, no explicit sign constraints:
Θ0(u) := inf {L0(x; u) : x ∈ Rn} and z∗dual := sup{
Θ0(u) : u ∈ Rm+}
... for z∗+ several options:
(Semi-)Lagrangian dual functions and problems
... for z∗, no explicit sign constraints:
Θ0(u) := inf {L0(x; u) : x ∈ Rn} and z∗dual := sup{
Θ0(u) : u ∈ Rm+}
... for z∗+ several options: full Lagrangian
Θ(u, v) := inf {L(x; u, v) : x ∈ Rn}with dual problem
z∗dual,+ := sup{
Θ(u, v) : (u, v) ∈ Rm+ × Rn+}
(Semi-)Lagrangian dual functions and problems
... for z∗, no explicit sign constraints:
Θ0(u) := inf {L0(x; u) : x ∈ Rn} and z∗dual := sup{
Θ0(u) : u ∈ Rm+}
... for z∗+ several options: full Lagrangian
Θ(u, v) := inf {L(x; u, v) : x ∈ Rn}with dual problem
z∗dual,+ := sup{
Θ(u, v) : (u, v) ∈ Rm+ × Rn+}
or Semi-Lagrangian dual
Θsemi(u) := inf{L0(x; u) : x ∈ Rn+
}
(Semi-)Lagrangian dual functions and problems
... for z∗, no explicit sign constraints:
Θ0(u) := inf {L0(x; u) : x ∈ Rn} and z∗dual := sup{
Θ0(u) : u ∈ Rm+}
... for z∗+ several options: full Lagrangian
Θ(u, v) := inf {L(x; u, v) : x ∈ Rn}with dual problem
z∗dual,+ := sup{
Θ(u, v) : (u, v) ∈ Rm+ × Rn+}
or Semi-Lagrangian dual
Θsemi(u) := inf{L0(x; u) : x ∈ Rn+
}with
z∗semi := sup{
Θsemi(u) : u ∈ Rn+}.
Semi-Lagrangian bounds are always tighter
General principle: for any (u, v) ∈ Rm+ × Rn+
Θ(u, v) = inf {L(x; u, v) : x ∈ Rn}
Semi-Lagrangian bounds are always tighter
General principle: for any (u, v) ∈ Rm+ × Rn+
Θ(u, v) = inf {L(x; u, v) : x ∈ Rn}
≤ inf{L(x; u, v) : x ∈ Rn+
}
Semi-Lagrangian bounds are always tighter
General principle: for any (u, v) ∈ Rm+ × Rn+
Θ(u, v) = inf {L(x; u, v) : x ∈ Rn}
≤ inf{L(x; u, v) : x ∈ Rn+
}= inf
{L0(x, u)− 2v>x : x ∈ Rn+
}
Semi-Lagrangian bounds are always tighter
General principle: for any (u, v) ∈ Rm+ × Rn+
Θ(u, v) = inf {L(x; u, v) : x ∈ Rn}
≤ inf{L(x; u, v) : x ∈ Rn+
}= inf
{L0(x, u)− 2v>x : x ∈ Rn+
}≤ inf
{L0(x, u) : x ∈ Rn+
}
Semi-Lagrangian bounds are always tighter
General principle: for any (u, v) ∈ Rm+ × Rn+
Θ(u, v) = inf {L(x; u, v) : x ∈ Rn}
≤ inf{L(x; u, v) : x ∈ Rn+
}= inf
{L0(x, u)− 2v>x : x ∈ Rn+
}≤ inf
{L0(x, u) : x ∈ Rn+
}= Θsemi(u) .
Semi-Lagrangian bounds are always tighter
General principle: for any (u, v) ∈ Rm+ × Rn+
Θ(u, v) = inf {L(x; u, v) : x ∈ Rn}
≤ inf{L(x; u, v) : x ∈ Rn+
}= inf
{L0(x, u)− 2v>x : x ∈ Rn+
}≤ inf
{L0(x, u) : x ∈ Rn+
}= Θsemi(u) .
Therefore in fact
z∗dual,+ ≤ z∗semi ≤ z
∗+ .
Semi-Lagrangian bounds are always tighter
General principle: for any (u, v) ∈ Rm+ × Rn+
Θ(u, v) = inf {L(x; u, v) : x ∈ Rn}
≤ inf{L(x; u, v) : x ∈ Rn+
}= inf
{L0(x, u)− 2v>x : x ∈ Rn+
}≤ inf
{L0(x, u) : x ∈ Rn+
}= Θsemi(u) .
Therefore in fact
z∗dual,+ ≤ z∗semi ≤ z
∗+ .
Both bounds have conic representations !
SDP formulation of Lagrangian bounds
Select (d+1)×(d+1) matrices Mi such that qi(x) = y>Miy = Mi•Ywhere Y = [1, x>]>[1, x>].
• > > >
∗ { • • ≤ ∈ • }
{ • • ≤ ∈ • }
SDP formulation of Lagrangian bounds
Select (d+1)×(d+1) matrices Mi such that qi(x) = y>Miy = Mi•Ywhere Y = [1, x>]>[1, x>].
Y is psd. and satisfies J0 • Y = 1 where J0 = [1, o>][1, o>]>.
SDP formulation of Lagrangian bounds
Select (d+1)×(d+1) matrices Mi such that qi(x) = y>Miy = Mi•Ywhere Y = [1, x>]>[1, x>].
Y is psd. and satisfies J0 • Y = 1 where J0 = [1, o>][1, o>]>. Thus
z∗ = infY psd.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m] , J0 • Y = 1, rk Y = 1} .
SDP formulation of Lagrangian bounds
Select (d+1)×(d+1) matrices Mi such that qi(x) = y>Miy = Mi•Ywhere Y = [1, x>]>[1, x>].
Y is psd. and satisfies J0 • Y = 1 where J0 = [1, o>][1, o>]>. Thus
z∗ = infY psd.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m] , J0 • Y = 1, rk Y = 1} .
SDP relaxation: drop rank constraint, keep psd. condition,
z∗SDP,primal = infY psd.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m], J0 • Y = 1} .
SDP formulation of Lagrangian bounds
Select (d+1)×(d+1) matrices Mi such that qi(x) = y>Miy = Mi•Ywhere Y = [1, x>]>[1, x>].
Y is psd. and satisfies J0 • Y = 1 where J0 = [1, o>][1, o>]>. Thus
z∗ = infY psd.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m] , J0 • Y = 1, rk Y = 1} .
SDP relaxation: drop rank constraint, keep psd. condition,
z∗SDP,primal = infY psd.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m], J0 • Y = 1} .
Dual SDP problem: slack matrix S(y) = M0 − y0J0 +m∑i=1
uiMi,
z∗SDP,dual := sup{y0 ∈ R : S(y) psd., y = [y0, u
>]> ∈ R× Rm+}.
COP formulation of Semi-Lagrangian bounds
Theorem [Shor <’83, ’87]:
z∗dual = z∗SDP,dual ≤ z∗SDP,primal ≤ z
∗ .
>
COP formulation of Semi-Lagrangian bounds
Theorem [Shor <’83, ’87]:
z∗dual = z∗SDP,dual ≤ z∗SDP,primal ≤ z
∗ .
Further, no duality/relaxation gap,
z∗dual = z∗ if and only if
COP formulation of Semi-Lagrangian bounds
Theorem [Shor <’83, ’87]:
z∗dual = z∗SDP,dual ≤ z∗SDP,primal ≤ z
∗ .
Further, no duality/relaxation gap,
z∗dual = z∗ if and only if S(y) is psd.
for y> = [q0(x), u>] at a KKT pair (x, u) ∈ F × Rm+ for z∗.
COP formulation of Semi-Lagrangian bounds
Theorem [Shor <’83, ’87]:
z∗dual = z∗SDP,dual ≤ z∗SDP,primal ≤ z
∗ .
Further, no duality/relaxation gap,
z∗dual = z∗ if and only if S(y) is psd.
for y> = [q0(x), u>] at a KKT pair (x, u) ∈ F × Rm+ for z∗.
... sufficient condition for global optimality of KKT point x.
COP formulation of Semi-Lagrangian bounds
Theorem [Shor <’83, ’87]:
z∗dual = z∗SDP,dual ≤ z∗SDP,primal ≤ z
∗ .
Further, no duality/relaxation gap,
z∗dual = z∗ if and only if S(y) is psd.
for y> = [q0(x), u>] at a KKT pair (x, u) ∈ F × Rm+ for z∗.
... sufficient condition for global optimality of KKT point x.
Theorem [B.’14]: For the Semi-Lagrangian bound we have
z∗semi = z∗COP,dual
with the copositive optimization representation
z∗COP,dual := sup{y0 ∈ R : S(y) copositive, y = [y0, u
>]> ∈ R× Rm+}.
Copositive optimization - conic primal/dual pair
Lifting again: if x ∈ Rn+ and y> = [1, x>], then
Y = yy> is completely positive (cp.)
Copositive optimization - conic primal/dual pair
Lifting again: if x ∈ Rn+ and y> = [1, x>], then
Y = yy> is completely positive (cp.), so reformulation of z∗+:
z∗+ = infY cp.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m], J0 • Y = 1, rk Y = 1} ,
Copositive optimization - conic primal/dual pair
Lifting again: if x ∈ Rn+ and y> = [1, x>], then
Y = yy> is completely positive (cp.), so reformulation of z∗+:
z∗+ = infY cp.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m], J0 • Y = 1, rk Y = 1} ,
and its natural relaxation: drop rank constraint but keep Y cp.
z∗COP,primal = infY cp.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m], J0 • Y = 1} .
Copositive optimization - conic primal/dual pair
Lifting again: if x ∈ Rn+ and y> = [1, x>], then
Y = yy> is completely positive (cp.), so reformulation of z∗+:
z∗+ = infY cp.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m], J0 • Y = 1, rk Y = 1} ,
and its natural relaxation: drop rank constraint but keep Y cp.
z∗COP,primal = infY cp.
{M0 • Y : Mi • Y ≤ 0, i∈ [1:m], J0 • Y = 1} .
The conic dual of this problem is exactly z∗COP,dual.
Strong duality for copositive primal/dual pair
Theorem [B.’14]: Consider the copositive relaxation of z∗+.
If Qi is strictly copositive for one i∈ [0:m] and if F ◦+ 6= ∅, then
z∗COP,dual = z∗COP,primal and both values are attained.
> > ∈ ׯ
Strong duality for copositive primal/dual pair
Theorem [B.’14]: Consider the copositive relaxation of z∗+.
If Qi is strictly copositive for one i∈ [0:m] and if F ◦+ 6= ∅, then
z∗COP,dual = z∗COP,primal and both values are attained.
Further, the Semi-Lagrangean/copositive relaxation is tight,
z∗semi = z∗+ if and only if
Strong duality for copositive primal/dual pair
Theorem [B.’14]: Consider the copositive relaxation of z∗+.
If Qi is strictly copositive for one i∈ [0:m] and if F ◦+ 6= ∅, then
z∗COP,dual = z∗COP,primal and both values are attained.
Further, the Semi-Lagrangean/copositive relaxation is tight,
z∗semi = z∗+ if and only if S(y) is copositive
for y> = [q0(x), u>] at a generalized KKT pair (x, u) ∈ F+ × Rm+:
Strong duality for copositive primal/dual pair
Theorem [B.’14]: Consider the copositive relaxation of z∗+.
If Qi is strictly copositive for one i∈ [0:m] and if F ◦+ 6= ∅, then
z∗COP,dual = z∗COP,primal and both values are attained.
Further, the Semi-Lagrangean/copositive relaxation is tight,
z∗semi = z∗+ if and only if S(y) is copositive
for y> = [q0(x), u>] at a generalized KKT pair (x, u) ∈ F+ × Rm+:
i.e., Hux = du + v with vjxj = 0 and uiqi(x) = 0 for all i, j,
but without requiring vj ≥ 0.
Strong duality for copositive primal/dual pair
Theorem [B.’14]: Consider the copositive relaxation of z∗+.
If Qi is strictly copositive for one i∈ [0:m] and if F ◦+ 6= ∅, then
z∗COP,dual = z∗COP,primal and both values are attained.
Further, the Semi-Lagrangean/copositive relaxation is tight,
z∗semi = z∗+ if and only if S(y) is copositive
for y> = [q0(x), u>] at a generalized KKT pair (x, u) ∈ F+ × Rm+:
i.e., Hux = du + v with vjxj = 0 and uiqi(x) = 0 for all i, j,
but without requiring vj ≥ 0.
Again have sufficient condition for global optimality of x.
Ok. But how to employ it ?
So we know, under weak conditions,
z∗dual,+ ≤ z∗semi = z∗COP,dual = z∗COP,primal ≤ z
∗+ .
Ok. But how to employ it ?
So we know, under weak conditions,
z∗dual,+ ≤ z∗semi = z∗COP,dual = z∗COP,primal ≤ z
∗+ .
[Dickinson/Gijben ’14]: the copositive cone C?d and its dual cone
Cd are both intractable, checking membership is NP-hard.
Ok. But how to employ it ?
So we know, under weak conditions,
z∗dual,+ ≤ z∗semi = z∗COP,dual = z∗COP,primal ≤ z
∗+ .
[Dickinson/Gijben ’14]: the copositive cone C?d and its dual cone
Cd are both intractable, checking membership is NP-hard.
But can employ approximation hierarchies
Drd ⊂ C?d for all r = 0,1,2, . . .
of tractable cones Drd.
Ok. But how to employ it ?
So we know, under weak conditions,
z∗dual,+ ≤ z∗semi = z∗COP,dual = z∗COP,primal ≤ z
∗+ .
[Dickinson/Gijben ’14]: the copositive cone C?d and its dual cone
Cd are both intractable, checking membership is NP-hard.
But can employ approximation hierarchies
Drd ⊂ Dr+1d ⊂ C?d for all r = 0,1,2, . . .
of tractable cones Drd.
Approximation hierarchies by tractable cones
Dual cones [Drd]? shrink towards cp. cone Cd, e.g., if
[Drd]? = {Y ∈ Lrd : Y is psd.}
with Lrd polyhedral cones of d×d matrices with no negative entries
such that∞⋂r=0
Lrd = Cd .
Approximation hierarchies by tractable cones
Dual cones [Drd]? shrink towards cp. cone Cd, e.g., if
[Drd]? = {Y ∈ Lrd : Y is psd.}
with Lrd polyhedral cones of d×d matrices with no negative entries
such that∞⋂r=0
Lrd = Cd .
Theorem [B.’14]: The tractable bounds
z∗D,r = sup{y0 ∈ R : S(y) ∈ Drn+1, y = [y0, u
>]> ∈ R× Rm+}
increase with r
Approximation hierarchies by tractable cones
Dual cones [Drd]? shrink towards cp. cone Cd, e.g., if
[Drd]? = {Y ∈ Lrd : Y is psd.}
with Lrd polyhedral cones of d×d matrices with no negative entries
such that∞⋂r=0
Lrd = Cd .
Theorem [B.’14]: The tractable bounds
z∗D,r = sup{y0 ∈ R : S(y) ∈ Drn+1, y = [y0, u
>]> ∈ R× Rm+}
increase with r and are tighter than the Lagrangian bounds.
Survey of approximation constructions
Name symbol mode method remarksB./de Klerk E outer LP rational grid for ∆n
Yıldırım Y outer LP Y ⊂ E, gridBundfuss/Dur B outer LP simplicial partitionB./Dur/Teo B(M) outer LP M⊃ K∗Bundfuss/Dur D inner LP simplicial partitionSponsel et al. D(M) inner LP M⊂ K∗
Parrilo et al. I inner LP coeff p(d)S ≥ o
Parrilo et al. S inner SDP p(d)S is a s.o.s.
Pena et al. Q inner SDP I ⊂ Q ⊂ SLasserre L(µ, T ) outer SDP µ-moments over TDickinson/Povh L(µ, T ;A) outer SDP L(µ, T ;A) ⊂ L(µ, T )
Survey of approximation constructions
Name symbol mode method remarks
Bundfuss/Dur D inner LP simplicial partitionSponsel et al. D(M) inner LP M⊂ K∗
Parrilo et al. I inner LP coeff p(d)S ≥ o
Parrilo et al. S inner SDP p(d)S is a s.o.s.
Pena et al. Q inner SDP I ⊂ Q ⊂ S
Improvement even at approximation level zero !
Starting hierarchy with r = 0: take
L0d =
{X = X>d× d : min
i,jXij ≥ 0
}.
Then D0d = P+N is cone of nonnegatively decomposable (NND)
matrices [Jarre/Lieder ’13] and its dual [ (NND)? = DNN ]
[D0d ]? =
{X = X>d× d psd. : min
i,jXij ≥ 0
}is familiar doubly nonnegative (DNN) cone P ∩N .
Theorem [B.’14]: Even at zero approximation level, with abovechoice,
z∗dual,+ ≤ z∗D,0 ≤ z
∗semi = z∗COP ≤ z
∗+ .
Hopefully [experiments needed] first inequality is significant.
Selected references in chronological order
[Shor ’87] Quadratic optimization problems, Izv. Akad. Nauk SSSR Tekhn.Kibernet. 22, 128–139.
[Hotho et al.’06] Information retrieval in folksonomies: Search and ranking,in The Semantic Web: Research and Applications, 411-426. Springer,Heidelberg.
[Burer ’09] On the copositive representation of binary and continuous non-convex quadratic programs, Math. Programming 120, 479–495.
[Saha et al.’10] Dense Subgraphs with Restrictions and Applications to GeneAnnotation Graphs, in Research in Computational Molecular Biology,456–472, Springer, Berlin.
[Bhaskara et al.’12] On quadratic programming with a ratio objective, inProc.ICALP’12, 109-120. Springer, Berlin.
[Jarre/Lieder ’13] Computing a nonnegative decomposition of a matrix,talk presented on 1 August 2013 at ICCOPT, Lisbon.
Selected references, continued
[Dickinson/Gijben ’14] On the computational complexity of membershipproblems for the completely positive cone and its dual,Comput. Optim. Appl. 57, 403–415.
[B. ’14] Copositive relaxation beats Lagrangian dual bounds in quadraticallyand linearly constrained QPs,Isaac Newton Institute Preprint series NI13064-POP, submitted.
[Amaral/B.’15] Copositivity-based approximations for mixed-integer fractio-nal quadratic optimization, Pacific J Opt., to appear.
[Pena/Vera/Zuluaga ’15] Completely positive reformulations for polynomialoptimization, Math.Programming, DOI 10.1007/s10107-014-0822-9, toappear.