Some Aspects of the Interplay Between Convex Analysis …plc/sestri/combettes2014.pdf · Some Aspects of the Interplay Between Convex Analysis and Monotone Operator Theory Patrick

Some Aspects of the Interplay BetweenConvex Analysis and Monotone Operator

Theory

Patrick L. Combettes

Laboratoire Jacques-Louis LionsFaculte de Mathematiques

Universite Pierre et Marie Curie – Paris 675005 Paris, France

Sestri Levante, 12 settembre 2014

Patrick L. Combettes Convex Functions and Monotone Operators 1/17

Overview

Our objective is to discuss certain aspects of the impor-tance of the theory of monotone operators and of non-expansive operators in the analysis and the numerical so-lution of problems in inverse problems and learning theory,even when those admit a purely variational formulation.

Special emphasis is placed on the role played by duality.


Duality in Hilbert spaces

Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)

Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)



Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )

Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)



Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )

Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)



Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )

Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)



Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)

Nonexpansive operators: R → −R (l R = 2J − Id)



Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)


Problems in duality

Functional setting (Fenchel, Moreau): f + g → f ∗∨ + g∗

Composite functional setting (Rockafellar):

f + g ◦ L→ f ∗∨ ◦ L∗ + g∗

Maximally monotone operator setting (Mosco 1972(for V.I.), Mercier 1980, Attouch-Thera 1996):

A + B ↔ −A−1 ◦ (−Id) + B−1.

Composite maximally monotone operator setting (Robin-son 1999, Pennanen 2000):

A + L∗BL↔ −LA−1(−L∗) + B−1

Open problem 1: (Firmly) nonexpansive operator set-ting?


Problems in duality



f + g ◦ L→ f ∗∨ ◦ L∗ + g∗


A + B ↔ −A−1 ◦ (−Id) + B−1.


A + L∗BL↔ −LA−1(−L∗) + B−1



Problems in duality



f + g ◦ L→ f ∗∨ ◦ L∗ + g∗


A + B ↔ −A−1 ◦ (−Id) + B−1.


A + L∗BL↔ −LA−1(−L∗) + B−1



Problems in duality



f + g ◦ L→ f ∗∨ ◦ L∗ + g∗


A + B ↔ −A−1 ◦ (−Id) + B−1.


A + L∗BL↔ −LA−1(−L∗) + B−1



Problems in duality



f + g ◦ L→ f ∗∨ ◦ L∗ + g∗


A + B ↔ −A−1 ◦ (−Id) + B−1.


A + L∗BL↔ −LA−1(−L∗) + B−1



Open problem 1: Duality for nonexpansiveoperators?

Consider the inclusion 0 ∈ Ax + Bx , call P its solution setand D the dual solution set. Then:

P ={

x ∈ H | (∃u ∈ D) − u ∈ Ax and u ∈ Bx}

andD =

{u ∈ H | (∃ x ∈P) x ∈ A−1(−u) and x ∈ B−1u

}P = dom (A ∩ (−B)) and D = dom (A−1 ◦ (−Id) ∩ B−1)

Suppose that A and B are maximally monotone, setTA,B = JA(2JB − Id) + Id− JB, and let γ > 0. Then:

TA,B is firmly nonexpansiveP = JγB

(Fix TγA,γB

)D = γB

(Fix TγA,γB

)P 6= Ø⇔ D 6= Ø⇔ Fix TγA,γB 6= Ø


Open problem 2: True Fenchel duality formonotone operators?

Suppose that 0 ∈ sri(dom f − dom g). Then

inf(f + g)(H) = −min(f∨∗ + g∗)(H)

We always have

zer(A + B) 6= Ø ⇔ zer(−A−1 ◦ (−Id) + B−1) 6= Ø.

For suitable “closure” or “enlargement” operations,can

zer(A + B

)\ 6= Ø ⇒ zer(− A−1 ◦ (−Id) + B−1)[ 6= Ø

recover Fenchel duality?




inf(f + g)(H) = −min(f∨∗ + g∗)(H)

We always have

zer(A + B) 6= Ø ⇔ zer(−A−1 ◦ (−Id) + B−1) 6= Ø.


zer(A + B

)\ 6= Ø ⇒ zer(− A−1 ◦ (−Id) + B−1)[ 6= Ø





inf(f + g)(H) = −min(f∨∗ + g∗)(H)

We always have

zer(A + B) 6= Ø ⇔ zer(−A−1 ◦ (−Id) + B−1) 6= Ø.


zer(A + B

)\ 6= Ø ⇒ zer(− A−1 ◦ (−Id) + B−1)[ 6= Ø



Splitting methods: Quick overview

Traditional splitting techniques were developed in thelate 1970s for inclusions involving the sum of two maxi-mally monotone operators:

0 ∈ Ax + Bx

Open problem 3: The continuous-time dynamics lead-ing to FB are reasonably understood (−x ′ ∈ Ax + Bx);what about FBF and DR?Main algorithms:

Forward-backward method (Mercier, 1979)Douglas-Rachford method (Lions and Mercier, 1979)Forward-backward-forward algorithm (Tseng, 2000).

Until recently, general splitting methods lacked for thecomposite problem

0 ∈ Ax + L∗BLx


Splitting methods: Composite problems

L : H → G linear and bounded, A : H → 2H, B : G → 2G

maximally monotoneSolve

0 ∈ Ax + L∗BLx

Main issue: 3 objects (A,B, L) to split... and a binaryrelation ∈ binding themWe need to reduce a problem to a 2-object problemin a larger spaceKey: recast the problem in the primal dual space


Kuhn-Tucker set of a composite inclusion

Primal solutions: P ={

x ∈ H | 0 ∈ Ax + L∗BLx}

Dual solutions: D ={

v∗ ∈ G | 0 ∈ −L ◦ A−1(−L∗v∗) + B−1v∗}

Kuhn-Tucker set

Z ={

(x , v∗) ∈ H ⊕ G | −L∗v∗ ∈ Ax and Lx ∈ B−1v∗}

Z is a closed convex setZ 6= Ø⇔P 6= Ø⇔ D 6= ØZ ⊂P ×D




x ∈ H | 0 ∈ Ax + L∗BLx}


v∗ ∈ G | 0 ∈ −L ◦ A−1(−L∗v∗) + B−1v∗}

Kuhn-Tucker set

Z ={






x ∈ H | 0 ∈ Ax + L∗BLx}


v∗ ∈ G | 0 ∈ −L ◦ A−1(−L∗v∗) + B−1v∗}

Kuhn-Tucker set

Z ={





Strategy: find a Kuhn-Tucker pair (x , v∗) by apply-ing a standard (e.g., Douglas-Rachford or forward-backward-forward) method to a monotone+skew de-composition of the problem in H⊕ G.[

00

]∈[A 00 B−1

]︸︷︷︸

M

[xv∗

]+

[0 L∗

−L 0

]︸︷︷︸

S

[xv∗

]

L. M. Briceno-Arias and PLC, A monotone+skew splittingmodel for composite monotone inclusions in duality, SIAM J.Optim., vol. 21, 2011.

Even in minimization problems, such a framework can-not be reduced to a functional setting: monotone op-erator splitting is required!Possible limitation: linear inversions (in DR) or knowl-edge of ‖L‖ (in FBF) are necessary



Strategy: find a Kuhn-Tucker pair (x , v∗) by apply-ing a standard (e.g., Douglas-Rachford or forward-backward-forward) method to a monotone+skew de-composition of the problem in H⊕ G.[

00

]∈[A 00 B−1

]︸︷︷︸

M

[xv∗

]+

[0 L∗

−L 0

]︸︷︷︸

S

[xv∗

]

L. M. Briceno-Arias and PLC, A monotone+skew splittingmodel for composite monotone inclusions in duality, SIAM J.Optim., vol. 21, 2011.

Even in minimization problems, such a framework can-not be reduced to a functional setting: monotone op-erator splitting is required!Possible limitation: linear inversions (in DR) or knowl-edge of ‖L‖ (in FBF) are necessary


Revisiting the proximal point algorithm

A maximally monotone, (γn)n∈N ∈ ]0,+∞[N, zer A 6= Ø,

xn+1 = JγnAxn

Classical results by Brezis&Lions (1978):If∑

n∈N γ2n = +∞, then xn ⇀ x ∈ zer A

If A = ∂f (f ∈ Γ0(H)) and∑

n∈N γn = +∞, then xn ⇀ x ∈zer A = Argmin f

Is the proximal point algorithm of any use?



A : H → 2H maximally monotone, V be a closed vec-tor subspace of HAV : partial inverse of A w.r.t. V (Spingarn, 1983)

gra AV ={

(PV x + PV⊥u,PV u + PV⊥x) | (x,u) ∈ gra A}

Then (Spingarn, 1983):AV is maximally monotonez ∈ zer AV ⇔ (PV z,PV⊥z) ∈ gra Ap = JAV z ⇔ PV p + PV⊥(z − p) = JAz

Aside – Open problem 4: Let f ∈ Γ0(H) and V = {0}.Then (∂f )V = ∂f ∗, but how to define a partial conju-gate of f ? (in general (∂f )V is not a subdifferential)



A : H → 2H maximally monotone, V be a closed vec-tor subspace of HAV : partial inverse of A w.r.t. V (Spingarn, 1983)

gra AV ={

(PV x + PV⊥u,PV u + PV⊥x) | (x,u) ∈ gra A}

Then (Spingarn, 1983):AV is maximally monotonez ∈ zer AV ⇔ (PV z,PV⊥z) ∈ gra Ap = JAV z ⇔ PV p + PV⊥(z − p) = JAz

Aside – Open problem 4: Let f ∈ Γ0(H) and V = {0}.Then (∂f )V = ∂f ∗, but how to define a partial conju-gate of f ? (in general (∂f )V is not a subdifferential)



Back to the Kuhn-Tucker set

Z ={


Define H = H⊕ G, A : (x , y) 7→ Ax × By , andV =

{(x , y) ∈ H ⊕ G | Lx = y

}Apply the (monotone operator) proximal point algo-rithm to AV

M. A. Alghamdi, A. Alotaibi, PLC, and N. Shahzad, A primal-dual method of partial inverses for composite inclusions, Op-tim. Lett., March 2014.



TheoremSet Q = (Id + L∗L)−1 and assume that zer(A + L∗BL) 6= Ø. Let(λn)n∈N ∈ ]0, 2[

N, such that∑

n∈N λn(2 − λn) = +∞, let x0 ∈ H,v∗0 ∈ G, and set y0 = Lx0, u0 = −L∗v∗0 , and

(∀n ∈ N)

pn ≈ JA(xn + un)qn ≈ JB(yn + v∗n )rn = xn + un − pnsn = yn + vn − qntn = Q(rn + L∗sn)wn = Q(pn + L∗qn)xn+1 = xn − λntnyn+1 = yn − λnLtnun+1 = un + λn(wn − pn)v∗n+1 = v∗n + λn(Lwn − qn).

Then xn−wn → 0, yn−Lwn → 0, un−rn+tn → 0, and v∗n−sn+Ltn → 0,(xn, v∗n )⇀ (x , v∗) ∈ K .


A strongly convergent splitting method

We have seen that the Kuhn-Tucker set

Z ={


is a closed and convexThe previous methods require efficient linear inversionsschmes or knowledge of ‖L‖, neither of which may beavailable in many situations (e.g., domain decompo-sition methods). In addition they require stringent ad-ditional hypotheses to guarantee strong convergence(to an unspecified Kuhn-Tucker point)We present a method which alleviates all these limita-tions.

Aside – Open problem 5: in infinite-dimensional prob-lems is weak convergence relevant?



We have seen that the Kuhn-Tucker set

Z ={


is a closed and convexThe previous methods require efficient linear inversionsschmes or knowledge of ‖L‖, neither of which may beavailable in many situations (e.g., domain decompo-sition methods). In addition they require stringent ad-ditional hypotheses to guarantee strong convergence(to an unspecified Kuhn-Tucker point)We present a method which alleviates all these limita-tions.Aside – Open problem 5: in infinite-dimensional prob-lems is weak convergence relevant?



An abstract Haugazeau scheme:Let C be a nonempty closed convex subset of H andlet x0 ∈H. Iterate

for n = 0, 1, . . .⌊take xn+1/2 ∈H such that C ⊂ H(xn,xn+1/2)xn+1 = P

H(x0,xn)∩H(xn,xn+1/2

)x0

Suppose that, for every x ∈ H and every strictly in-creasing sequence (kn)n∈N in N, xkn ⇀ x ⇒ x ∈ C . Then(xn)n∈N is well defined and xn → PCx0.Strategy: Apply this principle to C = Z , construct xn+1/2by suitably choosing points in gra A and gra B

A. Alotaibi, PLC, and N. Shahzad, Best approximation fromthe Kuhn-Tucker set of composite monotone inclusions, J.Nonlinear Convex Anal., to appear.



TheoremFix (x0, v∗0 ) ∈ H× G and ε ∈ ]0, 1[, and iterate for every n ∈ N

(γn, µn) ∈ [ε, 1/ε]2

an = JγnA(xn − γnL∗v∗n ), ln = Lxn, bn = JµnB(ln + µnv∗n )s∗n = γ−1

n (xn − an) + µ−1n L∗(ln − bn), tn = bn − Lan

τn = ‖s∗n‖2 + ‖tn‖2, λn ∈ [ε, 1]θn = λn

(γ−1

n ‖xn − an‖2 + µ−1n ‖ln − bn‖2)/τn

xn+1/2 = xn − θns∗n , v∗n+1/2 = v∗n − θntn

χn =⟨x0 − xn | xn − xn+1/2

⟩+⟨v∗0 − v∗n | v∗n − v∗n+1/2

⟩µn = ‖x0 − xn‖2 + ‖v∗0 − v∗n ‖2, νn = ‖xn − xn+1/2‖2 + ‖v∗n − v∗n+1/2‖2

ρn = µnνn − χ2n

if ρn > 0 and χnνn > ρn⌊xn+1 = x0 + (1 + χn/νn)(xn+1/2 − xn)v∗n+1 = v∗0 + (1 + χn/νn)(v∗n+1/2 − v∗n )

if[ ETC... easy steps]

Then (xn, v∗n )→ PK (x0, v∗0 )


Some Aspects of the Interplay Between Convex Analysis …plc/sestri/combettes2014.pdf · Some Aspects of the Interplay Between Convex Analysis and Monotone Operator Theory Patrick

Documents