Fast algorithms for nonconvex compressive sensing

Fast algorithms for nonconvexcompressive sensing

Rick Chartrand

Los Alamos National Laboratory

New Mexico Consortium

February 25, 2009

Slide 1 of 16

Operated by Los Alamos National Security, LLC for NNSA

Outline

Motivating example

Nonconvex compressive sensing

Algorithms

Summary

Slide 2 of 16


Motivating example

Sparse tomographySuppose we want to reconstruct animage from samples of its Fouriertransform. How many samples dowe need?

Shepp-Logan phantom

Consider radial sampling,such as in MRI or (roughly)CT.

Ω

Slide 3 of 16


Motivating example

Nonconvexity

Fewer measurements are needed with nonconvex minimization:

minu‖∇u‖pp, subject to (Fu)|Ω = (Fx)|Ω.

With p = 1, solution is u = x with 18 lines ( |Ω||x| = 6.9%).

With p = 1/2, 10 lines suffice ( |Ω||x| = 3.8%).

backprojection, 18 lines p = 1, 18 lines p = 12

, 10 lines p = 1, 10 lines

Slide 4 of 16


Motivating example

Nonconvexity





backprojection, 18 lines p = 1, 18 lines

p = 12


Slide 4 of 16


Motivating example

Nonconvexity





backprojection, 18 lines p = 1, 18 lines p = 12


Slide 4 of 16


Motivating example

New results

These are old results (Mar. 2006); what’s new?

I Reconstruction (to 50 dB) in 13 seconds (versus literature-best1–3 minutes).

I Exact reconstruction from 9 lines (3.5% of Fourier transform).

10 lines fastest 10-line re-covery

9 lines recovery fromfewest samples

Slide 5 of 16


Motivating example

New results

These are old results (Mar. 2006); what’s new?I Reconstruction (to 50 dB) in 13 seconds (versus literature-best

1–3 minutes).

I Exact reconstruction from 9 lines (3.5% of Fourier transform).



Slide 5 of 16


Motivating example

New results

These are old results (Mar. 2006); what’s new?I Reconstruction (to 50 dB) in 13 seconds (versus literature-best

1–3 minutes).I Exact reconstruction from 9 lines (3.5% of Fourier transform).



Slide 5 of 16



Optimization for sparse recovery

I Let x ∈ RN be sparse: ‖Ψx‖0 = K, K N .I Suppose Φ is an M ×N matrix, M N , with Φ and Ψ

incoherent. For example, Φ = (ϕij), i.i.d. ϕij ∼ N(0, σ2).

minu‖Ψu‖0, s.t. Φu = Φx.


minu‖Ψu‖pp, s.t. Φu = Φx,

Unique solution is u = x with opti-mally small M , but is NP-hard.

M ≥ 2K suffices w.h.p.

Can be solved efficiently; requiresmore measurements for reconstruc-tion. M ≥ CK log(N/K)

where 0 < p < 1. Solvable in prac-tice; requires fewer measurementsthan `1.M ≥ C1(p)K+pC2(p)K log(N/K)(with V. Staneva)

Slide 6 of 16



Optimization for sparse recovery

I Let x ∈ RN be sparse: ‖Ψx‖0 = K, K N .I Suppose Φ is an M ×N matrix, M N , with Φ and Ψ

incoherent. For example, Φ = (ϕij), i.i.d. ϕij ∼ N(0, σ2).



minu‖Ψu‖pp, s.t. Φu = Φx,

Unique solution is u = x with opti-mally small M , but is NP-hard.

M ≥ 2K suffices w.h.p.

Can be solved efficiently; requiresmore measurements for reconstruc-tion. M ≥ CK log(N/K)

where 0 < p < 1. Solvable in prac-tice; requires fewer measurementsthan `1.M ≥ C1(p)K+pC2(p)K log(N/K)(with V. Staneva)

Slide 6 of 16



The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 2:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.1p

Slide 7 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.2p

Slide 7 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.3p

Slide 7 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.4p

Slide 7 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.5p

Slide 7 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.1p

Slide 8 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.2p

Slide 8 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.3p

Slide 8 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.4p

Slide 8 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.5p

Slide 8 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.6p

Slide 8 of 16



The geometry of `p


x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.7p

Slide 8 of 16



The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.1p

Φu = Φx

Slide 9 of 16



The geometry of `p


x

|u1|p + |u2|p + |u3|p = 0.2p

Φu = Φx

Slide 9 of 16



The geometry of `p


x

|u1|p + |u2|p + |u3|p = 0.3p

Φu = Φx

Slide 9 of 16



The geometry of `p


x

|u1|p + |u2|p + |u3|p = 0.4p

Φu = Φx

Slide 9 of 16



The geometry of `p


x

|u1|p + |u2|p + |u3|p = 0.5p

Φu = Φx

Slide 9 of 16



The geometry of `p


x

|u1|p + |u2|p + |u3|p = 0.6p

Φu = Φx

Slide 9 of 16



The geometry of `p


x

|u1|p + |u2|p + |u3|p = 0.7p

Φu = Φx

Slide 9 of 16



The geometry of `p


x

|u1|p + |u2|p + |u3|p = 0.8p

Φu = Φx

Slide 9 of 16



The geometry of `p


x

|u1|p + |u2|p + |u3|p = 0.9p

Φu = Φx

Slide 9 of 16



The geometry of `p


x

|u1|p + |u2|p + |u3|p = 1p

Φu = Φx

Slide 9 of 16



Why might global minimization be possible?

Consider the ε-regularized, constraint-eliminated objective:

Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,

whereR(V ) = N (Φ). A moderate ε fills in the local minima.

Slide 10 of 16





Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,


ε = 0

Slide 10 of 16





Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,


ε = 1

Slide 10 of 16





Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,


ε = 0.1

Slide 10 of 16





Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,


ε = 0.01

Slide 10 of 16





Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,


ε = 0.001

Slide 10 of 16


Algorithms

Semiconvex regularization

Now we generalize an approach of J. Yang, W. Yin, Y. Zhang, and Y.Wang. Consider a mollified `p objective on R2:

ϕ(t) =

γ|t|2 if |t| ≤ α|t|p/p− δ if |t| > α

The parameters are chosen tomake ϕ ∈ C1.

ϕ(t)

t1α

Now we seek ψ such that

ϕ(t) = minw

ψ(w) + (β/2)|t− w|22

This can be found by convex duality, as |t|22/2−ϕ(t)/β is convex ifβ = αp−2.

Slide 11 of 16


Algorithms

Semiconvex regularization

Now we generalize an approach of J. Yang, W. Yin, Y. Zhang, and Y.Wang. Consider a mollified `p objective on R2:

ϕ(t) =

γ|t|2 if |t| ≤ α|t|p/p− δ if |t| > α

The parameters are chosen tomake ϕ ∈ C1.

ϕ(t)

t1α

Now we seek ψ such that

ϕ(t) = minw

ψ(w) + (β/2)|t− w|22

This can be found by convex duality, as |t|22/2−ϕ(t)/β is convex ifβ = αp−2.

Slide 11 of 16


Algorithms

A splitting approach

Now we consider an unconstrained `p minimization problem, andreplace

minu

N∑i=1

ϕ((Du)i) + (µ/2)‖Φu− b‖22

with the split version

minu,w

N∑i=1

ψ(wi) + (β/2)‖Du− w‖22 + (µ/2)‖Φu− b‖22,

which we solve by alternate minimization.

Slide 12 of 16


Algorithms

Easy iterations

Holding u fixed, the w-subproblem is separable, and its solutioncomes from the convex duality:

wi = max

0, |(Du)i| −

|(Du)i|p−1

β

(Du)i|(Du)i|

.

This generalizes shrinkage ( or soft thresholding ) to `p.

Holding w fixed, the u-problem is quadratic:

(βDTD + µΦTΦ)u = βDTw + µΦT b.

If Φ is a Fourier sampling operator, we can solve this in the Fourierdomain. This is very fast!

Slide 13 of 16


Algorithms

Easy iterations


wi = max

0, |(Du)i| −

|(Du)i|p−1

β

(Du)i|(Du)i|

.





Slide 13 of 16


Algorithms

Easy iterations


wi = max

0, |(Du)i| −

|(Du)i|p−1

β

(Du)i|(Du)i|

.





Slide 13 of 16


Algorithms

Enforcing equality

minu,w

N∑i=1

ψ(wi) + (β/2)‖Du− w‖22 + (µ/2)‖Φu− b‖22

Typically one enforces w = Du and Φu = b by iteratively growingβ and µ larger (continuation).

We get better results from Bregman iteration (generalizing T.Goldstein, S. Osher):

minu,w

N∑i=1

ψ(wi)+(β/2)‖Du−w−Bw‖22 +(µ/2)‖Φu−b−Bu‖22,

and update Bn+1w = Bnw + w −Du (inner loop),

Bm+1u = Bmu + b− Φu (outer loop, if desired).

Slide 14 of 16


Algorithms

Enforcing equality

minu,w

N∑i=1

ψ(wi) + (β/2)‖Du− w‖22 + (µ/2)‖Φu− b‖22

Typically one enforces w = Du and Φu = b by iteratively growingβ and µ larger (continuation).

We get better results from Bregman iteration (generalizing T.Goldstein, S. Osher):

minu,w

N∑i=1

ψ(wi)+(β/2)‖Du−w−Bw‖22 +(µ/2)‖Φu−b−Bu‖22,

and update Bn+1w = Bnw + w −Du (inner loop),

Bm+1u = Bmu + b− Φu (outer loop, if desired).

Slide 14 of 16


Algorithms

A surprise

I An early coding mistake led to the use of p = −1/2. This iswhat made the 9-line phantom reconstruction possible.

I Results decay slowly as p is decreased below −1/2.I Negative p-values were used by Rao and Kreutz-Delgado in an

iteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.

I A mollified IRLS approach (with Wotao Yin) apparently avoidslocal minima, but negative p results are not better than positivep results.

Slide 15 of 16


Algorithms

A surprise


I Results decay slowly as p is decreased below −1/2.

I Negative p-values were used by Rao and Kreutz-Delgado in aniteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.


Slide 15 of 16


Algorithms

A surprise





Slide 15 of 16


Algorithms

A surprise





Slide 15 of 16


Summary

Summary

I Nonconvex compressive sensing allows sparse signals to berecovered with even fewer measurements than “traditional”compressive sensing.

I Decreasing p also improves robustness to noise, and speedsup convergence.

I Regularizing the objective appears to keep algorithms fromconverging to nonglobal minima.

I For Fourier-sampling measurements, such as MRI, a very fastalgorithm is available.

math.lanl.gov/~rick

Slide 16 of 16


math.lanl.gov/~rick

Fast algorithms for nonconvex compressive sensing

Documents