Enhanced Compressed Sensing based on Iterative Support ...

Overview Theoretical Results Numerical Results Conclusions

Enhanced Compressed Sensingbased on Iterative Support Detection

Wotao Yin

Department of Computational and Applied MathematicsRice University

Joint work with Yilun Wang

Supported by ONR and NSF

Wotao Yin Enhanced Compressed Sensing based on Iterative Support Detection


Notation

x : sparse signal, has ≤ k nonzero entriesb = Ax : CS measurements

`0-problem: min ‖x‖0, s.t. Ax = b. Exact recovery needs m ≥ 2kfor Gaussian A`1-problem: min ‖x‖1, s.t. Ax = b. Needs a much bigger mAlso called Basis Pursuit



Notation

x : sparse signal, has ≤ k nonzero entriesb = Ax : CS measurements

`0-problem: min ‖x‖0, s.t. Ax = b. Exact recovery needs m ≥ 2kfor Gaussian A`1-problem: min ‖x‖1, s.t. Ax = b. Needs a much bigger mAlso called Basis Pursuit


Overview Theoretical Results Numerical Results Conclusions The Approach Simple Examples

Outline

1 OverviewThe ApproachSimple Examples

2 Theoretical ResultsSummaryThe Null Space PropertyRecoverability Improvement

3 Numerical ResultsNoiseless measurementsNoisy measurementsA failed case

4 Conclusions



Approach

Goal: to beat the `1-minimization, i.e., basis pursuitRecover x from less measurementsRemain computationally tractable

Iterative approach:If `1-minimization fails, detect good in the (wrong) solutionRemove the discoveries from the `1-norm.T : remaining entries, ‖xT‖1 =

∑i∈T |xi |,

T C : discoveries = correct ∪ wrong, out of `1-norm.t = |T |Solve

Truncate `1-problem: minx ‖xT‖1, s.t. Ax = b.



Approach


Iterative approach:If `1-minimization fails, detect good in the (wrong) solution

Remove the discoveries from the `1-norm.T : remaining entries, ‖xT‖1 =

∑i∈T |xi |,





Approach


Iterative approach:If `1-minimization fails, detect good in the (wrong) solutionRemove the discoveries from the `1-norm.

T : remaining entries, ‖xT‖1 =∑

i∈T |xi |,T C : discoveries = correct ∪ wrong, out of `1-norm.t = |T |Solve




Approach



∑i∈T |xi |,

T C : discoveries = correct ∪ wrong, out of `1-norm.t = |T |

Solve




Approach



∑i∈T |xi |,





A Simple ExampleSetup:

n = 200, k = 25, m = 2k = 50, A is Gaussian random

Basis pursuit result: x (1)

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2L1 Minimization

true signalrecovered signal





Basis pursuit result: x (1)

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2L1 Minimization

true signalcorrect recoveryfalse recovery





Basis pursuit result: x (1), threshold ε = ‖x (1)‖∞/3

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2L1 Minimization




A Thresholding Framework

Initialize: j ← 1 and T = {1,2, . . . ,n}.While not converged do

1 Truncated `1-minimization:

x (j) ← min ‖xT‖1 s.t. Ax = b.

2 Support detection by thresholding:

ε← ‖x (j)‖∞/3j ,

T ← {i : |x (j)i | < ε}.



Results of Iterative Thresholding

Basis pursuit result: x (1), threshold ε = ‖x (1)‖∞/3

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2L1 Minimization





Truncated `1-result: x (2), reduced threshold ε = ‖x (2)‖∞/32

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2Truncated L1 Minimization





Truncated `1-result: x (3), reduced threshold ε = ‖x (3)‖∞/33

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5






Truncated `1-result: x (4), exact recovery!

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5


true signalcorrect recovery



Robustness TestTry tighter thresholds:

ε = ‖x (j)‖∞/5j .

j = 1, basis pursuit result:

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

21st iteration. (total,det,good,bad)=(25,20,13,7), Err = 5.14e−001





ε = ‖x (j)‖∞/5j .

j = 2, truncated `1-minimization result:

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

22nd iteration. (total,det,good,bad)=(25,27,21,6), Err = 1.29e−001





ε = ‖x (j)‖∞/5j .

j = 3, truncated `1-minimization result:

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

23rd iteration. (total,det,good,bad)=(25,25,25,0), Err = 1.93e−015





ε = ‖x (j)‖∞/5j .

j = 4, truncated `1-minimization result: exact recovery!

0 50 100 150 200−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

24th iteration. (total,det,good,bad)=(25,25,25,0), Err = 6.56e−016

true signalcorrect recovery


Overview Theoretical Results Numerical Results Conclusions Summary The Null Space Property Recoverability Improvement

Outline




4 Conclusions



Summary of Results

Q: Condition for an exact recovery?

Truncated Null Space Property (T-NSP) holds with γ < 1.

Q: How good is a support discovery?

= γ(j) − γ(j+1). To have γ(j) > γ(j+1):(inc. correct discoveries) / (inc. wrong discoveries) > γ(j)

Q: Not knowing the exact solutionhow to have enough correct discoveries?thresholding for fast decaying signals. Ranking is not as robust.how to measure improvement?compute the size of tail.when to stop?tail is zero or small enough.



Summary of Results


Truncated Null Space Property (T-NSP) holds with γ < 1.Q: How good is a support discovery?





Summary of Results




Q: Not knowing the exact solutionhow to have enough correct discoveries?thresholding for fast decaying signals. Ranking is not as robust.

how to measure improvement?compute the size of tail.when to stop?tail is zero or small enough.



Summary of Results




Q: Not knowing the exact solutionhow to have enough correct discoveries?thresholding for fast decaying signals. Ranking is not as robust.how to measure improvement?compute the size of tail.

when to stop?tail is zero or small enough.



Summary of Results







Null Space Property

A sufficient condition for min{‖x‖1 : Ax = b} to yield the right x .

Observe {x : Ax = b} = x +N (A). We need ‖x‖1 < ‖x + v‖1 forall v ∈ N (A).Let S = {i : xi 6= 0}.

‖x + v‖1 = ‖xS + vS‖1 + ‖0 + vSC‖1

= (‖xS + vS‖1 − ‖xS‖1 + ‖vS‖1) + ‖xS‖1 + ‖vSC‖1 − ‖vS‖1

= (‖xS + vS‖1 − ‖xS‖1 + ‖vS‖1)︸︷︷︸≥0

+‖x‖1 + ‖vSC‖1 − ‖vS‖1︸︷︷︸≥0??

.

We need ‖vS‖1 < ‖vSC‖1.A necessary condition for uniform exact recovery for all|S|-sparse signals.



Null Space Property

A sufficient condition for min{‖x‖1 : Ax = b} to yield the right x .Observe {x : Ax = b} = x +N (A). We need ‖x‖1 < ‖x + v‖1 forall v ∈ N (A).

Let S = {i : xi 6= 0}.

‖x + v‖1 = ‖xS + vS‖1 + ‖0 + vSC‖1


= (‖xS + vS‖1 − ‖xS‖1 + ‖vS‖1)︸︷︷︸≥0

+‖x‖1 + ‖vSC‖1 − ‖vS‖1︸︷︷︸≥0??

.




Null Space Property

A sufficient condition for min{‖x‖1 : Ax = b} to yield the right x .Observe {x : Ax = b} = x +N (A). We need ‖x‖1 < ‖x + v‖1 forall v ∈ N (A).Let S = {i : xi 6= 0}.

‖x + v‖1 = ‖xS + vS‖1 + ‖0 + vSC‖1


= (‖xS + vS‖1 − ‖xS‖1 + ‖vS‖1)︸︷︷︸≥0

+‖x‖1 + ‖vSC‖1 − ‖vS‖1︸︷︷︸≥0??

.

We need ‖vS‖1 < ‖vSC‖1.

A necessary condition for uniform exact recovery for all|S|-sparse signals.



Null Space Property

A sufficient condition for min{‖x‖1 : Ax = b} to yield the right x .Observe {x : Ax = b} = x +N (A). We need ‖x‖1 < ‖x + v‖1 forall v ∈ N (A).Let S = {i : xi 6= 0}.

‖x + v‖1 = ‖xS + vS‖1 + ‖0 + vSC‖1


= (‖xS + vS‖1 − ‖xS‖1 + ‖vS‖1)︸︷︷︸≥0

+‖x‖1 + ‖vSC‖1 − ‖vS‖1︸︷︷︸≥0??

.




Null Space Property

Definition (Cohen-Dahmen-DeVore and others)

A ∈ Rm×n has the Null Space Property (NSP) with order L and γ > 0if

‖vS‖1 ≤ γ‖vSc‖1, ∀|S| ≤ L, v ∈ N (A).

Usage:A has NSP with L and γ < 1⇒

Uniform exact recovery of all L-sparse signalsFor k > L, a recovery error bound for k -sparse signals

Minimal γ is monotonic in LNSP is weaker than RIP and can be obtained from RIPNSP is more essential than RIP for basis pursuit (left multiplyingA by a nonsingular matrix changes RIP but not NSP)



Null Space Property



‖vS‖1 ≤ γ‖vSc‖1, ∀|S| ≤ L, v ∈ N (A).






Null Space Property



‖vS‖1 ≤ γ‖vSc‖1, ∀|S| ≤ L, v ∈ N (A).



Minimal γ is monotonic in L

NSP is weaker than RIP and can be obtained from RIPNSP is more essential than RIP for basis pursuit (left multiplyingA by a nonsingular matrix changes RIP but not NSP)



Null Space Property



‖vS‖1 ≤ γ‖vSc‖1, ∀|S| ≤ L, v ∈ N (A).



Minimal γ is monotonic in LNSP is weaker than RIP and can be obtained from RIP

NSP is more essential than RIP for basis pursuit (left multiplyingA by a nonsingular matrix changes RIP but not NSP)



Null Space Property



‖vS‖1 ≤ γ‖vSc‖1, ∀|S| ≤ L, v ∈ N (A).






Truncated Null Space Property

Definition (Y.-Wang)

A ∈ Rm×n has the Truncated Null Space Property (T-NSP) with t , L,and γ, written as T-NSP(t ,L, γ), if

‖vS‖1 ≤ γ‖vT\S‖1, ∀S ⊂ T , |S| ≤ L, |T | = t , v ∈ N (A).

Intuitively, T-NSP(t ,L, γ)⇔ all length-t subvectors of v ∈ N (A) satisfythe inequality of NSP(L, γ)

Theorem (Y.-Wang)

For T given , if A satisfies T-NSP(|T |,L, γ) where γ < 1, thentruncated `1-minimization over the support of T yields an exactrecovery.



Truncated Null Space Property

Definition (Y.-Wang)

A ∈ Rm×n has the Truncated Null Space Property (T-NSP) with t , L,and γ, written as T-NSP(t ,L, γ), if

‖vS‖1 ≤ γ‖vT\S‖1, ∀S ⊂ T , |S| ≤ L, |T | = t , v ∈ N (A).

Intuitively, T-NSP(t ,L, γ)⇔ all length-t subvectors of v ∈ N (A) satisfythe inequality of NSP(L, γ)

Theorem (Y.-Wang)

For T given , if A satisfies T-NSP(|T |,L, γ) where γ < 1, thentruncated `1-minimization over the support of T yields an exactrecovery.



Recoverability Improvement

Theorem (Y.-Wang)

Suppose A satisfies both T-NSP(t1,L1, γ1) and T-NSP(t2,L2, γ2)where t2 < t1 and γ1 and γ2 are minimal. Then,

L1 − L2

(t1 − t2)− (L1 − L2)> γ1 =⇒ γ2 < γ1.

Interpretation:t1 = |T 1|: numbers of entries in T before detectiont2 = |T 2|: numbers of entries in T after detectiont1 − t2 = decrease in |T | = increase in the total discoveriesL1 − L2 = increase in the correct discoveriesγ2 < γ1: recoverability improved (recall γ < 1⇒ exact recovery)To improve, it is sufficient to have(inc. corr. discoveries) / (inc. false discoveries) > γ1

Result is in dependent of support detectors.




Theorem (Y.-Wang)


L1 − L2

(t1 − t2)− (L1 − L2)> γ1 =⇒ γ2 < γ1.

Interpretation:t1 = |T 1|: numbers of entries in T before detectiont2 = |T 2|: numbers of entries in T after detection

t1 − t2 = decrease in |T | = increase in the total discoveriesL1 − L2 = increase in the correct discoveriesγ2 < γ1: recoverability improved (recall γ < 1⇒ exact recovery)To improve, it is sufficient to have(inc. corr. discoveries) / (inc. false discoveries) > γ1





Theorem (Y.-Wang)


L1 − L2

(t1 − t2)− (L1 − L2)> γ1 =⇒ γ2 < γ1.

Interpretation:t1 = |T 1|: numbers of entries in T before detectiont2 = |T 2|: numbers of entries in T after detectiont1 − t2 = decrease in |T | = increase in the total discoveries

L1 − L2 = increase in the correct discoveriesγ2 < γ1: recoverability improved (recall γ < 1⇒ exact recovery)To improve, it is sufficient to have(inc. corr. discoveries) / (inc. false discoveries) > γ1





Theorem (Y.-Wang)


L1 − L2

(t1 − t2)− (L1 − L2)> γ1 =⇒ γ2 < γ1.

Interpretation:t1 = |T 1|: numbers of entries in T before detectiont2 = |T 2|: numbers of entries in T after detectiont1 − t2 = decrease in |T | = increase in the total discoveriesL1 − L2 = increase in the correct discoveries

γ2 < γ1: recoverability improved (recall γ < 1⇒ exact recovery)To improve, it is sufficient to have(inc. corr. discoveries) / (inc. false discoveries) > γ1





Theorem (Y.-Wang)


L1 − L2

(t1 − t2)− (L1 − L2)> γ1 =⇒ γ2 < γ1.






Theorem (Y.-Wang)


L1 − L2

(t1 − t2)− (L1 − L2)> γ1 =⇒ γ2 < γ1.





Results for Random Sampling

Theorem (Y.-Wang, an extension to Candes-Tao and Zhang)

For Gaussian random A (or any rank-m matrix A such that BA> = 0where B ∈ R(n−m)×m is Gaussian random), a sufficient condition forexact recovery with high probability is

‖xT‖0 <C2

4m − d

1 + log n−dm−d

,

where d = n − |T | and C is an independent constant.

Application: Bound C and show that

−1 <∂RHS∂d

< 0,

leaving room for incorrect discoveries.



Results for Random Sampling

Theorem (Y.-Wang, an extension to Candes-Tao and Zhang)

For Gaussian random A (or any rank-m matrix A such that BA> = 0where B ∈ R(n−m)×m is Gaussian random), a sufficient condition forexact recovery with high probability is

‖xT‖0 <C2

4m − d

1 + log n−dm−d

,

where d = n − |T | and C is an independent constant.

Application: Bound C and show that

−1 <∂RHS∂d

< 0,

leaving room for incorrect discoveries.


Overview Theoretical Results Numerical Results Conclusions Noiseless measurements Noisy measurements A failed case

Outline




4 Conclusions



Numerical Results

Experiment 1: noiseless measurementsn = 100, m = 50k = 9, . . . ,21. Each k had 200 trials.x : sparse Gaussian signalsA: Gaussian randomSuccessful recovery declared if ‖x (j) − x‖∞ ≤ 10−3

Thresholds: ε = ‖x (j)‖∞/2j

Empirical exact recovery conditions:Basis pursuit:

k ≤ m5

With iterative support detection:

k ≤ m3



Numerical Results

Experiment 1: noiseless measurementsn = 100, m = 50k = 9, . . . ,21. Each k had 200 trials.x : sparse Gaussian signalsA: Gaussian randomSuccessful recovery declared if ‖x (j) − x‖∞ ≤ 10−3

Thresholds: ε = ‖x (j)‖∞/2j

Empirical exact recovery conditions:Basis pursuit:

k ≤ m5

With iterative support detection:

k ≤ m3



Numerical Results

Percentage of Successful Recoveries

8 10 12 14 16 18 20 22

0.4

0.5

0.6

0.7

0.8

0.9

1

Number of nonzeros k

Pro

babi

lity

of S

ucce

ssfu

l Rec

over

y[Gaussian,Gaussian]

plain L1 minimization1 iterations2 iterations4 iterations8 iterations



Numerical Results

Experiment 2: noisy measurementsn = 100, m = 50k = 9,11,15,19. Each k had 200 trials.x : sparse Gaussian signalsA: Gaussian randomb = Ax + z, where z ∼ N(0,0.001)

Logarithms of relative errors of x (j) to x are plottedThresholds: ε = ‖x (j)‖∞/2j



Numerical Results

0 50 100 150 20010

−3

10−2

10−1

200 trials

Rel

ativ

e er

ror

[Gaussian,Gaussian], k=9, σ=0.001

plain L1 minimization4 iterations

0 50 100 150 20010

−3

10−2

10−1

200 trialsR

elat

ive

erro

r





Numerical Results

0 50 100 150 20010

−3

10−2

10−1

100

200 trials

Rel

ativ

e er

ror



0 50 100 150 20010

−3

10−2

10−1

100

200 trialsR

elat

ive

erro

r





Numerical Results

Experiment 3: sparse signals with nonzero = ±1, noiselessmeasurements

0 50 100 150 200−1.5

−1

−0.5

0

0.5

1L1 Minimization


Excessive false detections!



Numerical ResultsExperiment 3: signals with Bernoulli nonzeros, noiselessmeasurements

8 10 12 14 16 18 20 22

0.4

0.5

0.6

0.7

0.8

0.9

1

Number of nonzeros k

Pro

babi

lity

of S

ucce

ssfu

l Rec

over

y

[Bernulli,Gaussian]

plain L1 minimization1 iterations2 iterations4 iterations8 iterations

Little improvement over basis pursuit.



Outline




4 Conclusions



Conclusions

ConclusionsEffective support detection improves CS recovery

In particular, iterative thresholding is effective on sparse signalswith fast decaying distribution of nonzero valuesComputationally tractable

one `1-minimization per iteration, can be warm-startedonly a small number of iterations are needed



Conclusions

ConclusionsEffective support detection improves CS recoveryIn particular, iterative thresholding is effective on sparse signalswith fast decaying distribution of nonzero values

Computationally tractableone `1-minimization per iteration, can be warm-startedonly a small number of iterations are needed



Conclusions

ConclusionsEffective support detection improves CS recoveryIn particular, iterative thresholding is effective on sparse signalswith fast decaying distribution of nonzero valuesComputationally tractable

one `1-minimization per iteration, can be warm-startedonly a small number of iterations are needed



On-going Work

On-going workother types of signals: images, video, etc.

other priors: ‖Φx‖p, p ≤ 1, and TV (x)

further theoretical analysis is underwayapply to greedy algorithms (OMP, ROMP, CoSaMP, ...).



On-going Work

On-going workother types of signals: images, video, etc.other priors: ‖Φx‖p, p ≤ 1, and TV (x)




On-going Work


further theoretical analysis is underway

apply to greedy algorithms (OMP, ROMP, CoSaMP, ...).



On-going Work





AcknowledgementsColleaguesRice: Rich Baraniuk, Volkan Cevher, Kevin Kelly, Yin ZhangColumbia: Donald Goldfarb, Shiqian Ma, Zaiwen WenUCLA: Stan Osher, Jerome Darbon, Bin Dong, Yu MaoLos Alamos: Rick Chartrand, Simon MorganAlabama: Weihong GuoStudents: Junfeng Yang, Yilun WangFunding Agencies: ONR, NSF

CS Resources: www.dsp.ece.rice.edu/cs

Our algorithms: www.caam.rice.edu/˜optimization/L1

Thank You!


www.dsp.ece.rice.edu/cs

www.caam.rice.edu/~optimization/L1

Enhanced Compressed Sensing based on Iterative Support ...

Documents