Toward 0-norm Reconstruction, and A Nullspace Technique for Compressive Sampling Christine Law Gary Glover Dept. of EE, Dept. of Radiology Stanford University.

Toward 0-norm Toward 0-norm Reconstruction, and A Reconstruction, and A Nullspace Technique Nullspace Technique

for Compressive for Compressive SamplingSamplingChristine LawChristine LawGary GloverGary Glover

Dept. of EE, Dept. of Dept. of EE, Dept. of RadiologyRadiology

Stanford UniversityStanford University

OutlineOutline

0-norm Magnetic Resonance 0-norm Magnetic Resonance Imaging (MRI) reconstruction.Imaging (MRI) reconstruction. HomotopicHomotopic Convex iterationConvex iteration

Signal separation exampleSignal separation example

1-norm deconvolution in fMRI. 1-norm deconvolution in fMRI. Improvement in cardinality Improvement in cardinality

constraint problem with nullspace constraint problem with nullspace technique.technique.

Shannon Shannon vsvs. Sparse . Sparse SamplingSampling

Nyquist/Shannon : sampling rate ≥ 2*max freqNyquist/Shannon : sampling rate ≥ 2*max freq

Sparse Sampling Theorem: (Candes/Donoho Sparse Sampling Theorem: (Candes/Donoho 2004)2004)

Suppose Suppose xx in in RRnn is is kk-sparse and we are given -sparse and we are given mm Fourier coefficients with frequencies selected Fourier coefficients with frequencies selected uniformly at random. If uniformly at random. If

mm ≥ ≥ kk log log22 (1 + (1 + n n / / kk))

then then

reconstructs reconstructs xx exactly with overwhelming exactly with overwhelming probability. probability.

1arg min s.t. =x x x y

n

k = 2

Rat dies 1 week after drinking poisoned wine

Example by Anna Gilbert

x1 x2 x3 x4 x5 x6 x7

y1

y2

y3

10011 01

0101011

0010111

x1 x2 x3 x4 x5 x6 x7

y1

y2

y3

1001 01

0101011

00

1

110 11

n=7m=3

k=1

Reconstruction by Reconstruction by OptimizationOptimization

Compressed Sensing theory (2004 Donoho, Candes): under certain conditions,

Candes et al. IEEE Trans. Information Theory 2006 52(2):489 Donoho. IEEE Trans. Information Theory 2006 52(4):1289



y are measurements (rats)

x are sensors (wine)

0-norm reconstruction0-norm reconstruction

Try to solve 0-norm directly.Try to solve 0-norm directly.

For For pp-norm, where 0 < -norm, where 0 < pp < 1 < 1

Chartrand (2006) demonstrated fewer samples Chartrand (2006) demonstrated fewer samples y y required than 1-norm formulation.required than 1-norm formulation.

Chartrand. IEEE Signal Processing Letters. 2007: 14(10) 707-710.Chartrand. IEEE Signal Processing Letters. 2007: 14(10) 707-710.

0arg min s.t. =u u u y

Trzasko (2007): Rewrite the Trzasko (2007): Rewrite the problem problem

0, .min lim s.t

i iu u u y

00, lim

i iu u

Trzasko Trzasko et alet al. IEEE SP 14th workshop on statistical signal processing. 2007. 176-180.. IEEE SP 14th workshop on statistical signal processing. 2007. 176-180.

0arg min s.t. =u u u y

where where is tanh, laplace, log, is tanh, laplace, log, etcetc. . such thatsuch that

Homotopic function in Homotopic function in 1D1D

Laplace function: 1 xe

Start as 1-norm problem, then reduce slowly and approach 0-norm function.

1000

1

Homotopic methodHomotopic method

2

2min ,

2iiE

uu u y

T H,

,

Find minimum of by zeroing Lagrangian gradient:

= 0

isdiag

u

E

E u u u y

d d uu

u

T

11 T

1

,

,

It's a contraction

Iterate Conjugate Gradient method until .

.

t t

t t

u u y

u u y

u u

DemonstrationDemonstration when is big (1when is big (1stst iteration), solving 1-norm problem. iteration), solving 1-norm problem. reduce reduce to approach 0-norm solution. to approach 0-norm solution.

061%x

02%x

original x∆

: image to solve

: k-space measurements (undersampled)

: Total Variation (TV) operator

: Fourier matrix (masked)

u

y

Example 1Example 1

F

F

original phantom - reconstructionerror 20 log

original phantom

original subsampled

Zero-filled Reconstruction Fourier sample mask

HomotopicHomotopic result: use 4% Fourier data error: -66.2 dB 85 seconds

1-norm result: use 4% Fourier data error: -11.4 dB 542 seconds

1-normrecon

homotopicrecon

Example 2Example 2

AngiographyAngiography 360x360, 27.5% radial samples360x360, 27.5% radial samples

original

reconstructionreconstruction1-norm method: 1-norm method: error: -24.7 dB, error: -24.7 dB, 1151 seconds1151 seconds

360x360360x360

reconstruction reconstruction homotopichomotopic method: method:

error: -26.5 dB, error: -26.5 dB, 101 seconds101 seconds

original

27.5% samples27.5% samples

2

2

0=

min ,2

arg min s.t. u y

E uu

u u

u y

0,1i

Chretien, An Alternating l1 approach to the compressed sensing problem, arXiv.org .Dattorro, Convex Optimization & Euclidean Distance Geometry, Meboo.

Convex Iteration

Convex IterationConvex Iteration2

2min ,

2E u

uu y

T H

Find minimum of by zeroing Lagrangian gradient:

sgn - 0

sgn

u

E

E u u y

uu

u

1T

111 T

1

It's a contraction

Iterate Conjugate Gradient method until .

.

t t

t t

u u y

u u y

u u

Convex Iteration demoConvex Iteration demo

zero-filled reconstruction Fourier sample mask

use 4% Fourier data error: -104 dB 96 seconds

reconstruction

Signal Separation by Convex Iteration

-1 -1D

D

Du u u

is Total Variation operator

D is Discrete Cosine Transform matrix

D

T

is inverse measurement in domain

real

: Gram-Schmidt orthogonalization of Fourier matrix

: binary sampling mask rank = rank

: ( )

=

y y P u u

P M

M P M

F

: F F w

F :

Dwant to find ,u u

D1 1D

-1 -1

D

minimize,

s.t. Dy P

1-norm formulation

as convex iteration

T TD

D

-1 -1

D

minimize ,,

s.t. Dy P

0 50 100 150 200 250 300-60

-40

-20

0

20

40

60

Signal Construction

0 50 100 150 200 250 300-40

-20

0

20

40

60

0 50 100 150 200 250 300-6

-4

-2

0

2

4

6

=

+

Cardinality(steps) = 7Cardinality(cosine) = 4

0 50 100 150 200 250 300-80

-60

-40

-20

0

20

40

60

0 50 100 150 200 250 300-50

-40

-30

-20

-10

0

10

-1

-1D DD

u

u

u

u∆

uD

D

Donoho, Tanner, 2005. Baron, Wakin et al., 2005

Minimum Sampling RateMinimum Sampling Rate

k/n

m/k

m measurementsk cardinalityn record length

0 50 100 150 200 250 3000

5

10

15

20

25

30

35

m = 28 measurementsm/n = 0.1 subsamplem/k = 2.5 sample ratek/n = 0.04 sparsity

0 50 100 150 200 250 300-60

-40

-20

0

20

40

60

Cardinality(steps) = 7Cardinality(cosine) = 4

M uF

uSignal Separationby Convex Iteration

0 50 100 150 200 250 300-50

0

50

100error = -23.189286 dB

orig signal

L1 est signal

0 50 100 150 200 250 300-50

0

50

100error = -241.880696 dB

orig signal

est signal

0 50 100 150 200 250 300-50

0

50

100error = -14.488360 dB

orig signal

est signal using TV alone

0 50 100 150 200 250 300-50

0

50

100error = -6.648319 dB

orig signal

est signal using DCT alone

1-norm

Convex iteration

TV only

DCT only

-241dB reconstruction error

0 50 100 150 200 250 300-40

-20

0

20

40

60

piecewise

est piecewise

0 50 100 150 200 250 300-6

-4

-2

0

2

4

6

cosine

est cosine

u∆

uD

functional Magnetic Resonance functional Magnetic Resonance Imaging (fMRI)Imaging (fMRI)

Haemodynamic Response Function Haemodynamic Response Function

(HRF)(HRF)

How to conduct How to conduct fMRI?fMRI?

Huettel, Song, Gregory. Functional Magnetic Resonance Imaging. Sinauer.

Stimulus Timing

How to conduct How to conduct fMRI?fMRI?

Stimulus Timing

What does fMRI What does fMRI measure?measure?

Neural activity Signalling Vascular response

Vascular tone (reactivity)Autoregulation

Metabolic signalling

BOLD signal

glia

arteriole

venule

B0 field

Synaptic signalling

Blood flow,oxygenationand volume

dendriteEnd bouton

fMRI signal originfMRI signal originrest

Oxygenated Hb

Deoxygenated Hb

task

Oxygenated Hb

Deoxygenated Hb

Haemodynamic Response Function Haemodynamic Response Function

(HRF)(HRF)

Stimulus Timing Canonical HRF Predicted Data

=

=

Time

Time

Which part of the brain is Which part of the brain is activated?activated?

http://www.fmrib.ox.ac.uk/

Time Time

Actual measurement Prediction

Sig

nal

Inte

nsi

ty

=

Stimulus Timing MeasurementActual HRF

HRF calibration

Variability of HRFVariability of HRF

Deconvolve HRF Deconvolve HRF hh

10

0

-10

1

0100 200 300 400 500

0

0.5

1

100 200 300 400 500

0

1

2

100 200 300 400 500-10

0

10

100 200 300 400 5000

0.5

1

100 200 300 400 500

0

1

2

100 200 300 400 500-10

0

10

100 200 300 400 5000

0.5

1

100 200 300 400 500

0

1

2

100 200 300 400 500-10

0

10

100 200 300 400 5000

0.5

1

100 200 300 400 500

0

1

2

100 200 300 400 500-10

0

10

y

DhD: convolution matrix

D(: , 1)

1 2

3

1

T

minimize

s.t.

(1) ( ) 0

0

hWh y Dh

h

h h n

E h

W: CoifletE: monotone cone

1 32

canonical HRFdeconvolved HRF

(d)

WhDiscrete wavelet transform

Deconvolve HRF Deconvolve HRF hh

10

0

-10

1

0100 200 300 400 500

0

0.5

1

100 200 300 400 500

0

1

2

100 200 300 400 500-10

0

10

100 200 300 400 5000

0.5

1

100 200 300 400 500

0

1

2

100 200 300 400 500-10

0

10100 200 300 400 500

0

0.5

1

100 200 300 400 500

0

1

2

100 200 300 400 500-10

0

10

y

stimulustiming

1 2

3

1

T

minimize

subject to

(1) ( ) 0

0

hWh y Dh

h

h h n

E h

W: Coiflet waveletE: monotone coneD: convolution matrix measurement

smoothness

100 200 300 400 5000

0.5

1

100 200 300 400 500

0

1

2

100 200 300 400 500-10

0

10

Dh

D( : ,

1)

Deconvolution resultsDeconvolution results

10 canonical canonical20 log /h h h

0 5 10 15 20 25 30 35-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

impulse (evenly spaced) = -10.6dBimpulse (jittered) = -6.2dBBlock = -11.8dBCanonical HRF

Low noise simulation: SNR= 6dB

0 5 10 15 20 25 30 35-1.5

-1

-0.5

0

0.5

1

1.5

impulse (evenly spaced) = -6.7dBimpulse (jittered) = -4.4dBBlock = -3.5dBCanonical HRF

High noise simulation: SNR = -10dB

in vivoin vivo deconvolution deconvolution resultsresults

0 5 10 15 20 25 30

-0.5

0

0.5

1

frame

RO

I val

ue

file: C:\Documents and Settings\THE SHEEP\My Documents\rawdata\Nov11/measure_hrf/hrf roi loc: 38 48 13 22 5:

frame

DICOMname = ../../../rawdata/Nov11/002/I ccfname = ../../../rawdata/Nov11/correl/hrf

p<0.350000 t-statistic = 0.5057 cc-statistic= 50.5 total # activated pixels = 162

0

DICOMname = ../../../rawdata/Nov11/002/I ccfname = ../../../rawdata/Nov11/correl/hrf


0

0 5 10 15 20 25 30

-0.5

0

0.5

1

frame

RO

I va

lue

DICOMname = ../../../rawdata/Nov10/002/I ccfname = ../../../rawdata/Nov10/correl/block


0 2 4 6

…

HRF calibration

…

HRF deconvolution

…

HRF deconvolution

Time (s) Time (s)

Cardinality Constraint Problem

3

5

Ab3

1

x5

1

A = 0.29 0.47 0.62 0.16 0.24

0.75 0.51 0.89 0.52 0.42

0.82 0.41 0.20 0.86 0.79

0.29 0.62 0.16

0.75 0.89 0.52

0.47 0.24

0.51 0.42

0.41 0.70. 982 0.20 0.86

b = A(:,2) * rand(1) + A(:,5) * rand(1)

Find Find xx with desired with desired cardinalitycardinality

e.g. e.g. kk = 2, = 2, wantwant 2

5

0

0

0

x

x

x

0want 2x

4

2

5

0

0

x

x

x

x = A\b 1arg min

s.t.

x x

Ax b

2

4

5

0

0

x

x

x

=

†A b 2

arg min

s.t.

x x

Ax b

1

2

3

4

5

x

x

x

x

x

m=3

n=5

Ab3

1

x5

1

Check every pair for k = 2Possible # solution:5

2

In general, !

! ( )!

n n

k k n k

From the Range perspective From the Range perspective ……

3

5

A

2

5

= 0

Nullspace PerspectivePerspective

T1ZT2ZT3ZT4ZT5Z

Z

Tpx Z xi ii Tp 0Z x ii

p

p p( )

x Z x

Ax A Z x Ax b

px = A\bParticular soln.

General soln.

Z = 0.57 0.34 -0.03 -0.26 -0.16 0.02 -0.68 -0.63 0.22 0.47

xp = 0.34 1.11 -0.30 0 0

-4 -2 0 2-1

0

1

2

3

4

5

6

1

2

1 2

1 2

1 2

1 2

1 2

0.57 + 0.34 + 0.34 0

-0.03 - 0.26 1.11 0

-0.16 + 0.02 0.30 0

-0.68 - 0.63 0 0

0.22 + 0.47 0 0

T

1...5

p

i

x Z xi ii

How to find intersection of How to find intersection of lines?lines?

Sum of normalized wedges

1T1

p

T

1n

Z

Z x

Z

(0, 0)

x = 0.34 1.11 -0.30 0 0

2

5

0

0

0

x

x

x

(-1.16, 4.51)

x = 0 0.68 0 0 0.51

Z = -0.59 -0.36 0.06 -0.55 0.15 0.36 0.74 -0.08 -0.27 0.66

Znew = -0.41 -0.18 -0.80 -0.30 0.48 0.19 -0.26 -0.07 1.00 0.37

-5 0 5-4

0

4

Znew = Z * randn(2);

-2 -1 00

2

4

6

8

SummarySummary

Ways to find 0-norm solutions other Ways to find 0-norm solutions other than than

1-norm (homotopic, convex iteration)1-norm (homotopic, convex iteration) fewer measurementsfewer measurements fasterfaster

In cardinality constraint problem, In cardinality constraint problem, convex iteration and nullspace convex iteration and nullspace technique success more often than 1-technique success more often than 1-norm.norm.