compressive sensing theory notes

8/12/2019 compressive sensing theory notes

1/118

Introduction to Compressive Sensing

Collection Editor:

Marco F. Duarte


2/118


3/118

Introduction to Compressive Sensing

Collection Editor:

Marco F. Duarte

Authors:

Mark A. Davenport

Ronald DeVore

Marco F. Duarte

Chinmay Hegde

Jason Laska

Michael A Lexa

Shriram Sarvotham

Mona Sheikh

Wotao Yin

Online:< http://cnx.org/content/col11355/1.2/ >

C O N N E X I O N S

Rice University, Houston, Texas


4/118


5/118


6/118


7/118

BA

BA:={fL2(R) :

f () = 0,|| A}.

f

f () := 1

2 Rf(t) eit dt.

f L1 f L2

f(t) := 1

2

R

f () eit d.

fBA f fnA

f(t) =nZ

f n

A

sinc ( (At n)) ,

sinc (t) = sint

t

A= 1 fBA=1

f(t) = 1

2

f () eit d.


8/118

F() 2

f

F() :=nZ

f (

2n) .

F()

F() =nZ

cnein,

cn

cn = 12

F() e

in d

= 12

f () ein d.

cn= 1

2f(n) .

F() = 1

2

nZ

f(n) ein.

f() = F() [,]= 1

2

nZ

f(n) ein[,],

F[,] = 12 sinc () F(g (t n)) =einF(g (t)) ,

f(t) =nZ

f(n) sinc ( (t n)) .

{sinc ( (t

n))

}n

Z

L2

f2L2= 2nZ

|f(n) |2

f

nZ

|sinc ( (t n)) | nZ

1

|t n| + 1


9/118

f () = 0 || , >0,

[,]

g ()

g

g

g

g

g () = 1 || g= 0 ||> g m1 g

|g (t) | C(|t| + 1)

m

C >0

g

f () = F()

g () = 12

nZ

f(n) ein

g () .

f(t) =nZ

f(n) g (t n) ,

[ , ]


10/118

[, ] [ , ] [,]

f(t) =

nZ f(n) sinc ( (t n)) ,f(t) =

nZ fn

g (t n) .

f BA f A

f f

f [T, T] T >0

f BA=1 [T, T] L [T, T]

BA:={fL2(R) :|

f() |= 0, || A}.

fBA f=n

f n

A

(At n) .

g()

1A 2A , f (x) = (x)x

> 1

f=

f n

A

g(At n) .


11/118

1A 2A

g() g()

|g(t) | c,k(1 + |t|)

k, k= 1, 2,...

g()

()

[T, T] [cT,cT]

t [T, T] [cT,cT]

c > 1 [T, T]

[T, T] T > 0 BA K

X

K

K X (E, D)E K D

X


12/118

d

d (K, E , D, X) := fKf D (Ef) X .

n (K, E) = fK#Ef #Ef Ef n fK

n(K, X) := (E,D){n (K, E) :d (K, E , D, X)} K

N d (K, E , D, X) E, D n (K, E)

N


13/118

K

>0 K K

KUNi=1b (fi, ) . N := {N : } N(K) K

X K N= N(K, X)

H(K, X) K X

H(K, X) = logN(K, X) .

KX n(K, X) =H(K, X)

f K N

K, X

logN

K, X


14/118

BA(M) ={fBA:|f(t) | M, t R} M

L [T, T] T > 0 BA(M) L BA(M)L [T, T]

nA

[T(1 + ) , T(1 + )]

BA

f () = 0 || A |f| M f = (T) > 1 > 0 k 2k1 < 2k f f f nA

nA [T(1 + ) , T(1 + )] (T)> 0 f

[T, T] f nA k + k0(T) f f

nA

f nA

k+ k0(T) f

nA

f nA

f nA

f n

A

2kk0 M.

g

f(t) = nNT

f nA g(At n) ,

NT :={n: T(1 + ) nA

T(1 + )}.

|f(t) f(t) | nNT f nA f nA |g(At n) |+| nA|>T(1+)f nA |g(At n) |


16/118


17/118

R3

N R

N R

N p

p[1, ]

xp={N

i=1 |xi|p 1p

, p[1, ) ;max

i=1,2,...,N|xi|, p=.

RN

< x, z >=zTx=Ni=1

xizi.

2 x2 =

< x, x > p p


18/118

x0 :=|supp(x) | supp(x) ={i: xi= 0} x

|supp(x) | supp(x) 0

limp0x0xp =|supp (x) |,

p p {x:xp= 1}, R

2 p < 1

R2

p p= 1, 2, p p= 1

2

1 2

p

x R2

A

p

x A x xp p A x p p x A

xA x p p p


19/118

R2

p

p= 1, 2,

p p= 1

2

1

2 p

={i}iI V V

V= RN I={1,...,N} x RN

x=iIa

ii,

ai =< x, i > {i}iI x

={i}iI

< i, j >= 0 i= j = T

{i}ni=1 Rd d < n Rdn x Rd

Ax2

2 T

x2

2Bx2

2

0 < A B 0 A B A B A= B A A = B = 1

> 0 i2 = i= 1,...,N = 1 d N A B T


20/118

x

x =

T = T =I

=

T1

A > 0 T

d= T

T1

x.

2 d2 2

x=

x K K x0K

K={x:x0K} K

x K x x= 0K


21/118

Xk =N1

n=0 xncosN

n + 12

k

k = 0, , N 1 xn


22/118

K p K K

K


23/118

K

x

x= ,

x x s

|s| C1sq

, s= 1, 2, ....

q


24/118

0 1 2 3 4 5 6

x 104

0

0.5

1

1.5

Sorted indices

Waveletcoefficientmagnitude

K N K K K K K(x)

K(x) = arg minK

x 2.


25/118

K(x)C2K1/2s.

K(x)2

Kr

i

ir+1/2 K

p

p x (n) p p p

xp=

i

|xi|p 1

p


26/118

p

p

p p p

x [n] p q > 1/p

p p1


27/118

x RN M

y= x,

M N y RM R

N N RM M N

x

x

x x

y

MN


28/118

N() ={z : z= 0}. x x

x, x K={x:x0K} x= x

x

x

y

x = x

x x = 0 x x 2K xK N() 2K

y RM xK y = x spark ()> 2K

y RM xK y = x spark () 2K 2K h N() h 2K h 2K h = x x x, x K h N()

x x

= 0 x = x x

K y = x

spark ()> 2K spark ()> 2K y x, x K

y = x = x

x x = 0 h = x x h = 0 spark ()> 2K 2K h= 0 x= x

spark()[2, M+ 1] M2K

N()

{1, 2, , N} c ={1, 2, , N} \

x N x c


29/118

M N c

K

C > 0

h2Chc1

K

h N() || K

h K hc1 = 0 h= 0 K N() h= 0

x : RM

R

N

(x) x2CK(x)1

K

x

K(x)p= min

xKx xp.

K K x

x

x

p p

K(x)1/

K K(x)2

2K

2K

: RN RM : RM RN (, ) 2K

||

M||


30/118

h N() 2K h 0 1 |0| =|1| = K x = h1+ hc x =h0 h= x

x

x

K x

= x h N()

h=

x x = 0 x = x x = (x)

h2 h2 =x x 2 =x (x) 2CK(x)1

K=

2C

hc12K

,

(x) x2CK(x)1

K,

K K(0, 1)

(1 K) x22 x22(1 + K) x22, xK={x:x0K}

2K K

x22 x22x22 0<

0

K=c1log (p)

p = /4

M c0log (p)2

=16c0K

c12 .

K K/2 Klog (N/K)

2K 2K | u1

u2sgn(u) 2 sgn(u) K 1 uK sgn(u) =

K

K u u

h h N()

h N()

1

2K h RN h= 0 0 {1, 2,...,N} |0| K 1 K hc

0 = 0 1

h2hc

01

K+

||h2

,

=

22K

1 2K, = 1

1 2K.

h

h N()

h N()

h2Chc1

K

2K h

0

K

h

h= 0

h2hc

01

K.


36/118

hc01

=h11+ hc1

Kh12+ hc1

h2

h12+hc1

K

.

h12 h2

(1 ) h2hc1

K.

2K 0 M = 2K 2K 2K 1 2K 2K>0 2K

NK

K RN

N K 2K 2K

E

2ij

= 1

M,

1/M

c > 0

E

eijtec2t2/2


37/118

t R

1/

M

c2 = E

2ij

=1M

M N ij ij c2 = 1/M Y = x x RN > 0 xRN

E Y22=x22

PY22 x22x222exp

M

2

= 2/ (1 log (2))6.52

(0, 1) M N ij ij c2 = 1/M

M1Klog

NK

,

K 1 2e2M

1 2 = 2/2 log (42e/) /1

x

M


38/118

NK

()

i j

() = max1i|i2j2

.

()

NMM(N1) , 1

N M ()1/M

N N

M

mij

1i, jN

N

di = di(ci, ri) 1iN ci= mii ri=

j=i|mij | G= T

spark ()1 + 1 ()

.

spark () {1,...,N} ||= p G= T

gii = 1 1ip |gij | () 1i, jp i=j

j=i|gij |


39/118

K 0 a=

X = [X1, X2,...,XN] Xi Xi Sub

c2

RN < X, > Sub c2 22 XiSSub

2

RN < X, >SSub 2 22

Xi

E

exp

tN

i=1 iXi

= EN

i=1 exp (tiXi)

=N

i=1 E (exp (tiXi))

Ni=1 exp

c2(it)2/2

=expN

i=1 2

i

c2

t2

/2

.

Xi c2 =2

E

< X, >2

= 2 22

X

Xi Xi

Sub(c) X2

X t > 0

P (Xt) E (X)t

.

f(x) X

E (X) =

0

xf(x)dxt

xf(x) dxt

tf(x) dx = tP (Xt) .

XSub c2 E

exp

X2/2c2 1

1 ,


42/118

[0, 1)

= 0 (0, 1) f(x) X X

exp (tx) f(x) dxexp c2t2/2

t R exp c2t2/2

exp

tx c2t2/2 f(x) dxexp c2t2 ( 1) /2 . t

exp

tx c2t2/2

dt

f(x) dx

exp

c2t2 ( 1) /2

dt,

1

c

2

exp

x2/2c2

f(x)dx 1c

2

1 .

X= [X1, X2,...,XM] Xi XiSub

c2

E

X2i

= 2

EX22= M 2.

(0, 1) c2/2, max 4 max 2/c2

PX22M 2exp M(1 )2/

PX22M2exp M( 1)2/ .

Xi

EX22= M

i=1

E

X2i

=

Mi=1

2 =M 2

PX22M2 = P exp X22exp M2

E(exp(X2

2))exp(M2)

=QMi=1

E(exp(X2i ))exp(M2) .


43/118

XiSub

c2

E

exp

X2i

= E

exp

2c2X2i/2c

2

1

1

2c2

.

Mi=1

E

exp

X2i 1

1 2c2M/2

PX22M2

exp22

1 2c2

M/2.

=

2

c2

2c22 (1 + ) .

PX22M2

2

c2exp

1

2

c2

M/2.

PX22M 2

2

c2exp

1

2

c2

M/2.

= max

4, 2

max2/c 12(max2/c 1) log (max2/c)

0, max2/c log ()( 1) 2( 1)

2

,

exp

( 1) 2( 1)2

.

= 2/c2 = 2/c2

M2 M c2/2 1 c2 = 2


44/118


45/118

MN

M

K(0, 1)

(1

K)

x

22

x

22

(1 + K)

x

22

xK K x K 2K > 0 M = 2K

2K 2K 1 2K 2K>0 2K

NK

K RN

N K 2K 2K 2K

MN ij ij SSub (1/M) Y = x x RN > 0 x RN

EY22=x22

PY22 x22x222expM2

= 2/ (1 log (2))6.52

E

2ij

= 1

M,

1/M

K

K

(0, 1) Q q2 = 1 q Q|Q| (3/)K x RK x2 = 1 qQ x q2


46/118

Q

q1 RK q12 = 1

Q

i

qi RK qi2 = 1

qi

qj

2 > j < i

x RK x2 = 1 q Q x q2 |Q| /2 Q 1 + /2 BK(r) r R

K

|Q| BK (/2) BK (1 + /2)

|Q| (BK(1+/2))

(BK(/2))

= (1+/2)K

(/2)K

(3/)

K.

(0, 1) M N ij ij SSub (1/M)

M1Klog

N

K

,

K 1 2e2M

1 > 1 2 = 2/2 1/1 log (42e/)

x2 = 1 T {1, 2,...,N} |T|= K XT K T QT QT XT q2 = 1 qQT xXT x2 = 1

minqQT

x q2/14.

QT |QT| (42/)K

T

QT

Q=

T:|T|=KQT.

NK

T

N

K

=

N(N 1) (N 2) (N K+ 1)K!

NK

K!

eN

K

K,

K! (K/e)K |Q| (42eN/K)K


47/118

= /

2

1

2(42eN/K)

KeM2/2 ,

1 /

2q22q22

1 + /

2q22, for all qQ.

M

log

42eN

K

KK

log

N

K

+ log

42e

M

1+ Mlog

42e

1 2e2M A

x

2

1 + A, for all x

K,

x

2 = 1.

A xK x2 = 1

qQ x q2/14 x qK xXT qQT XT x q2/14

x2 q2+ (x q)2

1 + /

2 +

1 + A /14.

A

1 + A

1 + /

2 +

1 + A /14

1 + A

1 + /

2

1 /14

1 + ,

x2 q2 (x q)2

1 /

2 1 + /14 1 ,

= I K K


48/118


49/118

x

y = x x

x

x=argminz

z0 subject to z B (y) ,

B (y) x y z0 =|supp(z) | z

B (y) ={z : z = y}

B (y) ={z :z y2 } x y

x x=

=argminz

z0 subject to z B (y)

B (y) ={z : z = y} B (y) ={z :z y2 } =

= I

xx2 =

cc2=

2

cc2 xx

0


50/118

0 1

x=argminz z1 subject to z B (y) .

B (y) B (y) ={z : z = y}

R2

1

p p= 1

2 1 p

1

1 p p < 1 1

1

1

1

1

1


51/118

1

1

1

x=argminz

z1 subject to z B (y) . B (y)

1

2K

2K 0

J H J(x) = x1 1 x H(x, y) = 12 xy22 2


62/118

x H x1 J()

J(x) = T V (x) + x1

1024 1024

1 J(x) =x1 O

N3

N

M

1

minx

x

1+ H(x) ,

H

H


63/118

shrink (t, ) ={

t ift > ,

0 if t, andt + ift < .

i= 1,...,N ith x (k+ 1)th

xk+1i = shrink

xk [U+25BD]Hxki,

>0 k xk+1 xk H() [U+25BD]H xk

J

H(x)

H(x) =y x22[U+25BD]H(x) = 2 (y x) .

xk+1i = shrink

xk [U+25BD]Hy xki,

y n

x

x 0 = 0 r= y k= 0

kk + 1 x x T r

xshrink (x, k ) ry x


64/118

x x

yk+1 =yk + y xkxk+1 =argmin J(x) + 2 x yk+1

2.

J(x) J(x) =x1 > 0

1

1

x

x y

minI

{|I|: y =iI

ixi},

I i = 1,...,N i ith

I

y


65/118


66/118

K

O (MN K) .

M=CKlogN K M=CKlogN

K

y

x

0 = 0 r= y = i= 0

ii + 1 b T r supp(T (b, 1))

x

i| x x i| C0 ry x i

x x i

K

1

r0 = y Trk1 kth

S= 10

O (KNlogN)


67/118

1 K K

i

xi1

eTr 2K e T supp xi1 b b|T Ty b|TC 0 b K xi ry xi

y K

K

x x

x 0 = 0 r= y i= 0

e T r supp (T (e, 2K)) T supp

x i 1

b| T T y b| T C

x

i T (b, K) ry x i

x x i

O (MN)

K


68/118

x0

xi+1 = T

xi+ T

y xi

, K

.

x

x

1

N

K

x

xi= 0

K

xi = 0

x ij j

th

ith

x

xi


69/118

i x

N= 232 x

y = x M N M N y

y

xi xi i y ith y xi y = x x x x

H h :{1,...,N} {1,...,m} H mN h H (h) m N j j = h (i) d h1,...,hd H M = md M N d

x y= x y {h1,...,hd} ith

y

h

yi

yi=

j:h(j)=i xj .

j yi xj i h

y jth

xj =minl

yi : hl(j) = i.

xj xj d = ClogN

m= 4/K

x

x x/K x x1, x K x 1


70/118

xj

xj =medianl

yi : hl(j) = i.

N x N x N

K

x

K

x

x

y = x x


71/118

x y x (i) y (i)

x

x x y

x (i) y (i)


72/118


73/118

N M MN M 1 y

y x yx

M N N

N

x

x

x M N {1, ....M} N > M x N y =

Mm=1 ixi = f

y

y= x + e e

e

x

e

(N M) N


74/118

= 0

y = y= x + e= e

e

e

e

y= y e x= y= y e

m K M < NCKlogN/K C

N

K

x xi= 0 K xi= 0

x ij j

th

ith x

xi

i

x

N= 232 x y= x M N MN y y

xi xi i y ith y xi y = x x x x


75/118

x (t) x (t) x (t) x (t)


76/118

x (t) {j(t)}Mj=1 t [0, T] M

y [j] =

T0

x (t) j(t) dt.

j(t)

M

x (t)

j(t)

M

y [j]

j(t)

j(t) M

j(t) j(t) =(t tj) {tj}Mj=1 M x (t)

[0, T]

M

x (t) 1 pc(t) Na Na x (t) 1/Ma Ma Na

y [j] =

j/Ma(j1)/Ma

pc(t) x (t)dt.


77/118


78/118

T = 1

=

1 +1 +1

1 +1 1+1 +1 1

+1 1 1

.

M N/M O (N) O (MN)

NN

M=O Klog2 (N/K) ,

pc[n] x (t) x (t) 1 K

MCKlog (N/K+ 1) C1.7

k/Na k

K

B

KB

y [j] =< x,j > N x N {j}Mj=1


79/118


80/118

N/2

{j} [n j]

M

N

x

x N= 256 256 M=N/50

N= 256 256 M=N/10

1


81/118

256256

M= 1300 50

256 256

M= 6500

j j {0, 1} N

N Wlog2N N N

W0 = 1 Wj

Wj = 1

2

Wj1 Wj1

Wj1 Wj1

.

1/N O (NlogN)

W1 = 1

2

1 1

1 1


82/118

W2 = 12

1 1 1 1

1 1 1 11 1 1 11 1 1 1

.

N = 2B WB I

M N I M IWB

WB WB D N N

=

1

2

N IWB+

1

2

D.

12

N IWB +

12 IWB

{0, 1

}

D WB

x y

f(x, y, )

f(x, y) ={f(x, y, )}


83/118


84/118

x y


85/118


86/118


87/118

= x,y=

[1, 1] x,y [1, 2] x,y [2, 1] x,y [2, 2] x,y

.

K K

K

N

RN


88/118

F K


89/118

K = 2 (1, 2) 1 2 RN

K M

RN

M = CKlog (N) K < M < < N M

RN

RM


90/118

RN

N


91/118

K

K

K

N


92/118

s s

SIR = 6

3 4


93/118

2

K M=O (KlogN)

M

M

M

x RN

f(x)

y = x M N p

f x f M


94/118

x

K

K N (KlogN)

M

M

kM n2

k

M


95/118

M

L

NL

M2/(2+1) M2


96/118

A

T

G

C

A

T

G

C


97/118

M

N

M N 1iM 1j N i j i,j j

xj i yi=N

j=1 i,jxj =ix j x y ={yi}i = 1,...,M y= x

x


98/118


99/118

[U+FFFD]

[U+FFFD]


100/118

[U+FFFD]

[U+FFFD] [U+FFFD]

[U+FFFD]

[U+FFFD] [U+FFFD]

[U+FFFD]

[U+FFFD] [U+FFFD]

[U+FFFD]

[U+FFFD]

[U+FFFD]

[U+FFFD]

[U+FFFD]

[U+FFFD]

[U+FFFD]

[U+FFFD]


101/118


102/118


103/118

[U+FFFD]


104/118


105/118


106/118


107/118


108/118


109/118


110/118


111/118


112/118


113/118


114/118


115/118


116/118


117/118


118/118

compressive sensing theory notes

Documents