E cient Message Passing-Based Inference in the Multiple Measurement Vector Problemschniter/pdf/asil11_mmv_slides.pdf · 2011-11-10 · E cient Message Passing-Based Inference in the

E�cient Message Passing-Based Inference in the MultipleMeasurement Vector Problem

Justin Ziniel Philip Schniter

Department of Electrical and Computer EngineeringThe Ohio State University

Asilomar Conference on Signals, Systems, and Computers, 2011

Work supported in part by NSF grant CCF-1018368 and DARPA/ONR grant N66001-10-1-4090

1 of 18

Outline

BackgroundThe Multiple Measurement Vector (MMV) ProblemExisting ApproachesSignal Model

Our Proposed MethodBelief Propagation-Based InferenceEM Parameter Learning

Empirical Study

Conclusion

2 of 18

The Multiple Measurement Vector (MMV) Problem

Consider a time-series of sparse, temporally correlated signal vectors thatshare a common support...

x(1)

x(2)

x(3)

x(4)

x(5)

3 of 18

The Multiple Measurement Vector (MMV) Problem

...observed through a noisy linear measurement process, Y = AX + E.

= +

Y A X E

Applications: Magnetoencephalogaphy, direction-of-arrival estimation, parallel MRI,...

4 of 18

Existing methods

� Greedy pursuit� M-BMP, M-OMP, M-ORMP [Cotter et al., '05]� S-OMP [Tropp et al., '06]� Subspace-augmented MUSIC* [Lee et al., '10]

� Mixed-norm (`1/`2) minimization� M-FOCUSS [Cotter et al., '05]� RX-penalty, RX-error [Tropp et al., '06]� JLZA [Hyder and Mahata, '10]� tMFOCUSS* [Zhang and Rao, '11a]

� Bayesian MMV� M-SBL [Wipf and Rao, '07]� JSSR-MP [Shedthikere and Chockalingam, '11]� T-MSBL*, T-SBL* [Zhang and Rao, '11b]

� Block-sparse single measurement vector� [Eldar and Mishali, '09]� bSBL [Zhang and Rao, '11b]

* = Accounts for temporal correlation in amplitudes5 of 18

Comparing Di�erent Approaches

Approach Speed Performance

Greedy Fast Fair

Mixed-norm Okay Good

Bayesian Slow Great

Why Bayesian?

� Modeling assumptions are made explicit

� Model parameters have meaningful interpretations

� Principled parameter learning

� Soft inference

6 of 18

Comparing Di�erent Approaches

Approach Speed Performance

Greedy Fast Fair

Mixed-norm Okay Good

Bayesian Slow Great

Why Bayesian?

� Modeling assumptions are made explicit

� Model parameters have meaningful interpretations

� Principled parameter learning

� Soft inference

6 of 18

A Model of Sparse Time-Evolving Signals

We write: x(t)n = s(t)n · θ(t)n for s(t)n ∈ {0,1} and θ(t)n ∼ CN (ζ,σ2).

Xs Θ

⊙ =

Amplitude Evolution

Treat {θ(t)n }Tt=1 as a Gauss-Markov

process: θ(t)n = (1− α)θ

(t−1)n + αw(t)

n ,

where w(t)n ∼ CN (0,ρ), and α conrols

the correlation in the random process.

7 of 18

The Factor Graph Representation

AMP

t

8 of 18

The Factor Graph Representation: Single Timestep

Signal

Mea

sure

men

ts

8 of 18

The Factor Graph Representation: Support Variables

AMP

t

8 of 18

The Factor Graph Representation: Amplitude Variables

AMP

t

8 of 18

Approximate Message Passing (AMP)M

easu

rem

ents

Signal

AMP

� Standard belief propagation is intractable here

� Simpli�cation: Approximate message passing(AMP), [Donoho, Maleki, and Montanari, '09, '10]

� Marginal for x(t)n : Bernoulli-Gaussian -

(1− π(t)n )δ(x(t)n ) + π

(t)n CN (x(t)n ;ξ(t)n ,ψ(t)

n )

� As M,N→∞, AMP behavior described precisely bystate evolution → MMSE-optimal estimates [Bayatiand Montanari, '10]

# of messages exchanged: O(N)Complexity per iteration: O(MN) (matrix-vectorproduct)

9 of 18

Parameter Learning via Expectation-Maximization

� Signal model governed by a number of parameters: Γ, {λ,ζ,σ2,α,ρ,σ2e }

� Parameters can be tuned automatically from the data using anexpectation-maximization (EM) algorithm

AMP-MMV EM Learning

{s,Θ}i

Γi+1

� Finds local maximizer of p(Y|Γ)� EM parameter estimation �ts naturally into the existing message passingprocedure� The E-step of the EM algorithm makes use of quantities available for free asa byproduct of AMP-MMV!

10 of 18

Empirical Study: Setup

� AMP-MMV w/ EM parameter learning was compared against 3 powerfulMMV algorithms, and an oracle-aided MMSE bound (support-awareKalman smoother)� Bayesian: MSBL and T-MSBL* [Zhang and Rao, '11b]� Greedy: Subspace-augmented MUSIC (SA-MUSIC*) [Lee et al., '10]

� Signals generated according to signal model; i.i.d. Gaussian A matrices;AWGN corrupting noise

* = Accounts for temporal correlation in amplitudes

11 of 18

Empirical Study: MSE vs. Normalized Sparsity Rate

1.5 2 2.5 3−30

−25

−20

−15

−10

−5

0

α = 0.1 | N = 5000, M = 1563, T = 4, SNR = 25 dB

Measurements−to−Active−Coefficients (M/K)

Tim

este

p−

Avera

ged N

orm

aliz

ed M

SE

(T

NM

SE

) [d

B]

T−MSBL

MSBL

SA−MUSIC

AMP−MMV

Oracle

1.5 2 2.5 310

0

101

102

103

104

105

α = 0.1 | N = 5000, M = 1563, T = 4, SNR = 25 dB

Measurements−to−Active−Coefficients (M/K)

Runtim

e [s]

T−MSBL

MSBL

SA−MUSIC

AMP−MMV

Correlation: 1− α = 0.9012 of 18

Empirical Study: NSER vs. Normalized Sparsity Rate

1.5 2 2.5 310

−3

10−2

10−1

100

α = 0.1 | N = 5000, M = 1563, T = 4, SNR = 25 dB

Measurements−to−Active−Coefficients (M/K)

Nor

mal

ized

Sup

port

Err

or R

ate

(NS

ER

)

T−MSBL

MSBL

SA−MUSIC

AMP−MMVAMP−MMV [p(s

n| y)]

1.5 2 2.5 310

0

101

102

103

104

105

α = 0.1 | N = 5000, M = 1563, T = 4, SNR = 25 dB

Measurements−to−Active−Coefficients (M/K)

Run

time

[s]

T−MSBL

MSBL

SA−MUSIC

AMP−MMV

Correlation: 1− α = 0.9013 of 18

Empirical Study: MSE vs. Normalized Sparsity Rate

1.5 2 2.5 3−30

−25

−20

−15

−10

−5

0

5

α = 0.01 | N = 5000, M = 1563, T = 4, SNR = 25 dB

Measurements−to−Active−Coefficients (M/K)

Tim

este

p−

Avera

ged N

orm

aliz

ed M

SE

(T

NM

SE

) [d

B]

T−MSBL

MSBL

SA−MUSIC

AMP−MMV

Oracle

1.5 2 2.5 310

0

101

102

103

104

105α = 0.01 | N = 5000, M = 1563, T = 4, SNR = 25 dB

Measurements−to−Active−Coefficients (M/K)

Runtim

e [s]

T−MSBL

MSBL

SA−MUSIC

AMP−MMV

Correlation: 1− α = 0.9914 of 18

Empirical Study: MSE vs. Signal Dimension

102

103

104

−30

−25

−20

−15

−10

−5

α = 0.05 | T = 4, N/M = 3, λ = 0.15, SNR = 25 dB

Signal Dimension (N)

Tim

este

p−

Avera

ged N

orm

aliz

ed M

SE

(T

NM

SE

) [d

B]

T−MSBL

MSBL

SA−MUSIC

AMP−MMV

Oracle

102

103

104

10−2

10−1

100

101

102

103

104

α = 0.05 | T = 4, N/M = 3, λ = 0.15, SNR = 25 dB

Signal Dimension (N)

Runtim

e [s]

T−MSBL

MSBL

SA−MUSIC

AMP−MMV

Correlation: 1− α = 0.9515 of 18

Empirical Study: MSE vs. Measurement Innovation

10−2

10−1

100

−30

−25

−20

−15

−10

−5

0

5

α = 0.01 | N = 5000, M = 167, T = 4, λ = 0.017, SNR = 25 dB

Innovation Rate, β

Tim

este

p−

Avera

ged N

orm

aliz

ed M

SE

(T

NM

SE

) [d

B]

AMP−MMV

Oracle

10−2

10−1

100

100

101

102

α = 0.01 | N = 5000, M = 167, T = 4, λ = 0.017, SNR = 25 dB

Innovation Rate, β

Runtim

e [s]

AMP−MMV

Time-varying measurement matrix: A(t) = (1− β)A(t−1) + βW(t)

Correlation: 1− α = 0.99 | Undersampling Rate (N/M): 30 | Normalized Sparsity (M/K): 216 of 18

Conclusion

� AMP-MMV� Works with temporally correlated signal amplitudes� Performance rivals an oracle-aided MMSE bound (support aware Kalmansmoother) over a wide range of problems

� Computational complexity scales linearly in all problem dimensions

� EM parameter learning� Principled method of learning signal model parameters� Closed-form updates using outputs of AMP-MMV

� Empirical study� Two orders-of-magnitude improvement in runtime� Major gains possible from matrix diversity

17 of 18

Empirical Study: MSE vs. Undersampling Rate

0 5 10 15 20 25−30

−28

−26

−24

−22

−20

−18

−16

−14

−12

−10

α = 0.25 | N = 5000, T = 4, M/K = 3 SNR = 25 dB

Unknowns−to−Measurements Ratio (N/M)

Tim

este

p−

Avera

ged N

orm

aliz

ed M

SE

(T

NM

SE

) [d

B]

T−MSBL

MSBL

SA−MUSIC

AMP−MMV

Oracle

0 5 10 15 20 2510

−1

100

101

102

103

α = 0.25 | N = 5000, T = 4, M/K = 3 SNR = 25 dB

Unknowns−to−Measurements Ratio (N/M)

Runtim

e [s]

T−MSBL

MSBL

SA−MUSIC

AMP−MMV

Correlation: 1− α = 0.7518 of 18