Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information

Information Geometry with Information Geometry with Applications in Components Applications in Components

Analysis and XAnalysis and X--Ray CT ImagingRay CT ImagingJoseph A. OJoseph A. O’’SullivanSullivan

Electronic Systems and Signals Research LaboratoryDepartment of Electrical and Systems Engineering

Washington [email protected]

http://essrl.wustl.edu/~jao

Supported by: ONR, ARO, NIHONR, ARO, NIH

J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004Information Geometry

2

CollaboratorsCollaborators

Michael D. DeVoreNatalia A. SchmidMetin OzRyan MurphyJasenka Benac

Donald L. SnyderWilliam H. SmithDaniel R. FuhrmannJeffrey F. WilliamsonBruce R. WhitingChrysanthe PrezaDavid G. PolitteG. James Blaine

Faculty Students and Post-Docs


3

OutlineOutline• Applications

- Components Analysis Information Value DecompositionHyperspectral Imaging

- Transmission Tomography- Maximum Likelihood Mixture

Model Estimation• Information Geometry

- Properties of I-Divergence- Projections

• Alternating Minimization Algorithms• Applications Revisited• Conclusions


4








5

Markov ApproximationsMarkov Approximations• X and Y RV’s on finite sets X and Y, p(x,y) unknown• Data: N i.i.d. pairs {Xi,Yi}

–Histogram is a matrix S

• Log likelihood function is log probability of S

• Unconstrained ML estimate

Nyxnyxs ),(),( =

∑ ∑

∑ ∑

∈ ∈

∈ ∈

=

=

X Y

X Y

x y

x y

yxpyxs

yxpyxnN

SPN

),(log),(

),(log),(1)(log1

),(),(1),( yxsyxnN

yxp ==


6

Equivalence of ML EstimationEquivalence of ML Estimationand Minimum Iand Minimum I--DivergenceDivergence

•• Define the information (I) divergenceDefine the information (I) divergence

•• Log likelihood function, Lagrange multiplierLog likelihood function, Lagrange multiplier

•• Constrained ML estimation: family of distributionsConstrained ML estimation: family of distributions

),(),(),(),(ln),()||( yxpyxs

yxpyxsyxsPSI

x y+−⎥

⎦

⎤⎢⎣

⎡= ∑ ∑

∈ ∈X Y

∑ ∑∑ ∑∈ ∈∈ ∈

−X YX Y x yx y

yxpyxpyxs ),(),(ln),(

F∈p


7

Markov Approximations:Markov Approximations:Components AnalysisComponents Analysis

•• Lower rank Markov approximation Lower rank Markov approximation X X M M YYMM in a set of cardinality in a set of cardinality KK

•• Factor analysis, contingency tables, economicsFactor analysis, contingency tables, economics•• Problem: Approximation of one matrix by another of Problem: Approximation of one matrix by another of

lower ranklower rank•• C. C. EckartEckart and G. Young, and G. Young, PsychometrikaPsychometrika, vol. 1, pp. , vol. 1, pp.

211211--218 1936.218 1936.•• SVD SVD IVDIVD

AP Φ=

)||(min ASIA ΦΦ

∑=

=K

kkk yaxyxp

1)()(),( φ


8

Spectral Components AnalysisSpectral Components Analysis• Data spectrum for each element (pixel)

• Model: linear combination of constituent spectra,

• Problem: Estimate constituents and proportions subject to nonnegativity; positivity of S assumed

• Ambiguity if α > 0, φ1 − α φ2 > 0 , − α φ1 + φ2 > 0

• Comments: Radiometric Calibration; Constraints Fundamental

∑=

=K

kkjkj as

1φ AS Φ=

Ijs +ℜ∈

0≥kja


9

Idealized Problem Statement:Idealized Problem Statement:MaximumMaximum--Likelihood Likelihood Minimum IMinimum I--divergencedivergence

•• Poisson distributed data Poisson distributed data loglikelihoodloglikelihood functionfunction

•• Maximization over Maximization over ΦΦ and A equivalent to and A equivalent to minimization of Iminimization of I--divergencedivergence

AS Φ=ˆ

∑∑ ∑∑= = == ⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

−⎥⎥⎦

⎤

⎢⎢⎣

⎡=Φ

I

i

J

j

K

kkjik

K

kkjikij aasASl

1 1 11ln)|( φφ

∑∑ ∑∑= =

== ⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

+−⎥⎥

⎦

⎤

⎢⎢

⎣

⎡=Φ

I

i

J

j

Kk kjikijK

k kjik

ijij as

a

ssASI

1 11

1

ln)||( φφ

•• Information Value Decomposition ProblemInformation Value Decomposition Problem


10

HyperspectralHyperspectral ImagingImagingat Washington Universityat Washington University

Donald L. SnyderDonald L. SnyderWilliam H. SmithWilliam H. SmithDaniel R. Daniel R. FuhrmannFuhrmannJoseph A. OJoseph A. O’’SullivanSullivanChrysantheChrysanthe PrezaPreza

WU Team

Digital Array Scanning Interferometer (DASI)


11

HyperspectralHyperspectral ImagingImaging• Scene Cube Data Cube• “Drink from a fire hose”• Filter wheel, interferometer,

tunable FPAs• Modeling and processing:

- data models- optimal algorithms- efficient algorithms


12

CT Imaging in Presence of High CT Imaging in Presence of High Density AttenuatorsDensity Attenuators

BrachytherapyBrachytherapy applicators applicators AfterAfter--loading loading colpostatscolpostats

for radiation oncologyfor radiation oncology

Cervical cancer: 50% survival rateCervical cancer: 50% survival rateDose prediction importantDose prediction important

ObjectObject--Constrained Computed Constrained Computed TomographyTomography (OCCT)(OCCT)


13

Filtered Back ProjectionFiltered Back Projection

Truth FBP

FBP: inverse Radon transform


14

Transmission TomographyTransmission Tomography• Source-detector pairs indexed by y; pixels indexed by x• Data d(y) Poisson, means g(y:µ), log likelihood function

• Mean unattenuated counts I0, mean background β• Attenuation function µ(x,E), E energies

• Maximize over µ or ci; equivalently minimize I-divergence

)(),(),(exp),():(

):():(ln)()):(|(

0 yExxyhEyIyg

ygygydgdl

E x

y

βµµ

µµµ

+⎟⎟⎠

⎞⎜⎜⎝

⎛−=

−=⋅

∑ ∑

∑

∈

∈

X

Y

∑=

=I

iii ExcEx

1)()(),( µµ


15

Maximum Likelihood Maximum Likelihood Minimum IMinimum I--DivergenceDivergence

Difficulties: log of sum, sums in exponent

)()()(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

10 yExcxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

E x

I

iii

y

y

βµµ

µµ

µ

µµµ

+⎟⎟⎠

⎞⎜⎜⎝

⎛−=

+−=⋅

−=⋅

∑ ∑ ∑

∑

∑

∈ =

∈

∈

X

Y

Y


16








17

Information Geometry:Information Geometry:Properties of IProperties of I--DivergenceDivergence

• I-divergence is nonnegative, convex in pair (p,q)• Generalization of relative entropy; not symmetric;

example of f-divergence (Csiszár)• Let P be a probability matrix. Then

• First projection property

∑ +−=i

iii

ii qp

qppqpI ln)||(

)||()||( qpIPqPpI ≤

)/||/()||()||( BqApAIBAIqpI

qBpAi

ii

i

+=

== ∑∑


18

Information Geometry:Information Geometry:Projections Using IProjections Using I--DivergenceDivergence

• Define the linear family of probability distributions

• Theorem. Suppose that q and L are given. Let p* in Lachieve

then for all p in L

{ }bAppbA n =ℜ∈= + :),(L

)||()||()||(

)||(minarg

**

),(

*

qpIppIqpI

qpIpbAp

+=

=∈L


19

Information Geometry:Information Geometry:Projections Using IProjections Using I--DivergenceDivergence

• Define the exponential family of probability distributions

• Theorem. Suppose that p and E are given. Let q* in E achieve

• then for all q in E , )||()||()||(

)||(minarg

**

),(

*

qqIqpIqpI

qpIqBq

+=

=∈ πE

( ){ }ννππ somefor ,exp:),( ∑=ℜ∈= + k kkiiin bqqBE


20

Comment on ProofsComment on ProofsDuality Theorem: the two problems below are (Fenchel) dual, with solutions q* = p*.

)||(min

)||(min

),(

),(

ππ

pI

qdI

dBBp

Bq

TTL

E

∈

∈

The resulting values of the objective functions satisfy

)||()||()||( ** ππ qIqdIdI +=


21

More Information GeometryMore Information Geometry……• Shun-ichi Amari, Imre Csiszár

• Two types of information geodesics:–Linear, m-projections–Exponential, e-projections

• Differential geometry on manifold of probability density functions

• Fisher Information is the Riemannian metric

• Exponential family e-flat manifold dually flat Riemannian space

• Dual parameterization: mean and exponential family parameter


22

VariationalVariational RepresentationsRepresentations•• Convex Decomposition LemmaConvex Decomposition Lemma. Let f be convex. Then

• Special Case: f is ln

• Basis for EM; see also De Pierro, Lange, Fessler

∑

∑ ∑≥=

≤

iii

i ii

irii

rr

xfrxf

0,1

)()( 1

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

=≥=

−=⎟⎟⎠

⎞⎜⎜⎝

⎛

∑

∑∑∈

iii

i i

ii

pii

ppp

qppq

1,0:

lnminln

P

P


23

Information Value Decomposition:Information Value Decomposition:Components Analysis

• Minimize over Φ and A

• Convex Decomposition Lemma

∑∑ ∑∑= =

== ⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

+−⎥⎥

⎦

⎤

⎢⎢

⎣

⎡=Φ

I

i

J

j

Kk kjikijK

k kjik

ijij as

a

ssASI

1 11

1

ln)||( φφ

kjikijijk

I

i

J

j

K

k kjik

ijijkijijk

PQ

PQ

asqasq

sq

ASQIASI

φφ

+−=

Φ=Φ

∑∑∑= = =∈

∈

|1 1 1

|| lnmin

)||(min)||( οο


24

Alternating Minimization AlgorithmAlternating Minimization Algorithm• Solution is result of double minimization over

exponential and linear families

∑ ∑∑

∑∑

∑

∑

= =+

=+

+

= ==

++

=

+

∈ΦΦ

=

==

=

Φ=Φ

Jj

Ii ij

lijk

Ii ij

lijkl

kj

J

jKk

ljk

lik

ijl

kjlik

J

jij

lijk

lik

Kk

ljk

lik

lkj

likl

ijk

PQAA

sq

sqa

a

sasq

a

aq

ASQIASI

1' 1 ')1(

'|

1)1(

|)1(

1 1')('

)('

)()(

1

)1(|

)1(

1')('

)('

)()()1(

|

,,)||(minmin)||(min

φφφ

φ

φ

οο


25

Comments on SolutionComments on Solution

• Blind estimation, Poisson data– T. Holmes, et al.– Katsaggelos, et al.

• Nonnegative matrix factorization–Lee and Seung, 1999, Nature.


26

Alternating Minimization AlgorithmsAlternating Minimization Algorithms

• Define problem as minq φ(q)• Derive variational representation: φ(q) = minp J(p,q)• J is an auxiliary function p is in auxiliary set P• Result: double minimization minq minp J(p,q)• Alternating minimization algorithm

),(minarg

),(minarg

)1()1(

)()1(

qpJq

qpJp

l

Qq

l

l

Pp

l

+

∈

+

∈

+

=

=

Comments: Guaranteed Monotonicity; J selected carefully


27

Alternating Minimization AlgorithmsAlternating Minimization Algorithms::II--Divergence, Linear, Exponential FamiliesDivergence, Linear, Exponential Families• Special Case of Interest: J is I-divergence• Families of Interest:

Linear Family L(A,b) = {p: Ap = b}Exponential Family E(π,B) = {q: qi = πi exp[Σj bij νj]}

)||(minarg

)||(minarg

)1(

)

)1(

)(

),(

)1(

qpIq

qpIp

l

B(q

l

l

bAp

l

+

∈

+

∈

+

=

=

,E

L

π

Csiszár and Tusnády; Dempster, Laird, Rubin; Blahut; Richardson; Lucy; Vardi, Shepp, and Kaufman; Cover;Miller and Snyder; O'Sullivan


28

Alternating Minimization ExampleAlternating Minimization Example• Linear family: p1 + 2 p2 = 2• Exponential family: q1 = exp (v), q2 = exp (-v)

0 1 2 3 4 50

1

2

3

4

5

p1, q1

p 2, q2

0 1 2 3 4 50

1

2

3

4

5

p1, q1

p 2, q2

0.6 0.8 1 1.2 1.40.2

0.4

0.6

0.8

1

1.2

p1, q1

p 2, q2

0.6 0.8 1 1.2 1.40.2

0.4

0.6

0.8

1

1.2

p1, q1

p 2, q2

)||(minmin qpILpEq ∈∈


29

Alternating Minimization AlgorithmsAlternating Minimization Algorithms• Projections and triangle equality

• Bounded sums (depending on initial condition)

• Monotonicity; limit points exist, form connected set

)||()||()||(

)||()||()||()()1()1()1()()1(

)()1()1()()()(

llllll

llllll

qqIqpIqpI

qpIppIqpI++++

++

+=

+=

∑

∑∞

=

+

∞

=

+

1

)()1(

1

)1()(

)||(

)||(

l

ll

l

ll

qqI

ppI


30

Information Value Decomposition:Information Value Decomposition:Changing Dimension

• Minimize over Φ and A:

• Change K to K+1 or K-1

∑∑ ∑∑= =

== ⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

+−⎥⎥

⎦

⎤

⎢⎢

⎣

⎡=Φ

I

i

J

j

Kk kjikijK

k kjik

ijij as

a

ssASI

1 11

1

ln)||( φφ

)||(

)||()||()()(

)()()()(

KK

KKKK

qpI

ASQIBSI

=

Φ= οο


31

Information Value Decomposition:Information Value Decomposition:Changing Dimension

L(S)

EKEK+1

EK-1p(K+1)

p(K)

p(K-1)

q(K+1)

q(K)

q(K-1)


32








33

S=ΦAGiven Φ and S, estimate A.

Alternating Minimization AlgorithmsAlternating Minimization Algorithmsfor for HyperspectralHyperspectral ImagingImaging

α1

α2

α3

+ PoissonProcess

C

AlternatingMinimization

Algorithm

InitialEstimated

Coefficients

EstimatedCoefficients


34

Alternating MinimizationsAlternating MinimizationsApplied to Applied to HyperspectralHyperspectral DataData

• Downloaded spectra from USGS website• 470 Spectral components• Randomly generated A with 2000 columns• Ran IVD on result


35

Alternating MinimizationsAlternating MinimizationsApplied to Applied to HyperspectralHyperspectral DataData


36

CT Minimum ICT Minimum I--Divergence FormulationDivergence FormulationMaximize log likelihood function over µ

minimize I-divergenceUse Convex Decomposition Lemmag is a marginal over q

∑ ∑ ∑

∑

∑

⎟⎟⎠

⎞⎜⎜⎝

⎛−=

+−=⋅

−=⋅

∈ =

∈

∈

E x

I

iii

y

y

ExcxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

X

Y

Y

10 )()(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

µµ

µµ

µ

µµµ


37

New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography

⎭⎬⎫

⎩⎨⎧

==

⎟⎟⎠

⎞⎜⎜⎝

⎛−=

+−=

∑

∑ ∑

∑ ∑

∈ =

∈∈

E

x

I

iii

y Eqp

ydEypEyp

ExcxyhEyIEyq

EyqEypEyqEypEypqpI

)(),(:),(

)()(),(exp),(),(

),(),(),(),(ln),()||(minmin

10

L

X

YL

µ

Data determine the linear familyExponential family parameters are image(s)


38

Interpretation: Compare predicted data to measured data via ratio of backprojectionsUpdate estimate using a normalization constant

Comments: Choice for constants; monotonic convergence;Linear convergence; Constraints easily incorporated

)(~)(ˆ

ln)(

1)(ˆ)(ˆ)(

)()()1(

xbxb

xZxcxc l

i

li

i

li

li −=+

New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography


39David G. PolitteOctober 31, 2002

Mini CT, AM Iteration 0000001
























































58

Iterative Algorithm with Known Iterative Algorithm with Known Applicator PoseApplicator Pose

Truth Known Pose


59

OCCT IterationsOCCT Iterations

Known Pose

OCCT


60

TruthTruth Known

StandardFBP

OCCT


61

Magnified views around Magnified views around brachytherapybrachytherapy applicatorapplicator

Truth OCCT

FBP


62

ConclusionsConclusions

•• Applications: Applications: Components AnalysisComponents AnalysisCT ImagingCT Imaging

•• Information GeometryInformation Geometry•• Alternating Minimization Alternating Minimization

AlgorithmsAlgorithms•• Information Value DecompositionInformation Value Decomposition


63


64

OO’’Sullivan Give a Math Talk?Sullivan Give a Math Talk?• Use Mathematics, interested, encourage

students,…

• Erdös mumber is 3

• Proof:– Szego Number is 2

Guido Weiss has Erdös number 2.

Information Geometry with Applications in Components ...jao/Talks/InvitedTalks/Duketalk033004.pdfInformation Geometry J. A. O’Sullivan. Duke Seminar, Mar. 30, 2004 21 More Information

Documents