Page 1
Information Signal ProcessingInformation Signal Processing
Joseph A. OJoseph A. O’’SullivanSullivan
Electronic Systems and Signals Research Laboratory
Center for Security TechnologiesDepartment of Electrical and Systems Engineering
Washington [email protected]
http://essrl.wustl.edu/~jaoSupported by: ONR, NSF, NIH, Boeing Foundation, DARPASpecial thanks to Naveen Singla
Page 2
J. A. O’Sullivan. 11/21/2003Information Signal Processing
2
CollaboratorsCollaborators
Jasenka BenacMichael D. DeVoreAndrew (Lichun) LiClayton MillerLee MontagninoRyan Murphy Natalia SchmidNaveen SinglaBrandon WestoverShenyu Yan
G. James BlaineRoger ChamberlainMark FranklinDaniel R. FuhrmannRonald S. IndeckChenyang LuPierre Moulin, UIUCMarcel MullerRobert PlessDavid G. PolitteChrysanthe PrezaAndrew Singer, UIUC Donald L. SnyderBruce R. WhitingJeffrey F. Williamson, VCULihao Xu
Faculty Current and Former Students
Page 3
J. A. O’Sullivan. 11/21/2003Information Signal Processing
4
OutlineOutline
• DSP ISP
- Data models, computational models,
algorithms
- Central role of information
• Graphical Data Models- X-ray CT imaging
- Iterative decoding
• Message Passing EM
Algorithms
• Applications Revisited
• Speculation on Trends• Conclusions
Page 4
J. A. O’Sullivan. 11/21/2003Information Signal Processing
5
Signal Processing
Information TheoryComputation and
Communication
FFTFFT
MultiresolutionMultiresolutionanalysisanalysis
Page 5
J. A. O’Sullivan. 11/21/2003Information Signal Processing
6
X(0)
X(1)
X(2)
X(3)
X(4)
X(5)
X(6)
X(7)
x(0)
x(4)
x(2)
x(6)
x(1)
x(5)
x(3)
x(7)
FFT
Page 6
J. A. O’Sullivan. 11/21/2003Information Signal Processing
7
Signal Processing
Information Theory
Numerical analysisNumerical analysis
Processors: parallel, Processors: parallel, ASIC, etc.ASIC, etc.
Systolic architecturesSystolic architectures
FFTWFFTW
Computation and
Communication
Page 7
J. A. O’Sullivan. 11/21/2003Information Signal Processing
8
Signal Processing
Information Theory
Numerical analysisNumerical analysis
Processors and Processors and architecturesarchitectures
FFT, FFTWFFT, FFTW
Transversal filtersTransversal filters
MRAMRA
Complexity theoryComplexity theory
Graphical modelsGraphical models
KalmanKalman filtersfilters
CompressionCompression
Computation and
Communication
Page 8
J. A. O’Sullivan. 11/21/2003Information Signal Processing
9
Computation and
Communication
Information Signal ProcessingInformation Signal Processing
Signal Processing
Information Theory
ComplexityComplexity--constrained processingconstrained processing
Signal processing on graphsSignal processing on graphs
Distributed signal processingDistributed signal processing
Distributed information theoryDistributed information theory
Distributed computation and Distributed computation and communicationcommunication
Optimal information extraction, Optimal information extraction, communication, computationcommunication, computation
Page 9
J. A. O’Sullivan. 11/21/2003Information Signal Processing
10
Distributed sensing,
communication, computation““The architecture for a The architecture for a
fully netted maritime forcefully netted maritime force””
Page 10
J. A. O’Sullivan. 11/21/2003Information Signal Processing
11
Wireless Sensor NetworksWireless Sensor Networks
http://www.greatduckisland.net/index.php
Great Duck Island Habitat
Page 11
J. A. O’Sullivan. 11/21/2003Information Signal Processing
14
OutlineOutline
• DSP ISP
- Data models, computational models,
algorithms
- Central role of information
• Graphical Data Models- X-ray CT imaging
- Iterative decoding
• Message Passing EM
Algorithms
• Applications Revisited
• Speculation on Trends• Conclusions
Page 12
J. A. O’Sullivan. 11/21/2003Information Signal Processing
15
Graphical Data ModelsGraphical Data Models
•• Model 1:Model 1: y = Hsy = Hs– y is n × 1, s is m × 1,
and H is n × m
– yj depends on sk if hjk 0
– Defines a graphical model
•• Model 2:Model 2:
– Neighborhood structure
– Bipartite graph model
•• Model 3:Model 3:– RVs on edges of graph
k
j
j
kj jksypp ))(,|()|( sy
k kj
kjk sxpp)(
)|()|( sx
j
jkj jkxypp ))(,|()|( xy
Page 13
J. A. O’Sullivan. 11/21/2003Information Signal Processing
16
Tomography
S
D
Nonrandom Graphs
Line integrals through patient
Quantization point spread function
weights on edges of graph
Helps organize computations
Siemens Somotom Emotion
Page 14
J. A. O’Sullivan. 11/21/2003Information Signal Processing
17
Computational ModelsComputational Models
System model accounts for:
• Information extraction
problem definition
• Compression of sensor data
• Network throughput
• Processor cycles per
instruction
• Size of processor local
memory
• Communication bandwidth
of each link
• etc.
Page 15
J. A. O’Sullivan. 11/21/2003Information Signal Processing
18
Computational ModelsComputational Models
• Local resources plus remote
• Communicate observation as well as classification- Human in the loop
- Remote contribution to classification when available
• Dynamic resource availability
• Sequence of partial classifications (an, n)
Page 16
J. A. O’Sullivan. 11/21/2003Information Signal Processing
19
Progress: Computational Graph Progress: Computational Graph Same as Data Model GraphSame as Data Model Graph
•• Message passing algorithmsMessage passing algorithms– Pearl’s belief propagation
– Iterative decoding
» Turbo-codes, parallel concatenated codes
» Low density parity check codes
» Repeat-accumulate codes, serial concatenated codes
– Iterative equalization and decoding
•• ExpectationExpectation--Maximization (EM) AlgorithmsMaximization (EM) Algorithms– Graphical models
– General problem
– Gaussian, Poisson (emission tomography, transmission tomography)
– Abstract examples on random graphs
Page 17
J. A. O’Sullivan. 11/21/2003Information Signal Processing
20
10011010101000
01010101010001
10100001010101
01100110010100
10010101001010
00101000101011
01001010100110
H Regular (3,6) n=14
Random Graphs
Comment: LDPC parity check matrix
Page 18
J. A. O’Sullivan. 11/21/2003Information Signal Processing
21
001010100000
100100000010
010001001000
000100010001
100000001100
001000100010
010001010000
000010000101
H Regular (2,3) n=12
Random Graphs
Page 19
J. A. O’Sullivan. 11/21/2003Information Signal Processing
22
10011010101000
01010101010001
10100001010101
10101010010100
10010101001010
00101000101101
01001001010110
H Irregular n=14
Random Graphs
Page 20
J. A. O’Sullivan. 11/21/2003Information Signal Processing
24
k
j
jk
jk
),(
),(
)1(
'
)1(
)(
'
)1(
stateg
statef
m
kjk
m
jk
m
jkj
m
jk
Page 21
J. A. O’Sullivan. 11/21/2003Information Signal Processing
27
Iterative Decoding Message PassingIterative Decoding Message Passing
kjk
m
jk
m
jk
\)('
)(
'
)1(
jkj
m
kjz
m
jk k
\)('
)1(
'
)1(
2tanh)1(
2tanh
Codeword Bit Nodes
zk
xj
Check Nodes
jk
jk
Page 22
J. A. O’Sullivan. 11/21/2003Information Signal Processing
28
Iterative Decoding Message PassingIterative Decoding Message Passing
-6
-5
-4
-3
-2
-1
0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
SNR [dB]
Bit e
rro
r ra
te,
Lo
g B
ase
10
ISI-free
BIAWGN Capacity
[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
Page 23
J. A. O’Sullivan. 11/21/2003Information Signal Processing
29
OutlineOutline
• DSP ISP
- Data models, computational models,
algorithms
- Central role of information
• Graphical Data Models- X-ray CT imaging
- Iterative decoding
• Message Passing EM
Algorithms
• Applications Revisited
• Speculation on Trends• Conclusions
Page 24
J. A. O’Sullivan. 11/21/2003Information Signal Processing
30
ML Problem: xsxxys
dpp )|()|(lnmax
EM Algorithm: )|()|(
),|(ln),|(minmin
sxxy
syxsyx
s pp
EM AlgorithmEM Algorithm
sk
yj
Hidden Data
Incomplete Data
xjk
Page 25
J. A. O’Sullivan. 11/21/2003Information Signal Processing
31
VariationalVariational RepresentationsRepresentations
•• Convex Decomposition LemmaConvex Decomposition Lemma. Let f be convex. Then
• Special Case: f is ln
• Basis for EM; see also De Pierro, Lange, Fessler
i
ii
i i
ii
rii
rr
xfrxf
0,1
)()(1
i
ii
i i
ii
i
iq
q
1,0:
lnminln
P
P
Page 26
J. A. O’Sullivan. 11/21/2003Information Signal Processing
32
EM AlgorithmEM Algorithm
')|'()'|(
)|()|(),|(
)(
)()()1(
xsxxy
sxxysyx
dpp
ppm
mmm
xsxsyxss
dp, mmm )|(ln)|(argmax )()1()1(
Assume the factorizations:
k kj
kjk sxpp)(
)|()|( sx
j
jkj jkxypp ))(,|()|( xy
These computations become local and thus message passing
In general, these are global computations
Page 27
J. A. O’Sullivan. 11/21/2003Information Signal Processing
33
Message Passing EM AlgorithmMessage Passing EM Algorithm
)('
'
)('
)(
'''
'),('
'
)('
)(
'''
1(
)|())(',|(
)|())(',|(
),|(
jk
jk
jk
m
kjkjkj
kkjk
jk
jk
m
kjkjkj
(m)
jjk
)m
dxsxpjkxyp
dxsxpjkxyp
yx
KK
KK
K
K
s
jkkjk
m
jjk
m
s
m
k dxsxp,yxsk
)|(ln)|(argmax )()1()1(s
sk
yj
Input Data
Measured Data
xjk
)( jkx
ks
Page 28
J. A. O’Sullivan. 11/21/2003Information Signal Processing
34
GaussianGaussian--MAPMAP
)2
,0(~
),0(~
;
0 Iw
Is
wHsy
NN
PN
)(
)(
0
)1( ˆ
|)(|2|)(|
ˆkj
l
jk
l
k x
j
NkP
Ps
)('
)1(
'
)1()1( ˆ|)(|
1ˆˆ
jk
l
kj
l
k
l
jk syj
sx
sk
yj
Input Data
Measured Data
xjk
jkx
ks
Page 29
J. A. O’Sullivan. 11/21/2003Information Signal Processing
35
GaussianGaussian--MAPMAP
[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
Page 30
J. A. O’Sullivan. 11/21/2003Information Signal Processing
36
Emission TomographyEmission Tomography
y ~ Poisson(H )
: Mean of emitted photons
)(
)()(
)1( ˆ|)(|
ˆˆ
kj
m
j
m
km
k qk
)('
)(
'
)(
ˆˆ
jk
m
k
jm
j
yq
k
yj
Pixels
Measured Data
jq
kˆ
Page 31
J. A. O’Sullivan. 11/21/2003Information Signal Processing
37
Emission TomographyEmission Tomography
[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
Page 32
J. A. O’Sullivan. 11/21/2003Information Signal Processing
38
Emission TomographyEmission Tomography
[50000,1000] regular (3,150) matrix[50000,1000] regular (3,150) matrix
Page 33
J. A. O’Sullivan. 11/21/2003Information Signal Processing
39
Transmission TomographyTransmission Tomography
y ~ Poisson(I0exp(-H ))
: photon attenuation )(
)(
)()()1( ln1ˆˆ
kj
m
j
kj
j
m
k
m
kq
y
z
)(
)(
0
)( ˆexpjk
m
k
m
j Iq
k
yj
Pixels
Measured Data
jq
kˆ
Page 34
J. A. O’Sullivan. 11/21/2003Information Signal Processing
40
[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
Transmission TomographyTransmission Tomography
Page 35
J. A. O’Sullivan. 11/21/2003Information Signal Processing
41
Comments on DetailsComments on Details
•• Information geometry basisInformation geometry basis
•• Easily extended to arbitrary Easily extended to arbitrary HH
•• Low density Low density sparsesparse
•• Constraints in iterative decoding vs. forward Constraints in iterative decoding vs. forward modelmodel
Page 36
J. A. O’Sullivan. 11/21/2003Information Signal Processing
42
OutlineOutline
• DSP ISP
- Data models, computational models,
algorithms
- Central role of information
• Graphical Data Models- X-ray CT imaging
- Iterative decoding
• Message Passing EM
Algorithms
• Applications Revisited- Iterative decoding
- X-ray CT imaging
• Speculation on Trends• Conclusions
Page 37
J. A. O’Sullivan. 11/21/2003Information Signal Processing
43
Science and technologyScience and technologypotentially yield potentially yield
6 Tb/in6 Tb/in22
courtesy R. S. courtesy R. S. IndeckIndeck
Patterned Magnetic Media
Page 38
J. A. O’Sullivan. 11/21/2003Information Signal Processing
44
Advanced Recording MediaAdvanced Recording MediaBluBlu--Ray DiscRay Disc
Next-generation Optical Disc Video Recording Format
Page 39
J. A. O’Sullivan. 11/21/2003Information Signal Processing
45
2D Intersymbol Interference2D Intersymbol Interference
1111
11
11
11
1111
21
21
11211
kkkk
k
xxx
x
xxx
1111110
110
12
1111110
100100
kkkkkk
kkkkkk
k
kk
k
rrrr
rrrr
r
rrrr
rrrw(i,j)
25.05.0
5.01h
Includes
Guard Band
jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.0
Singla et al., “Iterative decoding and equalization for 2-D recording channels,” IEEE Trans.
Magn., Sept. 2002.
Page 40
J. A. O’Sullivan. 11/21/2003Information Signal Processing
46
Full Graph Message PassingFull Graph Message Passing
Measured Data Nodes (r)
Codeword Bit Nodes (x)
Check Nodes (z)
jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.025.05.0
5.01h
Page 41
J. A. O’Sullivan. 11/21/2003Information Signal Processing
49
Full Graph Message Passing
-6
-5
-4
-3
-2
-1
0
0 0.5 1 1.5 2 2.5 3
SNR [dB]
Bit e
rro
r ra
te, L
og
Ba
se
10
ISI-free
Full Graph_50
Full Graph Message PassingFull Graph Message Passing
[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
Page 42
J. A. O’Sullivan. 11/21/2003Information Signal Processing
50
Full Graph AnalysisFull Graph Analysis
Length 4 cycles present which degrade performance Length 4 cycles present which degrade performance of messageof message--passing algorithm passing algorithm
x(i+2,j) x(i+2,j+1)
x(i+1,j) x(i+1,j+1)
x(i,j) x(i,j+1)
x(i+2,j+2)
x(i+1,j+2)
x(i,j+2)
r(i+1,j+1)
r(i,j+1)r(i,j)
r(i+1,j)
From
Check
Nodes
Kschischang et al., “Factor graphs and the sum-product algorithm,” IEEE Trans. Inform.
Theory, Feb. 2001.
Page 43
J. A. O’Sullivan. 11/21/2003Information Signal Processing
51
Ordered Subsets Message Passing Ordered Subsets Message Passing
From Imaging From Imaging –– Data set is grouped into subsets to Data set is grouped into subsets to increase rate of convergence of image increase rate of convergence of image reconstruction algorithmsreconstruction algorithms
For Decoding For Decoding –– Measured data is grouped into Measured data is grouped into subsets to eliminate short length cycles in the subsets to eliminate short length cycles in the Channel ISI graphChannel ISI graph
H. M. Hudson, and R. S. Larkin, “Accelerated image reconstruction using ordered subsets
of projection data,” IEEE Trans. Medical Imaging, Dec. 1994
Page 44
J. A. O’Sullivan. 11/21/2003Information Signal Processing
52
Labeling of data nodes into 4 subsetsLabeling of data nodes into 4 subsets
For each iteration use data nodes of one label onlyFor each iteration use data nodes of one label only
Labeled ISI GraphLabeled ISI Graph
J. A. O’Sullivan, and N. Singla, “Ordered subsets message-passing,” Int’l Symp. Inform.
Theory, Yokohama, Japan 2003.
Page 45
J. A. O’Sullivan. 11/21/2003Information Signal Processing
53
Ordered Subsets Message Passing
-6
-5
-4
-3
-2
-1
0
0 0.5 1 1.5 2 2.5 3
SNR [dB]
Bit e
rror
rate
in log10
ISI-free
Ordered Subsets_200
Full Graph_50
[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
Ordered Subsets Message PassingOrdered Subsets Message Passing
Page 46
J. A. O’Sullivan. 11/21/2003Information Signal Processing
54
CT Imaging in Presence of High CT Imaging in Presence of High Density Attenuators (J. Williamson, PI)Density Attenuators (J. Williamson, PI)
Brachytherapy applicators
After-loading colpostats
for radiation oncology
Cervical cancer: 50% survival rate
Dose prediction important
Object-Constrained Computed
Tomography (OCCT)
Page 47
J. A. O’Sullivan. 11/21/2003Information Signal Processing
55
Filtered Back ProjectionFiltered Back Projection
Truth FBP
FBP: inverse Radon transform
Page 48
J. A. O’Sullivan. 11/21/2003Information Signal Processing
56
Transmission TomographyTransmission Tomography• Source-detector pairs indexed by y; pixels indexed by x
• Data d(y) Poisson, means g(y: ), log likelihood function
• Mean unattenuated counts I0, mean background
• Attenuation function (x,E), E energies
• Maximize over or ci; equivalently minimize I-divergence
)(),(),(exp),():(
):():(ln)()):(|(
0 yExxyhEyIyg
ygygydgdl
E x
y
X
Y
I
i
ii ExcEx1
)()(),(
Page 49
J. A. O’Sullivan. 11/21/2003Information Signal Processing
57
MaximumMaximum--LikelihoodLikelihoodMinimum IMinimum I--divergencedivergence
• Poisson distribution
• Poisson distributed data loglikelihood function
• Maximization over equivalent to minimization of I-divergence
kk
kkI
kkkNP
ek
kNPk
ln)||(
!lnln)(ln
!)(
)(),(),(exp),():(
):()():(
)(ln)()):(||(
):():(ln)()):(|(
0 yExxyhEyIyg
ygydyg
ydydgdI
ygygydgdl
E x
y
y
X
Y
Y
Page 50
J. A. O’Sullivan. 11/21/2003Information Signal Processing
58
Maximum Likelihood Maximum Likelihood Minimum IMinimum I--DivergenceDivergence
Difficulties: log of sum, sums in exponent
)()()(),(exp),():(
):()():(
)(ln)()):(||(
):():(ln)()):(|(
1
0 yExcxyhEyIyg
ygydyg
ydydgdI
ygygydgdl
E x
I
i
ii
y
y
X
Y
Y
Page 51
J. A. O’Sullivan. 11/21/2003Information Signal Processing
59
Interpretation: Compare predicted data to measured data
via ratio of backprojections
Update estimate using a normalization constant
Comments: choice for constants; monotonic convergence;
linear convergence; fixed points satisfy Kuhn-
Tucker conditions; constraints easily incorporated
)(~
)(ˆln
)(
1)(ˆ)(ˆ
)(
)()()1(
xb
xb
xZxcxc
li
li
i
li
li
New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography
Page 52
J. A. O’Sullivan. 11/21/2003Information Signal Processing
60David G. Politte
October 31, 2002
Mini CT, AM Iteration 0000001
Page 53
J. A. O’Sullivan. 11/21/2003Information Signal Processing
61David G. Politte
October 31, 2002
Mini CT, AM Iteration 0000002
Page 54
J. A. O’Sullivan. 11/21/2003Information Signal Processing
62David G. Politte
October 31, 2002
Mini CT, AM Iteration 0000005
Page 55
J. A. O’Sullivan. 11/21/2003Information Signal Processing
63David G. Politte
October 31, 2002
Mini CT, AM Iteration 0000010
Page 56
J. A. O’Sullivan. 11/21/2003Information Signal Processing
64David G. Politte
October 31, 2002
Mini CT, AM Iteration 0000020
Page 57
J. A. O’Sullivan. 11/21/2003Information Signal Processing
65David G. Politte
October 31, 2002
Mini CT, AM Iteration 0000050
Page 58
J. A. O’Sullivan. 11/21/2003Information Signal Processing
66David G. Politte
October 31, 2002
Mini CT, AM Iteration 0000100
Page 59
J. A. O’Sullivan. 11/21/2003Information Signal Processing
67David G. Politte
October 31, 2002
Mini CT, AM Iteration 0000200
Page 60
J. A. O’Sullivan. 11/21/2003Information Signal Processing
68David G. Politte
October 31, 2002
Mini CT, AM Iteration 0000500
Page 61
J. A. O’Sullivan. 11/21/2003Information Signal Processing
69David G. Politte
October 31, 2002
Mini CT, AM Iteration 0001000
Page 62
J. A. O’Sullivan. 11/21/2003Information Signal Processing
70David G. Politte
October 31, 2002
Mini CT, AM Iteration 0002000
Page 63
J. A. O’Sullivan. 11/21/2003Information Signal Processing
71David G. Politte
October 31, 2002
Mini CT, AM Iteration 0005000
Page 64
J. A. O’Sullivan. 11/21/2003Information Signal Processing
72David G. Politte
October 31, 2002
Mini CT, AM Iteration 0010000
Page 65
J. A. O’Sullivan. 11/21/2003Information Signal Processing
73David G. Politte
October 31, 2002
Mini CT, AM Iteration 0020000
Page 66
J. A. O’Sullivan. 11/21/2003Information Signal Processing
74David G. Politte
October 31, 2002
Mini CT, AM Iteration 0050000
Page 67
J. A. O’Sullivan. 11/21/2003Information Signal Processing
75David G. Politte
October 31, 2002
Mini CT, AM Iteration 0100000
Page 68
J. A. O’Sullivan. 11/21/2003Information Signal Processing
76David G. Politte
October 31, 2002
Mini CT, AM Iteration 0200000
Page 69
J. A. O’Sullivan. 11/21/2003Information Signal Processing
77David G. Politte
October 31, 2002
Mini CT, AM Iteration 0500000
Page 70
J. A. O’Sullivan. 11/21/2003Information Signal Processing
78David G. Politte
October 31, 2002
Mini CT, AM Iteration 1000000
Page 71
J. A. O’Sullivan. 11/21/2003Information Signal Processing
79
Our Plans in CT ImagingOur Plans in CT Imaging
•• CTCT MultirowMultirow SinogramSinogram data:data:14081408 ×× 768768 ×× nndd ×× nnzz
– where nd is the number of detector rows and nz is the number of gantry rotations
•• Fully 3Fully 3--D Implementations for D Implementations for Quantitative CTQuantitative CT
•• SpeedSpeed--up: Ordered Subsets, up: Ordered Subsets, MultigridMultigrid Methods, Parallel Methods, Parallel Implementations on Clusters Implementations on Clusters of PCsof PCs
•• Future: PETFuture: PET--CTCT Siemens Somotom Emotion
Page 72
J. A. O’Sullivan. 11/21/2003Information Signal Processing
80
Slide and data from R. Laforest and M. Mintun.
PETCT-211
Page 73
J. A. O’Sullivan. 11/21/2003Information Signal Processing
81
PETCT-165
Slide and data from R. Laforest and M. Mintun.
Page 74
J. A. O’Sullivan. 11/21/2003Information Signal Processing
82
Additional Algorithm/DetectorAdditional Algorithm/DetectorModel DevelopmentModel Development
•• RegularizationRegularization
•• Energy integrating detectors Energy integrating detectors
•• Finite detector size, better source modelFinite detector size, better source model
•• Finite pixel, Finite pixel, voxelvoxel sizesize
•• Average integral or average exponentialAverage integral or average exponential(arithmetic vs. geometric average)(arithmetic vs. geometric average)
•• Partial volume effectsPartial volume effects
•• MotionMotion
•• ScatteringScattering
•• Limited angle tomographyLimited angle tomography
•• Region of interestRegion of interest
•• Scanner implementations: beam hardening Scanner implementations: beam hardening correction, sampling, etc.correction, sampling, etc.
),( EyEdN
Page 75
J. A. O’Sullivan. 11/21/2003Information Signal Processing
83
Computation and
Communication
Signal Processing
Information Theory
ComplexityComplexity--constrained processingconstrained processing
Signal processing on graphsSignal processing on graphs
Distributed signal processingDistributed signal processing
Distributed information theoryDistributed information theory
Distributed computation and Distributed computation and communicationcommunication
Optimal information extraction, Optimal information extraction, communication, computationcommunication, computation
Page 76
J. A. O’Sullivan. 11/21/2003Information Signal Processing
84
Limits of Information TheoryLimits of Information Theory
•• Information theory provides bounds on performance Information theory provides bounds on performance of communication, compression, and data analysisof communication, compression, and data analysis
– Channel coding theorem (capacity)
– Entropy, rate-distortion theory
– Fisher information
•• Open Problems in Information TheoryOpen Problems in Information Theory
– Broadcast channel p(y1,y2|x) capacity region of achievable (R1,R2)
» Depends only on p(y1|x) and p(y2|x); degraded channel known
– Distributed source compression achievable (R1,R2, D1,D2)
•• Algorithmic information theory (Algorithmic information theory (KolmogorovKolmogorovcomplexity)complexity)
p(x,yp(x,y))
XXnn
YYnn
XXnn
ffyy
ffxx
YYnngg ^
^
Page 77
J. A. O’Sullivan. 11/21/2003Information Signal Processing
85
Speculation on FutureSpeculation on Future
•• Distributed compressionDistributed compression– Compression with side information
– Analogies to information embedding
– Reduced communication rates
•• Broadcast channel models Broadcast channel models – Appropriate for motes
– Communication at different rates using a common signal
– Reduced communication rates
•• Tradeoffs in communication and computationTradeoffs in communication and computation
•• Mobile computing: cheap Mobile computing: cheap expensiveexpensive cheapcheap– mobile base station + network mobile
Page 78
J. A. O’Sullivan. 11/21/2003Information Signal Processing
86
ConclusionsConclusions
•• Information Signal ProcessingInformation Signal Processing
—— DSP, Information Theory, Computation andDSP, Information Theory, Computation and
CommunicationCommunication
•• Role of Graphical ModelsRole of Graphical Models
•• Message Passing EM AlgorithmsMessage Passing EM Algorithms
•• Iterative Equalization and DecodingIterative Equalization and Decoding
•• XX--Ray CT ImagingRay CT Imaging
Page 79
J. A. O’Sullivan. 11/21/2003Information Signal Processing
87
Future WorkFuture Work
Signal Processing
Information Theory
Fast algorithmsFast algorithms
Optimal communicationOptimal communication
Distributed information Distributed information theorytheory
Computation and
Communication