Topics in Multimedia Signal Processing 1 Inverse Problems and Machine Learning Julian Wörmann Research Group for Geometric Optimization and Machine Learning (GOL) Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 02.05.2014
106
Embed
Inverse Problems and Machine Learning · 1 Topics in Multimedia Signal Processing Inverse Problems and Machine Learning Julian Wörmann . Research Group for Geometric Optimization
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Topics in Multimedia Signal Processing 1
Inverse Problems and Machine Learning
Julian Wörmann Research Group for Geometric Optimization and Machine Learning (GOL)
Topics in Multimedia Signal Processing
Maschinelles Lernen und Inverse Probleme 02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 2
What are inverse problems?
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 3
Inverse Problems
cause/ excitation
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 4
Inverse Problems
cause/ excitation
System/ Process
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 5
Inverse Problems
cause/ excitation
effect/ measurement
System/ Process
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 6
Inverse Problems
cause/ excitation
effect/ measurement
System/ Process
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 7
Inverse Problems
cause/ excitation
effect/ measurement
System/ Process
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 8
Inverse Problems
cause/ excitation
effect/ measurement
System/ Process
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 9
Inverse Problems
cause/ excitation
effect/ measurement
System/ Process
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 10
Goal
02.05.2014
cause/ excitation
effect/ measurement
System/ Process
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 11
Goal
02.05.2014
cause/ excitation
effect/ measurement
System/ Process
System/ -1
Process
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 12
Goal
Model:
02.05.2014
cause/ excitation
effect/ measurement
System/ Process
System/ -1
Process
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 13
Goal
noise
Model:
02.05.2014
cause/ excitation
effect/ measurement
System/ Process
System/ -1
Process
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 14
Inverse Problems in Image Processing
?
Denoising Deblurring Inpainting
02.05.2014
Topics in Multimedia Signal Processing
1. Determine the model parameters
Maschinelles Lernen und Inverse Probleme 15
Tasks
02.05.2014
Topics in Multimedia Signal Processing
1. Determine the model parameters
Maschinelles Lernen und Inverse Probleme 16
Tasks
02.05.2014
Topics in Multimedia Signal Processing
1. Determine the model parameters
2. Reconstruct from
Maschinelles Lernen und Inverse Probleme 17
Tasks
02.05.2014
Topics in Multimedia Signal Processing
1. Determine the model parameters
2. Reconstruct from
Maschinelles Lernen und Inverse Probleme 18
Tasks
?
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 19
Approaches to solve inverse problems
02.05.2014
Topics in Multimedia Signal Processing
Maschinelles Lernen und Inverse Probleme 20
Least Squares approach
02.05.2014
Topics in Multimedia Signal Processing
• Problems: – ill-conditioned
Example: Signal Deconvolution/Deblurring
Maschinelles Lernen und Inverse Probleme 21
Least Squares approach
02.05.2014
Topics in Multimedia Signal Processing
• Problems: – ill-conditioned
Example: Signal Deconvolution/Deblurring – n ≠ m System under-/overdetermined
infinitely many/no solutions Example: Signal Inpainting
Maschinelles Lernen und Inverse Probleme 22
Least Squares approach
02.05.2014
Topics in Multimedia Signal Processing
• Problems: – ill-conditioned
Example: Signal Deconvolution/Deblurring – n ≠ m System under-/overdetermined
infinitely many/no solutions Example: Signal Inpainting
– No AWGN
Maschinelles Lernen und Inverse Probleme 23
Least Squares approach
02.05.2014
Topics in Multimedia Signal Processing
• Problems: – ill-conditioned
Example: Signal Deconvolution/Deblurring – n ≠ m System under-/overdetermined
infinitely many/no solutions Example: Signal Inpainting
– No AWGN • Solutions:
– Exploiting structures and properties of the data
Maschinelles Lernen und Inverse Probleme 24
Least Squares approach
02.05.2014
Topics in Multimedia Signal Processing
• Problems: – ill-conditioned
Example: Signal Deconvolution/Deblurring – n ≠ m System under-/overdetermined
infinitely many/no solutions Example: Signal Inpainting
– No AWGN • Solutions:
– Exploiting structures and properties of the data – Optimization under constraints
Maschinelles Lernen und Inverse Probleme 25
Least Squares approach
02.05.2014
Topics in Multimedia Signal Processing
Maschinelles Lernen und Inverse Probleme 26
Optimization under constraints
02.05.2014
Topics in Multimedia Signal Processing
Maschinelles Lernen und Inverse Probleme 27
Optimization under constraints
Constraint set encoded in function
02.05.2014
Topics in Multimedia Signal Processing
Maschinelles Lernen und Inverse Probleme 28
Optimization under constraints
Constraint set encoded in function
assumed noise energy
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 29
What are suitable constraints?
- Pixelvalues are always positive
- Images contain homogeneous regions, i.e.
neighbouring pixels often have the same value
- Signals can be composed of „Basissignals“ (e.g. sinusoids)
Maschinelles Lernen und Inverse Probleme 29 02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 30
Synthesis Operator (Dictionary) idealised
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 31
Synthesis Operator (Dictionary) idealised
atoms
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 32
Synthesis Operator (Dictionary) idealised
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 33
Synthesis Operator (Dictionary) idealised
atoms
…
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 34
Synthesis Operator (Dictionary) idealised
atoms
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
…
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 35
Synthesis Operator (Dictionary) idealised
=
atoms signal
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1
0 0
…
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 36
Synthesis Operator (Dictionary) idealised
=
atoms signal
0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
1
0 0
…
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 37
Synthesis Operator (Dictionary) idealised
=
atoms signal
0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
1
1 1
…
02.05.2014
Topics in Multimedia Signal Processing
Maschinelles Lernen und Inverse Probleme 38
Synthesis Model
02.05.2014
Topics in Multimedia Signal Processing
Dictionary
Maschinelles Lernen und Inverse Probleme 39
Synthesis Model
=
02.05.2014
Topics in Multimedia Signal Processing
Assumption: Signal has a sparse representation
Dictionary
Maschinelles Lernen und Inverse Probleme 40
Synthesis Model
=
02.05.2014
Topics in Multimedia Signal Processing
Assumption: Signal has a sparse representation
Dictionary
Maschinelles Lernen und Inverse Probleme 41
Synthesis Model
=
02.05.2014
Topics in Multimedia Signal Processing
Assumption: Signal has a sparse representation
Dictionary • redundant • Columns are called atoms
Maschinelles Lernen und Inverse Probleme 42
Synthesis Model
=
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 43
JPEG Compression
Natural images are compressible signals with a compressible representation in a DCT (JPEG) or Wavelet Basis (JPEG-2000)
Compressible Signals can be well approximated through sparse signals
Maschinelles Lernen und Inverse Probleme 43 02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 44
JPEG Compression
Maschinelles Lernen und Inverse Probleme 44 02.05.2014
Image from: Gregory K. Wallace, The JPEG Still Picture Compression Standard, IEEE Transactions on Consumer Electronics, vol. 38 no.1, Feb. 1992.
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 45 Maschinelles Lernen und Inverse Probleme 45 02.05.2014
What are appropriate Synthesis/Analysis Operators?
Maschinelles Lernen und Inverse Probleme 61 02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 62
Analysis Operator Learning
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 63
Operator Learning example for Image Processing
vectorised patches
Operator Learning
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 64
Analysis Operator Learning Basics
Required: N representative training signals
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 65
Analysis Operator Learning Basics
Required: N representative training signals
Sought: Analysis Operator, such that N analysed vectors are sparse
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 66
Analysis Operator Learning Basics
Required: N representative training signals
Sought: Analysis Operator, such that N analysed vectors are sparse
Analysis Operator Atoms
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 67
Analysis Operator Learning Basics
Required: N representative training signals
Sought: Analysis Operator, such that N analysed vectors are sparse
Analysis Operator Atoms
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 68
Analysis Operator Learning Basics
Required: N representative training signals
Sought: Analysis Operator, such that N analysed vectors are sparse
Constraint set to avoid trivial solution
Analysis Operator Atoms
02.05.2014
Topics in Multimedia Signal Processing
Maschinelles Lernen und Inverse Probleme 69
Geometric Analysis Operator Learning (GOAL)
02.05.2014
Topics in Multimedia Signal Processing
• Constraints
Maschinelles Lernen und Inverse Probleme 70
Geometric Analysis Operator Learning (GOAL)
02.05.2014
Topics in Multimedia Signal Processing
• Constraints 1. Atoms/rows of are normalised, i.e.
Maschinelles Lernen und Inverse Probleme 71
Geometric Analysis Operator Learning (GOAL)
02.05.2014
Topics in Multimedia Signal Processing
• Constraints 1. Atoms/rows of are normalised, i.e. 2. has full rank, i.e.
Maschinelles Lernen und Inverse Probleme 72
Geometric Analysis Operator Learning (GOAL)
02.05.2014
Topics in Multimedia Signal Processing
• Constraints 1. Atoms/rows of are normalised, i.e. 2. has full rank, i.e. 3. Rows are not trivially linear dependent,
Maschinelles Lernen und Inverse Probleme 73
Geometric Analysis Operator Learning (GOAL)
02.05.2014
Topics in Multimedia Signal Processing
• Constraints 1. Atoms/rows of are normalised, i.e. 2. has full rank, i.e. 3. Rows are not trivially linear dependent,
• From constraints 1+2 element of a special manifold efficient method to find a solution (e.g. Conjugate Gradient, Quasi-Newton)
Maschinelles Lernen und Inverse Probleme 74
Geometric Analysis Operator Learning (GOAL)
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 75
Example: Manifold Learning
• Normalised rows lie on the surface of a sphere (with radius = 1)
• Step along geodesics
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 76
Applied to solve inverse problems
?
Denoising Deblurring Inpainting
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 77
Applied to solve inverse problems
!
Denoising Deblurring Inpainting
02.05.2014
Topics in Multimedia Signal Processing
Demo: GOAL + Lena
02.05.2014 Maschinelles Lernen und Inverse Probleme 78
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 79
Bimodal signal reconstruction
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 80
Application: 3D Reconstruction in HD
3D scene analysis with high-resolution camera
and depth sensor
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 81
Application: 3D Reconstruction in HD
3D scene analysis with high-resolution camera
and depth sensor
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 82
Application: 3D Reconstruction in HD
3D scene analysis with high-resolution camera
and depth sensor
Bimodal Analysis Operator
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 83
Learning from bimodal signals
Intensity Depth
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 84
Learning from bimodal signals
Intensity Depth bright
dark
signal pair
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 85
Learning from bimodal signals
Intensity Depth bright
dark
Intensity operator 𝛀𝐼 Depth operator 𝛀𝐷
minimize𝛀𝐼,𝛀𝐷
𝐺 𝛀𝐼𝑺𝐼 ,𝛀𝐷𝑺𝐷
learn 𝛀𝑰 and 𝛀𝐷 such that both analyzed vectors 𝛀𝐼𝒔𝑖 and 𝛀D𝒔d are maximally sparse
signal pair
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 86
Bimodal reconstruction Unimodal
𝐬∗ ∈ arg min𝐬 ∈ ℝ𝑵
𝑔 𝛀𝐬 subject to 𝒜𝐬 − 𝒚 22 ≤ 𝜀
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 87
Bimodal reconstruction Unimodal
𝐬∗ ∈ arg min𝐬 ∈ ℝ𝑵
𝑔 𝛀𝐬 subject to 𝒜𝐬 − 𝒚 22 ≤ 𝜀
Bimodal
(𝒔𝐼∗, 𝒔𝐷∗ ) ∈ arg min𝒔𝐼,𝒔𝐷∈ ℝ𝑵
𝐺 𝛀𝐼𝒔𝐼 ,𝛀𝐷𝒔𝐷 subj. to 𝒜𝐼𝒔𝐼 − 𝒚𝐼 22 + 𝒜𝐷𝒔𝐷 − 𝒚𝐷 2
2 ≤ 𝜀
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 88
Bimodal reconstruction Unimodal
𝐬∗ ∈ arg min𝐬 ∈ ℝ𝑵
𝑔 𝛀𝐬 subject to 𝒜𝐬 − 𝒚 22 ≤ 𝜀
Bimodal
𝒔𝐷∗ ∈ arg min𝒔𝐷∈ ℝ𝑵
𝜆𝐺 𝒄,𝛀𝐷𝒔𝐷 + 𝒜𝐷𝒔𝐷 − 𝒚𝐷 22
𝒄 0 Intensity image fixed
(𝒔𝐼∗, 𝒔𝐷∗ ) ∈ arg min𝒔𝐼,𝒔𝐷∈ ℝ𝑵
𝐺 𝛀𝐼𝒔𝐼 ,𝛀𝐷𝒔𝐷 subj. to 𝒜𝐼𝒔𝐼 − 𝒚𝐼 22 + 𝒜𝐷𝒔𝐷 − 𝒚𝐷 2
2 ≤ 𝜀
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 89
Results of the bimodal reconstruction (JID)
bicubic interpolation NN interpolation JID
8x
Depth Map Super-Resolution
3D Scene Reconstruction
original bicubic interpolation JID
02.05.2014
Topics in Multimedia Signal Processing Maschinelles Lernen und Inverse Probleme 90
Analysis Based Blind Compressive Sensing
02.05.2014
Topics in Multimedia Signal Processing
• Only a few linear and non-adaptive measurements are sufficient to reconstruct the signal with high accuracy.
Maschinelles Lernen und Inverse Probleme 91
Concept of Compressive Sensing
02.05.2014
Topics in Multimedia Signal Processing
• Only a few linear and non-adaptive measurements are sufficient to reconstruct the signal with high accuracy.
• Exploitation of the sparse representation with a (analytically) given Dictionary or Operator.
Maschinelles Lernen und Inverse Probleme 92
Concept of Compressive Sensing
02.05.2014
Topics in Multimedia Signal Processing
• Reconstruction of the signals under the assumption that there exists a sparse representation
Maschinelles Lernen und Inverse Probleme 93
Analysis Based Compressive Sensing
02.05.2014
Topics in Multimedia Signal Processing
• Exploiting the property that learned operators admit a sparser representation
• Adaptive, signal dependent regularisation of the inverse problem under
consideration of the error model
Maschinelles Lernen und Inverse Probleme 94
Analysis Based Blind Compressive Sensing
02.05.2014
Topics in Multimedia Signal Processing
• Exploiting the property that learned operators admit a sparser representation
• Adaptive, signal dependent regularisation of the inverse problem under
consideration of the error model
Maschinelles Lernen und Inverse Probleme 95
Analysis Based Blind Compressive Sensing
02.05.2014
Topics in Multimedia Signal Processing
Image reconstruction
Maschinelles Lernen und Inverse Probleme 96
Analysis Based Blind Compressive Sensing
(1) ABCS (2) TV Operator
02.05.2014
Topics in Multimedia Signal Processing
Learned Analysis Operators
Maschinelles Lernen und Inverse Probleme 97
Analysis Based Blind Compressive Sensing
(1) Random Input (2) Barbara (3) Piecewise constant
02.05.2014
Topics in Multimedia Signal Processing
• Simultaneous reconstructing and learning allows one to find an operator that adaptively fits the underlying image structure
Maschinelles Lernen und Inverse Probleme 98
Analysis Based Blind Compressive Sensing
02.05.2014
Topics in Multimedia Signal Processing
• Simultaneous reconstructing and learning allows one to find an operator that adaptively fits the underlying image structure
• The Analysis Operator does not need to be learned before the
reconstruction
Maschinelles Lernen und Inverse Probleme 99
Analysis Based Blind Compressive Sensing
02.05.2014
Topics in Multimedia Signal Processing
• Simultaneous reconstructing and learning allows one to find an operator that adaptively fits the underlying image structure
• The Analysis Operator does not need to be learned before the
reconstruction
• Ability to reconstruct different signal/image classes by simply exchanging the error model
Maschinelles Lernen und Inverse Probleme 100
Analysis Based Blind Compressive Sensing
02.05.2014
Topics in Multimedia Signal Processing
Maschinelles Lernen und Inverse Probleme 101
Take Home Messages
02.05.2014
Topics in Multimedia Signal Processing
• Structure in data is extremely important and can be utilized to
regularize inverse problems
Maschinelles Lernen und Inverse Probleme 102
Take Home Messages
02.05.2014
Topics in Multimedia Signal Processing
• Structure in data is extremely important and can be utilized to
regularize inverse problems
• Sparsity is a valuable property of many signals
Maschinelles Lernen und Inverse Probleme 103
Take Home Messages
02.05.2014
Topics in Multimedia Signal Processing
• Structure in data is extremely important and can be utilized to
regularize inverse problems
• Sparsity is a valuable property of many signals
• Machine Learning can help to find such structures
Maschinelles Lernen und Inverse Probleme 104
Take Home Messages
02.05.2014
Topics in Multimedia Signal Processing
• Structure in data is extremely important and can be utilized to
regularize inverse problems
• Sparsity is a valuable property of many signals
• Machine Learning can help to find such structures
• Geometric aspects of a problem can be exploited in the optimization
Maschinelles Lernen und Inverse Probleme 105
Take Home Messages
02.05.2014
Topics in Multimedia Signal Processing
Weiterführende Literatur
02.05.2014 Maschinelles Lernen und Inverse Probleme 106
S. Hawe, M. Kleinsteuber, and K. Diepold. Analysis Operator Learning and its Application to Image Reconstruction. IEEE Transactions on Image Processing, vol.22, no.6, pp.2138-2150, June 2013. J. Wörmann, S. Hawe, and M. Kleinsteuber. Analysis Based Blind Compressive Sensing. IEEE Signal Processing Letters, 20(5) 491-494, 2013. M. Kiechle, S. Hawe, and M. Kleinsteuber. A Joint Intensity and Depth Co-Sparse Analysis Model for Depth Map Super-Resolution. IEEE International Conference on Computer Vision 2013. M. Aharon, M. Elad, and A. Bruckstein. K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation. IEEE Transactions on Signal Processing, vol. 54, no.11, 2006.