Page 1
ProbExplorer: Uncertainty-guided Exploration and Editing of Probabilistic Medical Image Segmentation
Ahmed Saad1,2, Torsten Möller1, and Ghassan Hamarneh2
1Graphics, Usability, and Visualization (GrUVi) Lab,2 Medical Image Analysis Lab (MIAL),
School of Computing Science, Simon Fraser University, Canada
Page 2
2Ahmed Saad ProbExplorer
Outline
• Medical image segmentation challenges• ProbExplorer framework• Case studies
– Highlight suspicious regions (e.g. tumors)– Correct misclassification results
• Uncertainty visualization using shape and appearance prior information
• Conclusion and future work
Page 3
3Ahmed Saad ProbExplorer
Medical image segmentation
• Partitioning the image into disjoint regions of homogeneous properties
• Useful for statistical analysis, diagnosis, and treatment evaluation
Medical Image Segmentation
Page 4
4Ahmed Saad ProbExplorer
Segmentation challenges
• Low signal-to-noise ratio• Partial volume effect• Anatomical shape variability• Multi-dimensional data
Magnetic Resonance Imaging Positron Emission Tomography
Page 5
5Ahmed Saad ProbExplorer
Segmentation challenges
• Low signal-to-noise ratio• Partial volume effect• Anatomical shape variability• Multi-dimensional data
Page 6
6Ahmed Saad ProbExplorer
Segmentation challenges
• Low signal-to-noise ratio• Partial volume effect• Anatomical shape variability• Multi-dimensional data
Patient 1 Patient 2 Patient 3 Patient 4
Page 7
Ahmed Saad ProbExplorer
Segmentation challenges
• Low signal-to-noise ratio• Partial volume effect• Anatomical shape variability• Multi-dimensional data
4D CT dPET DTMRI
7
Page 8
Ahmed Saad ProbExplorer
Segmentation output
Crisp Probabilistic (Fuzzy)
70%
20%10%
Putamen
White matterGrey matter
Putamen
8
Max
Page 9
9Ahmed Saad ProbExplorer
Outline
• Medical image segmentation challenges• ProbExplorer framework• Case studies
– Highlight suspicious regions (e.g. tumors)– Correct misclassification results
• Uncertainty visualization using shape and appearance prior information
• Conclusion and future work
Page 10
10Ahmed Saad ProbExplorer
Goal
• Given probabilistic segmentation results, we will allow expert users to visually examine regions of segmentation uncertainty to– Highlight suspicious regions (e.g. tumors)– Correct misclassification results without re-
running the segmentation
Page 11
11Ahmed Saad ProbExplorer
ProbExplorer
Preprocessing Selecting voxels EditingProbabilistic
segmentation
Change selectionCommit an editing action
Page 12
12Ahmed Saad ProbExplorer
ProbExplorer
Preprocessing Selecting voxels EditingProbabilistic
segmentation
Change selectionCommit an editing action
Page 13
13Ahmed Saad ProbExplorer
ProbExplorer
Preprocessing Selecting voxels EditingProbabilistic
segmentation
Change selectionCommit an editing action
Page 14
14Ahmed Saad ProbExplorer
ProbExplorer
Preprocessing Selecting voxels EditingProbabilistic
segmentation
Change selectionCommit an editing action
Before After
Page 15
15Ahmed Saad ProbExplorer
Preprocessing
• A probabilistic vector field
)](),....,()([)( 21 xPxPxPxP C
Sort maxP )(xPFBG
)(xM
Page 16
16Ahmed Saad ProbExplorer
Outline
• Medical image segmentation challenges• ProbExplorer framework• Case studies
– Highlight suspicious regions (e.g. tumors)– Correct misclassification results
• Uncertainty visualization using shape and appearance prior information
• Conclusion and future work
Page 17
17Ahmed Saad ProbExplorer
Renal dynamic SPECT
• 4D image of size 64 x 64 x 32 with 48 time steps with an isotropic voxel size of (2 mm)3
Raw data Crisp segmentation
Page 18
18Ahmed Saad ProbExplorer
Uncertainty interaction overview widget
?
Page 19
19Ahmed Saad ProbExplorer
Selection of normal behavior
Page 20
20Ahmed Saad ProbExplorer
Selection of abnormal behavior
Page 21
21Ahmed Saad ProbExplorer
Outline
• Medical image segmentation challenges• ProbExplorer framework• Case studies
– Highlight suspicious regions (e.g. tumors)– Correct misclassification results
• Uncertainty visualization using shape and appearance prior information
• Conclusion and future work
Page 22
22Ahmed Saad ProbExplorer
Uncertainty-based segmentation editing
Ground truth Overestimation Underestimation
Page 23
23Ahmed Saad ProbExplorer
Synthetic example
No noise no PVE
Ground truth
Observed = noise + PVE
Current segmentation
Page 24
Ahmed Saad ProbExplorer
Synthetic example: push action
Push action
Source set Destination set
24
Page 25
25Ahmed Saad ProbExplorer
Synthetic example: push action
is the first best guess
0.40.3
0.2Swap0.3 0.4
Page 26
26Ahmed Saad ProbExplorer
Dynamic PET brain
Page 27
27Ahmed Saad ProbExplorer
Overestimated putamen
Ground truth Overestimated Putamen
Page 28
28Ahmed Saad ProbExplorer
Uncertainty interaction overview widget
Page 29
29Ahmed Saad ProbExplorer
Dynamic PET brain
Page 30
Ahmed Saad ProbExplorer
Dynamic PET brain
Push actionPutamen
Background
Skull
Grey matter
Cerebellum
Source set Destination set
30
Page 31
31Ahmed Saad ProbExplorer
Dynamic PET brain
After 2 editing actions
Page 32
32Ahmed Saad ProbExplorer
More (Saad et al., EuroVis10)
Selection
Page 33
33Ahmed Saad ProbExplorer
Outline
• Medical image segmentation challenges• ProbExplorer framework• Case studies
– Highlight suspicious regions (e.g. tumors)– Correct misclassification results
• Uncertainty visualization using shape and appearance prior information
• Conclusion and future work
Page 34
Ahmed Saad ProbExplorer
• Maximum-a-posteriori principle
Bayesian perspective
Likelihood PriorPosterior
34
Page 35
Ahmed Saad ProbExplorer
Framework
Atlas construction Shape prior
Like
lihoo
d
Appearance prior
Like
lihoo
d
Images
Expert binarysegmentations
Probabilistic shape prior
Probabilistic appearance prior
Population representative image
New image New probabilistic segmentation
Image-to-Image registration
Alignedlikelihood
35
Page 36
Ahmed Saad ProbExplorer
• is a spatial location in • is a feature vector associated with that can
be constructed from intensity, gradient, etc.• can be decomposed into:
– is the shape prior– is the appearance prior
Mathematical notations
36
Page 37
Ahmed Saad ProbExplorer
Algorithm demonstration using synthetic example
Piecewise constant Blurring Noise
100 noise realizations and random translations
37
Page 38
Ahmed Saad ProbExplorer
• We adopt the method used by Hamarneh and Li [JIVC 09]• Alignment of binary shapes• Shape histogram
Atlas construction:Shape prior modeling
38
Page 39
Ahmed Saad ProbExplorer
• Alignment of binary shapes• Shape histogram • Distance transform DIST(X)
Atlas construction:Shape prior modeling
39
Page 40
Ahmed Saad ProbExplorer
• Alignment of binary shapes• Shape histogram • Distance transform DIST(X)• Probabilistic shape prior
Atlas construction:Shape prior modeling
40
Page 41
Ahmed Saad ProbExplorer
• Multivariate Gaussian fitting
Atlas construction:Appearance prior modeling
41
Page 42
Ahmed Saad ProbExplorer
• Mixture of Gaussians
• Other probabilistic segmentation techniques can be used, e.g. Random walker, Probabilistic SVM, etc.
Likelihood
42
Page 43
Ahmed Saad ProbExplorer
Abnormal cases
43
Page 44
Ahmed Saad ProbExplorer
Abnormal shapeData Maximum likelihood
Selection
44
Page 45
Ahmed Saad ProbExplorer
Abnormal shapeData
Selection
Maximum likelihood
45
Page 46
Ahmed Saad ProbExplorer
Abnormal appearanceData
Selection
Maximum likelihood
46
Page 47
Ahmed Saad ProbExplorer
Abnormal shape and appearanceData
Selection
Maximum likelihood
47
Page 48
Ahmed Saad ProbExplorer
Misclassification correction
Dice: 0.32 Dice: 0.75
48
Page 49
49Ahmed Saad ProbExplorer
More (Saad et al., IEEEVis10)
Page 50
50Ahmed Saad ProbExplorer
User evaluation
• Our clinical collaborators showed how ProbExplorer can be used to achieve highly accurate segmentation from a very noisy dSPECT renal study (Humphries et al. IEEE Nuclear Science Symposium/Medical Image Conference 2009)
Page 51
51Ahmed Saad ProbExplorer
Conclusion
• ProbExplorer: a framework for the analysis and visualization of probabilistic segmentation results
• We provided a number of visual data analysis widgets to reveal the different class interactions that are usually hidden by a simple crisp visualization
Page 52
52Ahmed Saad ProbExplorer
Future work
• Spatial dependency between voxels during interactive editing
• Investigate the behavior of the resulting probabilistic results from different segmentation techniques
• Multi-structure atlas• Registration uncertainty visualization
Page 53
53Ahmed Saad ProbExplorer
Acknowledgements
• Natural Sciences and Engineering Research Council of Canada (NSERC)
• Prof. Vesna Sossi, Prof. Anna Celler, Thomas Humphries, and Prof. Manfred Trummer
Page 54
54
Ahmed Saad
www.sfu.ca/~aasaad
[email protected]