Interaction Techniques in Medical Volume Visualization Bernhard Preim
Jan 03, 2016
Interaction Techniques in Medical Volume Visualization
Bernhard Preim
Interaction Tasks and Techniques
Interaction Tasks• Exploration of original data• Data reduction• Manipulation of transfer functions• Multiplanar reformatting (MPR)
Bernhard Preim 2/68
Interaction Tasks and Techniques: Exploration of Original Data
Bernhard Preim 3/68
• “Browsing” through the slice data• Simple contrast and brightness
control via mouse movement (windowing)
• Flexible definition of slices in a corresponding visualization
• Cine mode for animation impression
• Opening and closing of a legend in the viewer
Patient information (name, date of birth, gender, Id)Image information (modality, voxel size, recording date)Coordinate and value of the selected voxelOption: more or less detailed legend
• Synchronized display of two data sets
Example: Liver CT; first data set without contrast agent, second data set with CASynchronization related to windowing parameters and the displayed layer
• Selection of the viewing direction (coronary, sagittal, axial)
Bernhard Preim 4/68
Interaction Tasks and Techniques: Exploration of Original Data
Example for legends, data: Univ. Hospital Leipzig
Bernhard Preim 5/68
Interaction Tasks and Techniques: Exploration of Original Data
Interaction Tasks and Techniques: Exploration of Original Data
Change of contrast and brightness, data: Univ. Hospital Leipzig
Bernhard Preim 6/68
Interaction Tasks and Techniques: Exploration of Original Data
Browsing through the slices (interactive or as movie)
Bernhard Preim 7/68
Interaction Tasks and Techniques: Exploration of Original Data
Synchronized illustration. Left: original data, right: filtered data
Bernhard Preim 8/68
Moving of a cross line in communicated views (Peter Hastreiter, Uni Erlangen)
Bernhard Preim 9/68
Historical model:Drawings by Dürer
Interaction Tasks and Techniques: Exploration of Original Data
Why?
Focus on certain problemsReduction of the data volume (memory requirements, rendering speed)
How?
Data selection in a certain interval (e.g. iso-surface)Definition of a volume of interest (cuboid partial volume)Subsampling of data (e.g. reduction by factor 2 in x and y direction)
Bernhard Preim 10/68
Interaction Tasks and Techniques: Data Reduction
Definition of a VOI in orthogonal views
Bernhard Preim 11/68
Interaction Tasks and Techniques: Data Reduction
Transfer functions: Mapping of data onto presentation parameters (colors, gray values, transparency)
• Determine the visibility and perceptibility of structures• Parametrization of TFs is an essential interaction for the
exploration of volume data.
Challenges:• Exploration of data sets with unknown structures• Exploration of data sets with different structures of similar
intensity
Bernhard Preim 12/68
Interaction Tasks and Techniques: Transfer Functions
Three volume visualizations of one CT data set with different opacity transfer functions.
Bernhard Preim 13/68
Skin Bones Teeth
Interaction Tasks and Techniques: Transfer Functions
Requirements• Selection of predefined TFs (e.g. liver CT, lung CT)• Targeted search for suitable TFs• Correlation between adjustable parameters and characteristics
of the resulting images• Definition flexibility• Fast preview
Bernhard Preim 14/68
Interaction Tasks and Techniques: Transfer Functions
Typical transfer functions:• Windowing• Bi-/trilevel windowing• Inverse windowing• Piecewise linear functions• Polynoms of higher degree/splines
Problem: No recognizable relation between TF characteristics and visualization
Bernhard Preim 15/68
Interaction Tasks and Techniques: Transfer Functions
Thorax CT data set, emphasis of skeletal structures
Bernhard Preim 16/68
Interaction Tasks and Techniques: Transfer Functions
Bernhard Preim 17/68
Thorax CT data set, emphasis of blood vessels
Interaction Tasks and Techniques: Transfer Functions
Representation and application of TFs• Discrete representation in lookup tables• Size: e.g. 4096 entries with 32 bit (8 bit each for RGB and
alpha)• Hardware support for Lookup tables
Problem: hardware dependency w.r.t. size of color tables
Bernhard Preim 18/68
Interaction Tasks and Techniques: Transfer Functions
Interaction Tasks and Techniques: Manipulation of Transfer Functions
Sophisticated concepts:
• Stochastic generation of TFs that may be selected by the user (multilevel iterative search), presentation as thumbnails (He et al. [1996], König et al. [2001])
• Image-based TF design (Fang et al. [1998])
• Enhanced TF
Integration of image processing filters (e.g. edge recognition)Local TF
Multidimensional TF (illustration of derived data, e.g. gradient fields, Levoy [1988])
Bernhard Preim 19/68
Interaction Tasks and Techniques: Manipulation of Transfer Functions
Stochastic generation of TFs:
Iterative search process (He et al. [1996]):
1. Use of an initial TF library
2. "Mutation" of this function through a genetic algorithm (25 generations)3. Direct volume rendering (back then with VolVis 100x100 pixel, 10s)4. Subjective result analysis by the user
20/68Bernhard Preim
21/68Bernhard Preim
Source:
König et al. [2001]
Interaction Tasks and Techniques: Manipulation of Transfer Functions
Interaction Tasks and Techniques: Transfer Functions
Image-based TF design• Idea: Definition of the transfer function, image information
serve as context (Castro et al. [1998])
• Global histogram• Histogram along a layer• Histogram along a ray
Bernhard Preim 22/68
Interaction Tasks and Techniques: Transfer Functions
Histogram along the orange ray as context for TF specification
Bernhard Preim 23/68
Image: Dirk Bartz, Univ. Leipzig
eye ball (light)muscles (dark)
Interaction Tasks and Techniques: Transfer Functions
RGBAlpha and gray value Alpha TF (Peter Hastreiter, Uni Erlangen)
Bernhard Preim 24/68
Interaction Tasks and Techniques: Transfer Functions
Composition of a TF as weighted sum of component functions
Parameters of component functions:
Sb, Sc - inner sampling points, Sa, Sd - outer sampling points
Bernhard Preim 25/68
Interaction Tasks and Techniques: Transfer Functions
Adaptation of a trapezoid template to the local histogram of a rectangular region.
Bernhard Preim 26/68
Source: Castro et al. [1998]
• “Implicit” segmentation of the white brain substance through suitable transfer functions
• Emphasis of the histogram area between the maxima of gray and white brain substance
Bernhard Preim 27/68
Interaction Tasks and Techniques: Transfer Functions
Histogram TF (purple: opacity values, green: gray values)
Intersection of gray and white substance
• “Implicit” segmentation of the white brain substance through suitable transfer functions
• Emphasis of the histogram area between the maxima of gray and white brain substance
Interaction Tasks and Techniques: Transfer Functions
Bernhard Preim 28/68
The Transfer Function Bake-Off, Data: Sheep heart (IEEE CG&Application 5/6 2001)• Comparison of different TF specification techniques1. ISO rendering of the segmented raw data (sheep heart) 2. Trial&Error - (20 min) with VolumePro3. Without data model - ISO automatically selected according to the maximum
gradient magnitude4. 2D TF with data model – automatic distance map, semiautom. opacity, manual color
map
Bernhard Preim 29/66
Interaction Tasks and Techniques: Transfer Functions
Interaction Tasks and Techniques: Transfer Functions
Data-based TechniquesSelection of a transfer function that emphasizes the edges.Edge model:
Perfect intersection between 2 structures is "blurred" through an error function. Assumption: Blurring through an isotropic Gaussian function. -> fits to CT data well
Source: Kindlman, Durkin [1998]
Bernhard Preim 30/68
Interaction Tasks and Techniques: Transfer Functions
Data-based Techniques: edge enhancementEdge criteria: strong gradient g, very small second derivative
h (zero crossing):
-h(v)p(v) =
g(v)
Data values along an edge, 1st and 2nd derivative
Bernhard Preim 31/68
Interaction Tasks and Techniques: Transfer Functions
Determination of g (v) and h (v) via average determination from all first and second derivatives of all voxels with value v.Internal representation:Histogram volume H:
x-axis → f (v)y-axis → f“(v)z-axis → f´(v)
Algorithm:1. Determine min. and max. valuesfor f‘‘(v) and the maximum for f´(v). Minimum for f´(v) is assumed to be 0.
2. Fill H, whereas the values are scaled such that min and max are depicted from f´ and f´´ to 0 and 256.
32/68Bernhard Preim
Interaction Tasks and Techniques: Transfer Functions
Data-based Techniques: edge enhancement• What can be determined from the histogram volume?
Edge positions w.r.t. the data• What can be entered by the user?
• A selection of the "peaks" that shall be depicted• Form of the depicted peaks via boundary emphasis
function (bef)• Typical forms of bef()
Bernhard Preim 33/68
Interaction Tasks and Techniques: Transfer Functions
Data-based Techniques: edge enhancement
Applied 2D opacity functionand volume rendering of the Visible Woman data set (TF automatically determined).• The small image indicates the 2DHistogram (intensity values vs. • Gradient magnitude)Brightness indicates frequency of.
Source: Kindlmann, Durkin (1998)
Bernhard Preim 34/68
Interaction Tasks and Techniques: Transfer Functions
Data-based Techniques: edge enhancementComparison of edge-enhancing direct volume rendering and iso-
surface rendering
Illustration of a spiny dendrite based on microscopy data
Source: Kindlmann and Durkin 1998
Bernhard Preim 35/68
Interaction Tasks and Techniques: Transfer Functions
Data-based Techniques: edge enhancement• Preconditions for successful application:
Existence of clear object boundariesHomogenous dataOnly little noise, no "outliers"Medicine: CT data (if CA is applied, it must be equally distributed)
Bernhard Preim 36/68
Interaction Tasks and Techniques: Transfer Functions
Data-based Techniques• Use of a once specified TF as reference• Goal: "Re-use" of an empirically specified TF
Application: targeted illustration of a structure in a modality (e.g. aneurysms in MR)Procedure:
Selection of a reference data set Dref and a TF Tref(v)Use of the normalized histograms of the data sets H(Dref )
and H(Dstudy)Non-linear transformation t of the intensity values of Dstudy, such that H (Dstudy) ~ H (t(Dref))Hence, Tstudy (v) = Tref (v)
Bernhard Preim 37/68
Interaction Tasks and Techniques: Transfer Functions
Data-based Techniques: reference TF• Determination of the similarity of the histograms
1. Idea: minimization of the histogram distances2. Better idea: use of the p-function by Kindlmann (considers also f‘(v) and f‘‘(v))In case of comparable data sets the p-values are similar to the histograms
Literature: Rezk-Salama et al., VMV [2000]
Bernhard Preim 38/68
Interaction Tasks and Techniques: Transfer Functions
Data-based Techniques: Reference TF
• Visualization of blood vessels in the brain with CT angiography, left: no adaptation, middle: illustration of the first idea (histogram transformation), right: adaptation of the p-function Source: Rezk-Salama et al. [2000]
Bernhard Preim 39/68
Interaction Tasks and Techniques: Transfer Functions
Multidimensional TFs• 1D TF: Map data onto opacity/colors• Multidimensional TFs: Use additionally derived information,
e.g. strength of the gradient or the second derivative
• Typical example: Adaptation of the opacity to the strength of the gradient, emphasis of data
intersections• Advantage: Additional degrees of freedom to generate
high-quality images• Disadvantage:High interaction costs
Bernhard Preim 40/68
Interaction Tasks and Techniques: Transfer Functions
Multidimensional TFs• Consideration of the 2nd derivative (1st derivative of a scalar
field → vector, 2nd derivative → matrix)
• Hessian Matrix:
• Criterion (scalar value) for the 2nd derivative: largest eigenvalue of the Hessian Matrix and strength of the 2nd derivative, respectively in direction to the gradient (instead of the Hessian Matrix)
Bernhard Preim 41/68
Interaction Tasks and Techniques: Transfer Functions
Multidimensional TFs• Gradient calculation usually via central differences
• Mapping of the gradient size to the opacity (gradient magnitude weighted transparency)
Bernhard Preim 42/68
Interaction Tasks and Techniques: Transfer Functions
Multidimensional TFs• Volume visualization with a
gradient-dependent TF for opacity, accord. to Levoy [1988]) (Visible Human CT data set)
Bernhard Preim 43/68
Interaction Tasks and Techniques: 2D Transfer Functions
Starting point for a simple specification: gradient intensity histograms. Filtering is important. Goal: accentuation of intersections.
Bernhard Preim 44/68
Interaction Tasks and Techniques: 2D Transfer Functions
Bernhard Preim 45/68
Dense tissue and bone parts with additional gradient emphasis (green marking)Image courtesy: Hoen-Oh Shin and Benjamin King,MH Hannover [2004]
Interaction Tasks and Techniques: 2D Transfer Functions
Bernhard Preim 46/68
More dense soft tissue (yellow marking)Image courtesy: Hoen-Oh Shin and Benjamin King,MH Hannover [2004]
Interaction Tasks and Techniques: 2D Transfer Functions
Bernhard Preim 47/68
Regions with high gradients are visualized (red marking)Image courtesy: Hoen-Oh Shin and Benjamin King, MH Hannover [2004]
Interaction Tasks and Techniques: 2D Transfer Functions
Bernhard Preim 48/68
Source: Stölzl [2004]
Interaction Tasks and Techniques: 2D Transfer Functions
Edge detector as input to define arcs
Bernhard Preim 49/68
Source: Stölzl [2004]
Interaction Tasks and Techniques: Transfer Functions
Local TFs• Motivation: Often, global TFs enable no sufficient
differentiation• Example: Division of a lookup table into 4 segments for 4
different illustrations
Caution: Interpolation beyond segment borders is not allowed!
Bernhard Preim 50/68
Source: Rezk-Salama [2002]
Interaction Tasks and Techniques: Transfer Functions
Blood vessels in the lung lobes are displayed with separate local TFs.
Bernhard Preim 51/68
Interaction Tasks and Techniques: Transfer Functions
Template-based specification of 2D TFs1D templates:
2D templates:
Bernhard Preim 52/68
Source: Tappenbeck [2006]
Source: Castro et al. [1998]
Interaction Tasks and Techniques: Transfer Functions
Template-based specification of 2D TFs:• Simplification of the interaction is even more important than in
the 1D case• Discretization in a Lookup table• Sufficient size required, at least 256x256• Applicable to arbitrary 2D domains (intensity: gradient
strength, intensity: distance to a target structure, …)
Bernhard Preim 53/68
Interaction Tasks and Techniques: Transfer Functions
Representation of 2D TFs in a rectilinear grid as basis for discretization in an LUT
Bernhard Preim 54/68
Source: Tappenbeck [2006]
Distance-dependent TFs:• Additional entries:
Segmented target structure (tagged volume)Distance transformation w.r.t. the target
• Use of an editor for 2D TF• Sample applications:
Fade-in of safety margins around tumorsOpacity control in case of large organs
Bernhard Preim 55/68
Interaction Tasks and Techniques: Transfer Functions
Interaction Tasks and Techniques: Transfer Functions
Target structure: lung surfaceSelection of interesting structures by intensity and distances
Bernhard Preim 56/68
Source: Tappenbeck [2006]
Resulting Volume Visualization (quite useless example; but appropriate illustration of the concept). Distance to the lung is used to assign different colors and opacity values.
Bernhard Preim 57/68
Source: Tappenbeck [2006]
Interaction Tasks and Techniques: Transfer Functions
Useful example: Emphasis of blood vessels in certain distances around a tumor via distance-dependent TFs
Bernhard Preim 58/68
Source: Tappenbeck [2006]
Interaction Tasks and Techniques: Transfer Functions
Interaction Tasks and Techniques: Multiplanar Reformatting
MPR illustration of an MRT data set of the head.Left: The cutting plane indicates which slice is cut out from the original data.
Bernhard Preim 59/68
• Combination of the exploration of slice data with an MPR illustration
Bernhard Preim 60/68
Interaction Tasks and Techniques: Multiplanar Reformatting
• Direct manipulative control of the MPR via Jack Manipulator
Bernhard Preim 61/68
Interaction Tasks and Techniques: Multiplanar Reformatting
Application for vessel diagnostics • MPR is automatically oriented
orthogonal to the vessel centerline • Integrated view of cross section and
3D visualization
Bernhard Preim 62/68
Interaction Tasks and Techniques: Multiplanar Reformatting
Local MPR:Rotation around a locally interesting structure (tumor, vessel centerline)
Bernhard Preim 63/68
Interaction Tasks and Techniques: Multiplanar Reformatting
Anatomical Reformatting:
Idea: Use of segmentation results for a reformatting during which layers with a constant
distance to an anatomical structure (e.g. an organ surface) arise.
Procedure: - Lines of the data set are shifted against each other in such a way that voxels on the surface are located in a layer vertically to the viewing direction.
- The originally curved slices are viewed in layers → organ-specific coordinate system.
Feature: Anatomically reformatted layers show only voxels with same distances to the organ boundary.
Bernhard Preim 64/68
Interaction Tasks and Techniques: Multiplanar Reformatting
Bernhard Preim 65/68
Anatomical reformatting based on a lung lobe segmentation.Round lesions in the mediastinum.Source: Dicken et al., BVM 2003, Data: Prof. Günther (RWTH Aachen)
Interaction Tasks and Techniques: Multiplanar Reformatting
Summary
• Suitable interaction techniques are crucial for the practical application of medical visualization techniques.
• On the one hand, the interaction should be simple and clear. On the other hand, it should be flexible enough.
• Presets, or automatically adapting presets, are often a good basis.
• Transfer functions: Image galleries and gradient-dependent TFs are the standard.
Bernhard Preim 66/68
Literature
Silvia Castro, Andreas König, Helwig Löffelmann, and Eduard Gröller, Transfer Function Specification for the Visualization of Medical Data, Technical Report, Institute of Computer Graphics and Algorithms, Vienna University of Technology, 1998, ftp://ftp.cg.tuwien.ac.at/pub/TR/98/TR-186-2-98-12Paper.ps.gz
V. Dicken, B. Wein, H. Schubert et al. Projektionsansichten zur Vereinfachung der Diagnose von multiplen Lungenrundherden in CT-Thorax-Aufnahmen, Bildverarbeitung für die Medizin, Springer, Reihe Informatik aktuell, März 2003
S. Fang, T. Biddlecome, and M. Tuceryan. Image-Based Transfer Function Design for Data Exploration in Volume Visualization. In Proc. IEEE Visualization, 1998, http://www.cs.iupui.edu/~tuceryan/research/Microscopy/vis98.pdf
T. He, L. Hong, A. Kaufman, and H. Pfister, Generation of Transfer Functions with Stochastic Search Techniques, in Proceedings of Visualization '96, October 1996, http://www.merl.com/people/pfister/pubs/vis96.pdf
Gordon Kindlmann and James W. Durkin. Semi-Automatic Generation of Transfer Functions for Direct Volume Rendering, In IEEE Symposium On Volume Visualization, 1998, http://www.cs.utah.edu/~sci/publications/vv98glk-paper.pdf
Bernhard Preim 67/68
LiteratureA. König and E. Gröller. Mastering Transfer Function Specification by Using VolumePro
Technology, In Proc. Spring Conference on Computer Graphics, 2001,http://www.cg.tuwien.ac.at/research/TR/00/TR-186-2-00-07Paper.pdf
M. Levoy, Display Surfaces from Volume Data, IEEE Computer Graphics and Applications, 25 (1988) http://www-graphics.stanford.edu/papers/volume-cga88
C. Rezk-Salama, Peter Hastreiter, J. Scherer, G. Greiner: Automatic adjustment of transfer functions for 3d volume rendering. In Proc. of Vision, Modelling and Visualization, S. 357-364, 2000.
C. Rezk-Salama, Volume Rendering Techniques for General Purpose Graphics Hardware, Dissertation, Philipp-Alexander Universität Erlangen-Nürnberg
D. Stölzel. Entwurf gradientenabhängiger 2D-Transferfunktionen für die medizinische Volumenvisualisierung. Master's thesis, Dept. of Computer Science, 2004. http://www.vismd.de/lib/exe/fetch.php?media=files:master_thesis:stoelzel_2004_thesis.pdf
A. Tappenbeck, B. Preim, and V. Dicken.Distance-Based Transfer Function Design: Specification Methods and Applications. In Simulation und Visualisierung, pages 259-274. SCS-Verlag, 2006 http://www.vismd.de/lib/exe/fetch.php?media=files:volume_rendering:tappenbeck_2006_simvis.pdf
Bernhard Preim 68/68