Automatic Brain Tumor segmentation
Automatic Brain Tumor segmentationProblem:
The task in this problem is to automatically detect the presence
of tumors in MR images of the brain, and segment the abnormal
pixels from the normal pixels. Traditionally, the task has tried to
segment the metabolically active 'enhancing' area of the tumor,
which appears hyper-intense in T1 weighted images after the
injection of gadolinium. Several recent methods have focused on
additionally segmenting non-enhancing regions, as well as tumors
that may only partially enhance or do not enhance at all. Several
recent methods have also focused on the related task of segmenting
edema (swelling) associated with tumors.
Segmentation of completely enhancing or 'border enhancing'
tumors is a relatively easy problem, while more work is still
required for the task of segmenting tumors that do not have these
characteristics. This is an interesting problem, since it is a task
that humans can learn to do very well, while developing algorithms
to do the same task has proven challenging. Some of the challenges
associated with this task include:
Local Noise
Inter-Slice Intensity Variations
Partial Volume Artifacts
Intra-Volume Intensity Inhomogeneity
Inter-Volume Intensity Inhomogeneity
Integration of multi-spectral (potentially unaligned) data
Intensity Overlap between normal and abnormal areas
Tumor heterogeneity
Changes in normal structures observed due to the presence of
tumorsAutomated Tumour Segmentation, using Machine Learning
Most of our efforts to date have focused on locating tumour
volumes within a patient's brain, based on a set of MR images -- a
task known as 'tumour segmentation'. Our automated segmentation
program (ASP) takes as input a 3 * k grid of images, where each
image is an MR image of a patient's brain; see first 3 columns of
Figure 1. (This shows only k=5 of the typically around k=21 slices
taken.) Note all images in each row corresponds to the same axial
slice of the patient, at some height, and the first 3 columns are
(respectively) images that are weighted T1, T1c, (T1 weighted after
injection of the gadolinium contrast agent), and T2. The ASP output
is a 3D volume corresponding to its assessment of the Gross Tumour
Volume (GTV), encoded as a sequence of k images; see the 4th column
of Figure 1. Our goal is to produce an accurate volume,
automatically -- i.e., without any human assistance
Figure 1: Each row corresponds to the same axial slice through a
patient's brain. The first three columns are (resp.) T1, T1c, T2
weighted images, and the 4th is the region that ASP labels as
GTV.
ASP uses a classifier to label each voxel in the complete volume
as either GTV or not, based on properties associated with that
voxel. The obvious properties would be simply the 3 intensity
values (T1, T1c and T2). Unfortunately, this is not sufficient, as
there are 'intensity triples' that are tumour in one part of the
brain, but normal in another (Schmidt, 2005). We therefore need to
find other properties for each voxel, besides the intensities, and
then to find the most effective combination of these features.
Feature Sets
Other researchers have produced brain atlases (templates), that
specify voxel-level properties of a typical brain - e.g., providing
the probability of grey matter at location (18 down, 200 left, 52
back), as well as every other coordinate (Schmidt, 2005). This can
be extremely useful, as it can help specify what T1/T1c/T2
intensities to expect at each location; large deviations from the
expectation can help identify possible tumours. Of course, this
assumes we can relate the locations in the patient image to
locations in the template. Unfortunately, these locations will
typically be different - e.g., position (18, 200, 52) in the
template might correspond to, say, (19, 207, 55) in the patient -
as the patient's head may not be the 'typical' size and shape, and
moreover it may have been tilted when the image was captured. A
linear transformation can adjust for such simple scaling and
rotation; a more serious problem arises when large tumours distort
the anatomy within the brain itself - e.g., by shrinking the
ventricles, or pushing them to new locations; see Figure 2. We
solved this by following the linear registration with a
highly-regularized non-linear morphing (Schmidt, 2005).
For each voxel, we augmented the 3 intensity values (T1, T1c and
T2) with 15 other values, 12 based on various templates, and 3
based on an alignment-based measure, 'symmetry', which is the
difference of intensity between current position and its reflection
about the mid-saggital plane. (Note this requires us to identify
this mid-saggital plane, which is 'x=0' in the aligned image but
need not be 'x=0' in the initial images.) We then considered 4
different 'resolutions', to take into account some texture effects
(see below), for a total of (3+12+3) 4 = 72 features; see Figure
3.
Figure 3: Each box shows one of the 72 features used by ASP
system. That is, the label for the voxel at each location depends
on 72 feature values, one from the corresponding location in these
boxes.It is also important for the intensities to 'mean the same
thing' throughout the volume, and also across patients (see below).
However, due to various artifacts (as well as differences in magnet
strength, etc), a T1 intensity of 122 might correspond to 'normal
white matter' at one location of a patient but to 'tumour' at
another. We addressed this by a combination of noise reduction and
intensity standardization steps, based on MIPAV. Figure 4 shows the
results of these processes.
At a high level, this requires
Preprocessing
Intensity Normalization to make intensities more consistent
within and between volumes
Spatial Normalization to allow comparisons between spatial
locations in different modalities and with locations in the
template/coordinates
Segmentation
Feature extraction to represent pixel-level measurements related
to the image, coordinate system, and template; see 72-features.
Supervised Pixel Classification to Assign the pixels a class
(tumour or normal) by combining the feature values
Label Relaxation to correct misclassifications using spatial
label dependencies
Figure 4: Results of pre-processing steps. Top row: original
images (T1, T1c, T2). Middle row: after Noise Reduction. Bottom
row: after intensity standardization.
Best Combination of Features -- Learning a Classifier
Using the ideas discussed above, we can assign a set of features
to each voxel in the volume. The real challenge, however, is
determining which combination of these features corresponds to GTV
-- e.g., perhaps a voxel is part of a GTV if
(T1c intensity)>0.5 and (diff from symmetry) > 0.2 and
(prob of grey-matter) < 0.3
or alternatively, maybe it is a GTV if
0.3 * (T1c intensity) + -0.1 * (diff from symmetry) + 13 * (prob
of grey-matter ) - 6 is positive
or some other variant. If we, or any of our colleagues, knew
which combination was appropriate, we could just implement this
function directly. Unfortunately, no one knows this best function.
However, we do have access to many images of other patients, which
experts have labeled -- i.e., explicitly identified which voxels
are GTV and which are normal. The field of Machine Learning has
provided a number of algorithms that can use these labeled images
to learn a classifier, which can then assign a label to a new
(unlabeled) voxel of a novel patient (Mitchell, 1997). In essence,
these learners find the patterns in the data that identify when a
voxel belongs to the class of GTV voxels, versus the class of
normal voxels. The current version of ASP used a standard algorithm
for learning a 'Support Vector Machine' (Cristianini and
Shawe-Taylor, 2000); we found this worked effectively and is an
accurate classifier (Zhang et al., 2004; Zijdenbos et al., 2002;
Garcia and Morena, 2004. For 'inter-patient training' -- when the
learner is trained on one set of patients, but then tested on a
different patient -- we obtained an average Jaccard score of 0.732
over 11 patients, with a variety of tumours (grade 2 astrocytoma,
anaplastic astrocytoma, glioblastoma multiforme, oligodendroglioma)
at different stages of treatment, and based on two 1.5 T scanners;
see (Schmidt et al., 2005).
We see that this 'learning approach' is important - indeed,
essential! - in this situation, where the 'correct' answer is not
known. It also adds flexibility to the system, as it means we can
produce a new segmentation system: the classifier learned depends
on the labels of the training data. For the GTV-classifier, the
expert labeled voxel as '1' if it is a GTV voxel, and '0'
otherwise. If the expert, instead, labeled each training voxel with
'1' if it was 'Tumour+Edema' area (and '0' otherwise), then the
learner will produce a 'Tumour+Edema-classifier'. Our inter-patient
score on this task was 0.77. See Figure 5.
Figure 5: The bottom row shows results of three different
classifiers: gross tumour volume, enhancing region, and
tumour+edema. Each classifier takes as input all 3 images (T1, T1c
and T2) shown in the top row, as well as information based on
various templates; see Figure 3Notice this methodology also allows
ASP to accommodate different magnet strengths or other contexts. We
would just need to train on images produced by this alternative
MRI, and then labeled by an expert.
Our initial ASP system used only the features associated with a
voxel to assign a label to that voxel. This means it does not
directly exploit another source of information: the fact that there
are correlations between spatially adjacent voxels, in that they
tend to have the same label - e.g., if the voxel at (10, 21, 37) is
a tumour, then we would anticipate that the voxel at (10, 21, 38)
is tumour as well. (Actually the large-resolution features, based
on a region around the current voxel, begin to address this, as
neighbouring voxels will often have many very similar feature
values. ASP also includes a post-processing step to reclassify
small regions that are initially labeled as tumours, as normal. But
these are indirect approaches.)
The fact that in spatial domains, the feature values of
neighbouring locations are not independent has led many researchers
to explore 'Markov Random Fields' and 'Conditional Random Fields'
(Lafferty et al., 2001), (Kumar and Hebert, 2003) for
classification using spatial data. These methods attempt to
correlate the labels of neighbouring locations. We have begun to
experiment with an extension of these ideas, by implementing a
random field system based on Support Vector Machines, SVRF. This
has produced very encouraging results, improving on our prior
results; see (Lee et al., 2005).
Proposed Future Work
Better Algorithms:
We plan to continue exploring our SVRF approach, and consider
other related 'random field' technologies. One issue, with all of
these approaches, is efficiency: these systems require a great deal
of time to run (as well as considerable time to train). We are now
investigating more efficient versions of these algorithms.
Learning Classifiers for Anatomical Brain StructuresAs noted
above, as our system is able to train its classifier, we can use
this approach to produce other classifiers; we need only provide
images that have been (hand)labeled. We can therefore train a
classifier to identify eyes, the brain stem or other anatomical
features, and then use the resulting 'eye-classifier' to
automatically identify the eye region in new images.
Integrating other type of data:1. The current system uses only
the axial slices. We plan to build a more accurate 3D model by also
using coronal and sagittal slices from the MRI data.
2. We have access to other information, about the patient, the
tumour history, patient records, histology, etc. We plan to find
ways to incorporate this information into ASP, to improve its
segmentation accuracy. Fortunately, we anticipate that this should
be fairly straightforward, as we are already using a probabilistic
representation that can easily accommodate these various types of
information - ranging from general information about the patient
and tumour, to specific information about the specific voxels.
3. While we have access to a number of MR images of brains with
tumours, there are a much larger number of MR images of tumour-free
brains (LONI). We have begun to use this information, to help our
system better identify features and high-level textures that
distinguish tumour from non-tumour.
4. We also plan to use the other modalities, including Magnetic
Resonance Spectroscopy (MRS), Positron Emission Tomography (PET),
and Diffusion Tensor Imaging (DTI) -- viewing their values at each
voxel as a feature; this set of features will then be combined with
the other features (Figure 3) and used to classify each voxel.
Software:
Our software is currently not publicly available. For more
information on using the software, potential collaboration, and
sharing data please contact us. For information about possible
licensing opportunities
Oracle Segmentation Program
The Oracle Segmentation Program was developed to allow expert
segmentation of brain tumours. The program which functions as a
client server interface also allows us a secure and convenient way
to store and access our MRI data.Features include:
MRI data management based on patient-study-series-pathology
information
Visualization of different MRI modalities
Visualization of axial, coronal, and sagittal slices
Tumour and Edema overlays on all modalities
Viewing multiple modalities simultaneously
Management of expertly segmented volumes (number of slices
segmented/still to be segmented)
Creation of tumour and edema regions of interest
Image histogram visualization
Window/Level imaging controls
Ability to zoom, rotate, and flip images
Secure access to confidential material
Editing (stretch, move, cut, split) regions of interest
Create regions of interest using splines
Slideshows of slices
Automatic Segmentation Visualization Program
Figure 1.1 Screenshot of the OSP tool as a human expert segments
a tumour.
The Automatic Segmentation Visualization Program was created to
allow easy visualization of the results of our Automatic Brain
Tumour Segmentation pipeline. This program allows the visualization
of all the steps in our pipeline. It also provides a means to
organize the large repository of data for convenient
viewing.Features include:
Managment of large amounts of MRI data
Visualization of different MRI modalities
Visualize each individual step of the pipeline separately
View previous steps in addition to current step
Zoom images
Select modalities to visualize
Transparent overlays of edema or tumour labels
View image difference after each pipeline step
Overlay aligned templates
Jaccard measure calculation and confusion matrix calculation per
slice or per volume.
Automatic Brain Tumour Segmentation Pipeline:The process of
automatic segmentation involves several preprocessing steps to
regularize the data. These steps are necessary for the segmentor to
be able to identify abnormalities due to physiology rather than
noise in images or systematic MRI inconsistencies.
The Automatic Segmentation Program consists of the following
pipeline of steps that each dataset is put through:
Preprocessing:
Noise Reduction 2D Local Noise Reduction SUSAN Noise Reduction
Filter
Inter-Slice Intensity Variation Reduction Weighted Linear
Regression
Intensity Inhomogeneity Reduction Nonparametric Nonuniform
intensity Normalization
3D Local Noise Reduction SUSAN Noise Reduction Filter
Spatial Registration
Inter-Modality Coregistration Maximum Normalized Mutual
Information Rigid-Body Transformation
Linear Template Alignment Maximum a Posteriori 12-parameter
Affine Transformation
Non-Linear Template Warping Maximum a Posteriori Combination of
Basis Function Warps
Spatial Interpolation High Order Polynomial -Spline
Intensity Standardization
Template-Based Intensity Standardization Weighted Linear
Regression Segmentation
Feature Extraction
Image-Based Features Intensities, Textures, Normal Intensity
Distances
Coordinate-Based Features Spatial Tissue Probabilities, Brain
Area Probability, Expected Spatial Intensities
Registration-Based Features: Template Intensities, Bi-Lateral
Symmetry
Feature-Based Features Regional Characterizations of Image-,
Coordinate-, and Registration-Based Features
Classification
Binary Pixel Classification Support Vector Machine
Relaxation
Relaxation of Pixel Label Median Root Filter and Morphological
Hole Filling
Segmenting Brain Tumors using
Pseudo{Conditional Random Fields1 Introduction
Segmenting brain tumors in magnetic resonance (MR) images
involves classifying
each voxel as tumor or non-tumor [1{3]. This task, a
prerequisite for treating
brain cancer using radiation therapy, is typically done by hand
by expert medical
doctors, who nd this process laborious and time-consuming.
Replacing this
manual eort with a good automated classier would save doctors
time; the
resulting labels may also be more accurate, or at least more
consistent.
We treat this as a binary classication task, using a classier to
map each
MR image voxel described as a vector of values x 2