Top Banner
Automatic Brain Tumor segmentation Problem: The task in this problem is to automatically detect the presence of tumors in MR images of the brain, and segment the abnormal pixels from the normal pixels. Traditionally, the task has tried to segment the metabolically active 'enhancing' area of the tumor, which appears hyper-intense in T1 weighted images after the injection of gadolinium. Several recent methods have focused on additionally segmenting non-enhancing regions, as well as tumors that may only partially enhance or do not enhance at all. Several recent methods have also focused on the related task of segmenting edema (swelling) associated with tumors. Segmentation of completely enhancing or 'border enhancing' tumors is a relatively easy problem, while more work is still required for the task of segmenting tumors that do not have these characteristics. This is an interesting problem, since it is a task that humans can learn to do very well, while developing algorithms to do the same task has proven challenging. Some of the challenges associated with this task include: Local Noise Inter-Slice Intensity Variations Partial Volume Artifacts
140
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript

Automatic Brain Tumor segmentation

Automatic Brain Tumor segmentationProblem:

The task in this problem is to automatically detect the presence of tumors in MR images of the brain, and segment the abnormal pixels from the normal pixels. Traditionally, the task has tried to segment the metabolically active 'enhancing' area of the tumor, which appears hyper-intense in T1 weighted images after the injection of gadolinium. Several recent methods have focused on additionally segmenting non-enhancing regions, as well as tumors that may only partially enhance or do not enhance at all. Several recent methods have also focused on the related task of segmenting edema (swelling) associated with tumors.

Segmentation of completely enhancing or 'border enhancing' tumors is a relatively easy problem, while more work is still required for the task of segmenting tumors that do not have these characteristics. This is an interesting problem, since it is a task that humans can learn to do very well, while developing algorithms to do the same task has proven challenging. Some of the challenges associated with this task include:

Local Noise

Inter-Slice Intensity Variations

Partial Volume Artifacts

Intra-Volume Intensity Inhomogeneity

Inter-Volume Intensity Inhomogeneity

Integration of multi-spectral (potentially unaligned) data

Intensity Overlap between normal and abnormal areas

Tumor heterogeneity

Changes in normal structures observed due to the presence of tumorsAutomated Tumour Segmentation, using Machine Learning

Most of our efforts to date have focused on locating tumour volumes within a patient's brain, based on a set of MR images -- a task known as 'tumour segmentation'. Our automated segmentation program (ASP) takes as input a 3 * k grid of images, where each image is an MR image of a patient's brain; see first 3 columns of Figure 1. (This shows only k=5 of the typically around k=21 slices taken.) Note all images in each row corresponds to the same axial slice of the patient, at some height, and the first 3 columns are (respectively) images that are weighted T1, T1c, (T1 weighted after injection of the gadolinium contrast agent), and T2. The ASP output is a 3D volume corresponding to its assessment of the Gross Tumour Volume (GTV), encoded as a sequence of k images; see the 4th column of Figure 1. Our goal is to produce an accurate volume, automatically -- i.e., without any human assistance

Figure 1: Each row corresponds to the same axial slice through a patient's brain. The first three columns are (resp.) T1, T1c, T2 weighted images, and the 4th is the region that ASP labels as GTV.

ASP uses a classifier to label each voxel in the complete volume as either GTV or not, based on properties associated with that voxel. The obvious properties would be simply the 3 intensity values (T1, T1c and T2). Unfortunately, this is not sufficient, as there are 'intensity triples' that are tumour in one part of the brain, but normal in another (Schmidt, 2005). We therefore need to find other properties for each voxel, besides the intensities, and then to find the most effective combination of these features.

Feature Sets

Other researchers have produced brain atlases (templates), that specify voxel-level properties of a typical brain - e.g., providing the probability of grey matter at location (18 down, 200 left, 52 back), as well as every other coordinate (Schmidt, 2005). This can be extremely useful, as it can help specify what T1/T1c/T2 intensities to expect at each location; large deviations from the expectation can help identify possible tumours. Of course, this assumes we can relate the locations in the patient image to locations in the template. Unfortunately, these locations will typically be different - e.g., position (18, 200, 52) in the template might correspond to, say, (19, 207, 55) in the patient - as the patient's head may not be the 'typical' size and shape, and moreover it may have been tilted when the image was captured. A linear transformation can adjust for such simple scaling and rotation; a more serious problem arises when large tumours distort the anatomy within the brain itself - e.g., by shrinking the ventricles, or pushing them to new locations; see Figure 2. We solved this by following the linear registration with a highly-regularized non-linear morphing (Schmidt, 2005).

For each voxel, we augmented the 3 intensity values (T1, T1c and T2) with 15 other values, 12 based on various templates, and 3 based on an alignment-based measure, 'symmetry', which is the difference of intensity between current position and its reflection about the mid-saggital plane. (Note this requires us to identify this mid-saggital plane, which is 'x=0' in the aligned image but need not be 'x=0' in the initial images.) We then considered 4 different 'resolutions', to take into account some texture effects (see below), for a total of (3+12+3) 4 = 72 features; see Figure 3.

Figure 3: Each box shows one of the 72 features used by ASP system. That is, the label for the voxel at each location depends on 72 feature values, one from the corresponding location in these boxes.It is also important for the intensities to 'mean the same thing' throughout the volume, and also across patients (see below). However, due to various artifacts (as well as differences in magnet strength, etc), a T1 intensity of 122 might correspond to 'normal white matter' at one location of a patient but to 'tumour' at another. We addressed this by a combination of noise reduction and intensity standardization steps, based on MIPAV. Figure 4 shows the results of these processes.

At a high level, this requires

Preprocessing

Intensity Normalization to make intensities more consistent within and between volumes

Spatial Normalization to allow comparisons between spatial locations in different modalities and with locations in the template/coordinates

Segmentation

Feature extraction to represent pixel-level measurements related to the image, coordinate system, and template; see 72-features.

Supervised Pixel Classification to Assign the pixels a class (tumour or normal) by combining the feature values

Label Relaxation to correct misclassifications using spatial label dependencies

Figure 4: Results of pre-processing steps. Top row: original images (T1, T1c, T2). Middle row: after Noise Reduction. Bottom row: after intensity standardization.

Best Combination of Features -- Learning a Classifier

Using the ideas discussed above, we can assign a set of features to each voxel in the volume. The real challenge, however, is determining which combination of these features corresponds to GTV -- e.g., perhaps a voxel is part of a GTV if

(T1c intensity)>0.5 and (diff from symmetry) > 0.2 and (prob of grey-matter) < 0.3

or alternatively, maybe it is a GTV if

0.3 * (T1c intensity) + -0.1 * (diff from symmetry) + 13 * (prob of grey-matter ) - 6 is positive

or some other variant. If we, or any of our colleagues, knew which combination was appropriate, we could just implement this function directly. Unfortunately, no one knows this best function. However, we do have access to many images of other patients, which experts have labeled -- i.e., explicitly identified which voxels are GTV and which are normal. The field of Machine Learning has provided a number of algorithms that can use these labeled images to learn a classifier, which can then assign a label to a new (unlabeled) voxel of a novel patient (Mitchell, 1997). In essence, these learners find the patterns in the data that identify when a voxel belongs to the class of GTV voxels, versus the class of normal voxels. The current version of ASP used a standard algorithm for learning a 'Support Vector Machine' (Cristianini and Shawe-Taylor, 2000); we found this worked effectively and is an accurate classifier (Zhang et al., 2004; Zijdenbos et al., 2002; Garcia and Morena, 2004. For 'inter-patient training' -- when the learner is trained on one set of patients, but then tested on a different patient -- we obtained an average Jaccard score of 0.732 over 11 patients, with a variety of tumours (grade 2 astrocytoma, anaplastic astrocytoma, glioblastoma multiforme, oligodendroglioma) at different stages of treatment, and based on two 1.5 T scanners; see (Schmidt et al., 2005).

We see that this 'learning approach' is important - indeed, essential! - in this situation, where the 'correct' answer is not known. It also adds flexibility to the system, as it means we can produce a new segmentation system: the classifier learned depends on the labels of the training data. For the GTV-classifier, the expert labeled voxel as '1' if it is a GTV voxel, and '0' otherwise. If the expert, instead, labeled each training voxel with '1' if it was 'Tumour+Edema' area (and '0' otherwise), then the learner will produce a 'Tumour+Edema-classifier'. Our inter-patient score on this task was 0.77. See Figure 5.

Figure 5: The bottom row shows results of three different classifiers: gross tumour volume, enhancing region, and tumour+edema. Each classifier takes as input all 3 images (T1, T1c and T2) shown in the top row, as well as information based on various templates; see Figure 3Notice this methodology also allows ASP to accommodate different magnet strengths or other contexts. We would just need to train on images produced by this alternative MRI, and then labeled by an expert.

Our initial ASP system used only the features associated with a voxel to assign a label to that voxel. This means it does not directly exploit another source of information: the fact that there are correlations between spatially adjacent voxels, in that they tend to have the same label - e.g., if the voxel at (10, 21, 37) is a tumour, then we would anticipate that the voxel at (10, 21, 38) is tumour as well. (Actually the large-resolution features, based on a region around the current voxel, begin to address this, as neighbouring voxels will often have many very similar feature values. ASP also includes a post-processing step to reclassify small regions that are initially labeled as tumours, as normal. But these are indirect approaches.)

The fact that in spatial domains, the feature values of neighbouring locations are not independent has led many researchers to explore 'Markov Random Fields' and 'Conditional Random Fields' (Lafferty et al., 2001), (Kumar and Hebert, 2003) for classification using spatial data. These methods attempt to correlate the labels of neighbouring locations. We have begun to experiment with an extension of these ideas, by implementing a random field system based on Support Vector Machines, SVRF. This has produced very encouraging results, improving on our prior results; see (Lee et al., 2005).

Proposed Future Work

Better Algorithms:

We plan to continue exploring our SVRF approach, and consider other related 'random field' technologies. One issue, with all of these approaches, is efficiency: these systems require a great deal of time to run (as well as considerable time to train). We are now investigating more efficient versions of these algorithms.

Learning Classifiers for Anatomical Brain StructuresAs noted above, as our system is able to train its classifier, we can use this approach to produce other classifiers; we need only provide images that have been (hand)labeled. We can therefore train a classifier to identify eyes, the brain stem or other anatomical features, and then use the resulting 'eye-classifier' to automatically identify the eye region in new images.

Integrating other type of data:1. The current system uses only the axial slices. We plan to build a more accurate 3D model by also using coronal and sagittal slices from the MRI data.

2. We have access to other information, about the patient, the tumour history, patient records, histology, etc. We plan to find ways to incorporate this information into ASP, to improve its segmentation accuracy. Fortunately, we anticipate that this should be fairly straightforward, as we are already using a probabilistic representation that can easily accommodate these various types of information - ranging from general information about the patient and tumour, to specific information about the specific voxels.

3. While we have access to a number of MR images of brains with tumours, there are a much larger number of MR images of tumour-free brains (LONI). We have begun to use this information, to help our system better identify features and high-level textures that distinguish tumour from non-tumour.

4. We also plan to use the other modalities, including Magnetic Resonance Spectroscopy (MRS), Positron Emission Tomography (PET), and Diffusion Tensor Imaging (DTI) -- viewing their values at each voxel as a feature; this set of features will then be combined with the other features (Figure 3) and used to classify each voxel.

Software:

Our software is currently not publicly available. For more information on using the software, potential collaboration, and sharing data please contact us. For information about possible licensing opportunities

Oracle Segmentation Program

The Oracle Segmentation Program was developed to allow expert segmentation of brain tumours. The program which functions as a client server interface also allows us a secure and convenient way to store and access our MRI data.Features include:

MRI data management based on patient-study-series-pathology information

Visualization of different MRI modalities

Visualization of axial, coronal, and sagittal slices

Tumour and Edema overlays on all modalities

Viewing multiple modalities simultaneously

Management of expertly segmented volumes (number of slices segmented/still to be segmented)

Creation of tumour and edema regions of interest

Image histogram visualization

Window/Level imaging controls

Ability to zoom, rotate, and flip images

Secure access to confidential material

Editing (stretch, move, cut, split) regions of interest

Create regions of interest using splines

Slideshows of slices

Automatic Segmentation Visualization Program

Figure 1.1 Screenshot of the OSP tool as a human expert segments a tumour.

The Automatic Segmentation Visualization Program was created to allow easy visualization of the results of our Automatic Brain Tumour Segmentation pipeline. This program allows the visualization of all the steps in our pipeline. It also provides a means to organize the large repository of data for convenient viewing.Features include:

Managment of large amounts of MRI data

Visualization of different MRI modalities

Visualize each individual step of the pipeline separately

View previous steps in addition to current step

Zoom images

Select modalities to visualize

Transparent overlays of edema or tumour labels

View image difference after each pipeline step

Overlay aligned templates

Jaccard measure calculation and confusion matrix calculation per slice or per volume.

Automatic Brain Tumour Segmentation Pipeline:The process of automatic segmentation involves several preprocessing steps to regularize the data. These steps are necessary for the segmentor to be able to identify abnormalities due to physiology rather than noise in images or systematic MRI inconsistencies.

The Automatic Segmentation Program consists of the following pipeline of steps that each dataset is put through:

Preprocessing:

Noise Reduction 2D Local Noise Reduction SUSAN Noise Reduction Filter

Inter-Slice Intensity Variation Reduction Weighted Linear Regression

Intensity Inhomogeneity Reduction Nonparametric Nonuniform intensity Normalization

3D Local Noise Reduction SUSAN Noise Reduction Filter

Spatial Registration

Inter-Modality Coregistration Maximum Normalized Mutual Information Rigid-Body Transformation

Linear Template Alignment Maximum a Posteriori 12-parameter Affine Transformation

Non-Linear Template Warping Maximum a Posteriori Combination of Basis Function Warps

Spatial Interpolation High Order Polynomial -Spline

Intensity Standardization

Template-Based Intensity Standardization Weighted Linear Regression Segmentation

Feature Extraction

Image-Based Features Intensities, Textures, Normal Intensity Distances

Coordinate-Based Features Spatial Tissue Probabilities, Brain Area Probability, Expected Spatial Intensities

Registration-Based Features: Template Intensities, Bi-Lateral Symmetry

Feature-Based Features Regional Characterizations of Image-, Coordinate-, and Registration-Based Features

Classification

Binary Pixel Classification Support Vector Machine

Relaxation

Relaxation of Pixel Label Median Root Filter and Morphological Hole Filling

Segmenting Brain Tumors using

Pseudo{Conditional Random Fields1 Introduction

Segmenting brain tumors in magnetic resonance (MR) images involves classifying

each voxel as tumor or non-tumor [1{3]. This task, a prerequisite for treating

brain cancer using radiation therapy, is typically done by hand by expert medical

doctors, who nd this process laborious and time-consuming. Replacing this

manual eort with a good automated classier would save doctors time; the

resulting labels may also be more accurate, or at least more consistent.

We treat this as a binary classication task, using a classier to map each

MR image voxel described as a vector of values x 2