Large Scale Analysis of micro-vascular lesions in the brain and retina using MRI and colour fundus imaging for early detection of CVD Prof. Ramamohanarao.

Large Scale Analysis of micro-vascular lesions

in the brain and retina using MRI and colour

fundus imaging for early detection of CVD

Prof. Ramamohanarao Kotagiri

Department of Computing and Information Systems

The University of Melbourne, Australia 3010

Email: kotagiri@unimelb.edu.au

1

Cardiovascular Diseases

•CVD (Cardiovascular Disease) (heart disease & stroke) is the most common cause of death in the developed world• 36% all deaths in Australia

(2004)• Kills one Australian every

10 minutes.

Condition Australia of about 23

Million Population

Diabetes 1.7M(0.9M diagnosed)

Pre-diabetes 1.5M

Hypertension 3.7M

Coronary Heart Disease (CHD)

26,000 Deaths/Year

400,000 Deaths in the USA in

2010

Stroke 12,500 Deaths/Year

All CVD 3.7M Cases50,000

Deaths/Year

Present day Diagnostic Tools• Currently available prediction tools based on regular

assessment of traditional risk factors, e.g. – Blood pressure– Cholesterol– smoking history– MRI after the symptoms of a stroke/heat attack

• These account for only 50% of CVD cases and MRI in most case is too late and very expensive.

Need for improved early detectiondiagnostic tools

Retina

• Only place blood vessels can be viewed & monitored non-invasively

• Damage to the retinal circulation may reflect impact of both recognised & unrecognised risk factors, and susceptibility

• Indicator of structural vascular damage

Retinal Circulation = ‘Window’ to Brain Circulation

Retinal vascular signs can predict Cardiovascular Diseases and Diabetes!

Retinal Imaging

• A retinal camera is used to capture image• A picture is taken showing the optic nerve (i.e., disc), fovea,

surrounding vessels, and the retinal layer

A retinal camera (left) retinal fundus image (right)

Human Retina

New Diagnostic Tools

Traditional CVD Risk Factors

e.g. blood pressure

Unknown CVD Risk Factors

e.g. genetic factors

Sub clinical vascular damage

CARDIOVASCULAR DISEASE AND DEATH

Retinal Vascular Signs

Presentation Outline Motivation Brief Background

(Magnetic Resonance Imaging) MRI Retinal Imaging Brain microvascular lesions

White matter lesions (WMLs) Brain infarcts

Retinal microvascular lesions Arteriovenus nicking (AV nicking) Focal arteriolar narrowing (FAN)

Objective Proposed Method

Segmentation and quantification of WMLs Detection and quantification of AV nicking Detection and quantification of FAN

Summary

8

Retinal Vascular Sign

Observations Associations

Retinal vascular calibre

Arteriolar narrowing

2.6 fold increased risk of incident hypertension2 fold increased risk of incident CHD

Venular dilatation

2 fold increased risk of incident stroke3 fold increased risk of incident CHD

Retinal Signs Predict CVD

COMPUTER-BASED RETINAL IMAGING PROGRAM FOR

IDENTIFICATION OF CARDIOVASCULAR DISEASE RISK

Patient’s retina photographed

Photos transmitted to reading centre

Retinal grading

Retinal vascular scan report generated

Retinal Blood VesselAnalysisBlood Vessels• Artery (Red)• Vein (Blue)

Magnetic Resonance Imaging (MRI)

MRI is used for non invasively visualizing internal structures of the body .

Does not have any side effects of radiation .

Property of nuclear magnetic resonance (NMR) is used to image nuclei of atoms inside the body.

Provides higher details about soft-tissues

12

Magnetic Resonance Imaging (MRI)

Different modalities are used to characterize and discriminate among tissues

T1 weighted MRI is effective for visualizing various anatomical structures such as White matter, Gray matter and CSF very useful for identifying the location of tissues

T2 weighted MRI is effective for visualizing pathologies such as lesions and tumors

Fluid attenuated inversion recovery (FLAIR) is effective in suppressing CSF (Cerebral Spinal Fluid) and enhancing lesions.

13

T1 MRI

White Matter

WMLs

CSF(Cerebral Spinal Fluid)

Gray Matter

Anatomical Structures are clearly identifiable.

14

T2 MRI

WMLs

Pathology (WMLs) looks more clearer.

15

Flair MRI

WMLs

Intensity of CSF is suppressed

16

Relationship between WMLs and Risk of Stroke1

[1] Vermeer, Sarah E., et al. "Silent brain infarcts and white matter lesions increase stroke risk in the general population The Rotterdam Scan Study." Stroke 34.5 (2003): 1126-1129.

Motivation

17

Correlation between severity of Sub-Cortical WMLs and AV nicking2

Sub-cortical WMLs Load• Q# is the severity scale of sub-cortical WMLs. • # Reference group; * p<0.05; § p<0.01.[2] Qiu, Chengxuan, et al. "Retinal and cerebral microvascular signs and diabetes the age, gene/environment susceptibility-reykjavi

study." Diabetes 57.6 (2008): 1645-1650.

Motivation

18

Correlation between severity of Periventricular WMLs and AV nicking2

Periventricular WMLs Load • T# is the severity scale of peri-ventricular WMLs. • # Reference group; * p<0.05; § p<0.01.[2] Qiu, Chengxuan, et al. "Retinal and cerebral microvascular signs and diabetes the age, gene/environment susceptibility-reykjavi

study." Diabetes 57.6 (2008): 1645-1650.

Motivation

19

Correlation between severity of Sub-Cortical WMLs WMLs and FAN2

Sub-cortical WMLs Load

• Q# is the severity scale of sub-cortical WMLs. • # Reference group; * p<0.05; § p<0.01. [2 ] Qiu, Chengxuan, et al. "Retinal and cerebral microvascular signs and diabetes the age,

gene/environment susceptibility-reykjavi study." Diabetes 57.6 (2008): 1645-1650.

Motivation

20

Correlation between severity of Sub-Cortical WMLs WMLs and FAN2

• T# is the severity scale peri-ventricular WMLs. • # Reference group; * p<0.05; § p<0.01. [2 ]Qiu, Chengxuan, et al. "Retinal and cerebral microvascular signs and diabetes the age,

gene/environment susceptibility-reykjavi study." Diabetes 57.6 (2008): 1645-1650.

Motivation

21

Periventricular WMLs Load

MotivationRetinal microvascular lesions (AV nicking and FAN) can be an important bio-marker to predict the severity of WMLs load.

Current approach of quantification and correlation analysis is Manual

Limitations Highly subjective Expensive Time consuming High intra and inter-grader variability

22

Objective

Automatic segmentation and quantification of

WMLs

Automatic detection and quantification of AV

nicking

Automatic detection and quantification of FAN

Quantify the correlation between retinal and brain

microvascular lesions

Develop a retinal image based brain

microvascular lesions prediction model

23

Automated WMLs segmentation method

T1 MRIFlair MRI

Pre-processing

Feature extraction

Classification of WMLs

Post-processing using MRF

24

Automated WMLs segmentation method

Pre-processing for noise reduction and spatial normalization Co-registration of T1 and Flair MRI using Statistical

parametric mapping (SPM8)3

Brain skull removal using Brain Extraction Tool (BET) 4

Intensity normalization using dynamic maximum boundary5

Feature extraction Multimodal MRI (T1 and Flair) intensity Tissue probability mask (PWM, PGM, PCSF) Normalized spatial coordinate (X, Y, Z) Global neighbourhood based contrast

[3] J. Ashburner and K. J. Friston, “Unified segmentation,” Neuroimage, vol. 26, no. 3, pp. 839–851, 2005.[4] V. Popescu, M. Battaglini, W. Hoogstrate, S. Verfaillie, I. Sluimer, R. Van Schijndel, B. van Dijk, K. Cover, D. Knol, M. Jenkinson et al., “Optimizing parameter choice for fsl-brain extraction tool (bet) on 3d t1 images in multiple sclerosis,” Neuroimage, vol. 61, no. 4, pp. 1484– 1494, 2012.[5] Liang, Xi, Kotagiri Ramamohanarao et al. "Nat. ICT Australia (NICTA), Eveleigh, SA, Australia." Digital Image Computing Techniques and Applications (DICTA), 2012 International Conference on. IEEE, 2012.

25

Proposed automated WMLs segmentation method

Multimodal MRI (T1 and Flair) intensity

T1 MRI Flair MRI 26

Automated WMLs segmentation method Tissue probability mask

Multiple atlas construction from healthy T1 MRI using FAST6 Non-linear registration of atlases with input T1 MRI Tissue mask construction based on multi atlas voting

PWMPGM PCSF

[6] Y. Zhang, M. Brady, and S. Smith, “Segmentation of brain mr images through a hidden markov random field model and the expectationmaximization algorithm,” Medical Imaging, IEEE Transactions on, vol. 20, no. 1, pp. 45–57, 2001.

27

Automated WMLs segmentation method• Normalized spatial coordinate

• Subject patient’s T1 MRI is linearly registered with Montreal Neurological Institute (MNI ) space

• The voxel wise corresponding Montreal Neurological Institute (MNI) space is warped back to the experimental subject space.

• This results in normalized X, Y and Z space comparable between patients.

X Y Z 28

Automated WMLs segmentation method• Global neighbourhood based

contrast • Create global neighbours

for each pixel• Mask neighbours of

candidate pixel using PWM.• A box filter of size m is

applied on candidate and its neighbour pixels

• Global neighbourhood based contrast (GNC) is computed using

m

mk

m

mlljkiji x

mmX ,, *

1

N

n

n

X

XX

Nc

1

1

Here, X represent candidate pixel and Xn represents its neighbours.

29

Automated WMLs segmentation method

Classification of Lesion using Random Forest

30

Automated WMLs segmentation methodPost processing using Markov random field (MRF)

)1..().........,,,(),(

1maxargˆ jj

i iNjiiii

yxyxyxy

Zy

represents lesion probability computed by RF classifier represents edge potential computed by Eq. 2. xi and yi represent ith pixel intensity and RF given class label. is the penalty constant. Loopy belief propagation (LBP) 6 is used to infer the outcome

of MRF

(.)(.)

Neighbourhood structure

[6] K. P. Murphy, Y. Weiss, and M. I. Jordan, “Loopy belief propagation for approximate inference: An empirical study,” in Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc., 1999, pp. 467–475. 31

Automated WMLs segmentation method

Segmentation outcome after applying MRF

Initial Segmentation MRF given Segmentation Ground Truth Segmentation

32

Automated WMLs segmentation method

Classification Parameters Classifier : Random Forest Number of trees in RF : 200

Data Number of Subject : 24 Source of Data: ENVISion study7 Size : 256 × 256 × 36 Resolution : 0.94 × 0.94 × 4 mm3

Training and Testing procedures 4 Fold cross-validation (4 times)

2 fold for training 1 fold for parameter selection 1 Fold for testing

• [7] C. M. Reid, E. Storey, T. Y.Wong, R. Woods, A. Tonkin, J. J. Wang, A. Kam, A. Janke, R. Essex,W. P. Abhayaratna et al., “Aspirin for the prevention of cognitive decline in the elderly: rationale and design of a neuro-vascular imaging study (envis-ion),” BMC neurology, vol. 12, no. 1, p. 3, 2012. 33

Training of Random Forest Classifier

• Number of subjects : • Resolution of each Subject• Number of data points (voxels) after skull stripping

• Number of features for each data point: 26• The number of training sample is very large

= 810,00*12 = 9.7 million voxels• To reduce the training time we randomly sample

20,000 voxels to build a random forest tree and build about 200 random forest trees. This method provides adequate accuracy!

36256256 24

Features Description

Local Features T1 , FLAIR intensityTissue probability map of White matter, Gray matter and Cerebral spinal fluid Normalized spatial coordinates

Global neighbourhood based contrast feature (GNCF)

Intensity of the lesion is enhanced and normalized based on the contrast of global neighbourhood points

Histogram based features MeanVarianceSkewnessKurtosis

Gray level co-occurrence matrix (GLCM) [1]

ContrastCorrelation EnergyHomogeneity

Run length matrix (RLM) [2] Short run emphasis inverse momentsLong run emphasis momentsGray level non uniformity Run length non uniformity

Isotropic undecimated wavelet transform (IUWT) [3]

Wavelet coefficients for five different scales

[2] M. M. Galloway, “Texture analysis using gray level run lengths,” Computer graphics and image processing, vol. 4, no. 2, pp. 172–179, 1975 .[3] P. Bankhead, C. N. Scholfield, J. G. McGeown, and T. M. Curtis, “Fast retinal vessel detection and measurement using wavelets and edge location refinement,” PloS one, vol. 7, no. 3, p. e32435, 2012.

[1] A. Baraldi and F. Parmiggiani, “An investigation of the textural characteristics associated with gray level cooccurrence matrix statistical parameters,” Geoscience and Remote Sensing, IEEE Transactions on, vol. 33, no. 2, pp. 293–304, 1995.

Automated WMLs segmentation method

Evaluation metrics Sensitivity (SEN)

Positive predictive value (PPV)

Dice Similarity Index (SI)

FNTP

TPSEN

FPTP

TPPPV

FNFPTP

TPSI

2

2

36

Automated WMLs segmentation method

WMLs Load Category

High Lesion Load (HLL [>10ml])

Medium Lesion Load (MLL [between 6 ml to 10 ml])

Low Lesion Load (LLL [between 1 ml to 5 ml ])

37

Proposed automated WMLs segmentation method

Results

38

Proposed automated WMLs segmentation method

Comparison with state of the art methods

For both methods p-value< 0.05.

[8] M. D. Steenwijk, P. J. Pouwels, M. Daams, J. W. van Dalen, M. W. Caan, E. Richard, F. Barkhof, and H. Vrenken, “Accurate white matter lesion segmentation by k nearest neighbor classification with tissue type priors (knn-ttps),” NeuroImage: Clinical, vol. 3, pp. 462–469, 2013.[9] P. Schmidt, C. Gaser, M. Arsic, D. Buck, A. F¨orschler, A. Berthele, M. Hoshi, R. Ilg, V. J. Schmid, C. Zimmer et al., “An automated tool for detection of flair-hyperintense white-matter lesions in multiple sclerosis,” Neuroimage, vol. 59, no. 4, pp. 3774–3783, 2012.

8

9

39

Automatic quantification of AVN and FAN

Retinal Image Pre-processing Feature extraction

Quantification of AVN and FAN

40

Automatic quantification of AVN and FAN

Pre-processing

Vessel Segmentation

Vessel Cross-over point detection

Candidate vessel region selection

Vessel width measurement

41

First to Analyze the Vessel: Segmentation

Although many methods have been proposed, significantimprovement is still a necessity due to the limitations in state of the-

art methods, which include:

Poor segmentation in the presence of central reflex (i.e., bright strip along the centre of a vessel).

Poor segmentation at bifurcation and crossover points.

The merging of close vessels.

The missing of small vessels.

False vessel detection at the optic disk and pathological regions.

42

Some Limitations of Existing Methods(a) a portion of a retinal

image showing the

presence of central reflex

vessels (white solid

arrows), close vessels

(white dashed arrows),

artery-vein crossing

regions (black solid

arrows), and small vessels

(black dashed arrows) and

segmentations obtained

by

(b) Staal et al. method;

(c) Soares et al. method;

(d) Ricci-line method ;

(e) Ricci-svm method

(f) Our proposed

method.

Uyen et. all (2013), Pattern Recognition

43

Our Proposed Method

A linear combination of line detectors at different scales to produce the vessel segmentation for each retinal image.

A basic line detector uses a set of approximated rotated straight lines to detect the vessels at different angles.

The difference between the average gray level of the winning line (the line with maximum average gray level) and the average gray level of the surrounding window provides a measure of ‘vesselness’ of each image pixel.

44

Vessel Segmentation Accuracy

Segmentation results on some selective regions showing theimprovements of the proposed method over existing methods in terms of segmentation quality: (a) original image; segmentations of (b) Staal et al. method; (c) Soares et al. method; (d) Ricci-line method; and (e) proposed method. 45

Vessel Segmentation Accuracy

46

Vessel Segmentation Accuracy (Cont.)

47

Vessel Crossover Point Detection

- Artery-vein crossover points in a retinal image are the locations where retinal artery and vein cross each other- Necessary for Individual Vessel segment Identification

Crossover points (left) often complicated to detect due to intensity variation (right)

48

Our Proposed Crossover Point Detection

Framework for Crossover Point Detection49

Crossover Point Detection

Analyzing branch and crossover points to detect crossover points

50

Crossover Point Detection

A retinal image with all crossover points detected by the proposed method marked as white spots.

51

Accuracy on Crossover Point Detection

52

Vessel Width Measurement

To Find mean Arterial and Venular Width in the Image and Find Association with CVD or Hypertension

Analyze the Vessel Width with Tortuosity or other Features

53

Vessel Width Measurement

Method for identifying vessel edge points representing vessel width at a centre point.

54

Width Measurement Accuracy

55

Feature for AVN severity grading

MW

CrMWF nn

Multi-scale measurement of narrowing Fn near the cross over point is used to classify AVN.

where,

c is the middle point of the vessel centreline and Wi represent the vein widths.

In this study, we have used 3 scales for Fn with n = 10,20,30.

n

iin W

nCr

1

1

56

Accuracy AVN severity classification

Severity Level Prediction Accuracy

Proposed Method Uyen et al.

0 78.43 % 70.58 %

1 29.41 % 17.64 %

2 42.85 % 14.28 %

3 81.81 % 45.45 %

Uyen T. V. Nguyen, Alauddin Bhuiyan, Laurence A.F. Park, Ryo Kawasaki, Tien Y Wong, Jie Jin Wang, Paul Mitchell and Kotagiri Ramamohanarao, An Automated Method for Retinal Arteriovenous Nicking Quantification From Color Fundus Images, IEEE Transactions on Biomedical Engineering, vol. 60, number.11, pp. 3194–3203, 2013.

57

Feature for FAN severity grading

Lmin and Lmax represents the set of local minima and local maxima for the vessel widths D ,

For each , the closest right and left local maxima and is computed using

is used as a feature to classify FAN

11minmin :LLi iiii DDDD

1maxLi

11minmax :LLi iiii DDDD

iminL

1maxLi

2

)()(V

minmaxminmaxi

11 iiiiLLLL

)max(Vi

58

Accuracy FAN severity classification

Severity Level Prediction Accuracy

0 87%

1 78%

59

A novel WMLs segmentation method. A novel AV nicking quantification method A novel FAN quantification method In future: Proposed method will be applied on a large dataset

to quantify the correlation between brain and retinal micro-vascular lesions.

Quantified correlation will be used to develop a retinal image based brain microvascular lesion severity prediction model.

Summary

60

Thank you!

61