BIRS 2016: Opening the analysis black box: Improving robustness and interpretation Matthew Brown, PhD University of Alberta, Canada.

BIRS 2016:Opening the analysis black box:

Improving robustness and interpretation

Matthew Brown, PhDUniversity of Alberta, Canada

Overview

1. About us2. Preprocessing quality assurance3. Interpretation of group vs. individual

differences4. Trial type fMRI signatures

Dept. PsychiatryDept. Computing Science

ComputationalPsychiatryGroup

• Diagnosis– What disease?

• Prognosis– Predict patient response to treatment options

Clinical decision-making

What are we detecting?• 10 psychosis patients, 10 controls, fMRI• Highly diagnostic Fourier power distribution

from voxels IN THE EYES• Eye movement disturbances in psychosis

ADHD-200 and ABIDE datasets• n=1000 approx.• ADHD patients or autism patients• Structural MRI, resting state fMRI• Simple diagnosis

– Classify patients vs. controls– Accuracy 50-70% in various papers

• Some papers reported higher 75%+ accuracyBUT cherry-picking sites?

ADHD-200 Global Competition• Best-performing algorithm, but did not win• Used only non-imaging features:

– Age, gender, handedness, IQ, site of scan– 3-class classification (ADHD-c, ADHD-i, control)– 63% hold-out accuracy (vs. 54% chance)

Using non-imaging features

Brown et al. 2012

Chance accuracyValid

ation

Accu

racy

(%)

Histogram of oriented gradient (HOG) featuresImage from Ghiassianet al. under review.

Also see Dalal and Triggs 2005. IEEE Computer Society Conference on. vol. 1. IEEE, p.886–893.

ADHD-200 and ABIDE datasets• Ghiassian et al. under review• State of the art (as of 1.5 years ago)• 2-class classification (patients vs. controls)

ADHD-200 ABIDEChance 55% 51%Non-imaging 69% 60%Non-imaging + Structural MRI

70% 64%

Non-imaging + Functional MRI

64% 65%

Overview

1. About us2. Preprocessing quality assurance3. Interpretation of group vs. individual

differences4. Trial type fMRI signatures

Registration failure Subject 1 Subject 15

Fixed ->

Standard preprocessing methods failed for 1 of 21 subjects.

Inter-site variability

Sen et al. in preparationPCA Component 1

PCA

Com

pone

nt 2

ADHD-200 Subjects Projected onto PCA component space

Each colour is a different scanning site.

Even with standard normalization procedures, inter-site structure remains in the data.

Overview

1. About us2. Preprocessing quality assurance3. Interpretation of group vs. individual

differences4. Trial type fMRI signatures

Clinical research

Huntington’sImage from Wikipedia

Healthy

One goal: Associate disease with biological features

ADHD-200 resting state fMRI functional connectivity analysis

ICA

Brown et al. 2012

ADHD patients vs. controls

“Default mode” network Patients vs. controls

Brown et al. 2012

“Desired” simple interpretation: “Patients are different from controls. This difference tells us something about the disease.”

Group vs. individual differences

PatientsControls

Statistically significant group differences, but substantial overlap between individual patients and controls.

Brown et al. 2012

Interpretation

• Simple interpretation “patients are different from controls”

• Overlap precludes simple interpretation• Yet many papers provide precisely and only

the simple interpretation

PatientsControls

Brown et al. 2012

Overview

1. About us2. Preprocessing quality assurance3. Interpretation of group vs. individual

differences4. Trial type fMRI signatures

Black box analysis

AnalysisSoftware

General linear model regression

Model voxel i’s timecourse

Model matrix for trial type k

Two different models for hemodynamic response function

SPM canonicalmodel

Finite impulseresponse model

Check deconvolved timecourses

Basically agree on shape (but not statistical differences in this case)

SPM canonical model

Finite impulse responsemodel, same region

Check deconvolved timecoursesSPM canonical model

Finite impulse responsemodel, same region

Noise in deconvolvedtimecourses

Another exampleSPM canonical model

Finite impulse responsemodel, same region

Noise in deconvolvedtimecourses

GLM analysis

• Check deconvolved timecourses• What is the model fitting

– Noise vs. signal• Model selection

– regularization

Summary

Quality check everythingVisualization

Intermediate steps and final resultsParticularly important for non-technicalend-users

Acknowledgements

People: Azad, Benoit, Dursun, Ghiassian, Greenshaw, Greiner, Juhas, Purdon, Ramasubbu, Rish, Sen, Silverstone

Funding: AICML, AIHS, CIHR, Norlien Foundation, AHS, AMHB, UAlberta

Questions?

Invitation

Continue informing other researchers about analysis pitfalls and caveats.

Questions?

Title