This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
The Statistician and the ScientistThe Statistician and the Scientist
When asked, the statistician says that he’d like to When asked, the statistician says that he’d like to give one give one last lecture on his theory of statistics. lecture on his theory of statistics.
When the scientist is asked, he says, “I’d like to be When the scientist is asked, he says, “I’d like to be shot first!”shot first!”
Rob Tibshirani @ S-Plus Users conference Oct., 1999. Rob Tibshirani @ S-Plus Users conference Oct., 1999.
The Functional Image Analysis Competition 1 The Functional Image Analysis Competition 1
Examine the perisylvian Examine the perisylvian language network using a language network using a repetition-priming design with repetition-priming design with spoken sentencesspoken sentences
3T whole-body Bruker3T whole-body Bruker
T2-weighted EPI, TR=2.5s, 30 T2-weighted EPI, TR=2.5s, 30 x 4 mm slicesx 4 mm slices
Epochs of 20s ON & 9s OFFEpochs of 20s ON & 9s OFF
2 x 2 design for 4 conditions2 x 2 design for 4 conditions
4 epochs/condition x 2 runs4 epochs/condition x 2 runs
Same Same sentence, sentence, same same speakerspeaker
Same sentence, Same sentence, different speakerdifferent speaker
Different Different sentence, sentence, same same speakerspeaker
Different sentence, Different sentence, different speakerdifferent speaker
Poline JB, Strother SC, Dehaene-Lambertz G, Egan GF, Lancaster JL. Poline JB, Strother SC, Dehaene-Lambertz G, Egan GF, Lancaster JL. Motivation and synthesis of Motivation and synthesis of the FIAC experiment: The reproducibility of fMRI results across expert analyses.the FIAC experiment: The reproducibility of fMRI results across expert analyses. (in press, special (in press, special issue Hum Brain Mapp)issue Hum Brain Mapp)
Abstract:Abstract: “… the FIAC … helped identify new activation “… the FIAC … helped identify new activation regions in the test-base data, and …, regions in the test-base data, and …, it illustrates the it illustrates the significant methods-driven variability that potentially exists significant methods-driven variability that potentially exists in the literature.in the literature. Variable results from different methods Variable results from different methods reported here should provide a cautionary note, and reported here should provide a cautionary note, and motivate the Human Brain Mapping community to explore motivate the Human Brain Mapping community to explore more thoroughly the methodologies they use for analysing more thoroughly the methodologies they use for analysing fMRI data.”fMRI data.”
Poline JB, Strother SC, Dehaene-Lambertz G, Egan GF, Lancaster JL. Motivation and synthesis of the FIAC experiment: The reproducibility of fMRI results across expert analyses. (in press, special issue Hum Brain Mapp)
The main effects of sentence repetition (in red) and of speaker repetition (in blue). 1: Meriaux et al, Madic; 2: Goebel et al, Brain voyager; 3: Beckman et al, FSL; and 4: Dehaene-Lambertz et al, SPM2.
Simulation & ROC curvesSimulation & ROC curves1.1. Skudlarski P., et al., Skudlarski P., et al., NeuroimageNeuroimage. 9(3):311‑329, 1999.. 9(3):311‑329, 1999.2.2. Della-Maggiore V., et al., Della-Maggiore V., et al., NeuroimageNeuroimage 17:19–28, 2002. 17:19–28, 2002.3.3. Lukic AS., et al., Lukic AS., et al., IEEE Symp. Biomedical ImagingIEEE Symp. Biomedical Imaging, 2004., 2004.4.4. Beckmann CF & Smith SM. Beckmann CF & Smith SM. IEEE Trans. Med. Img.IEEE Trans. Med. Img. 23:137-152, 2004. 23:137-152, 2004.
Data-Driven:Data-Driven:
1.1. GLM DiagnosticsGLM Diagnostics1.1. SPMd, SPMd, Luo W-L, Nichols T. Luo W-L, Nichols T. NeuroImageNeuroImage 19:1014-32, 2003 19:1014-32, 2003
2.2. Minimize p-valuesMinimize p-values1.1. Hopfinger JB, et al., Hopfinger JB, et al., Neuroimage,Neuroimage, 11:326-333, 2000. 11:326-333, 2000.
2.2. Tanabe J, et al. Tanabe J, et al. Neuroimage,Neuroimage, 15:902-907, 2002. 15:902-907, 2002.
3.3. Model Selection:Model Selection: Classical hypothesis testing, maximum likelihood, Classical hypothesis testing, maximum likelihood, Akaike’s information criterion (AIC), Minimum DescriptionLength, Bayesian Akaike’s information criterion (AIC), Minimum DescriptionLength, Bayesian Information Criterion (BIC) & Model Evidence, Information Criterion (BIC) & Model Evidence, Cross ValidationCross Validation
4.4. Replication/ReproducibilityReplication/Reproducibilitya.a. Empirical ROCsEmpirical ROCs – mixed multinomial model – mixed multinomial model
1.1. Genovese CR., et al., Genovese CR., et al., Magnetic Resonance in MedicineMagnetic Resonance in Medicine, 38:497–507, 1997., 38:497–507, 1997.2.2. Maitra, R., et al., Maitra, R., et al., Magnetic Resonance in Medicine, 48, Magnetic Resonance in Medicine, 48, 62 –70, 2002.62 –70, 2002.3.3. Liou M., et al., Liou M., et al., J. Cog. NeuroscienceJ. Cog. Neuroscience, 15:935-945, 2003., 15:935-945, 2003.
b.b. Empirical ROCsEmpirical ROCs – lower bound on ROC – lower bound on ROC1.1. Nandy RR & Cordes D. Nandy RR & Cordes D. Magnetic Resonance in MedicineMagnetic Resonance in Medicine 49:1152–1162, 2003. 49:1152–1162, 2003.
5.5. Prediction Error/AccuracyPrediction Error/Accuracy1.1. Kustra R & Strother SC. Kustra R & Strother SC. IEEE Trans Med ImgIEEE Trans Med Img 20:376-387, 2001. 20:376-387, 2001.2.2. Carlson, T.A., et al., Carlson, T.A., et al., J Cog NeuroscienceJ Cog Neuroscience, 15:704–717, 2003., 15:704–717, 2003.3.3. Hanson,S.J., et al., Hanson,S.J., et al., NeuroImageNeuroImage 23:156– 166, 2004 23:156– 166, 2004
• Hopfinger JB, Buchel C, Holmes AP, Friston KJ, A study of analysis parameters that influence the sensitivity of event related fMRI analyses, Neuroimage, 11:326-333, 2000.
• Tanabe J, Miller D, Tregellas J, Freedman R, Meyer FG. Comparison of detrending methods for optimal fMRI preprocessing. Neuroimage, 15:902-907, 2002.
Does not imply a stronger likelihood of getting the same result in another replication of the same experiment!
Optimization Framework 3Optimization Framework 3Model Selection: An attempt to formulate some
traditional problems in the methodology of science in a rigorous way.
Standard methods: (Classical hypothesis testing, maximum likelihood, Akaike’s information criterion (AIC), Minimum DescriptionLength, Bayesian Information Criterion (BIC) & Model Evidence, Cross Validation) compensate for errors in the estimation of model parameters.
All tradeoff fit with simplicity (least # parameters), but give simplicity different weights.
All favor more complex (less simple) models with more data.
Forster MR. Key concepts in model selection: Performance and Generalizability. J Math Psych 44:205-231, 2000
• replication is a fundamental criterion for a result to be considered scientific;
• smaller p values do not necessarily imply a stronger likelihood of repeating the result;
• for “good scientific practice” it is necessary, but not sufficient, to build a measure of replication into the experimental design and data analysis;
• Genovese CR, Noll DC, Eddy WF. Genovese CR, Noll DC, Eddy WF. Estimating test-retest reliability Estimating test-retest reliability in functional MR imaging. I. Statistical methodology.in functional MR imaging. I. Statistical methodology. Magnetic Magnetic Resonance in MedicineResonance in Medicine, 38:497–507, 1997., 38:497–507, 1997.
• Maitra, R., Roys, S. R., & Gullapalli, R. P. Maitra, R., Roys, S. R., & Gullapalli, R. P. Test–retest reliability Test–retest reliability estimation of functional MRI data.estimation of functional MRI data. Magnetic Resonance in Magnetic Resonance in Medicine, 48, Medicine, 48, 62 –70, 2002.62 –70, 2002.
• Liou M, Su H-R, Lee J-D, Cheng PE, Huang C-C, Tsai C-H. Liou M, Su H-R, Lee J-D, Cheng PE, Huang C-C, Tsai C-H. Bridging Functional MR Images and Scientific Inference: Bridging Functional MR Images and Scientific Inference: Reproducibility Maps.Reproducibility Maps. J. Cog. NeuroscienceJ. Cog. Neuroscience, 15:935-945, 2003., 15:935-945, 2003.
Stone, M. Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. B 36: 111–147. 1974
Hastie T, Tibshirani R, Friedman J. Hastie T, Tibshirani R, Friedman J. The The elements of statistical learning theory.elements of statistical learning theory. Springer-Verlag, New York, 2001Springer-Verlag, New York, 2001
Principal Component AnalysisL. K. Hansen, et al. Neuroimage, vol. 9, no. 5, pp. 534-44, 1999.Prediction via GLM, Split-Half ReproducibilityJ. V. Haxby, et al. Science, vol. 293, no. 5539, pp. 2425-30, 2001.Linear Discriminant Analysis/Canonical Variates AnalysisR. Kustra and S. Strother, IEEE Trans Med Imaging, vol. 20, no. 5, pp. 376-87, 2001.T. A. Carlson, et al. J Cogn Neurosci, vol. 15, no. 5, pp. 704-17, 2003.J. D. Haynes and G. Rees, Nat Neurosci, vol. 8, no. 5, pp. 686-91, 2005.Y. Kamitani and F. Tong, Nat Neurosci, vol. 8, no. 5, pp. 679-85, 2005.A.J. O'Toole, et al. J Cogn Neurosci, vol. 17, no. 4, pp. 580-90, 2005.Support Vector Machines (and LDA)D. Cox and R. L. Savoy, Neuroimage, vol. 19, no. 2 Pt 1, pp. 261-70, 2003.S. LaConte, et al. Neuroimage, vol. 26, no. 2, pp. 317-29, 2005.J. Mourao-Miranda, et al. Neuroimage, vol. 28, no. 4, pp. 980-95, 2005.Artificial Neural NetworksB. Lautrup, et al. in Proceedings of the Workshop on Supercomputing in Brain Research: From
Tomography to Neural Networks, H. J. Hermann, et al., Eds. Ulich, Germany: World Scientific, pp. 137-144, 1994.
N. Morch, et al. Lecture Notes in Computer Science 1230, J. Duncan and G. Gindi, Eds. New York: Springer-Verlag, pp. 259-270, 1997.
S. J. Hanson, et al. Neuroimage, vol. 23, no. 1, pp. 156-66, 2004.S. M. Polyn, et al. Science, vol. 310, no. 5756, pp. 1963-6, 2005.
NPAIRS Uses NPAIRS Uses “split-half” resampling to combine: resampling to combine:• Prediction & Reproducibility Metrics
• PCA-based reproducibility measures of:PCA-based reproducibility measures of:− uncorrelated signal and noise SPMs;uncorrelated signal and noise SPMs;− reproducible SPMs (rSPM) on a Z-score scale;reproducible SPMs (rSPM) on a Z-score scale;− multivariate dimensionality.multivariate dimensionality.
• Combined prediction and reproducibility metrics for:Combined prediction and reproducibility metrics for:− data-driven ROC-like curves;data-driven ROC-like curves;− optimizing bias-variance tradeoffs of pipeline interactions.optimizing bias-variance tradeoffs of pipeline interactions.
• Other Measures:Other Measures:− empirical random effects correction;empirical random effects correction;− measures of individual observation influence.measures of individual observation influence.
• Design matrix (G) “brain states” = discriminant classes.−prediction metric = posterior probability of class membership.−maximizes a multivariate signal-to-noise ratio:
Optimization of fMRI Static Force fMRI Optimization of fMRI Static Force fMRI Sixteen subjects with 2 runs/subject Sixteen subjects with 2 runs/subject Acquisition:
• Whole-brain, interleaved 1.5T BOLD-EPI;• 30 slices = 1 whole-brain scan;• 1 oblique slice = 3.44 x 3.44 x 5 mm3;• TR/TE = 4000 ms/70 ms
Experimental Design:
Analyzed with NPAIRS, GLM and PCA/CVA:• Dropped initial non-equilibrium and state-transition scans;Dropped initial non-equilibrium and state-transition scans;• 2-class single-subject;2-class single-subject;• 11-class 16-subject, group analysis;11-class 16-subject, group analysis;• NPAIRS/CVA, GLM-CVA comparison across preprocessing pipelines.NPAIRS/CVA, GLM-CVA comparison across preprocessing pipelines.
Preprocessing for Static ForcePreprocessing for Static Force All runs/subject(s) passed initial quality control:All runs/subject(s) passed initial quality control:
• movement (AIR 5) < 1 voxel;movement (AIR 5) < 1 voxel;• no artifacts in functional or structural scans;no artifacts in functional or structural scans;• no obvious outliers in PCA of centered data matrix.no obvious outliers in PCA of centered data matrix.
Alignment (AIR 5):Alignment (AIR 5):• Within-Subject:Within-Subject: across runs to 1st retained scan of run one; across runs to 1st retained scan of run one;
• Between-Subject:Between-Subject: 1 1st (Affine)st (Affine), 3, 3rdrd, 5, 5thth and 7 and 7thth order polynomials; order polynomials;
• Tri-linear and sinc (AIR 05) interpolation.Tri-linear and sinc (AIR 05) interpolation.
ROC-Like: Prediction vs. Reproducibility ROC-Like: Prediction vs. Reproducibility 2-Class Static Force, Single Subject A Bias-Variance Tradeoff.
As model complexity increases (i.e., As model complexity increases (i.e., #PCs 10 →100), prediction of #PCs 10 →100), prediction of design matrix’s class labels design matrix’s class labels improves and reproducibility improves and reproducibility
Optimizing Performance.Like an ROC plot there is a single Like an ROC plot there is a single point, (1, 1), on this prediction vs. point, (1, 1), on this prediction vs. reproducibility plot with the best reproducibility plot with the best performance; at this location the performance; at this location the model has perfectly predicted the model has perfectly predicted the design matrix while extracting an design matrix while extracting an infinite SNR.infinite SNR.
LaConte S, et. al. LaConte S, et. al. Evaluating preprocessing Evaluating preprocessing choices in single-subject BOLD-fMRI choices in single-subject BOLD-fMRI studies using data-driven performance studies using data-driven performance metricsmetrics. . Neuroimage Neuroimage 18:10-23, 2003
Pipeline Meta-models: Data Analysis 2Pipeline Meta-models: Data Analysis 2 Part of science by Strong Inference:Part of science by Strong Inference:
• for a scientifically interesting observation enumerate all alternative for a scientifically interesting observation enumerate all alternative hypotheses that can account for the observation, based on present hypotheses that can account for the observation, based on present knowledgeknowledge
− Jewett DL. What’s wrong with a single hypothesis. The Scientist, 19(21):10, 2005Jewett DL. What’s wrong with a single hypothesis. The Scientist, 19(21):10, 2005− Platt JR. Strong inference. Science, 146:347-353, 1964Platt JR. Strong inference. Science, 146:347-353, 1964
Comparing univariate GLM versus multivariate CVA Comparing univariate GLM versus multivariate CVA data analysis is a simple means of implementing multi-data analysis is a simple means of implementing multi-hypothesis tests in neuroimaging:hypothesis tests in neuroimaging:• test localizationist versus network theories of brain function!test localizationist versus network theories of brain function!• account for differences in data-analysis sensitivity and specificity!account for differences in data-analysis sensitivity and specificity!• test different interactions with preprocessing pipeline choices!test different interactions with preprocessing pipeline choices!
Simple Motor-Task Replication at 4.0TSimple Motor-Task Replication at 4.0T
t-test Fisher Linear Discriminant = 2-class CVA
L R
C. Tegeler, S. C. Strother, J. R. Anderson, and S. G. Kim, "Reproducibility of BOLD-based functional MRI obtained at 4 T," Hum Brain Mapp, vol. 7, no. 4, pp. 267-83, 1999.
AcknowledgementsAcknowledgementsUniversity of MinnesotaUniversity of Minnesota
International Neuroimaging Consortium & International Neuroimaging Consortium & VAMC: http://neurovia.umn.edu/incweb
Jon R. Anderson, M.Sc., Sally Frutiger, Ph.D., Kelly Rehm, Ph.D., David Rottenberg, M.D., Jon R. Anderson, M.Sc., Sally Frutiger, Ph.D., Kelly Rehm, Ph.D., David Rottenberg, M.D.,
Kirt Schaper, M.Sc., John Sidtis, Ph.D., Jane Zhang, Ph.D.Kirt Schaper, M.Sc., John Sidtis, Ph.D., Jane Zhang, Ph.D.
Seong-Ge Kim, Ph.D., Essa Yacob, Ph.D., Seong-Ge Kim, Ph.D., Essa Yacob, Ph.D., CMRR & Biomed. Eng.CMRR & Biomed. Eng.
James Ashe, M.D., Ph.D., James Ashe, M.D., Ph.D., Neurology & VAMCNeurology & VAMC
Suraj A. Muley, M.D., Suraj A. Muley, M.D., Neurology & VAMCNeurology & VAMC
Emory UniversityEmory University University of TorontoUniversity of Toronto