Page 1
ANewGenerationofBrain-ComputerInterfacesDrivenbyDiscoveryofLatentEEG-fMRILinkagesUsingTensorDecomposition
HYPOTHESISANDTHEORYARTICLEFrontiers inNeuroscience, 07June 2017
XiaopingHuProfessor ofBiomedical EngineeringUniversity ofCalifornia, Riverside
Andrej CichockiSkolkovo InstituteofScience andTechnology(Skoltech), Moscow, RussiaNicolaus Copernicus University (UMK), Torun,PolandSystemsResearchInstitute,PolishAcademyofScience,Warsaw,Poland
LukeOedingDepartmentofMathematicsandStatisticsAuburn University
Rangaprakash DeshpandeDepartmentofPsychiatry andBiobehavioral SciencesUCLA
Gopikrishna DeshpandeAUMRIResearchCenterAUDepartmentofElectricalandComputer EngineeringAUDepartmentofPsychologyAlabamaAdvanced ImagingConsortium
Page 2
BrainComputerInterface
https://en.wikipedia.org/wiki/File:Brain-computer_interface_(schematic).jpg
Page 3
EEGBasedBCI
• +Non-invasive• +Hightemporalresolutionforreal-timeinteraction• +Inexpensive,lightweight,andhighlyportable• - Poorspatialspecificity• - EEGSignalsindifferentchannelsarehighlycorrelated,reducingabilitytodistinguishneurologicalprocesses.• - Longtrainingtimerequired
Page 4
Real-timefMRIBasedBCI
• +Highspatialspecificity-- moreaccuracy• - Highcost• - Non-portable• - Lowtemporalresolution• - Restrictiveenvironment
Page 5
SimultaneousEEG– fMRIdataacquisition
http://www.ant-neuro.com/sites/default/files/styles/product_imageshow/public/85500991.jpg?itok=0WPQPi1c
Page 6
EEGData
Electrodeposition measurements viaPolhemus Fastrak 3DDigitizerSystemEEGdatawereepoched withrespecttoRpeaks ofEKG
signaland averagedovertrials.Theballistocardiogram (BCG)artifactintheEEGsignalobtained inside MRscannerisremoved.
Page 7
Objective:DiscoveryofLatentLinkagesbetweenEEGandfMRIandimproveBCI• Hypothesis:latentlinkagesbetweenEEGandfMRIcanbeexploitedtoestimatefMRI-likefeaturesfromEEGdata.• ThiscouldallowanindependentlyoperatedEEG-BCItodecodebrainstatesinrealtime,withbetteraccuracyandlowertrainingtime.• Hypothesis:Featuresfromasub-setofsubjectscanbegeneralizedtonewsubjects(forahomogeneoussetofsubjects).
Page 8
Strategies• ObtainfMRIdatawithhightemporalresolution:
• Usemultibandecho-planarimaging(M-EPI)[Feinberg,etal.2010]toachievewholebraincoveragewithsampling intervals(TR)asshortas200ms.• ViewfMRIasconvolutionofHDF(Hemodynamicresponsefunction)andneuronalstates.UsecubatureKalman filterbasedblinddeconvolutionoffMRI[Havlicek,etal.2011]torecoverdrivingneuronalstatevariableswithhighereffectivetemporalresolution.
• ObtaincleanEEGdata:• EEGsignalsampledat5000Hztoensureaccurategradientartifactremoval,thendownsampled to250Hztomakedatasetmoremanageable.
• UsethecomplexMorlet wavelet[Teolis,1998]togiveatime-frequencyrepresentationofbothEEGandfMRIforeachtrial.
Page 9
DiscoverlatentlinkagesbetweenEEGandfMRI• SimultaneousEEG/fMRIdatacollectedusingaP300spellerbasedparadigm.• EEGmodalities:trial–time–frequency–channel
• 4msupdates,63+1channels,4trials
• fMRImodalities:trial–time–neuronalstate–voxel• 200msupdateswithwholebraincoverageand3mmvoxels
• ApplyOrthogonalDecompositiontoeachEEGandfMRI.[ZhouandCichocki 2012]• Thefirstdimensionof“trials”isthesameforbothtensors,permittingtheapplicationofHOPLS.ThisimportantpropertyallowsbothEEGandfMRItobesampledatdifferentrates.• ItisnotrequiredtodownsampleEEGtofMRI’stemporalresolution,asdonebymostresearchersintheEEG-fMRIcomparisonliterature(Goldman,etal.2002)(Hinterberger,Veit,etal.2005),whichwillleadtolossofvitaltemporalinformation.
Page 10
• Assumptions:EEGdataistheindependentvariableX,anddeconvolvedfMRI(neuronalstates)dataisthedependentvariableY.• Reasonableassumptionbecausethehemodynamic/metabolic activityisasecondaryresponsetotheelectricalactivity.
• Goal:GivenX andY overmanytrials,andassumingF(X)=Y,discoverF.• HigherOrderMultilinearSubspaceRegression/HigherOrderPartialLeastSquares(HOPLS)[Q.Zhao,etal.2011]topredictthedependentvariable(deconvolvedfMRI)fromtheindependentvariable(EEG).• HOPLSparameters(latentvariables,coretensorsandtensorloadings)arelikelytoprovideinformationonlatentEEG-fMRIrelationships acrossthedimensionsconsidered.
DiscoverlatentlinkagesbetweenEEGandfMRI
Page 11
PartialLeastSquares(PLS)
𝑋 = 𝑇𝑃% +𝐸 =( 𝑡*
+
*,-
𝑝*% + 𝐸
𝑌 = 𝑈𝑄% + 𝐹 = ( 𝑢*
+
*,-
𝑞*% + 𝐹
𝑇 = 𝑡-,𝑡7,⋯ , 𝑡+ and𝑈 = 𝑢-,𝑢7,⋯ , 𝑢+ arematricesofR extractedlatentvariables fromX andY,respectively.U will havemaximum covariancewithT column-wise.P andQ arelatentvectorsubspace base loadings.E andF areresiduals.
TherelationbetweenT andU canbeapproximatedas𝑈 ≈ 𝑇𝐷whereD isanR×R diagonalmatrixofregression coefficients.
Partialleastsquares: Predictsasetofdependent variablesY fromasetofindependent variablesX.Attemptstoexplain asmuch aspossible thecovariancebetweenXand Y.
PLSoptimizationobjective istomaximizepairwise covarianceofasetoflatentvariables byprojecting bothXandY ontonewsubspaces.
(3)
Page 12
LeastSquares(UndergraduateLinearAlgebra)
• Givenalineartransformation𝑃 → 𝑄 wewanttosimultaneouslypredictthesubspacesℝR ⊂ 𝑃 andℝT ⊂ 𝑄 sothattherestrictedmapℝR → ℝTgivesagoodapproximationofthemapping.
• Givenanunderdeterminedmatrixequation𝐴�⃗� = 𝑏,wecanattempttosquarethesystemandsolve:𝐴\𝐴�⃗� = 𝐴\𝑏• PerhapsuseQR.
• Thestandardleast-squaressolutionis𝑥] = 𝐴\𝐴 ^-𝐴\𝑏.• ProjecttoalinearsubspaceandreplaceAwithafullrankmatrixusingSVD.
Page 13
SingularValueDecomposition• TheSingularValueDecomposition𝐴 = 𝑈Σ𝑉\
• Thequasi-diagonalmatrixofsingularvalues𝜎-, 𝜎7,… canbetruncatedtothelargestr singularvaluestogivethebestrankr approximation.
• TheorthogonalmatricesU andV(calledloadings)givetheembeddings oftherespectivesubspacesonwhichA isbestapproximatedtorankr.
• Thepseudoinverse ofA is𝐴† = 𝑉Σ†𝑈\,whereΣ† = diag(𝜎-^-, 𝜎7^-, …)
• Theminimalnormsolutionto𝐴�⃗� = 𝑏 is𝑥] = 𝐴†𝑏 = Σd,-* ef
ghif𝑣d.
Page 14
SVDandPLS• Givenm dataobservationsofnparticipantsstoredinadatamatrixX(independentvariables).• Givenk responsesofthen participantsstoredinadatamatrixY(dependentvariables).• FindalinearfunctionF thatexplainsthemaximumcovariancebetweenXandY.
𝑌 = 𝑋𝐹+ 𝐸• CenterandnormalizebothX andY.• ComputetheCovarianceMatrix𝑅 = 𝑌\𝑋• PerformSVD:R = 𝑈Σ𝑉\ (compactform,iterativealgorithm)• ThelatentvariablesofX andY areobtainedbyprojections:
𝐿n = 𝑋𝑉 𝐿o = 𝑌𝑈• U andVgivetheembeddingsofthesubspaces(theloadingsofthevariables)
Page 15
Structuredvariables• InthesituationofEEG-fMRIdata,evenafterawaveletdecompositionofthedata,westillhaveextrastructureinthedependentvariables(fMRI)YandintheindependentvariablesX.• EEGmodalities:trial–time–frequency–channel
• 4msupdates,63+1channels,4trials• SoXcouldbe4trialsx100waveletsx63channels
• fMRImodalities:trial–time–neuronalstate–voxel• 200msupdateswithwholebraincoverageand3mmvoxels• SoYcouldbe4trialsx200waveletsx36,000voxels
• Don’tthinkof100x63as6,300,don’tthinkof200x36,000as7.2×10u• Unfoldingleadsto“smallp largen”problemandalossofinformation.
Page 16
ModalProductsforTensors• For𝒜 ∈ ℝxy×xz×⋯×x{ and𝑈 ∈ ℝ|}×x} the𝑛��modetensor-matrixproductis
𝒜×T𝑈 ∈ ℝxy×xz×⋯×x}�y×|}×x}�y×⋯×x{
𝒜×T𝑈 ∶= ( 𝑎dy,dz,…,d},…d{𝑢�},d}d}∈x}
• Thismodalproductgeneralizesthematrixproductandvectorouterproductandreplacestranspose.
If𝐴 ∈ ℝxy×xz and𝐵 ∈ ℝxy×|z then𝐴×-𝐵 = 𝐵\𝐴 ∈ ℝ|z×xz
Ifx ∈ ℝ-×T andy ∈ ℝ-×R then �⃗�\𝑥 ∈ ℝR×T
Page 17
MatrixSVDusingmodalproductnotation
• SVDTheorem:Everycomplex𝐼-×𝐼7 matrix𝐹 hasanexpression𝐹 = 𝑆×-𝑈(-)×7𝑈(7)
with𝑈(-) aunitary𝐼-×𝐼-matrix𝑈(7) aunitary𝐼7×𝐼7 matrix𝑆 pseudodiagonal𝐼-×𝐼7 matrix,𝑆 = diag(𝜎- , 𝜎7,… , 𝜎��� xy,xz )Thesingularvaluesareordered:𝜎- ≥ 𝜎7 ≥ ⋯ ,≥ 𝜎��� xy,xz ≥ 0
Page 18
TensorSVD(OrthogonalTuckerDecomposition)
• Every𝐼-×𝐼7×⋯×𝐼� array𝒜 canbewrittenasaproduct:𝒜 = 𝑆×-𝑈(-)×7𝑈(7)⋯×�𝑈(�)
• Each𝑈(T)isaunitary𝐼T×𝐼T matrix.
• 𝑆isa𝐼-×𝐼7×⋯×𝐼� complextensorwithsliceshavingnorm 𝑆d},d = 𝜎d(T),
then-modesingularvaluesof𝒜• Foreachnthesingularvaluesareordered𝜎-
(T) ≥ 𝜎7T ≥ ⋯ ≥ 𝜎x}
T ≥ 0• Theslices𝑆d},d areall-orthogonal:
𝑆d},�,𝑆d},� = 0∀𝛼 ≠ 𝛽∀𝑛
• Computethen-modesingularmatrix𝑈(T) andn-modesingularvaluesbythematrixSVDofthen-th unfoldingofsize𝐼T×𝐼7𝐼�⋯𝐼T^-𝐼T�-𝐼�.• Siscomputedby𝑆 = 𝒜×-𝑈 - ∗
×7𝑈 7 ∗⋯×�𝑈 � ∗
Theorem:[DeLauthawer 2005, Zhao-Cichocki 2013]
Page 19
TuckerDecompositionforEEG--fMRI• Takean𝑁-×𝑁7×𝑁�×𝑁�tensorandexpressitasa(small)coretensorofsize𝐿-×𝐿7×𝐿�×𝐿�togetherwithchangesofbases(loadings)toputthecorebackintothelargertensorspace.• Let𝑋 and𝑌 betensorsofEEGanddeconvolvedfMRI,respectivelywithmodalities:trials,voxels/channels,timeandfrequency.• ObtainnewtensorsubspacesviatheTuckermodelforeachtrial:
• Approximate𝑋 withasumofmultilinear rank-(1, 𝐿7,𝐿�,𝐿�)terms• Approximate𝑌 withasumofmultilinear rank-(1,𝐾7, 𝐾�,𝐾�) terms
• ThecoretensorsmodeltheunderlyingbiophysicsandaredifferentforEEGandfMRI.• PerformHOSVDonthe𝐿7×𝐿�×𝐿�×𝐾7×𝐾�×𝐾� contraction𝑋×- 𝑌
Page 20
HigherorderPartialLeastSquares(HOPLS)TheHOPLSisexpressed as
𝑌 = (𝐷*
+
*,-
×-𝑡*×7𝑄*(7)×�𝑄*
(�)×�𝑄*� +𝐹
𝑋 =( 𝐺*
+
*,-
×-𝑡*×7𝑃*(7)×�𝑃*
(�)×�𝑃*� +𝐸
whereR isthenumber oflatentvectors,𝑡* istherth latentvector,𝑃*(T) and𝑄*
(R) areloadingmatricescorrespondingtolatentvector𝑡* onmode-n andmode-m,respectively,𝐺* and𝐷* arecoretensors,and×� istheproduct inthekth mode.
Compute the𝑡* astheleadingleftsingular vectorofanunfolding, deflate,andrepeat.
Page 21
FeasibilityStudy
• WeperformedthesimultaneousEEG/fMRIexperimentandEEG-onlyBCIusingtheP300spellerparadigmin4right-handedmalesubjects(meanage:21.5years)withnohistoryofneurologicalorotherillness.• fMRIdatawereacquiredusingtheM-EPIsequence(TR=200ms,multibandfactor=8,3mmisotropicvoxels,fullcoverage)anddeconvolvedusingthecubatureKalman filterapproach.• Theanalyseswerecarriedoutona highperformancecomputerwithIntel®Core™i7-3820(QuadCore,10MBCache)overclockedupto4.1GHzprocessorwithatopofthelineNVidiaGPUQuadro Plex 7000.• WeobtainedsignificantlyhighcorrelationusingboththefullandthesignificantHOPLSmodels,withthelatterprovidingbetteraccuracywithruntimesunderasecond.
Page 23
PreliminaryResultsOffline analysis of
Simultaneous EEG/fMRI ExptFull HOPLS forward model Significant HOPLS forward model
Correlation between deconvolved fMRI data and
that predicted from EEG0.76 ± 0.17 0.84 ± 0.13
Approximate run time for ‘prediction module’ in sec
1.4 0.8
Table.2 Prediction of deconvolved fMRI from simultaneously acquired EEG using offline analysis
Off line analysis of simultaneous
EEG/fMRI Expt
Original fMRI MVPA
fMRI predicted with significant
HOPLS + MVPA
fMRI predicted with full HOPLS
+ MVPA
SVM based on EEG tensors
(from sequential model)
SVM based on fMRI
tensors(from sequential model)
SVM based on ERP amplitude
and latency
Letter decoding accuracy
1 trial block
0.97±0.03 0.94 ± 0.04 0.93±0.05 0.84±0.10 0.86±0.12 0.68 ± 0.17
8 trial blocks
1 1 1 0.98±0.02 0.98±0.02 0.84 ± 0.11
Run time per letter decoded (sec)
0.9 1.8 2.4 0.13 0.24 0.08
Table.3 Letter decoding accuracy from post-hoc analysis of simultaneous EEG/fMRI data
Page 24
Preliminaryresults
Online analysis of EEG-only BCI dataParameters from same subject’s EEG/fMRI run
Parameters from random prior
subject’s EEG/fMRI run
Parameters learned from all prior subjects’ EEG/fMRI run
Subject-2 Subject-3 Subject-4
Letter decoding accuracy from fMRI predicted with significant
HOPLS + MVPA
1 trial block 0.93 ± 0.04 0.87 ± 0.11 0.86 0.91 0.93
8 trial blocks 1 0.94 ± 0.04 0.93 0.93 0.95
Table.4 Letter decoding accuracy from real-time analysis of EEG data using predicted fMRI (from significant HOPLS) as features for MVPA
In spite of these encouraging results, we stress the fact that they are derived from a small, homogeneous sample of 4 subjects. We need to do more trials to demonstrate more broad generalizability.
Page 25
(ExtraSlide) HigherOrderPartialLeastSquares• Thesubspacetransformationisoptimizedusingthefollowingobjectivefunction,yieldingthecommonlatentvariable𝑡* insteadof2latentvariables.
• 𝑚𝑖𝑛{¡ } ,¢ } } 𝑋 − [𝐺; 𝑡,𝑃7 ,𝑃 � ,𝑃 � ]
7+ 𝑌 − [𝐷;𝑡,𝑄 7 ,𝑄 � , 𝑄 � ]
7
suchthat 𝑃 T %𝑃 T = 𝐼¨}𝑎𝑛𝑑 𝑄R %𝑄 R = 𝐼ª«
• Simultaneousoptimizationofsubspacerepresentationsandlatentvariable𝑡*.SolutionscanbeobtainedbyMultilinearSingularValueDecomposition(MSVD)(see[Q.Zhao,etal.2011])• Minimizingtheerrorsisequivalenttomaximizingthenorms 𝐺 and 𝐷simultaneously(accountingforthecommonlatentvariable).Todothiswemaximizetheproduct 𝐺 7 ⋅ 𝐷 7.• Computethelatentvariables𝑡* asleadingleftsingularvectorsandthendeflate.Repeatuntilyoureachtheerrorboundsyouwant.