Decoding Patterns of Human Brain Activity · 2014-11-04 · Abstract Considerable information about mental states can be decoded from noninvasive measures of human brain activity.

PS63CH19-Tong ARI 19 November 2011 9:21

Decoding Patterns of HumanBrain ActivityFrank Tong and Michael S. PrattePsychology Department and Vanderbilt Vision Research Center, Vanderbilt University,Nashville, Tennessee 37240; email: [email protected]

Annu. Rev. Psychol. 2012. 63:483–509

First published online as a Review in Advance onSeptember 19, 2011

The Annual Review of Psychology is online atpsych.annualreviews.org

This article’s doi:10.1146/annurev-psych-120710-100412

Copyright c© 2012 by Annual Reviews.All rights reserved

0066-4308/12/0110-0483$20.00

Keywords

fMRI, multivoxel pattern analysis, MVPA

Abstract

Considerable information about mental states can be decoded fromnoninvasive measures of human brain activity. Analyses of brain activ-ity patterns can reveal what a person is seeing, perceiving, attendingto, or remembering. Moreover, multidimensional models can be usedto investigate how the brain encodes complex visual scenes or abstractsemantic information. Such feats of “brain reading” or “mind read-ing,” though impressive, raise important conceptual, methodological,and ethical issues. What does successful decoding reveal about the cog-nitive functions performed by a brain region? How should brain signalsbe spatially selected and mathematically combined to ensure that de-coding reflects inherent computations of the brain rather than thoseperformed by the decoder? We highlight recent advances and describehow multivoxel pattern analysis can provide a window into mind-brainrelationships with unprecedented specificity, when carefully applied.However, as brain-reading technology advances, issues of neuroethicsand mental privacy will be important to consider.

483

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Decoding: neuraldecoding involvesdetermining whatstimuli or mentalstates are representedby an observed patternof neural activity

Contents

INTRODUCTION . . . . . . . . . . . . . . . . . . 484BRIEF TUTORIAL ON

MULTIVOXEL PATTERNANALYSIS . . . . . . . . . . . . . . . . . . . . . . . . 486

REVIEW OF FUNCTIONALMAGNETIC RESONANCEIMAGING STUDIES . . . . . . . . . . . . . 487Decoding Visual Features . . . . . . . . . . 487Decoding Visual Perception . . . . . . . . 489Decoding Visual Objects . . . . . . . . . . . 489Identifying and Reconstructing

Novel Visual Scenes . . . . . . . . . . . . 491Decoding Top-Down Attentional

Processes . . . . . . . . . . . . . . . . . . . . . . . 492Decoding Imagery and Working

Memory . . . . . . . . . . . . . . . . . . . . . . . . 492Decoding Episodic Memory . . . . . . . . 493Extracting Semantic Knowledge . . . . 494Decoding Phonological

Representations andLanguage Processing . . . . . . . . . . . . 495

Decoding Decisions in the Brain . . . . 496CONCEPTUAL AND

METHODOLOGICAL ISSUES . . 496What Is Being Decoded? . . . . . . . . . . . 497Where in the Brain to

Decode From? . . . . . . . . . . . . . . . . . . 498At What Spatial Scales of Cortical

Representation Is DecodingMost Useful? . . . . . . . . . . . . . . . . . . . 500

ETHICAL AND SOCIETALCONSIDERATIONS . . . . . . . . . . . . . 500

CONCLUDING REMARKS . . . . . . . . . 503

INTRODUCTION

Imagine that it is the future, an unknown yearin the twenty-first century. A participant isbrought into a neuroimaging lab and askedto lie back comfortably on a padded bed ta-ble, which is slowly glided into a brain scan-ner. The participant watches a brightly coloreddisplay as it provides a virtual tour of everypainting in the Musee d’Orsay. All the while,

noninvasive measures of that person’s brain ac-tivity are discretely taken, and the arrays ofnumbers are quickly transferred to the mem-ory banks of a high-speed digital computer.After hours of brain scanning and computeranalysis, the real scientific test begins. A ran-domly drawn painting is shown again to theobserver. The computer analyzes the incom-ing patterns of brain activity from the partic-ipant’s visual cortex and makes the followingprediction with 99% confidence: She is look-ing at painting #1023, Cezanne’s Still Life withApples and Oranges. The experimenter turns tolook at the computer screen, and indeed, theparticipant is looking at a plateful of pastel-colored red and yellow apples, and ripe or-anges stacked in a porcelain bowl, all care-fully arranged in the thick folds of a tousledwhite tablecloth. Another randomly drawn pic-ture is shown, and the computer correctly pre-dicts Landscape with Green Trees by MauriceDenis.

What does this remarkable scientificdemonstration reveal—successful mind read-ing? Have the neuroscientists effectivelycracked the brain’s internal code for vision,such that they now understand how featuresand objects are represented in the mind’sinternal eye? We will refer to this as ScienceFiction Story #1.

The lab volunteer has kindly offered to par-ticipate in a second experiment. This time sheis shown two paintings in quick succession(Bedroom in Arles, The White Horse) and then isasked to pick one and hold that image in mindfor several seconds. She imagines a horse stand-ing in a shallow river, head bent low as if look-ing at its own reflection in the slowly flowingstream. The computer quickly scans the matrixof numbers streaming in. Although brain activ-ity levels are substantially weaker as she gazessteadily at the blank screen, compared to mo-ments ago, a pattern begins to emerge from hervisual cortex. The computer announces, with85% confidence, that the participant is imag-ining the second painting, The White Horse.Would successful decoding in this case indi-cate that the neural codes for imagination and

484 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


internal visual thoughts have been successfullydecoded? More generally, what would such ademonstration reveal about the visual and cog-nitive functions performed by the brain? Wewill refer to this as Science Fiction Story #2.

In reality, these stories represent more factthan fiction. A simplified version of ScienceFiction Story #1 was carried out at the start ofthe twenty-first century in a pioneering studyby Haxby and colleagues (2001). The authorsused functional magnetic resonance imaging(fMRI) to measure patterns of blood leveloxygen–dependent (BOLD) activity, focusingon object-responsive regions in the ventraltemporal cortex. By comparing the similarityof brain activity patterns between the first andsecond half of the experiment, the authorsshowed that these high-level object areas couldaccurately predict whether participants wereviewing pictures of faces, houses, chairs, cats,bottles, shoes, scissors, or scrambled stimuli(Figure 1a). The use of more sophisticatedpattern-classification algorithms (Figure 1b)greatly improved researchers’ ability to predictwhat object categories people were viewing(Carlson et al. 2003, Cox & Savoy 2003). Sub-sequently, Kamitani & Tong (2005) discoveredthat it was possible to decode orientation- anddirection-selective responses with surprisingaccuracy (Figure 2), even though such feature-selective information is primarily organized atthe scale of submillimeter columns in the visualcortex. Thus, fMRI pattern analysis could re-veal cortical information that would otherwisefail to be detected. Perhaps the most strikingdemonstration of Science Fiction Story #1comes from the work of Kay et al. (2008). Theypresented more than 1,000 natural images toobservers and then characterized the responsepreferences of each voxel in the visual cortex,specifying their selectivity for retinotopic posi-tion, spatial frequency, and orientation. Whenthe observers were shown a new set of 120pictures, each of a different real-world scene,the authors could accurately predict which newimage was being viewed by finding the bestmatch between the observed pattern of activity

and the predicted activity of these modeledvoxels.

These studies reveal an unprecedented abil-ity to predict the basic visual features, com-plex objects, or natural scenes that are beingviewed by the participant. By combining fMRIwith sensitive pattern-analysis methods, accu-rate predictions about the viewed stimulus canbe made. Yet it would be a mistake to con-sider such feats as examples of mind reading.Why? Because the experimenter does not needa mind-reading device to achieve this perfor-mance. The same result could be achieved bysimply looking over the participant’s shoul-der, “Oh, she is looking at painting #1023,Cezanne’s Still Life with Apples and Oranges.”Put another way, one could perform these samefeats by reading out the activity patterns formedon the retina even though conscious processingof the image has yet to take place. Activity pat-terns on the retina would remain robust evenif the person were anesthetized or fell into adeep coma. So instead, Science Fiction Story#1 should be considered an example of brainreading.

Science Fiction Story #2 can be better justi-fied as a demonstration of mind reading. Here,information that is fundamentally private andsubjective is being decoded from the person’sbrain; the only alternative would be to ask theparticipant directly about what she is thinkingand to hope for an honest reply. Ongoing re-search is just beginning to probe the possibili-ties and limits of reading out subjective infor-mation from the human brain.

In this review, we discuss recent advancesin brain reading and mind reading, and weconsider important conceptual and method-ological issues regarding how to apply thesetechniques to the study of human cognition.The brain reading approach has revealedhow different types of stimulus informationare represented in specific brain areas, andsome studies provide clues to the functionalorganization of these representations. Patternanalysis of brain activity can also be adaptedto perform feats of mind reading to extractinformation about a person’s subjective mental

www.annualreviews.org • Decoding Patterns of Human Brain Activity 485

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Encoding: how astimulus or mentalstate is encoded orrepresented by specificpatterns of neuralactivity

state or cognitive goal. We consider whethersuch feats of mind reading should be likened tofancy parlor tricks that require the assistance ofa brain scanner or whether these methods canbe used to genuinely advance our understand-ing of brain function. Studies employing thismind-reading approach have revealed how par-ticular representations are activated or calledupon during conscious perception, attentionalselection, imagery, memory maintenance andretrieval, and decision making. As will beseen, careful consideration of experimentaldesign, analysis, and interpretation of the datais essential when adopting powerful patternanalysis algorithms to probe the functions thatmight be carried out by a brain area. As thesemethodologies continue to advance, it willbecome increasingly important to consider theethical implications of this technology.

There have been previous reviews on thetopic of fMRI decoding (sometimes calledmultivoxel pattern analysis, or MVPA (Haynes& Rees 2006, Norman et al. 2006), as well asmore in-depth reviews on the technical aspectsof decoding and encoding (Kriegeskorte 2011,Naselaris et al. 2011, O’Toole et al. 2007,Pereira et al. 2009). In this review, we highlightrecent studies and discuss key issues regardinghow fMRI pattern analysis can be used toadvance understanding of the bases of humancognition.

BRIEF TUTORIAL ONMULTIVOXEL PATTERNANALYSIS

Traditional methods of fMRI analysis treat eachvoxel as an independent piece of data, using sta-tistical tests to determine whether that voxel re-sponded more in some experimental conditionsthan in others. Such analyses are univariate: theanalysis of one voxel has no impact on the anal-ysis of any other. By contrast, multivariate pat-tern analysis extracts the information containedin the patterns of activity among multiple vox-els so that the relative differences in activitybetween voxels can provide relevant informa-tion. Whereas univariate statistical analyses are

designed to test whether some voxels respondmore to one condition than another, multivari-ate analyses are designed to test whether two(or more) experimental conditions can be dis-tinguished from one another on the basis ofthe activity patterns observed in a set of vox-els. Critically, multivariate methods might beable to tell apart the activity patterns for twodifferent conditions even if the average level ofactivity does not differ between conditions.

Figure 1b illustrates the simplest exampleof multivariate pattern analysis involving twoexperimental conditions (shown in red andgreen) and just two voxels, with the responseamplitude of each voxel shown on separateaxes. Each dot corresponds to a single activitypattern or data sample, with its positionindicating the strength of the response forvoxels 1 and 2. The Gaussian density plots inthe margins indicate that either voxel alonedoes a rather poor job of separating the twoexperimental conditions. Nevertheless, the twoconditions can be well separated by consideringthe pattern of responses to both voxels, asindicated by the separating boundary line.In this particular example, the responses ofvoxels 1 and 2 are positively correlated, and theclassification boundary helps to remove thiscorrelated “noise” to better separate the twoexperimental conditions. If there were threevoxels, a third dimension would be added;the red dots and greens dots would form twolargely separated (but still overlapping) cloudsof points, and the classification boundarywould consist of a linear plane that best dividesthose two clouds. Typically, anywhere from afew dozen to several thousand voxels might beused for fMRI pattern analysis, so an activitypattern with N voxels would be representedin an N-dimensional space, and clouds of dotsrepresenting the two classes would be separatedby a linear hyperplane. (Multiclass classifi-cation analysis involves calculating multiplehyperplanes to carve up this multidimensionalspace among three or more conditions.)

The goal of linear pattern classificationalgorithms, such as support vector machines(SVM), linear discriminant analysis (LDA),

486 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


or logistic regression, is to find the linear hy-perplane that best separates the two (or more)conditions in this multidimensional voxel space.The accuracy of classification performance isusually assessed using cross-validation, whichinvolves dividing the full set of data samplesinto separate sets for training and testing theclassifier. Typically, an entire fMRI run orperhaps just one sample from each conditionis reserved for the test set. The classifier istrained with the remaining data to obtain theclassification boundary, which is then used topredict the class of each data sample (e.g., “red”or “green”) in the test set. This procedure canbe done iteratively so that every sample in thedata set is tested and an overall measure of clas-sification accuracy is obtained. Classificationaccuracy reflects the amount of informationavailable in a set of voxels for discriminatingbetween the experimental conditions tested.

Here, we focus on linear pattern classifica-tion, since the performance of nonlinear classi-fiers applied to a brain region could potentiallyreflect computations performed by the classifierrather than by brain itself (Kamitani & Tong2005). For example, if one were to apply suffi-ciently complex nonlinear classifiers to the pat-terns of activity observed on the retina, it wouldbe possible to construct the functional equiva-lent of receptive fields with position-invarianttuning to visual orientation, curved lines, sharpcorners, or even a smiley face cartoon of BartSimpson, despite the lack of any such patterndetectors in the human retina. All brain pro-cesses essentially reflect a series of nonlinearcomputations; therefore, to characterize the in-formation processed by a brain region, we be-lieve it is important to avoid adding additionalnonlinear steps.

The reliability of linear classification per-formance depends on several factors: (a) thedegree of separation between the two classesof data samples (i.e., pattern separability orsignal-to-noise ratio), (b) the number of datasamples available for analysis, since havingmore samples will allow for better estimationof the optimal classification hyperplane, (c) thechoice of classification algorithm and its suit-

ability for the data set to be analyzed (Misakiet al. 2010), and (d ) the voxels used for patternanalysis. Adding more voxels should leadto better classification performance if thosevoxels contain some relevant information thatcan be used to better distinguish between thetwo conditions. However, if these additionalvoxels are uninformative, they may simply addnoise or unwanted variability to the activitypatterns and could thereby impair classificationperformance (Yamashita et al. 2008).

REVIEW OF FUNCTIONALMAGNETIC RESONANCEIMAGING STUDIES

Decoding Visual Features

In their original study of orientation decod-ing, Kamitani & Tong (2005) found thatactivity patterns in early visual areas couldpredict which of several oriented gratingswas being viewed with remarkable accuracy(Figure 2a). How was this possible, given thatBOLD responses were sampled from the visualcortex using 3mm-wide voxels, whereas orien-tation columns are organized at submillimeterspatial scales (Obermayer & Blasdel 1993,Yacoub et al. 2008)? The authors performedsimulations to show that random local vari-ations in cortical organization could lead toweak orientation biases in individual voxels. Bypooling the information available from manyindependent voxels, a pattern classifier couldachieve robust predictions of what orientationwas being presented in the visual field. Insubsequent work, high-resolution functionalimaging studies of the cat and human visualcortices have provided support for this hypoth-esis (Swisher et al. 2010). These experimentsshow that orientation information exists atmultiple spatial scales, extending from thatof submillimeter cortical columns to severalmillimeters across the cortex (Figure 2b). Ineffect, variability in columnar organization at asubmillimeter scale appears to lead to modestfeature biases at coarser spatial scales on theorder of millimeters. It should be noted that


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


studies find the presence of some global pref-erence for orientations radiating outward fromthe fovea as well (Freeman et al. 2011, Sasakiet al. 2006), but when such radial biases arecontrolled for, substantial orientation informa-tion can still be extracted from the visual cortex(Harrison & Tong 2009, Mannion et al. 2009).

These orientation-decoding studies suggestthat pattern analysis can be used to detectsignals of columnar origin by pooling weaklyfeature-selective signals that can be found at thescale of millimeters, presumably due to variabil-ity in the organization of the columns. Thus,fMRI pattern analysis could be used to revealhidden signals originating from fine-scale cor-tical columns that would otherwise be difficultor impossible to isolate with noninvasive imag-ing. Previously, researchers had to rely on fMRImeasures of visual adaptation to assess the fea-ture selectivity of responses in the human vi-sual cortex (Boynton & Finney 2003, Engel &Furmanski 2001).

This decoding approach has been used to in-vestigate cortical responses to many basic visualfeatures. Studies have revealed how the humanvisual system responds selectively to motiondirection (Kamitani & Tong 2006), color(Brouwer & Heeger 2009, Goddard et al. 2010,Sumner et al. 2008), eye-of-origin information(Haynes et al. 2005, Shmuel et al. 2010), andbinocular disparity (Preston et al. 2008). Thereliability of feature decoding depends onthe strength of the sensory signal; for example,the orientation of high-contrast gratings canbe decoded more readily than low-contrastgratings (Smith et al. 2011). Moreover, the am-plitude of the stimulus-driven BOLD responseserves as a good predictor of how much feature-selective information can be extracted from thedetailed pattern of activity found in a given vi-sual area. Pattern classification has also revealedsensitivity to more complex visual features. Forexample, sensitivity to orientations defined bymotion boundaries and by illusory contourshas been found in early visual areas, includingthe human primary visual cortex (Cliffordet al. 2009). It has also been used to showthat motion patterns that are more difficult

to see, namely second-order texture-definedmotion, lead to direction-selective patternsof activity in the human visual cortex that aresimilar to basic first-order motion (Hong et al.2011).

The feature-decoding approach has alsobeen used to test for selectivity to conjunc-tions of features (Seymour et al. 2009, 2010).For example, Seymour et al. (2009) tested forsensitivity to conjunctions of color and motionby presenting observers with compound dis-plays consisting of red dots moving clockwiseand overlapping with green dots moving coun-terclockwise, or green dots moving clockwisepaired with red dots moving counterclockwise.Activity patterns in early visual areas could dis-criminate between these different combinationsof color and motion, implying that these areascontain neurons sensitive to the conjunction ofthese features. These findings inform currenttheories of perceptual binding, which have de-bated whether top-down attentional processesare required to represent conjunctions of fea-tures (Treisman 1996).

What are the underlying neural sources ofthese feature-selective responses in the humanvisual cortex? In the case of orientation oreye-of-origin signals, these feature-selectiveresponses appear to reflect local biases incolumnar organization to a considerableextent (Shmuel et al. 2010, Swisher et al.2010). In other cases, feature selectivity mightreflect random variations in the distribution offeature-selective neurons (Kamitani & Tong2006) or more global biases such as a preferencefor radial patterns or radial motions acrossthe retinotopic visual cortex (Clifford et al.2009, Sasaki et al. 2006). For example, opticalimaging has revealed the presence of oculardominance columns, orientation columns, andcolor-sensitive blobs in the primary visualcortex (V1) of monkeys, but no evidence ofdirection-selective columns (Lu et al. 2010).Nonetheless, it is possible to decode strongdirection-selective responses from human V1(Kamitani & Tong 2006). Multiple factorscan contribute to the spatial distribution ofthese feature preferences in the cortex, and

488 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


these factors could have a strong impact onthe efficacy of fMRI pattern analysis. In manycases, future studies using high-resolutionfMRI in humans or optical imaging in animalswill be required to map the feature-selectiveproperties of the visual cortex.

Decoding Visual Perception

In the study by Kamitani & Tong (2005), a ma-jor goal was to extend the pattern-classificationapproach from the problem of brain readingto that of mind reading, which had not beendemonstrated before. They reported the resultsof a visual mind-reading experiment, showingthat it is possible to decode whether an observeris covertly attending to one set of oriented linesor the other when viewing an ambiguous plaiddisplay. Activity patterns in early visual areas(V1–V4) allowed for reliable prediction of theobserver’s attentional state (∼80% accuracy).Moreover, decoding of the attended orienta-tion was successful even in V1 alone, indicatingthat feature-based attention can bias orienta-tion processing at the earliest possible corticalsite.

Encouraged by these findings, severalresearch groups began to pursue fMRI patternclassification methods to investigate the neuralunderpinnings of subjective perceptual andcognitive states. Haynes & Rees (2005b)showed that fMRI pattern classification caneffectively decode which of two stimuli are per-ceptually dominant during binocular rivalry,with perceptual alternations occurring everyseveral seconds. Similarly, they found thatorientation-selective responses were disruptedby backward visual masking, although a smallamount of orientation information could stillbe detected in V1 for unseen visual orientations(Haynes & Rees 2005a). Perhaps most striking,they were able to apply these methods toextract monocular responses in the lateralgeniculate nucleus (Figure 3) and showed thatbinocular rivalry leads to modulations at thisvery early site of visual processing (Hayneset al. 2005, see also Wunderlich et al. 2005).This latter study provided novel evidence to

inform neural models of binocular rivalry(Blake & Logothetis 2002, Tong et al. 2006).Other research groups demonstrated that theperception of ambiguous motion displays couldbe decoded at greater-than-chance levels fromhuman motion area MT+ and other dorsalvisual areas (Brouwer & van Ee 2007, Serences& Boynton 2007b).

An intriguing study by Scolari & Serences(2010) revealed that these feature-selectiveresponses can also be linked to the accuracyof behavioral performance. The researchersfirst characterized the very modest orientationpreference of every voxel in the visual cortex.Next, they tested whether voxel responses to aparticular orientation might be boosted on tri-als in which observers correctly discriminate asmall change in visual orientation, as comparedto incorrect trials. When observers correctlydiscriminated a change in orientation centeredaround, say, 45◦, responses in V1 were notenhanced for voxels tuned specifically to 45◦;instead, they were enhanced for voxels thatpreferred neighboring orientations (∼10◦ and80◦). This counterintuitive result is predictedby models of optimal visual coding, whichpropose that discrimination performance willbe most improved by enhancing neighboringoff-channel responses.

Decoding Visual Objects

The pioneering work by Haxby et al. (2001)suggested that categorical information aboutobjects is represented in a distributed mannerthroughout the ventral temporal lobe. Activitypatterns in this region could accurately dis-criminate between multiple object categories,even when the most strongly category-selectivevoxels were removed from the analysis. Ineffect, the authors could perform “virtuallesions” on these activity patterns, and theythereby revealed the distributed nature ofobject information (but see also Spiridon& Kanwisher 2002). Curiously, however,subsequent studies found that activity patternsin low-level visual areas could outperformhigh-level object areas at telling apart viewed


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


objects (Cox & Savoy 2003). How was thispossible, given that low-level visual areas areprimarily tuned to the retinotopic position oflow-level features? These results indicate thatthe images in each object category differedin some of their low-level properties and thatthese low-level confounds can persist evenwhen multiple images are shown in a stimulusblock. Although low-level confounds can bereduced by manipulations of object size or 3Dvantage point, they might not be eliminated,as indicated by the fact that early visual areascan still classify an object across changes in sizeand 3D viewpoint (Eger et al. 2008).

These findings reveal a core challenge forfMRI decoding studies. Pattern classifiers arequite powerful and will try to leverage any dis-criminating information that is present in brainactivity patterns. Even if a brain area can dis-tinguish between certain object images, howcan one go further to show that a brain area isgenuinely sensitive to object properties and notsimply the low-level features of those objects?

Work by Kanwisher and colleagues hasprovided several lines of evidence linking theactivity patterns in object-selective areas toobject perception. In a study of backward visualmasking, they found that activity patterns inobject-selective areas were severely disruptedon trials in which the observer failed torecognize a briefly presented target (Williamset al. 2007). By contrast, activity patterns re-mained stable in early visual areas, despite theparticipant’s impaired performance. Anotherstudy manipulated the physical similarity ofsimple 2D shapes and estimated the perceptualsimilarity between pairs of stimuli based onthe confusion errors that participants madewith visually masked stimulus presentations.Multivariate pattern analysis revealed a strikingdissociation: Activity patterns in the lateraloccipital area reflected the physical similarityof the images, whereas those in the ventraltemporal cortex correlated with perceptualsimilarity (Haushofer et al. 2008). However,other studies have found that activity patternsin the lateral occipital area reflect the perceived3D shape of “bumps” and “dimples” conveyed

by shape-from-shading cues, even when thephysical image is greatly altered by changesin the source of illumination (Gerardin et al.2010).

Activity patterns in the lateral occipital andventral temporal cortices show strong position-invariant selectivity and remain quite stablefor a particular object across changes in retinalposition (Schwarzlose et al. 2008). However,these areas show some evidence of positionselectivity as well. Face- and body-selective ar-eas can better discriminate between pictures ofdifferent body parts if those parts are presentedat a familiar location (Chan et al. 2010). Forexample, a front-view image of a person’s rightshoulder will lead to more reliable activitypatterns if the stimulus appears to the left offixation, as it would if one were looking at thehead or chest, than if it appears to the rightof fixation. It is also possible to decode theretinotopic position of an object from activitypatterns in high-level object areas. Moreover,perceptual illusions that lead to shifts inapparent position are better predicted by theposition information contained in the activitypatterns in high-level object areas than thosein the early visual areas (Fischer et al. 2011).

When objects are subliminally presented toan observer, activity in object-selective areasis greatly attenuated, but somewhat greater-than-chance-level decoding is still possible,indicating the presence of some unconsciousvisual information in these areas (Sterzer et al.2008). Subliminal stimuli also appear to evokemore variable patterns of activity in object-selective areas across repeated presentations,which partly accounts for the poorer decodingof subliminal stimuli (Schurger et al. 2010).

A major challenge in object recognitionconcerns the ability to distinguish a partic-ular exemplar from other items in the samecategory. In an ambitious study, Kriegeskorteand colleagues (2008) presented 92 images ofdifferent real-world objects and assessed whichimages tended to evoke more similar patternsof activity. Images of animate and inanimatestimuli led to broadly distinctive patterns ofactivity in the human ventral temporal cortex,

490 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


and a similar animate/inanimate distinctionwas observed when analyzing neuronal activitypatterns obtained from single-unit recordingsin monkeys (Kiani et al. 2007). This study alsofound evidence of exemplar-specific activity.Activity patterns in the human inferotemporalcortex were better at discriminating betweenimages of different human faces than betweenthe faces of nonhuman primates, whereas atrend toward the opposite pattern of resultswas observed in the monkey data.

Attempts to isolate exemplar-specific in-formation from small cortical regions havemet with limited success, with decoding per-formance reaching levels just slightly greaterthan chance (Kaul et al. 2011, Kriegeskorteet al. 2007). When large portions of the ventraltemporal cortex are pooled for analysis, thenconsiderably better decoding of specific facescan be obtained (Kriegeskorte et al. 2008, Natuet al. 2010). However, it remains to be seenwhether these large-scale distributed repre-sentations are truly important for representingindividual faces or whether the diverse shapecodes throughout this region simply providemore information for the classifier to capitalizeupon when performing these subtle discrim-inations. Single-unit recordings from isolatedface-selective patches in the monkey indicatethat a cluster of a few hundred neighboringneurons can provide remarkably detailed infor-mation for distinguishing between individualfaces (Freiwald et al. 2009, Freiwald & Tsao2010, Tsao et al. 2006). However, currentfMRI technology cannot isolate information atthis level of detail.

Identifying and ReconstructingNovel Visual Scenes

Decoding algorithms can classify a person’sbrain state as belonging to the same categoryas a previously recorded brain state, but thesemethods lack the flexibility to identify novelbrain states. To address this, Kay and col-leagues (2008) devised a visual encoding modelto predict how early visual areas should re-spond to novel pictures of complex real-world

scenes. First, they presented 1,750 differentimages to observers, and from the resultingfMRI data, they were able to characterize theresponse preferences of each voxel in visualcortex, specifying its preference for particularretinotopic locations, spatial frequencies, andorientations. When the observers were latershown a new set of 120 pictures, the modelpredicted how these voxels should respond toeach new image. By comparing the predictedand actual patterns of activity, the modelcorrectly identified 110 out of 120 test imagesfor one participant. In a follow-up experiment,the observer was tested with 1,000 new images,of which 820 were correctly identified.

This level of identification performance isakin to Science Fiction Story #1, identifyingwhich painting the participant is viewing at theMusee d’Orsay. An even loftier goal would beto reconstruct the painting, using only the brainactivity that results from viewing that work ofart. An early attempt at fMRI reconstructionmet with limited success—only small portionsof the simple shapes that were viewed couldbe reconstructed with some degree of accuracy(Thirion et al. 2006). In a more recent fMRIstudy, observers were presented with hundredsof different random patterns of flickeringchecks placed within a 10x10 square grid, andpattern analysis was used to predict whetheror not any given tile of the grid was flickering(Miyawaki et al. 2008). Using this model, theauthors could effectively reconstruct novelstimuli shown to the participant, includ-ing simple shapes and letters (Figure 4a).Moreover, the authors could reconstruct theviewed stimulus from single brain volumes toshow how this information evolved over thetime course of the BOLD response (Figure 4b).Extending the work of Kay et al. (2008),Naselaris et al. (2009) attempted to reconstructcomplex natural scenes using local-featuremodels and were able to capture regions ofhigh contrast and some of the “blurry” low-spatial-frequency components of the image(Figure 4c). By incorporating the category-specific information available in higher-levelobject areas, they could also select an image


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


(from a set of 6 million possible images) thatbest matched the visual features and categoryproperties evoked by the original viewed image(“natural image prior” condition).

Decoding Top-DownAttentional Processes

The ability to decode feature-selective re-sponses has helped advance the study of visualattention and, in particular, feature-basedattention. Kamitani & Tong (2005) showedthat the activity patterns evoked by single ori-entations can predict which of two overlappingorientations is being attended by an observer.Similar results were obtained in studies of atten-tion to overlapping motion stimuli (Kamitani& Tong 2006). These findings indicate thattop-down attention can bias the strengthof feature-selective responses in early visualareas, consistent with models of early atten-tional selection. Serences & Boynton (2007a)demonstrated that attending to one of twooverlapping sets of moving dots leads to biaseddirection-selective responses not only at thesite of the attended stimulus, but also inunstimulated portions of the visual field. Suchspatial spreading of feature-based attention isconsistent with neurophysiological studies inmonkeys (Treue & Maunsell 1996). A recentstudy found that spatial and feature-basedattention can lead to distinct effects in thevisual cortex ( Jehee et al. 2011). When spatialattention was directed to one of two laterallypresented gratings, overall BOLD activity wasenhanced for the attended stimulus, and yetthe orientation-selective component of theseresponses improved only when observers fo-cused on discriminating the orientation of thestimulus rather than its contrast. This suggeststhat enhanced processing of a specific visualfeature may depend more on feature-basedattention than on spatial attention ( Jehee et al.2011, but see also Saproo & Serences 2010).

Recent studies have also investigated thepossible top-down sources of these attentionalsignals. Activity patterns in posterior parietalareas and the frontal eye fields contain reliable

information about whether participants areattending to features or spatial locations(Greenberg et al. 2010) and can even discrimi-nate which of two features or locations is beingattended (Liu et al. 2011). These parietal andfrontal areas could serve as plausible sources ofattentional feedback to early visual areas.

Multivariate pattern analysis has also beenused to quantify the extent to which spatialattention can bias activity in category-selectiveobject areas, for example, when face andhouse stimuli are simultaneously presented indifferent locations (Reddy et al. 2009). Whenobservers view overlapping face-house stimuli,it is possible to decode the focus of object-basedattention from activity patterns in high-levelobject areas as well as in early visual areas,indicating that top-down feedback serves toenhance the local visual features belonging tothe attended object (E.H. Cohen & F. Tong,manuscript under review). Interestingly, at-tending to objects in the periphery leads topattern-specific bias effects in the foveal repre-sentation of early visual areas, perhaps suggest-ing some type of remapping of visual infor-mation or reliance on foveal representations torecognize peripheral stimuli (Williams et al.2008). Pattern classification has also beenused to investigate visual search for objects incomplex scenes. Activity patterns in the lateraloccipital complex can reveal what object cate-gory participants are actively searching for, aswell as those occasions when the target objectbriefly appears at an attended or unattendedlocation (Peelen et al. 2009). Overall, fMRIpattern classification has greatly expandedthe possibilities for studies of visual attentionby providing an effective tool to measureattention-specific signals in multiple brainareas, including parietal and frontal areas.

Decoding Imagery andWorking Memory

In an early fMRI study of mental imagery,O’Craven & Kanwisher (2000) showed thatit was possible to predict with 85% accuracywhether a person was imagining a famous face

492 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


or place by inspecting the strength of activityin the fusiform face area and parahippocampalplace area. A more recent study used MVPA andfound that activity patterns in the ventral tem-poral cortex could predict whether participantswere imagining famous faces, famous buildings,tools, or food items with reasonable accuracy(Reddy et al. 2010). Similar results have been re-ported in studies of working memory for faces,places, and common objects (Lewis-Peacock &Postle 2008). It is also possible to decode the im-agery of simple shapes such as an X or O fromthese object-sensitive visual areas (Stokes et al.2009). In these studies, the activity patternsobserved during imagery or working memorywere very similar to those observed during per-ception, consistent with perception-based the-ories of imagery (Kosslyn et al. 2001). Interest-ingly, it is also possible to distinguish silent clipsof movies that imply distinctive sounds (e.g.,howling dog, violin being played) from activ-ity patterns in the auditory cortex, presumablybecause these visual stimuli elicit spontaneousauditory imagery (Meyer et al. 2010).

Although early visual areas have been impli-cated in visual imagery (Kosslyn & Thompson2003), these areas typically show little evidenceof sustained BOLD activity during visualworking memory tasks (Offen et al. 2008).However, recent fMRI decoding studies haveprovided novel evidence to suggest that earlyvisual areas are important for retaining preciseinformation about visual features (Harrison& Tong 2009, Serences et al. 2009). Serencesand colleagues cued participants in advance toremember either the color or orientation of agrating, and after a 10-second delay, presenteda second grating to evaluate working memoryfor the cued feature. They found that activitypatterns in V1 allowed for prediction of thetask-relevant feature (∼60% accuracy) butnot of the task-irrelevant feature; informationin extrastriate visual areas proved unreliable.Harrison & Tong (2009) used a postcueingmethod to isolate memory-specific activity bypresenting two near-orthogonal gratings at thebeginning of each trial, followed by a cue indi-cating which orientation to retain in working

memory [for a timeline of trial events, refer toFigure 5a]. Activity patterns in areas V1–V4allowed for reliable decoding of the remem-bered orientation (mean accuracy of 83%),and reliable working memory information wasfound in each visual area, including V1 (∼70%to 75% accuracy). Moreover, they foundevidence of a striking dissociation between theoverall amplitude of BOLD activity and thedecoded information contained at individualfMRI time points. Whereas BOLD activity fellover time (Figure 5a), information about theremembered grating was sustained throughoutthe delay period (Figure 5b). In half of theirparticipants, activity in V1 fell to baselinelevels, equivalent to viewing a blank screen, yetdecoding of the retained orientation provedas effective for these participants as for thosewho showed significantly elevated activity latein the delay period. These results suggest thatvisually precise information can be retainedin early visual areas with very little overallincrease in metabolic activity, due to subtleshifts in the patterns of activity in these areas.

Decoding Episodic Memory

Although long-term memories are storedvia modified synaptic connections in thehippocampus and cortex in their inactive state,it is possible to decode these memories whenthey are actively recalled or reinstated bythe participant (for an in-depth review, seeRissman & Wagner 2012). Polyn et al. (2005)had participants study images of famous faces,famous places, and common objects in theMRI scanner and trained pattern classifiers onwhole-brain activity to discriminate betweenthese categories. When participants were laterasked to freely recall these items, the classifierreadily tracked the category that was being re-called from memory (Figure 5c). Remarkably,this category-selective activity emerged severalseconds before participants switched to report-ing items from a new category, suggesting thatthis categorical information might have servedas a reinstated contextual cue to facilitatememory retrieval (Howard & Kahana 1999,


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Tulving & Thomson 1973). Evidence of con-textual reinstatement has even been observedwhen participants fail to recall the studiedcontext ( Johnson et al. 2009). Whole-brainactivity patterns could predict which of threedifferent encoding tasks was performed on anitem at study, based on the reinstated patternsof activity that were later observed duringa recognition memory test. Task-specificpatterns of activity were found for correctlyrecognized items, and this proved true evenfor items that were rated as merely familiar,despite participants’ reports that they could notrecollect any details surrounding the time ofstudying the target item. These findings argueagainst proposed dissociations between con-scious recollection and feelings of familiarity,and further suggest that cortical reinstatementof the studied context might not be sufficientfor experiencing explicit recollection (McDuffet al. 2009). Decoding can also reliably predictwhether an item will be judged as old or new.When participants performed a recognitionmemory task involving faces, multiple brain re-gions responded more strongly to items judgedas old than new, including the lateral andmedial prefrontal cortex and posterior parietalcortex (Rissman et al. 2010). The pooledinformation from these regions could reliablydistinguish between correctly recognized orcorrectly rejected items with 83% mean accu-racy but failed to distinguish missed items fromcorrectly rejected items. Explicit performanceof these recognition memory judgments wasnecessary for decoding, as the classifier couldno longer distinguish between old and newitems when participants instead performed agender discrimination task. The studies de-scribed above reveal how fMRI pattern analysiscan provide a powerful tool for investigatingitem-specific memory processing at the time ofstudy and test and how such data can be used toaddress prevalent theories of memory function.

Decoding can also be used to isolatecontent-specific information from fine-scaleactivity patterns in the human hippocampus.After participants learn the spatial layoutof a virtual environment, decoding applied

to hippocampal activity can reveal somereliable information about the participant’scurrent location in that learned environment(Morgan et al. 2011, Rodriguez 2010). It hasalso been shown that activity patterns in thehippocampus can predict which of three shortmovie clips a participant is engaged in recall-ing from episodic memory (Chadwick et al.2010). Although decoding performance for thehippocampus was modest (∼60% accuracy),activity patterns in this region were found toperform significantly better than neighboringregions of the entorhinal cortex or the posteriorparahippocampal gyrus. The ability to targetspecific episodic memories in the hippocampusmay greatly extend the possibilities for futurestudies of human long-term memory.

Extracting Semantic Knowledge

Semantic knowledge is fundamentally multi-dimensional and often multimodal, consistingof both specific sensory-motor associations andmore abstracted knowledge. For example, weknow that a rose is usually red, has soft petals butsharp thorns, smells sweetly fragrant, and thatthe flowers of this plant make for an excellentgift on Valentine’s Day. Given the multidimen-sional nature of semantic information, multi-variate pattern analysis might be well suited toprobe its neural bases.

An early fMRI study demonstrated that itwas possible to decode whether participantswere viewing words belonging to 1 of 12 pos-sible semantic categories, such as four-leggedanimals, fish, tools, or dwellings (Mitchell et al.2003). Subsequent studies have consistentlyfound that animate and inanimate visual objectslead to highly differentiated patterns of activ-ity in the ventral temporal cortex (Kriegeskorteet al. 2008, Naselaris et al. 2009). Remarkably,people who have been blind since birth exhibita similar animate/inanimate distinction in theventral temporal cortex when presented withtactile objects (Mahon et al. 2009, Pietrini et al.2004), leading to the proposal that this seman-tic differentiation might be innately determinedrather than driven by visual experience (Mahon& Caramazza 2011).

494 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


How might one characterize the broader se-mantic organization of the brain or predict howthe brain might respond to any item based onits many semantic properties? Mitchell et al.(2008) developed a multidimensional seman-tic feature model to address this issue. Theytried to predict brain responses to novel nounsby first quantifying how strongly these nounswere associated with a basis set of semantic fea-tures, consisting of 25 verbs (e.g., see, hear,touch, taste, smell, eat, run). In essence, thesesemantic features served as intermediate vari-ables to map between novel stimuli and pre-dicted brain activity (cf. Kay et al. 2008). Thestrength of the semantic association betweenany noun and these verbs could be estimated onthe basis of their frequency of co-occurrence,from analyzing a trillion-word text corpus pro-vided by Google Inc. Using fMRI activity pat-terns elicited by 60 different nouns, the authorscharacterized the distinct patterns of activityassociated with each verb and could then pre-dict brain responses to novel nouns by assum-ing that the resulting pattern of activity shouldreflect a weighted sum of the noun’s associa-tion to each of the verbs (Figure 6). Usingthis method, Mitchell et al. could predict whichof two nouns (excluded from the training set)was being viewed with 77% accuracy and couldeven distinguish between two nouns belong-ing to the same semantic category with 62%accuracy. The activity patterns for particularverbs often revealed strong sensorimotor asso-ciations. For example, “eat” predicted positiveactivity in frontal regions associated with mouthmovements and taste, whereas “run” predictedactivity in the superior temporal sulcus asso-ciated with the perception of biological mo-tion. These findings are quite consistent withthe predictions of neural network models of se-mantic processing, in which specific items arelinked to multiple associated features throughlearning, and semantically related items are rep-resented by more similar patterns of activatedfeatures (McClelland & Rogers 2003).

Decoding has also been applied to otherdomains of knowledge such as numericalprocessing. One study found that activity

patterns in the parietal cortex reflected notonly spatial attention directed to the left orright side of space, but this spatial bias alsocould be used to predict whether participantswere engaged in a subtraction or additiontask (Knops et al. 2009). Another study foundthat activity patterns in the parietal cortexcould distinguish between different numbers,whether conveyed by digit symbols or dotpatterns (Eger et al. 2009). In general, thesestudies are consistent with the proposal thatnumber representations are strongly associatedwith the parietal lobe and may be representedaccording to an implicit spatial representationof a number line (Hubbard et al. 2005).

Decoding PhonologicalRepresentations andLanguage Processing

Some recent studies have begun to use fMRIdecoding methods to investigate the neuralunderpinnings of phonological and languageprocessing. In one study, participants were pre-sented with audio clips of three different speak-ers uttering each of three different vowel sounds(Formisano et al. 2008). Activity patterns in theauditory cortex could successfully discriminatewhich vowel was heard even when the classifierwas tested on a voice not included in the trainingset. Likewise, pattern classifiers could identifythe speaker at above-chance levels even whentested with vowels not included in the trainingset. Another study showed that activity patternsin the auditory cortex can distinguish betweennormal speech and temporally reorderedversions of these stimuli, implying sensitivityto speech-specific content (Abrams et al. 2011).

Another fruitful approach has been to inves-tigate the role of experience in the developmentof phonological representations. An analysis ofactivity patterns in the auditory cortex revealedbetter discrimination of the syllables /ra/ or /la/in native English speakers than in Japanese par-ticipants who often have difficulty distinguish-ing between these phonemes (Raizada et al.2010). Moreover, the authors found evidenceof a correlation within each group, between an


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


individual’s decoding performance and his orher behavioral ability to distinguish betweenthese phonemes, suggesting that fMRI decod-ing may be sensitive to individual differences inlanguage processing. A recent study of readingability provides further evidence for this view(Hoeft et al. 2011). The authors instructed chil-dren with dyslexia to perform a phonologicalprocessing task in the scanner and later assessedwhether their reading skills had improved twoand a half years later. Although purely behav-ioral measures taken in the first session failed topredict which children would improve in read-ing skills over time, a pattern classifier trainedon the whole-brain data was able to predict im-provement with over 90% accuracy. These re-sults raise the exciting possibility of using fMRIpattern analysis for diagnostic purposes withrespect to language processing.

Decoding Decisions in the Brain

Decoding has revealed that it is possible to pre-dict the decisions that people are likely to make,even in advance of their actual choices. For ex-ample, activity in the anterior cingulate cortex,medial prefrontal cortex, and the ventral stria-tum is predictive of the participants’ choicesin a reward-learning paradigm (Hampton& O’Doherty 2007). Here, one of two stimuliis associated with a higher likelihood of rewardand the other with a lower likelihood, butthese reward probabilities are reversed atunpredictable times. Activity in these areas ishighly predictive of whether participants willswitch their choice of stimulus on a given trial,and activity on the trial prior to a switch is alsosomewhat predictive, indicating an accrual ofinformation over time regarding whether thecurrent regime should be preferred or not. Suchvaluation responses can also be observed in theinsula and medial prefrontal cortex for unat-tended stimuli, and these decoded responsescorrespond quite well to the participants’valuation of as item, such as a particular modelof car (Tusche et al. 2010). fMRI decoding caneven predict participants’ choices of real-worldproducts at greater-than-chance levels. In these

experiments, participants were offered the op-portunity to purchase or decline to purchasea variety of discounted items ranging in valuefrom $8 to $80, with the foreknowledge thattwo of their purchase choices would be realizedat the end of the experiment (Knutson et al.2007). In studies of arbitrary decisions, suchas deciding to press a button with one’s leftor right hand at an arbitrary time, participantsshow evidence of preparatory activity in motorand supplementary motor areas a few seconds inadvance of their action. Remarkably, however,a small but statistically reliable bias in activitycan be observed in the frontopolar cortex up to10 seconds prior to the participant’s response,suggesting some form of preconscious bias inthe decision-making process (Soon et al. 2008).

CONCEPTUAL ANDMETHODOLOGICAL ISSUES

Whenever a new methodology is developed,important conceptual and methodological is-sues can emerge regarding how the data shouldbe analyzed, interpreted, and understood. Pat-tern classification algorithms are statisticallypowerful and quite robust. However, thesevery strengths can pose a challenge, as thealgorithms are designed to leverage whateverinformation is potentially available in a brainregion to make better predictions about a stim-ulus, experimental condition, mental state, orbehavioral response. An example of unwantedleveraging was apparent in one of the reportedresults of the 2006 Pittsburgh Brain Competi-tion (http://pbc.lrdc.pitt.edu/), an open com-petition that was designed to challenge researchgroups to develop state-of-the-art analyticmethods for the purposes of brain reading andmind reading. This competition assessed theaccuracy of decoding the presence of particularactors, objects, spatial locations, and periodsof humor from the time series of fMRI datacollected while participants watched episodesof the TV series “Home Improvement.” Todecode scenes containing humorous events,it turned out that the ventricles proved to bethe most informative region of the brain—this

496 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.

http://pbc.lrdc.pitt.edu/


high-contrast region in the functional imagestended to jiggle whenever the participant felt anurge to laugh. Despite the remarkable accuracyof decoding periods of mirth from this region,it would clearly be wrong to conclude thatthis brain structure has a functional role in thecognitive processing of humorous information.If the accuracy of decoding is not sufficient forestablishing function, then how can one deter-mine precisely what information is processedby a brain region? Below, we consider these andother conceptual and methodological issues.

What Is Being Decoded?

A long-standing problem in fMRI research con-cerns the potential pitfalls of reverse inference.As an example, it is well established that the hu-man amygdala responds more strongly to fear-related stimuli than to neutral stimuli, but itdoes not logically follow that if the amygdala ismore active in a given situation that the personis necessarily experiencing fear (Adolphs 2010,Phelps 2006). If the amygdala’s response variesalong other dimensions as well, such as the emo-tional intensity, ambiguity, or predictive valueof a stimulus, then it will be difficult to makestrong inferences from the level of amygdalaactivity alone.

A conceptually related problem emerges infMRI decoding studies when one identifies abrain region that can reliably discriminate be-tween two particular sensory stimuli or two cog-nitive tasks. For example, Haxby et al. (2001)showed that activity patterns in the humanventral temporal cortex were reliably differentwhen participants viewed images of differentobject categories. The authors interpreted thisdecoding result to suggest that the ventral tem-poral object areas are sensitive to complex ob-ject properties. However, subsequent studiesrevealed that early visual areas could discrim-inate between the object categories just as wellas or better than the high-level object areasbecause of the pervasiveness of low-level dif-ferences between the object categories (Cox &Savoy 2003). Therefore, successful decoding ofa particular property from a brain region, such

as object category, does not necessarily indi-cate that the region in question is truly selectivefor that property. The inferences one can makewith multivariate pattern analysis still dependon strong experimental design, and in manycases multiple experiments may be needed torule out potential confounding factors.

One approach for determining the func-tional relevance of a particular brain area isto test for links between behavioral perfor-mance and decoding performance. For exam-ple, if one compares correct versus incorrecttrials in a fine-grained orientation discrimina-tion task, greater activity in the primary visualcortex is found specifically in those voxels tunedto orientations neighboring the target orienta-tion (Scolari & Serences 2010). Similarly, de-coding of object-specific information from thelateral occipital complex is much better on tri-als with successful than unsuccessful recogni-tion (Williams et al. 2007). Related studies havefound that functional activity patterns in theventral temporal object areas are more reliableand reproducible when a stimulus can be con-sciously perceived than when it is subliminallypresented (Schurger et al. 2010). Interestingly,when participants must study a list of items onmultiple occasions, items that evoke more sim-ilar activity patterns across repeated presenta-tions are also more likely to be remembered(Xue et al. 2010).

Because of the high-dimensional nature ofvisual input, it is possible to investigate thesimilarity of cortical activity patterns acrossa variety of stimulus conditions to assess theproperties they might be attuned to. For ex-ample, similar orientations evoke more similaractivity patterns in early visual areas (Kamitani& Tong 2005), and similar colors have beenfound to do so in visual area V4 (Brouwer& Heeger 2009). However, the similarityrelationships of responses to objects are quitedifferent in early visual areas and high-levelobject areas, with the object areas exhibiting asharp distinction in their activity patterns foranimate and inanimate objects (Kriegeskorteet al. 2008, Naselaris et al. 2009). Studies ofolfactory perception have revealed comparable


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


findings in the posterior piriform cortex, withmore similar odors leading to more similarpatterns of fMRI activity (Howard et al. 2009).Thus, if neural activity patterns share thesimilarity structure of perceptual judgments,this can provide strong evidence to implicatethe functional role of a brain area.

One can further investigate the functionaltuning properties of a brain area by assessinggeneralization performance: Do the activitypatterns observed in a brain area generalize tovery different stimulus conditions or behavioraltasks? In Harrison & Tong’s (2009) study ofvisual working memory, the authors traineda classifier on visual cortical activity patternselicited by unattended gratings and testedwhether these stimulus-driven responses mightbe able to predict which of two orientationswas being maintained in working memorywhile participants viewed a blank screen.Successful generalization was found despitethe differences in both stimulus and task acrossthe experiments, thereby strengthening theinference that orientation-specific informationwas being maintained in the visual cortexduring the working memory task. In a studyof auditory perception, classifiers trained usingphonemes pronounced by one speaker couldsuccessfully generalize to the correspondingphonemes spoken by another speaker, despitechanges in the auditory frequency content(Formisano et al. 2008). Perhaps the mostrigorous test of generalization performancecomes from demonstrations of the ability topredict brain responses to novel stimuli, ashas been shown by Kay and Gallant’s visualencoding model and Mitchell et al.’s semanticencoding model (Kay et al. 2008, Mitchellet al. 2008). Successful generalization can be aneffective tool for ruling out potential low-levelstimulus confounds or task-related factors.

In studies of high-level cognition, isolatingthe specific function of a brain area may be morechallenging if the experimental design focuseson discriminating between two cognitive tasks.When participants perform cognitive tasks dif-fering in the stimuli, task demands, and be-havioral judgments required, almost the entire

cerebral cortex can show evidence of reliablediscriminating activity (Poldrack et al. 2009).Differential activity can result from many fac-tors, including differences in low-level sensorystimulation, working memory load, languagedemands, or the degree of response inhibitionrequired for the task. Even when two tasks arequite closely matched, such as performing ad-dition or subtraction (Haynes et al. 2007) ordirecting attention to features or spatial loca-tions (Greenberg et al. 2010), it is importantto consider potential confounding factors. Ifone task is slightly more difficult or requiresa bit more processing time for a given partic-ipant, then larger or longer fMRI amplitudescould occur on those trials, which could allowdecoding to exceed chance-level performance.This potential confound has sometimes beenaddressed by performing decoding on the av-erage amplitude of activity in a brain region tosee if overall activity is predictive or whethermore fine-grained information is needed for re-liable decoding. Another approach might be toattempt to assess decoding of fast versus slowreaction times using the same brain region andto test whether these activity patterns resemblethose that distinguish the two tasks.

Where in the Brain to Decode From?

Many fMRI decoding studies have focused onthe human visual system, which contains manywell-defined visual areas. In addition, it is com-mon to map the particular region of visual spacethat will be stimulated in an experiment so thatonly the corresponding voxels in the retinotopicvisual cortex are used for decoding analysis.There are several advantages to applying pat-tern analysis to well-defined functional areas.First, localization of function is possible, andthe information contained in each functionalregion can be independently assessed and com-pared to other regions. Second, there is reducedconcern that decoding performance might re-flect information combined across functionallydistinct areas. Finally, decoding performancecan be compared to other known functionalproperties of that brain area to ask whether

498 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


the results seem reasonable and readily inter-pretable. Focused investigations of the humanhippocampus have also benefitted from havinga targeted anatomical locus (Chadwick et al.2010, Hassabis et al. 2009).

In studies of higher-level cognition, prede-fined regions of interest usually are not avail-able, and multiple distributed brain areas mightbe involved in the cognitive task. Many of thesestudies rely on decoding of whole-brain activity,sometimes first selecting the most active voxelsin the task or applying a method to reduce thedimensionality of the data (e.g., principal com-ponents analysis) prior to classification analy-sis. [When selecting a subset of voxels priorto the decoding analysis, it is important to en-sure that the selection process is independentof the property to be decoded so it will notbias decoding performance to be better thanit should (Kriegeskorte et al. 2009).] The ad-vantage of the whole-brain approach lies in itsability to reveal a majority of the informationavailable throughout the brain. Moreover, it ispossible to inspect the pattern of “weights” inthe classifier and to project these onto the cor-tex to reveal how this information is distributedthroughout the brain. For example, Polyn andcolleagues (2005) found that that fusiform facearea was one of the regions most active dur-ing the free recall of famous faces, whereas theparahippocampal place area and retrosplenialcortex were most active during the recall of fa-mous places. Thus, decoding of whole-brain ac-tivity can reveal what information is present inthe brain and where in the brain such informa-tion is most densely concentrated.

However, classification analysis implicitlyassumes a “readout mechanism,” in whichrelative differences between the strengths ofparticular brain signals are calculated andleveraged to compute useful information.It is not clear whether the brain is actuallycomparing or combining the neural signals thatare being analyzed by the classifier, especiallywhen information from distinct brain regionsis combined. For example, a semantic modelmight find that the word “rose” leads to whole-brain activity that is well predicted by the

patterns associated with “smell,” “plants,” and“seeing” vivid colors such as red. Should eachof the respective components of this activitybe considered part of a single unified represen-tation or as entirely separate components thatare being unified outside of the brain by theclassifier (Mahon & Caramazza 2009, Mitchellet al. 2008)? This distinction can be mademore vivid with a somewhat different example.Assume it is possible to decode whether some-thing smells “floral” or “citrus” from activitypatterns in the olfactory piriform cortex, andit is also possible to decode whether the color“red” or “yellow” is being perceived from thevisual cortex. Now, if decoding of whole-brainactivity can tell apart a floral-scented redrose from one that smells like lemon or haslemon-colored petals, can it be argued that thebrain contains a unified representation of thecolor and smell of roses? According to a recentfMRI study of perceptual binding (Seymouret al. 2009), establishing evidence of a conjointrepresentation of color and smell would requiredemonstrating that brain activity patterns candistinguish between a floral-scented red rosepaired with a citrus-scented yellow rose asdistinct from a citrus-scented red rose pairedwith a floral-scented yellow rose. This issuealso points to a longstanding debate regard-ing whether the brain relies on modular ordistributed representations for informationprocessing (Haxby et al. 2001, Op de Beecket al. 2008). Recent fMRI studies indicate thatmany types of information are distributed quitewidely throughout the brain but that therealso exist highly stimulus-selective modulesthat may form a more local, exclusive network(Moeller et al. 2008, Tsao et al. 2006).

An alternative to decoding whole-brain ac-tivity is to perform a searchlight analysis, inwhich decoding is iteratively performed on lo-cal activity patterns sampled throughout thecortex (Kriegeskorte et al. 2006). This typi-cally involves using a moveable searchlight tosample a local “sphere” of voxels (say a 5 ×5 × 5 voxel cubic region) from each point inthe cortex. This approach reveals the informa-tion contained in local activity patterns, which


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


reduces the extent to which information willbe combined across distinct functional areas. Apotential concern is that brain signals fromdisparate areas may sometimes be combinedacross a sulcus, so this approach could be fur-ther strengthened by analyzing activity patternsbased on a flattened representation of the cor-tical surface. A disadvantage of this approachis the need to correct for multiple comparisonsfor each iteration of the search, which reducesstatistical power. For these reasons, searchlightanalyses are often combined with group-levelstatistical analyses to evaluate whether reliableinformation is consistently found in a particularregion of the brain across participants.

At What Spatial Scales of CorticalRepresentation Is DecodingMost Useful?

MVPA may serve different purposes depend-ing on whether the sought-after informationresides at fine or coarse spatial scales in thebrain. At the finest scale, multivoxel patternclassification may be particularly advantageousat detecting signals arising from variability inthe spatial arrangement of cortical columns,which can lead to locally biased signals onthe scale of millimeters (Swisher et al. 2010).Pattern analysis of fine-scale signals has proveneffective not only in the visual cortex but also inhigh-resolution fMRI studies of the hippocam-pus (Hassabis et al. 2009). Such fine-grainedinformation would otherwise be very difficult orimpossible to detect using traditional univariatemethods of analysis. At a somewhat coarserscale, pattern classifiers are also very effective atextracting category-selective information fromthe ventral temporal cortex, which reveals astrong functional organization at spatial scalesof several millimeters to centimeters (Haxbyet al. 2001). These methods can be helpful forpooling distributed information about objectsor semantic categories, particularly when thereis no single “hotspot” of functional selectivityavailable in the broad cortical region to beanalyzed. Decoding has also been applied toactivity patterns of large spatial scale, including

whole-brain activity, even when differen-tially activated regions can be seen usingtraditional univariate analyses such as statisticalparametric mapping. For example, one canattain much better predictions of an ob-server’s near-threshold perceptual judgmentsregarding fearful versus nonfearful faces bypooling information across multiple activatedregions (Pessoa & Padmala 2007). Beyondthe benefits of signal averaging, combiningsignals from multiple regions of interest can bebeneficial if each region contains some uniqueinformation. Another example of whole-braindecoding comes from a recognition memorystudy, which compared participants’ behavioralperformance at old-new judgments with thediscriminating performance of the patternclassifier (Rissman et al. 2010). Although thepatterns picked up by the classifier closelyresembled the statistical maps, the decodinganalysis revealed a compelling relationshipbetween subjective ratings of memory confi-dence and differential brain responses to oldversus new items on individual trials. Theseexamples illustrate how decoding can be usefulwhen applied at large spatial scales. Neverthe-less, interpreting the combined results fromdisparate brain areas can be challenging andmay warrant careful consideration of exactlywhat is being decoded, as we have describedabove.

ETHICAL AND SOCIETALCONSIDERATIONS

What are the potential implications of humanneuroimaging and brain-reading technologiesas this rapidly growing field continues toadvance? Over the past decade, there has beensteadily growing interest in neuroethics, whichfocuses on the current and future implicationsof neuroscience technology on ethics, society,and law (Farah 2005, Roskies 2002). Althoughsome had thought these concerns to bepremature, the intersection between law andneuroscience (sometimes called neurolaw) hasrapidly evolved in recent years ( Jones & Shen2012).

500 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


In October 2009, Dr. Kent Kiehl appearedat a Chicago court hearing to find out whetherthe fMRI scans he had collected of BrianDugan’s brain might be admissible as evidencein a high-profile death penalty case. Dugan,who had already served more than 20 years inprison for two other murders, had recently con-fessed to murdering a 10-year-old girl in 1983,following the discovery of DNA evidence link-ing him to the crime.

On November 5, 2009, the fMRI scans of adefendant’s brain were considered as evidencein the sentencing phase of a murder trial, forwhat appears to be the first time (Hughes 2010).Dr. Kiehl provided expert testimony, describ-ing the results of two psychiatric interviews andthe unusually low levels of activity in several re-gions of Dugan’s brain, similar to levels of manyother criminal psychopaths when they wereshown pictures of violent or morally wrong ac-tions (Harenski et al. 2010). He pointed to theseregions on cartoon drawings of the brain, as thejudge had decided that the presentation of ac-tual brain pictures might unduly influence thejury (Weisberg et al. 2008). Expert testimonyfrom the prosecution refuted the brain imagingdata on two grounds: Dugan’s brain might havebeen very different 26 years ago, and Dr. Kiehl’sneuroimaging studies of criminal psychopathsshowed average trends in the data and werenot designed for individual diagnosis. After lessthan an hour of deliberation, the jury initiallyreached a mixed verdict (10 for and 2 against thedeath penalty), but then asked for more time,switching to a unanimous verdict in favor ofthe death penalty the next day. Dugan’s lawyernoted that although the verdict was unfavor-able, Kiehl’s testimony “turned it from a slamdunk for the prosecution into a much toughercase.”

If courts are primarily concerned that neu-roimaging evidence appears unreliable for indi-vidual diagnosis, then recent advances in brainclassification methods for diagnosing neurolog-ical disorders could lead to the increasing preva-lence of such evidence in courtrooms. Recentstudies have shown that pattern-classificationalgorithms applied to structural MRI scans or

functional MRI scans can distinguish whetheran individual is a normal control or is a patientsuffering from schizophrenia (Nenadic et al.2009), depression (Craddock et al. 2009), orpsychopathy (Sato et al. 2011), with reportedaccuracy levels ranging from 80% to 95%. Inthe context of a court case, these accuracy lev-els might be high enough to influence a jury’sdecision. For example, a diagnosis of para-noid schizophrenia might influence decisionsregarding whether a defendant was likely tohave been psychotic at the time of the crime.Although a diagnosis of psychopathy might beunlikely to affect the determination of whethera defendant should be considered guilty basedon his or her actions, such evidence could proveto be an influential mitigating factor duringthe sentencing phase of the trial. As neuro-science continues to advance our understandingof the neural mechanisms that lead to decisionsand actions, neuroscientists and perhaps soci-ety more generally may feel motivated to re-consider our traditional definitions of free willand personal responsibility (for discussions ofthis issue, see Greene & Cohen 2004, Roskies2006, Sapolsky 2004).

Brain classification methods for individualdiagnosis could have strong ethical implica-tions in medical settings as well, especiallyconcerning disorders of consciousness. Somepatients who partially recover from comaare diagnosed as being in a vegetative stateif they exhibit periods of wakefulness butappear to lack awareness or any purpose intheir motor actions. Despite this apparent lackof awareness, it was recently discovered thatsome vegetative-state patients are capable ofvoluntarily performing mental imagery tasks(Owen et al. 2006). When asked to imagineeither playing tennis or walking around ahouse, differential patterns of activity can beobserved in their brains. Recently, this imageryparadigm has been combined with fMRI de-coding to obtain reliable yes/no responses froma patient to questions such as “Is your father’sname Alexander?” (Monti et al. 2010). If highlyreliable communication can be establishedwith such patients, this could lead to uncharted


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


territories in terms of the ethical and legalconsiderations regarding, for example, anymedical requests made by the patient.

Perhaps the strongest ethical concerns havebeen raised regarding the potential applicationof fMRI decoding to detect lies or the pres-ence of guilty knowledge (Bizzi et al. 2009).Much attention has focused on recent studiesof lie detection and their claims, as well as theefforts made by private companies to developand market this nascent technology. In a studyby Langleben and colleagues, participants weregiven two cards in an envelope and asked inadvance to lie whenever they were asked if theyhad one card and to tell the truth about the othercard (Davatzikos et al. 2005). Pattern classifi-cation applied to whole-brain activity revealedthat truths and lies could be distinguished inthis task with 88% accuracy on individual tri-als because of the greater activity observed forlies in multiple areas, including the prefrontalcortex, anterior cingulate, and insula. On thebasis of these findings, some rather bold claimswere made about the prospects of future fMRIlie detection technology. However, it is criticalto note that it is not lying per se that is beingdecoded from these brain areas but rather thecognitive and emotional processes that are as-sociated with lying (Spence et al. 2004). Thus,lie-detection technology suffers the same prob-lem of reverse inference that we have discussedpreviously. Although lying typically leads to theactivation of a certain set of brain areas, the acti-vation of these brain areas does not necessarilyindicate lying. In real world settings, such aswhen a defendant is strongly suspected of com-mitting a crime or feels guilty for having wit-nessed the crime, any questions about the crimemight elicit strong emotional and cognitive re-sponses akin to those evoked by lying. It is alsonot clear whether criminals, particularly thosewith psychopathy, would show the same activ-ity patterns during lying. Other fMRI studieshave shown that brain activity patterns differfor prepared lies and spontaneous lies (Ganiset al. 2003) and that fMRI lie-detection tech-nology can be subverted by covertly engagingin a separate cognitive task during brain scan-

ning (Ganis et al. 2011). These major short-comings bring into serious question whether itwill be possible to develop an ecologically validand reliable fMRI lie detector anytime in thenear future.

However, this has not prevented the recentefforts of private companies to market suchtechnology or to prepare for their use in court-rooms. In May 2010, the first Daubert hearingwas held in Tennessee to determine whetherfMRI lie detection might be considered admis-sible as scientific evidence (Miller 2010). Dr.Steven Laken, CEO of Cephos, a company thatprovides fMRI lie-detection services, presentedevidence in favor of admitting the brain scans hehad performed on the defendant, which accord-ing to him, indicated innocence on the chargesof fraud. The prosecution invited expert testi-mony from neuroscientist Marcus Raichle andstatistician Peter Imrey to dispute the reliabil-ity of the current technology. In the end, thejudge determined that fMRI lie-detection tech-nology was supported by peer-reviewed pub-lications but had not gained wide acceptanceamong scientists. Moreover, its reliability andaccuracy had yet to be validated in real-worldsettings, and a well-standardized protocol forimplementing such tests had yet to be estab-lished (Shen & Jones 2012).

It remains to be seen whether fMRI liedetection will ever improve enough to meetgeneral scientific acceptance or gain admissioninto courts. Nevertheless, it would be prudentto consider the potential ethical and societalramifications of such technology should itimprove to the point that detection accuracy isno longer the primary concern. There wouldbe obvious benefits in a legal setting if accuracywere extremely high. However, mental privacycould face enormous new challenges, in bothlegal settings and beyond, as there has been noprecedent for being able to look into the mindof another human being. Although DNA canbe obtained as evidence from a suspect on thebasis of a court order, brain reading of thoughtsmight fall under the category of testimony,in which case defendants would be pro-tected by the Fifth Amendment. Even so, if the

502 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


technology were ever to develop to near-perfectlevels of accuracy, a refusal to voluntarily submitto fMRI lie detection might be interpreted as animplicit admission of guilt by some juries evenwhen instructed not to make such an interpre-tation. In the worlds of business and personalrelationships, the availability of such technol-ogy could have far-reaching consequences, par-ticularly in situations involving employers andemployees, business partners, or even spouses.Just the existence of such technology and thepressure of being asked to undergo testingcould lead people to disclose information thatthey otherwise would have declined to share.

Given the conceptual challenges of develop-ing reliable fMRI lie detection and the fact thatpeople can use countermeasures to alter theirpatterns of brain activity, we are doubtful thatthe technology will progress to being truly re-liable and ecologically valid. Nonetheless, it isimportant to consider potential implications incase such progress is ever made.

CONCLUDING REMARKS

In recent years, fMRI pattern classification hasled to rapid advances in many areas of cog-nitive neuroscience, encompassing perception,attention, object processing, memory, seman-tics, language processing, and decision making.These methods have allowed neuroimaging re-searchers to isolate feature-selective sensory re-sponses, neural correlates of conscious percep-tion, content-specific activity during attentionand memory tasks, and brain activity patternsthat are predictive of future decisions.

Furthermore, multivariate analyses canbe used to characterize the multidimensionalnature of neural representations, such as thefunctional similarity between object repre-sentations, scene representations, or semanticrepresentations, allowing one to predict howthe brain should respond to novel stimuli.Looking forward, the enhanced sensitivity andinformation content provided by these meth-ods should greatly facilitate the investigationof mind-brain relationships by revealing bothlocal and distributed representations of mentalcontent, functional interactions between brainareas, and the underlying relationships betweenbrain activity and cognitive performance.

Despite, or perhaps because of, the statisticalpower of these analytic tools, careful exper-imentation and interpretation are requiredwhen making inferences about successful de-coding of a stimulus, task, or mental state fromhuman brain activity. The extension of thesemethods into real-world applications couldprove very useful for medical diagnoses andneuroprostheses (Hatsopoulos & Donoghue2009). However, there are major concernsregarding the reliability and ecological validityof current attempts to perform real-worldlie detection. Much more research will beneeded to determine whether such methodsmight be valid or not. Strong ethical consid-erations also revolve around the prospect ofdeveloping reliable lie detection technology,and it would be prudent to consider howmental privacy would be protected if suchtechnology were allowed to gain prominentuse.

DISCLOSURE STATEMENT

The authors are unaware of any affiliation, funding, or financial holdings that might be perceivedas affecting the objectivity of this review.

ACKNOWLEDGMENTS

The authors would like to thank Owen Jones, Yukiyasu Kamitani, Sean Polyn, Elizabeth Counter-man, and Jascha Swisher for helpful comments on earlier versions of this manuscript. The authorswere supported by grants from the National Eye Institute (R01EY017082), the National ScienceFoundation (BCS-0642633), and the Defense Advanced Research Projects Agency.


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


LITERATURE CITED

Abrams DA, Bhatara A, Ryali S, Balaban E, Levitin DJ, Menon V. 2011. Decoding temporal structure in musicand speech relies on shared brain resources but elicits different fine-scale spatial patterns. Cereb. Cortex21:1507–18

Adolphs R. 2010. What does the amygdala contribute to social cognition? Ann. N.Y. Acad. Sci. 1191:42–61Bizzi E, Hyman SE, Raichle ME, Kanwisher N, Phelps EA, et al. 2009. Using Imaging to Identify Deceit: Scientific

and Ethical Questions. Cambridge, MA: Am. Acad. Arts Sci.Blake R, Logothetis NK. 2002. Visual competition. Nat. Rev. Neurosci. 3:13–21Boynton GM, Finney EM. 2003. Orientation-specific adaptation in human visual cortex. J. Neurosci. 23:8781–

87Brouwer GJ, Heeger DJ. 2009. Decoding and reconstructing color from responses in human visual cortex.

J. Neurosci. 29:13992–4003Brouwer GJ, van Ee R. 2007. Visual cortex allows prediction of perceptual states during ambiguous structure-

from-motion. J. Neurosci. 27:1015–23Carlson TA, Schrater P, He S. 2003. Patterns of activity in the categorical representations of objects. J. Cogn.

Neurosci. 15:704–17Chadwick MJ, Hassabis D, Weiskopf N, Maguire EA. 2010. Decoding individual episodic memory traces in

the human hippocampus. Curr. Biol. 20:544–47Chan AW, Kravitz DJ, Truong S, Arizpe J, Baker CI. 2010. Cortical representations of bodies and faces are

strongest in commonly experienced configurations. Nat. Neurosci. 13:417–18Clifford CW, Mannion DJ, McDonald JS. 2009. Radial biases in the processing of motion and motion-defined

contours by human visual cortex. J. Neurophysiol. 102:2974–81Cox DD, Savoy RL. 2003. Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and

classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage 19:261–70Craddock RC, Holtzheimer PE 3rd, Hu XP, Mayberg HS. 2009. Disease state prediction from resting state

functional connectivity. Magn. Reson. Med. 62:1619–28Davatzikos C, Ruparel K, Fan Y, Shen DG, Acharyya M, et al. 2005. Classifying spatial patterns of brain

activity with machine learning methods: application to lie detection. NeuroImage 28:663–68Eger E, Ashburner J, Haynes JD, Dolan RJ, Rees G. 2008. fMRI activity patterns in human LOC carry

information about object exemplars within category. J. Cogn. Neurosci. 20:356–70Eger E, Michel V, Thirion B, Amadon A, Dehaene S, Kleinschmidt A. 2009. Deciphering cortical number

coding from human brain activity patterns. Curr. Biol. 19:1608–15Engel SA, Furmanski CS. 2001. Selective adaptation to color contrast in human primary visual cortex.

J. Neurosci. 21:3949–54Farah MJ. 2005. Neuroethics: the practical and the philosophical. Trends Cogn. Sci. 9:34–40Fischer J, Spotswood N, Whitney D. 2011. The emergence of perceived position in the visual system.

J. Cogn. Neurosci. 23:119–36Formisano E, De Martino F, Bonte M, Goebel R. 2008. “Who” is saying “what”? Brain-based decoding of

human voice and speech. Science 322:970–73Freeman J, Brouwer GJ, Heeger DJ, Merriam EP. 2011. Orientation decoding depends on maps, not columns.

J. Neurosci. 31:4792–804Freiwald WA, Tsao DY. 2010. Functional compartmentalization and viewpoint generalization within the

macaque face-processing system. Science 330:845–51Freiwald WA, Tsao DY, Livingstone MS. 2009. A face feature space in the macaque temporal lobe.

Nat. Neurosci. 12:1187–96Ganis G, Kosslyn SM, Stose S, Thompson WL, Yurgelun-Todd DA. 2003. Neural correlates of different

types of deception: an fMRI investigation. Cereb. Cortex 13:830–36Ganis G, Rosenfeld JP, Meixner J, Kievit RA, Schendan HE. 2011. Lying in the scanner: covert countermea-

sures disrupt deception detection by functional magnetic resonance imaging. NeuroImage 55:312–19Gerardin P, Kourtzi Z, Mamassian P. 2010. Prior knowledge of illumination for 3D perception in the human

brain. Proc. Natl. Acad. Sci. USA 107:16309–14

504 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Goddard E, Mannion DJ, McDonald JS, Solomon SG, Clifford CW. 2010. Combination of subcortical colorchannels in human visual cortex. J. Vis. 10:25

Greenberg AS, Esterman M, Wilson D, Serences JT, Yantis S. 2010. Control of spatial and feature-basedattention in frontoparietal cortex. J. Neurosci. 30:14330–39

Greene J, Cohen J. 2004. For the law, neuroscience changes nothing and everything. Philos. Trans. R. Soc.Lond. B Biol. Sci. 359:1775–85

Hampton AN, O’Doherty JP. 2007. Decoding the neural substrates of reward-related decision making withfunctional MRI. Proc. Natl. Acad. Sci. USA 104:1377–82

Harenski CL, Harenski KA, Shane MS, Kiehl KA. 2010. Aberrant neural processing of moral violations incriminal psychopaths. J. Abnorm. Psychol. 119:863–74

Harrison SA, Tong F. 2009. Decoding reveals the contents of visual working memory in early visual areas.Nature 458:632–35

Hassabis D, Chu C, Rees G, Weiskopf N, Molyneux PD, Maguire EA. 2009. Decoding neuronal ensemblesin the human hippocampus. Curr. Biol. 19:546–54

Hatsopoulos NG, Donoghue JP. 2009. The science of neural interface systems. Annu. Rev. Neurosci. 32:249–66Haushofer J, Livingstone MS, Kanwisher N. 2008. Multivariate patterns in object-selective cortex dissociate

perceptual and physical shape similarity. PLoS Biol. 6:e187Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. 2001. Distributed and overlapping

representations of faces and objects in ventral temporal cortex. Science 293:2425–30Haynes JD, Deichmann R, Rees G. 2005. Eye-specific effects of binocular rivalry in the human lateral genic-

ulate nucleus. Nature 438:496–99Haynes JD, Rees G. 2005a. Predicting the orientation of invisible stimuli from activity in human primary

visual cortex. Nat. Neurosci. 8:686–91Haynes JD, Rees G. 2005b. Predicting the stream of consciousness from activity in human visual cortex.

Curr. Biol. 15:1301–7Haynes JD, Rees G. 2006. Decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7:523–34Haynes JD, Sakai K, Rees G, Gilbert S, Frith C, Passingham RE. 2007. Reading hidden intentions in the

human brain. Curr. Biol. 17:323–28Hoeft F, McCandliss BD, Black JM, Gantman A, Zakerani N, et al. 2011. Neural systems predicting long-term

outcome in dyslexia. Proc. Natl. Acad. Sci. USA 108:361–66Hong SW, Tong F, Seiffert AE. 2011. Direction-selective patterns of activity in human visual cor-

tex reveal common neural substrates for different types of motion. Neuropsychologia. In pressdoi:10.1016/j.neuropsychologia.2011.09.016

Howard JD, Plailly J, Grueschow M, Haynes JD, Gottfried JA. 2009. Odor quality coding and categorizationin human posterior piriform cortex. Nat. Neurosci. 12:932–38

Howard MW, Kahana MJ. 1999. Contextual variability and serial position effects in free recall. J. Exp. Psychol.:Learn. Mem. Cogn. 25:923–41

Hubbard EM, Piazza M, Pinel P, Dehaene S. 2005. Interactions between number and space in parietal cortex.Nat. Rev. Neurosci. 6:435–48

Hughes V. 2010. Science in court: head case. Nature 464:340–42Jehee JF, Brady DK, Tong F. 2011. Attention improves encoding of task-relevant features in the human visual

cortex. J. Neurosci. 31:8210–19Johnson JD, McDuff SG, Rugg MD, Norman KA. 2009. Recollection, familiarity, and cortical reinstatement:

a multivoxel pattern analysis. Neuron 63:697–708Jones OD, Shen FX. 2012. Law and neuroscience in the United States. In International Neurolaw: A Comparative

Analysis, ed. TM Spranger. New York: Springer. In pressKamitani Y, Tong F. 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci.

8:679–85Kamitani Y, Tong F. 2006. Decoding seen and attended motion directions from activity in the human visual

cortex. Curr. Biol. 16:1096–102Kaul C, Rees G, Ishai A. 2011. The gender of face stimuli is represented in multiple regions in the human

brain. Front. Hum. Neurosci. 4:238


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Kay KN, Naselaris T, Prenger RJ, Gallant JL. 2008. Identifying natural images from human brain activity.Nature 452:352–55

Kiani R, Esteky H, Mirpour K, Tanaka K. 2007. Object category structure in response patterns of neuronalpopulation in monkey inferior temporal cortex. J. Neurophysiol. 97:4296–309

Knops A, Thirion B, Hubbard EM, Michel V, Dehaene S. 2009. Recruitment of an area involved in eyemovements during mental arithmetic. Science 324:1583–85

Knutson B, Rick S, Wimmer GE, Prelec D, Loewenstein G. 2007. Neural predictors of purchases. Neuron53:147–56

Kosslyn SM, Ganis G, Thompson WL. 2001. Neural foundations of imagery. Nat. Rev. Neurosci. 2:635–42Kosslyn SM, Thompson WL. 2003. When is early visual cortex activated during visual mental imagery?

Psychol. Bull. 129:723–46Kriegeskorte N. 2011. Pattern-information analysis: from stimulus decoding to computational-model testing.

NeuroImage 56:411–21Kriegeskorte N, Formisano E, Sorger B, Goebel R. 2007. Individual faces elicit distinct response patterns in

human anterior temporal cortex. Proc. Natl. Acad. Sci. USA 104:20600–5Kriegeskorte N, Goebel R, Bandettini P. 2006. Information-based functional brain mapping. Proc. Natl. Acad.

Sci. USA 103:3863–68Kriegeskorte N, Mur M, Ruff DA, Kiani R, Bodurka J, et al. 2008. Matching categorical object representations

in inferior temporal cortex of man and monkey. Neuron 60:1126–41Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI. 2009. Circular analysis in systems neuroscience: the

dangers of double dipping. Nat. Neurosci. 12:535–40Lewis-Peacock JA, Postle BR. 2008. Temporary activation of long-term memory supports working memory.

J. Neurosci. 28:8765–71Liu T, Hospadaruk L, Zhu DC, Gardner JL. 2011. Feature-specific attentional priority signals in human

cortex. J. Neurosci. 31:4484–95Lu HD, Chen G, Tanigawa H, Roe AW. 2010. A motion direction map in macaque V2. Neuron 68:1002–13Mahon BZ, Anzellotti S, Schwarzbach J, Zampini M, Caramazza A. 2009. Category-specific organization in

the human brain does not require visual experience. Neuron 63:397–405Mahon BZ, Caramazza A. 2009. Concepts and categories: a cognitive neuropsychological perspective. Annu.

Rev. Psychol. 60:27–51Mahon BZ, Caramazza A. 2011. What drives the organization of object knowledge in the brain? Trends Cogn.

Sci. 15:97–103Mannion DJ, McDonald JS, Clifford CW. 2009. Discrimination of the local orientation structure of spiral

Glass patterns early in human visual cortex. NeuroImage 46:511–15McClelland JL, Rogers TT. 2003. The parallel distributed processing approach to semantic cognition.

Nat. Rev. Neurosci. 4:310–22McDuff SG, Frankel HC, Norman KA. 2009. Multivoxel pattern analysis reveals increased memory targeting

and reduced use of retrieved details during single-agenda source monitoring. J. Neurosci. 29:508–16Meyer K, Kaplan JT, Essex R, Webber C, Damasio H, Damasio A. 2010. Predicting visual stimuli on the basis

of activity in auditory cortices. Nat. Neurosci. 13:667–68Miller G. 2010. Science and the law. fMRI lie detection fails a legal test. Science 328:1336–37Misaki M, Kim Y, Bandettini PA, Kriegeskorte N. 2010. Comparison of multivariate classifiers and response

normalizations for pattern-information fMRI. NeuroImage 53:103–18Mitchell TM, Hutchinson R, Just MA, Niculescu RS, Pereira F, Wang X. 2003. Classifying instantaneous

cognitive states from fMRI data. AMIA Annu. Symp. Proc. 2003:465–69Mitchell TM, Shinkareva SV, Carlson A, Chang KM, Malave VL, et al. 2008. Predicting human brain activity

associated with the meanings of nouns. Science 320:1191–95Miyawaki Y, Uchida H, Yamashita O, Sato MA, Morito Y, et al. 2008. Visual image reconstruction from

human brain activity using a combination of multiscale local image decoders. Neuron 60:915–29Moeller S, Freiwald WA, Tsao DY. 2008. Patches with links: a unified system for processing faces in the

macaque temporal lobe. Science 320:1355–59Monti MM, Vanhaudenhuyse A, Coleman MR, Boly M, Pickard JD, et al. 2010. Willful modulation of brain

activity in disorders of consciousness. N. Engl. J. Med. 362:579–89

506 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Morgan LK, Macevoy SP, Aguirre GK, Epstein RA. 2011. Distances between real-world locations are repre-sented in the human hippocampus. J. Neurosci. 31:1238–45

Naselaris T, Kay KN, Nishimoto S, Gallant JL. 2011. Encoding and decoding in fMRI. NeuroImage 56:400–10Naselaris T, Prenger RJ, Kay KN, Oliver M, Gallant JL. 2009. Bayesian reconstruction of natural images

from human brain activity. Neuron 63:902–15Natu VS, Jiang F, Narvekar A, Keshvari S, Blanz V, O’Toole AJ. 2010. Dissociable neural patterns of facial

identity across changes in viewpoint. J. Cogn. Neurosci. 22:1570–82Nenadic I, Sauer H, Gaser C. 2009. Distinct pattern of brain structural deficits in subsyndromes of schizophre-

nia delineated by psychopathology. NeuroImage 49:1153–60Norman KA, Polyn SM, Detre GJ, Haxby JV. 2006. Beyond mind-reading: multi-voxel pattern analysis of

fMRI data. Trends Cogn. Sci. 10:424–30Obermayer K, Blasdel GG. 1993. Geometry of orientation and ocular dominance columns in monkey striate

cortex. J. Neurosci. 13:4114–29O’Craven KM, Kanwisher N. 2000. Mental imagery of faces and places activates corresponding stiimulus-

specific brain regions. J. Cogn. Neurosci. 12:1013–23Offen S, Schluppeck D, Heeger DJ. 2008. The role of early visual cortex in visual short-term memory and

visual attention. Vis. Res. 49:1352–63Op de Beeck HP, Haushofer J, Kanwisher NG. 2008. Interpreting fMRI data: maps, modules and dimensions.

Nat. Rev. Neurosci. 9:123–35O’Toole AJ, Jiang F, Abdi H, Penard N, Dunlop JP, Parent MA. 2007. Theoretical, statistical, and practical

perspectives on pattern-based classification approaches to the analysis of functional neuroimaging data.J. Cogn. Neurosci. 19:1735–52

Owen AM, Coleman MR, Boly M, Davis MH, Laureys S, Pickard JD. 2006. Detecting awareness in thevegetative state. Science 313:1402

Peelen MV, Fei-Fei L, Kastner S. 2009. Neural mechanisms of rapid natural scene categorization in humanvisual cortex. Nature 460:94–97

Pereira F, Mitchell T, Botvinick M. 2009. Machine learning classifiers and fMRI: a tutorial overview.NeuroImage 45:S199–209

Pessoa L, Padmala S. 2007. Decoding near-threshold perception of fear from distributed single-trial brainactivation. Cereb. Cortex 17:691–701

Phelps EA. 2006. Emotion and cognition: insights from studies of the human amygdala. Annu. Rev. Psychol.57:27–53

Pietrini P, Furey ML, Ricciardi E, Gobbini MI, Wu WH, et al. 2004. Beyond sensory images: object-basedrepresentation in the human ventral pathway. Proc. Natl. Acad. Sci. USA 101:5658–63

Poldrack RA, Halchenko YO, Hanson SJ. 2009. Decoding the large-scale structure of brain function byclassifying mental states across individuals. Psychol. Sci. 20:1364–72

Polyn SM, Natu VS, Cohen JD, Norman KA. 2005. Category-specific cortical activity precedes retrievalduring memory search. Science 310:1963–66

Preston TJ, Li S, Kourtzi Z, Welchman AE. 2008. Multivoxel pattern selectivity for perceptually relevantbinocular disparities in the human brain. J. Neurosci. 28:11315–27

Raizada RD, Tsao FM, Liu HM, Kuhl PK. 2010. Quantifying the adequacy of neural representations for across-language phonetic discrimination task: prediction of individual differences. Cereb. Cortex 20:1–12

Reddy L, Kanwisher NG, VanRullen R. 2009. Attention and biased competition in multi-voxel object repre-sentations. Proc. Natl. Acad. Sci. USA 106:21447–52

Reddy L, Tsuchiya N, Serre T. 2010. Reading the mind’s eye: decoding category information during mentalimagery. NeuroImage 50:818–25

Rissman J, Greely HT, Wagner AD. 2010. Detecting individual memories through the neural decoding ofmemory states and past experience. Proc. Natl. Acad. Sci. USA 107:9849–54

Rissman J, Wagner AD. 2012. Distributed representations in memory: insights from functional brain imaging.Annu. Rev. Psychol. In press

Rodriguez PF. 2010. Neural decoding of goal locations in spatial navigation in humans with fMRI.Hum. Brain Mapp. 31:391–97


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Roskies A. 2002. Neuroethics for the new millenium. Neuron 35:21–23Roskies A. 2006. Neuroscientific challenges to free will and responsibility. Trends Cogn. Sci. 10:419–23Sapolsky RM. 2004. The frontal cortex and the criminal justice system. Philos. Trans. R. Soc. Lond. B Biol. Sci.

359:1787–96Saproo S, Serences JT. 2010. Spatial attention improves the quality of population codes in human visual cortex.

J. Neurophysiol. 104:885–95Sasaki Y, Rajimehr R, Kim BW, Ekstrom LB, Vanduffel W, Tootell RB. 2006. The radial bias: a different

slant on visual orientation sensitivity in human and nonhuman primates. Neuron 51:661–70Sato JR, de Oliveira-Souza R, Thomaz CE, Basilio R, Bramati IE, et al. 2011. Identification of psychopathic

individuals using pattern classification of MRI images. Soc. Neurosci. May 14:1–13Schurger A, Pereira F, Treisman A, Cohen JD. 2010. Reproducibility distinguishes conscious from noncon-

scious neural representations. Science 327:97–99Schwarzlose RF, Swisher JD, Dang S, Kanwisher N. 2008. The distribution of category and location infor-

mation across object-selective regions in human visual cortex. Proc. Natl. Acad. Sci. USA 105:4447–52Scolari M, Serences JT. 2010. Basing perceptual decisions on the most informative sensory neurons.

J. Neurophysiol. 104:2266–73Serences JT, Boynton GM. 2007a. Feature-based attentional modulations in the absence of direct visual

stimulation. Neuron 55:301–12Serences JT, Boynton GM. 2007b. The representation of behavioral choice for motion in human visual cortex.

J. Neurosci. 27:12893–99Serences JT, Ester EF, Vogel EK, Awh E. 2009. Stimulus-specific delay activity in human primary visual

cortex. Psychol. Sci. 20:207–14Seymour K, Clifford CW, Logothetis NK, Bartels A. 2009. The coding of color, motion, and their conjunction

in the human visual cortex. Curr. Biol. 19:177–83Seymour K, Clifford CW, Logothetis NK, Bartels A. 2010. Coding and binding of color and form in visual

cortex. Cereb. Cortex 20:1946–54Shen FX, Jones OD. 2011. Brain scans as evidence: truths, proofs, lies, and lessons. Mercer Law Rev. 62:In

pressShmuel A, Chaimow D, Raddatz G, Ugurbil K, Yacoub E. 2010. Mechanisms underlying decoding at 7

T: ocular dominance columns, broad structures, and macroscopic blood vessels in V1 convey informationon the stimulated eye. NeuroImage 49:1957–64

Smith AT, Kosillo P, Williams AL. 2011. The confounding effect of response amplitude on MVPA perfor-mance measures. NeuroImage 56:525–30

Soon CS, Brass M, Heinze HJ, Haynes JD. 2008. Unconscious determinants of free decisions in the humanbrain. Nat. Neurosci. 11:543–45

Spence SA, Hunter MD, Farrow TF, Green RD, Leung DH, et al. 2004. A cognitive neurobiological accountof deception: evidence from functional neuroimaging. Philos. Trans. R. Soc. Lond. B Biol. Sci. 359:1755–62

Spiridon M, Kanwisher N. 2002. How distributed is visual category information in human occipito-temporalcortex? An fMRI study. Neuron 35:1157–65

Sterzer P, Haynes JD, Rees G. 2008. Fine-scale activity patterns in high-level visual areas encode the categoryof invisible objects. J. Vis. 8:101–2

Stokes M, Thompson R, Cusack R, Duncan J. 2009. Top-down activation of shape-specific population codesin visual cortex during mental imagery. J. Neurosci. 29:1565–72

Sumner P, Anderson EJ, Sylvester R, Haynes JD, Rees G. 2008. Combined orientation and colour informationin human V1 for both L-M and S-cone chromatic axes. NeuroImage 39:814–24

Swisher JD, Gatenby JC, Gore JC, Wolfe BA, Moon CH, et al. 2010. Multiscale pattern analysis of orientation-selective activity in the primary visual cortex. J. Neurosci. 30:325–30

Thirion B, Duchesnay E, Hubbard E, Dubois J, Poline JB, et al. 2006. Inverse retinotopy: inferring the visualcontent of images from brain activation patterns. NeuroImage 33:1104–16

Tong F, Meng M, Blake R. 2006. Neural bases of binocular rivalry. Trends Cogn. Sci. 10:502–11Treisman A. 1996. The binding problem. Curr. Opin. Neurobiol. 6:171–78Treue S, Maunsell JH. 1996. Attentional modulation of visual motion processing in cortical areas MT and

MST. Nature 382:539–41

508 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Tsao DY, Freiwald WA, Tootell RB, Livingstone MS. 2006. A cortical region consisting entirely of face-selective cells. Science 311:670–74

Tulving E, Thomson DM. 1973. Encoding specificity and retrieval processes in episodic memory. Psychol.Rev. 80:352–73

Tusche A, Bode S, Haynes JD. 2010. Neural responses to unattended products predict later consumer choices.J. Neurosci. 30:8024–31

Weisberg DS, Keil FC, Goodstein J, Rawson E, Gray JR. 2008. The seductive allure of neuroscience expla-nations. J. Cogn. Neurosci. 20:470–77

Williams MA, Baker CI, Op de Beeck HP, Shim WM, Dang S, et al. 2008. Feedback of visual object informationto foveal retinotopic cortex. Nat. Neurosci. 11:1439–45

Williams MA, Dang S, Kanwisher NG. 2007. Only some spatial patterns of fMRI response are read out intask performance. Nat. Neurosci. 10:685–86

Wunderlich K, Schneider KA, Kastner S. 2005. Neural correlates of binocular rivalry in the human lateralgeniculate nucleus. Nat. Neurosci. 8:1595–602

Xue G, Dong Q, Chen C, Lu Z, Mumford JA, Poldrack RA. 2010. Greater neural pattern similarity acrossrepetitions is associated with better memory. Science 330:97–101

Yacoub E, Harel N, Ugurbil K. 2008. High-field fMRI unveils orientation columns in humans. Proc. Natl.Acad. Sci. USA 105:10607–12

Yamashita O, Sato MA, Yoshioka T, Tong F, Kamitani Y. 2008. Sparse estimation automatically selects voxelsrelevant for the decoding of fMRI activity patterns. NeuroImage 42:1414–29


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


r = -0.12 r = -0.10 r = 0.55r= 0.45

Even

Runs

Odd

Runs

a

Voxel 1

Vo

xe

l 2

b

Figure 1Correlation and classification approaches to decoding brain activity patterns. (a) Average activity patterns for chairs and shoes in theventral temporal cortex, calculated separately for even and odd runs. Correlations between these spatial patterns of activity werecalculated between even and odd runs. Pairwise classifications between any two object categories were considered correct if thecorrelations were higher within an object category than between the two object categories. Adapted with permission from Haxby et al.(2001). (b) Hypothetical responses of two voxels to two different experimental conditions, denoted by red and green points. Densityplots in the margins indicate the distribution of responses to the two conditions for each voxel considered in isolation. The dividing linebetween red and green data points shows the classification results from a linear support vector machine applied to these patterns ofactivity; any points above the line would be classified as red, and those below would be classified as green.

www.annualreviews.org • Decoding Patterns of Human Brain Activity C-1

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


a

b

Figure 2Decoding the orientation of viewed gratings from activity patterns in the visual cortex. (a) Blue curvesindicate the distribution of predicted orientations shown on polar plots, with thick black lines indicating thetrue orientations. Note that common values are plotted at symmetrical directions because stimulusorientation repeats every 180◦. Reproduced with permission from Kamitani & Tong (2005). (b) Spatialdistribution of weak orientation preferences in the visual cortex, measured using high-resolution functionalmagnetic resonance imaging with 1mm isotropic voxels and plotted on an inflated representation of thecortical surface. Reproduced with permission from Swisher et al. (2010).

C-2 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


L

fMR

I sig

nal

-10 0 10 20

-0.5

0

0.5

-10 0 10 20-1

0

1R LR

a b

Figure 3Eye-specific modulation of activity in the lateral geniculate nucleus (LGN) during binocular rivalry.(a) Distribution of weak monocular preferences in the LGN of a representative participant. (b) Time courseof the decoded eye-specific signal from these LGN activity patterns is correlated with fluctuations inperceptual dominance during rivalry between left-eye and right-eye stimuli. Reproduced with permissionfrom Haynes et al. (2005).


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Presented image

Reconstructed

image

Presented

Time (2 sec /image)

Reconstructed

a

b

c Target image Flat prior Sparse prior Image prior

Figure 4Reconstruction of viewed images from activity patterns in the visual cortex, based on averaged fMRI activity patterns (a) and singlefMRI volumes acquired every 2 seconds (b). Reproduced with permission from Miyawaki et al. (2008). (c) Reconstruction of naturalscenes from visual cortical activity. Various methods are used to reconstruct the image’s high-contrast regions (flat prior) orlow-spatial-frequency components (sparse prior), or to select the most visually and semantically similar image to the target from adatabase of 6 million predefined images (image prior). Reproduced with permission from Naselaris et al. (2009).

C-4 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

−0.2

0

0.2

0.4

0.6

0.8

1M

RI s

igna

l cha

nge

(%)

Time (s)

V3AV4

V1V2V3

sample gratings

test grating

cue

a

Visual areas

0 2 4 6 8 10 12 1440

50

60

70

80

Cla

ssifi

catio

n ac

cura

cy (

%)

Time (s)

chance level

sample gratings

test grating

cue

b

V1-V4

Area V1

Time course of BOLD activity Time course of working memory decoding

10 20 30 40 50 60 70 80 900

0.2

0.4

0.6

0.8

1

Time (Scans)

Stu

dy /

Tes

t Mat

ch

Face estimateFace recall

Location estimateLocation recall

Object estimateObject recall

c

Figure 5Decoding item-specific information over time during working memory or free recall from long-term memory. (a) Average time courseof BOLD activity during a visual working memory task in which two oriented gratings were briefly shown, followed by a postcueindicating which orientation to retain until test. Although the mean BOLD signal steadily declined during the memory retentioninterval, decoding accuracy for the retained orientation remained elevated (b) throughout the delay period. Adapted with permissionfrom Harrison & Tong (2009). (c) Classification of the reinstated context during a participant’s free recall of famous faces, famousplaces, and common objects. Dots indicate when the participant verbally reported an item from a given category. Curves showestimates of match between fMRI activity patterns at each time point during free recall, using classifiers trained on activity patternsfrom the prior study period with each of the three categories. Reproduced with permission from Polyn et al. (2005).


Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Predicted

“Celery” = + 0.35 0.84

Predicted Activity

Pattern for “Celery”:

“eat” “taste”

+ 0.32 + …

“fill”

high

below

average

average Observed Pattern

Figure 6Semantic encoding model used to predict brain activity patterns to novel nouns. Neural responses to viewedobjects and their name, such as “celery,” were modeled as the sum of weighted activity patterns tointermediate semantic features consisting of 25 different verbs. Examples of activity patterns for threesemantic features (“eat,” “taste,” and “fill”) are shown, and the weight of their contribution to the predictedactivity pattern reflects their frequency of co-occurrence with the target word. Predicted activity patterns arethen compared with the observed activity for celery. Adapted with permission from Mitchell et al. (2008).

C-6 Tong · Pratte

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.

PS63-FrontMatter ARI 10 November 2011 9:52

Annual Review ofPsychology

Volume 63, 2012 Contents

Prefatory

Working Memory: Theories, Models, and ControversiesAlan Baddeley � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 1

Developmental Psychobiology

Learning to See WordsBrian A. Wandell, Andreas M. Rauschecker, and Jason D. Yeatman � � � � � � � � � � � � � � � � � � � � �31

Memory

Remembering in Conversations: The Social Sharingand Reshaping of MemoriesWilliam Hirst and Gerald Echterhoff � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �55

Judgment and Decision Making

Experimental PhilosophyJoshua Knobe, Wesley Buckwalter, Shaun Nichols, Philip Robbins,

Hagop Sarkissian, and Tamler Sommers � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �81

Brain Imaging/Cognitive Neuroscience

Distributed Representations in Memory: Insights from FunctionalBrain ImagingJesse Rissman and Anthony D. Wagner � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 101

Neuroscience of Learning

Fear Extinction as a Model for Translational Neuroscience:Ten Years of ProgressMohammed R. Milad and Gregory J. Quirk � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 129

Comparative Psychology

The Evolutionary Origins of FriendshipRobert M. Seyfarth and Dorothy L. Cheney � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 153

Emotional, Social, and Personality Development

Religion, Morality, EvolutionPaul Bloom � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 179

vi

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Adulthood and Aging

Consequences of Age-Related Cognitive DeclinesTimothy Salthouse � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 201

Development in Societal Context

Child Development in the Context of Disaster, War, and Terrorism:Pathways of Risk and ResilienceAnn S. Masten and Angela J. Narayan � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 227

Social Development, Social Personality, Social Motivation, Social Emotion

Social Functionality of Human EmotionPaula M. Niedenthal and Markus Brauer � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 259

Social Neuroscience

Mechanisms of Social CognitionChris D. Frith and Uta Frith � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 287

Personality Processes

Personality Processes: Mechanisms by Which Personality Traits“Get Outside the Skin”Sarah E. Hampson � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 315

Work Attitudes

Job AttitudesTimothy A. Judge and John D. Kammeyer-Mueller � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 341

The Individual Experience of UnemploymentConnie R. Wanberg � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 369

Job/Work Analysis

The Rise and Fall of Job Analysis and the Future of Work AnalysisJuan I. Sanchez and Edward L. Levine � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 397

Education of Special Populations

Rapid Automatized Naming (RAN) and Reading Fluency:Implications for Understanding and Treatment of Reading DisabilitiesElizabeth S. Norton and Maryanne Wolf � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 427

Human Abilities

IntelligenceIan J. Deary � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 453

Research Methodology

Decoding Patterns of Human Brain ActivityFrank Tong and Michael S. Pratte � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 483

Contents vii

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.


Human Intracranial Recordings and Cognitive NeuroscienceRoy Mukamel and Itzhak Fried � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 511

Sources of Method Bias in Social Science Researchand Recommendations on How to Control ItPhilip M. Podsakoff, Scott B. MacKenzie, and Nathan P. Podsakoff � � � � � � � � � � � � � � � � � � � � 539

Neuroscience Methods

Neuroethics: The Ethical, Legal, and Societal Impact of NeuroscienceMartha J. Farah � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 571

Indexes

Cumulative Index of Contributing Authors, Volumes 53–63 � � � � � � � � � � � � � � � � � � � � � � � � � � � 593

Cumulative Index of Chapter Titles, Volumes 53–63 � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 598

Errata

An online log of corrections to Annual Review of Psychology articles may be found athttp://psych.AnnualReviews.org/errata.shtml

viii Contents

Ann

u. R

ev. P

sych

ol. 2

012.

63:4

83-5

09. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by V

ande

rbilt

Uni

vers

ity o

n 11

/30/

11. F

or p

erso

nal u

se o

nly.

Decoding Patterns of Human Brain Activity · 2014-11-04 · Abstract Considerable information about mental states can be decoded from noninvasive measures of human brain activity.

Documents