Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects

Exp Brain Res (2007) 179:85–95

RESEARCH ARTICLE

Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects

Ryan A. Stevenson · Marisa L. Geoghegan · Thomas W. James

Received: 5 July 2006 / Accepted: 17 October 2006 / Published online: 16 November 2006© Springer-Verlag 2006

Abstract Evidence from neurophysiological studieshas shown the superior temporal sulcus (STS) to be asite of audio-visual integration, with neuronal responseto audio-visual stimuli exceeding the sum of indepen-dent responses to unisensory audio and visual stimuli.However, experimenters have yet to elicit superaddi-tive (AV > A+V) blood oxygen-level dependent(BOLD) activation from STS in humans using non-speech objects. Other studies have found integration inthe BOLD signal with objects, but only using less strin-gent criteria to deWne integration. Using video clipsand sounds of hand held tools presented at psycho-physical threshold, we were able to elicit BOLD activa-tion to audio-visual objects that surpassed the sum ofthe BOLD activations to audio and visual stimuli pre-sented independently. Our Wndings suggest that theproperties of the BOLD signal do not limit our abilityto detect and deWne sites of integration using stringentcriteria.

Keywords Multisensory · Integration · fMRI · Object recognition · Audio–visual

Introduction

All vertebrates have the ability to extract more thanone type of sensory information from the world aroundthem, and have evolved mechanisms for integrating

across those diVerent sensory modalities, therebyenhancing their ecological success. Humans are not anexception, and our brains easily integrate multiple sen-sory inputs into a single consistent perceptual experi-ence under normal circumstances. In the laboratory,however, sensory stimuli can be presented such thattwo inputs are incongruent and more diYcult to inte-grate. Behavioral studies using this cue-conXict para-digm have uncovered several interesting perceptualphenomena, including “fusion,” for which the two sen-sory inputs are blended into a third distinct percept(McGurk and MacDonald 1976), and “capture”, whichis when one sensory input dominates the other and italone is perceived (MateeV et al. 1985).

Integration can also be studied with congruent sen-sory inputs by measuring behavioral performance withthe combined multisensory inputs and comparing it toperformance with each unisensory input. Evidence ofenhanced performance with combined stimuli has beenfound using a variety of behavioral performance mea-sures (for examples, see Hershenson 1962; Morrell1968). More recently, researchers have become inter-ested in the brain regions and the neural mechanismsthat underlie these behavioral enhancements (Mere-dith and Stein 1983, 1986). Integration has beenstudied using anatomical and neurophysiological tech-niques in non-human primates (Benevento et al. 1977;Bruce et al. 1981; Hikosaka et al. 1988; Barracloughet al. 2005), as well as neuroimaging techniques inhumans (Calvert et al. 2000, 2001; Beauchamp et al.2004a, b; Beauchamp 2005).

Classifying an area of cortex or a subcortical struc-ture as a site of multisensory convergence requiresoperational deWnitions. Meredith (2002) distinguishedbetween two types of convergence, areal and neuronal.

R. A. Stevenson (&) · M. L. Geoghegan · T. W. JamesDepartment of Psychological and Brain Sciences, Indiana University, 1101 East Tenth Street, Room 293, Bloomington, IN 47405, USAe-mail: [email protected]

123

86 Exp Brain Res (2007) 179:85–95

Areal convergence is when two sensory inputs con-verge on one structure, but the information from thoseinputs does not interact, or does not interact to a sig-niWcant degree. Neuronal convergence, or what we willcall integration, is when two sensory inputs convergeon one structure and the information from those inputsdoes interact. This deWnition mirrors what has beenfound in behavioral studies, that combining sensoryinputs produces either a change in the perceptual expe-rience or a change in performance that is diVerent fromwhat would be predicted if the two inputs did not inter-act (Stein et al. 1988).

The distinction between areal convergence and inte-gration is not diYcult to make when recordings fromsingle neurons are available. Many studies utilizing sin-gle-unit recordings in non-human primates operation-alize multisensory integration using the maximum rule.The maximum rule sets the criterion for integration asthe maximum spike count produced by either of thetwo unisensory inputs (S1S2 > S1 \ S2). If the multisen-sory combination stimulus produced a spike count thatexceeds the criterion, then the neuron is considered tobe integrating the two sensory inputs. Brain structurescan be described by the percentage of neurons that arefound to exceed the criterion or the percentage of“multisensory” neurons. Although there are solid the-oretical grounds for using the maximum rule as a crite-rion, most studies investigating multisensory regionssuch as the superior temporal sulcus (STS) and supe-rior colliculus have found augmentation for multisen-sory stimuli in a subset of cells that not only surpassedthe maximum criterion, but that actually surpassed themore stringent superadditive (S1S2 > S1 + S2) criterion(Meredith and Stein 1983, 1986; Barraclough et al.2005).

Because single-unit recordings are diYcult to obtainfrom humans, to extend the investigation of integrationto humans, the Weld has turned to neuroimaging, par-ticularly functional magnetic resonance imaging(fMRI). We will not provide an exhaustive list here,but will instead focus on studies that investigatedaudio-visual integration in the STS, a cortical area thatis known to contain multisensory neurons in non-human primates. Studies of integration in STS havefound enhancements of the blood oxygen-level depen-dent (BOLD) activation that exceed the superadditivecriterion with speech stimuli (Calvert et al. 2000) andwith nonsense stimuli (Calvert et al. 2001), and thatexceed the maximum or mean rule (S1S2 > (S1 + S2)/2)criteria with non-speech objects (Beauchamp et al.2004a, b; Beauchamp 2005). All attempts to Wnd super-additivity in BOLD signals using non-speech objectshave been unsuccessful, and it has been suggested that

the superadditive criterion may be too conservative fornon-speech object stimuli (Beauchamp 2005; Laurientiet al. 2005).

One possible explanation for the lack of superaddi-tive Wndings in BOLD is that fMRI must record from apopulation of neurons and lacks the capability torecord from single neurons. This is a problem becausethe population within STS is not exclusively multisen-sory, but also includes unisensory neurons (Beneventoet al. 1977; Bruce et al. 1981; Hikosaka et al. 1988; Bar-raclough et al. 2005). Because fMRI measures asummed signal across all types of neurons, this mayallow the activation associated with unisensory neu-rons to overpower that of multisensory neurons. Also,the enhanced activation with some multisensory neu-rons may be cancelled out by the attenuated activationwith other multisensory neurons (Beauchamp 2005).This concept has previously been modeled in the supe-rior colliculus (Laurienti et al. 2005). In that model, theBOLD activation with audio-visual stimuli was pre-dicted to exceed the maximum rule, but fall short ofthe superadditivity criterion.

While the neuronal population distribution makessuperadditivity diYcult to obtain using fMRI, it alsomakes the distinction between using superadditivity ormore liberal criterion (e.g., maximum or mean rule)very important in determining if integration occurs. Ifactivation associated with a population is positive withaudio and visual stimuli, where the audio and visualstreams do not interact but act independently, then thearea would not be integrative. In such an area, the acti-vation with audio-visual stimuli would be the linear sumof the activations with audio and visual stimuli pre-sented independently. Therefore, even if the audio andvisual signals do not interact, it would be expected thatthe activation with audio-visual stimuli would exceedthe maximum rule merely by linear summation (Calvert2001). The same is true for the use of the mean rule,which is by deWnition equal to or lower than the maxi-mum rule. As such, showing that a population exceedsthe maximum rule or the mean rule without achievingsuperadditivity provides no evidence of integration.

In contrast, Wnding a population that is superaddi-tive would provide evidence of integration. If the audioand visual signals do not interact, the activation withaudio-visual stimuli would be the linear sum of the acti-vations with audio and visual stimuli, and thus wouldnot achieve superadditivity. Conversely, if superaddi-tive activation was found, the audio and visual signalscould not be orthogonal, and as such would provideevidence that the two signals were interacting. Thisshows that the same population-level measurements ofBOLD that make it diYcult to meet the conservative

123

Exp Brain Res (2007) 179:85–95 87

criteria of superadditivity also make it essential to usesuperadditivity as opposed to more liberal criteria toidentify a multisensory area as a site of integration withBOLD.

A second possible reason that past fMRI studieshave been unable to meet superadditivity is a potentialvascular ceiling eVect in the BOLD signal. If there isstrong activation to both audio and visual stimuli whenthey are presented separately, it is likely that the sumof these two signals will be greater than the level atwhich the BOLD signal asymptotes, even if the under-lying neuronal activity is actually superadditive. Anexample can be seen in a study by Beauchamp and col-leagues (2004a) where BOLD activation with bothaudio and visual unisensory stimuli was approximately2% signal change in multisensory regions. With suchstrong unisensory activations, achieving superadditivitywould require 4% or greater BOLD activation, whichis highly unlikely in a brain region that is not a primarysensory area. In that case, the experimenters would nothave seen superadditivity in the BOLD signal even ifmultisensory neurons in the area were superadditive,but this would have been due to only the constraints ofthe BOLD signal itself.

Despite the potential diYculty of meeting the super-additive criterion using fMRI, superadditivity has beenfound in BOLD activation in two studies to date (Cal-vert et al. 2000, 2001), one with audio-visual speechstimuli and one with audio-visual nonsense stimuli.Thus, the only recognizable stimuli for which superad-ditivity has been found are speech stimuli, but speechstimuli have been shown to elicit unique behavioral(McGurk and MacDonald 1976) and neural properties(Narain et al. 2003); therefore it should not be assumedthat the results can be generalized to other stimulusclasses. This distinction between linguistic and non-lin-guistic information is common, and has been speciW-cally outlined in reference to integration within STSpreviously (Calvert 2001). The problem of generalizingto other stimulus classes is further compounded by thefact that fMRI studies of STS using non-speech objectshave only reported non-superadditive BOLD activa-tions (Beauchamp et al. 2004a, b; Beauchamp 2005).

In the preceding paragraphs, we described two diY-culties with attempting to use a superadditive criterionwith BOLD activation measures. In an attempt toovercome these diYculties, we designed an fMRI studyusing threshold audio and visual stimuli. The Wrst diY-culty was that neural activity from unisensory neuronsmay exert a relatively larger inXuence over the popula-tion neural activity than neural activity from multisen-sory neurons. In single-unit physiology, integrativeresponse ampliWcation is greater with threshold stimuli

than that with supra-threshold stimuli (Meredith andStein 1983, 1986; Perrault et al. 2005; Stanford et al.2005) and is called inverse eVectiveness. Extrapolatingto neural populations suggests that the relative inXu-ence of the multisensory neurons on BOLD activationwill increase as the discriminability of the stimulidecreases. This would potentially increase the chancesof Wnding superadditive BOLD activation with thresh-old stimuli compared with supra-threshold stimuli.

The second diYculty was the vascular ceiling eVectimposed by the BOLD signal. Low-contrast unisensoryvisual stimuli produce less BOLD activation relative tohigh-contrast visual stimuli (Boynton et al. 1996). Weexpect that using visual and auditory stimuli at psycho-physical threshold will produce smaller BOLD activa-tions for both unisensory and multisensory stimuli. Ifunisensory stimuli produce BOLD activations that arewell below vascular ceiling, then it is likely that themultisensory stimuli will also produce an activationthat is below the level of the vascular ceiling. Bringingall of the signals below ceiling will allow a better assess-ment of the superadditive nature of the signals.

Our goal is to obtain superadditive BOLDresponses with non-speech object stimuli in a puta-tively multisensory brain region, the STS. In particular,we will use hand-held tools as stimuli, because theseare similar to stimuli used in previous fMRI studies(Beauchamp et al. 2004a, 2005), as well as non-humanprimate studies (Hikosaka 1988; Barraclough et al.2005), and as such, will allow us to relate our Wndings toprevious multisensory studies of STS.

Methods

Participants

Participants included eight subjects (Wve females, meanage = 23). All subjects were right-handed nativeenglish speakers. Experiment protocol was approvedby the Indiana University Institutional Review Board.

Stimuli

Stimuli consisted of two second digital audio-videorecordings of manual tools (e.g., hammer, saw). Scram-bled versions of the visual stimuli were created. Eachframe was parceled into 20 £ 20 pixel squares andexchanging each square with that which had the closestmean luminosity preserving the spatial distribution ofluminance. Scrambled versions of the audio stimuliwere also created. Each audio stimulus was partitionedand 50% of the waveform was Xipped, scrambling the

123

88 Exp Brain Res (2007) 179:85–95

sound but preserving the temporal dynamics of thenoise.

Procedures

All stimuli were presented using MATLAB 5.2(MATHWORKS Inc., Natick, MA) software with thePsychophysics Toolbox extensions (Brainard 1997;Pelli 1997), running on a Macintosh computer. Visualstimuli were projected onto a frosted glass screen usinga Mitsubishi XL30U projector. Visual stimuli were200 £ 200 pixels and subtended 10.3 £ 10.3 of visualangle.

Participants’ individual psychophysical thresholdswere found while in an MRI simulator designed tomimic the actual fMRI scanner. A three-down-one-upstaircase procedure was used to Wnd participants’ visualand auditory 79% thresholds independently. Partici-pants were presented with both intact and scrambledstimuli and asked to discriminate between the two. Forthe visual task, a grid was placed over both intact andscrambled images to prevent discrimination based onartiWcial contrast gradients produced by scrambling.During the audio task, pre-recorded scanner noise wasplayed at an equal decibel level to the actual scanner.For visual and auditory tasks, dynamic noise (standarddeviation = 0.1 and 0.0118, respectively) overlaid thevisual and auditory stimuli, and was held constantacross staircase levels. Participants completed 200 trialsin each modality, and threshold was determinedaccording to the median level of the Wnal 50 trials ineach modality. Participants then completed 50 trials atthreshold to familiarize themselves with the imagingparadigm.

Each imaging session included two phases: func-tional localizers and experimental scans. Functionallocalizers consisted of non-degraded supra-thresholdintact stimuli presented in a blocked stimulus designwhile participants completed a one-back matchingtask. Runs began with the presentation of a Wxationcross for 12 s followed by six blocks of audio-only (A),visual-only (V), or audio–visual (AV) stimuli. AVstimuli were always congruent with audio and videocomponents presented at the same threshold level.Each run included two blocks of each stimulus type,with blocks consisting of eight, two second stimuli pre-sentations, separated by 0.1 s inter-stimuli intervals(ISI). New blocks began every 24 s separated by Wxa-tion. Runs ended with 12 s of Wxation. Block orderswere counterbalanced across runs and participants.Seven participants completed four localizer runs, whileone participant completed only two due to timerestraints.

During experimental scans, threshold-level intactand scrambled stimuli were presented in a fast event-related design in which participants discriminatedbetween intact and scrambled stimuli. Runs began withthe presentation of a Wxation cross for 12 s, followed byseven trials of each stimulus type, for a total of 49 trialsper run. Stimulus types included intact and scrambledA, V, and AV, as well as Wxation. For the seven trials ofeach stimulus type, four trials were preceded by a twosecond ISI, two preceded by a four second ISI, and oneby a six second ISI. Runs concluded with 12 s of Wxa-tion. Trial and ISI orders were counterbalanced acrossruns and run order was counterbalanced across partici-pants. Seven participants completed six fast event-related runs, and one participant completed only Wvedue to time restraints.

Imaging parameters and analysis

Imaging was carried out using a Siemens MagnetronTrio 3-T whole body scanner, and collected on aneight-channel phased-array head coil. The Weld of viewwas 22 £ 22 £ 9.9 cm, with an in plane resolution of64 £ 64 pixels and 33 axial slices per volume (wholebrain), creating a voxel size of 3.44 £ 3.44 £ 3 mm.Images were collected using a gradient echo EPI(TE = 30 ms, TR = 2000 ms, Xip angle = 70°) forBOLD imaging. High-resolution T1-weighted anatom-ical volumes were acquired using Turbo-Xash 3-D(TI = 1,100 ms, TE = 3.93 ms, TR = 14.375 ms, FlipAngle = 12°) with 160 sagittal slices with a thickness of1 mm and Weld of view of 224 £ 256 (voxelsize = 1 £ 1 £ 1 mm).

Imaging data were pre-processed using Brain Voy-ager™ 3-D analysis tools. Anatomical volumes weretransformed into a common stereotactic space (Talara-ich and Tournoux 1988). Functional data were alignedto the Wrst volume of the run closest in time to the ana-tomical data collection. Each functional run was thenaligned to the transformed anatomical volumes, trans-forming the functional data to a common stereotacticspace across participants. Functional data underwent alinear trend removal, 3-D spatial Gaussian Wltering(FWHM 6 mm), slice scan time correction, and 3-Dmotion correction.

Imaging data were analyzed using Brain Voyager™multi-study general linear model (GLM) procedure.Event-related averages (ERA) were created based onstimulus type for both the localizer and the experimen-tal study using only trials in which subjects respondedaccurately. A deconvolution analysis was also per-formed, resulting in the same pattern of activation asthe ERA analysis. Hemodynamic peaks were deWned

123

Exp Brain Res (2007) 179:85–95 89

as a simple moving average of the time course (2–6 and6–16 s after stimulus presentation for experimental andlocalizer scans respectively,). Findings in the left andright hemisphere exhibited the same pattern of activa-tion, with all Wgures depicting those in the right hemi-sphere.

Results

Each participants’ stimuli contrast level was manipu-lated in the simulator such that accuracy for both Aand V stimuli was 79%. Participants’ accuracy wasrecorded during the experimental scanning in order toensure threshold levels found in the simulator werevalid in the MRI. Participants’ mean accuracy foraudio threshold stimuli during the experimental scan-ning was 77.4%, (SEM = 0.04), and for the visualthreshold stimuli was 78.9% (SEM = 0.03). These didnot signiWcantly diVer from 79% target accuracy(t = 0.43, P = 0.68; t = 0.01, P = 0.99). Mean accuracywith AV trials was 94.5% (SEM = 0.91) in the simula-tor and 93.5% (SEM = 1.84) in the MRI, which did notsigniWcantly diVer (t = 0.51, P = 0.62). These resultsverify that participants were performing similarly inthe simulator and in the MRI.

Three functional regions of interests (ROI) weredeWned on an individual subject basis by performing awhole-brain SPM analysis on the localizer runs (inwhich supra-threshold stimuli were used) and anatomi-cal landmarks (Fig. 1). The visual and auditory ROIswere deWned on a map created by contrasting A and Vstimuli. This comparison was designed to uncover largeareas of cortex that activated more with either A or Vstimuli (i.e., unisensory areas). Within the large areathat activated more with A than V stimuli, the auditoryROI was deWned as a 1,320 mm3 region of cortex(x = 55, y = ¡14, z = 8) along the middle of the supe-rior temporal gyrus (STG), which likely corresponds toprimary and secondary auditory cortex (Semple andScott 2003). Within the large area that activated morewith V than A stimuli, the visual ROI was deWned as a1,204 mm3 region of cortex (x = 42, y = ¡62, z = 0)

along the anterior inferior occipital gyrus, which likelycorresponds to a portion of the lateral occipital com-plex (Malach et al. 1995). Multisensory ROIs weredeWned on a map created by examining the overlap ofregions that activated with A and V stimuli. Within thelarge area that activated with both A and V stimuli,individual multisensory ROIs were deWned as a regionof cortex on the upper bank of STS (Seltzer andPandya 1978; Ungerleider and Desimone 1986). TheseROIs had a mean volume of 735 mm3.

During the experimental runs in which thresholdstimuli were presented, BOLD activations with intactA, V, and AV stimuli were extracted from the multi-sensory ROIs (Fig. 2a). Peak BOLD activation withmultisensory AV stimuli was found to be signiWcantlygreater than the summed peaks with unisensory A andV stimuli in the right (t = 2.33, P = 0.05) (Fig. 2b), andleft (t = 2.58, P < 0.04) hemispheres. Activation withAV was greater than the summed activations with Aand V for seven out of eight participants. Area underthe curve was also analyzed and the result was thesame in the right (t = 2.62, P < 0.04) and left (t = 2.78,P < 0.04) hemispheres.

In addition to the ROI analysis, a random-eVectswhole-brain SPM analysis was also conducted on theexperimental runs, and a superadditive contrast(A+V > AV) was applied. After correcting for multiplecomparisons, there were no clusters that passed statis-tical threshold. Although Wnding a cluster in the STSwould have shown that the eVect described in the ROIanalysis was extremely robust, we did not expect thisWnding, because of the diminished power of whole-brain SPM analyses compared to ROI analyses (Saxeet al. 2006) This Wnding is similar to previous Wndingswhere a whole-brain SPM was also not sensitiveenough to detect superadditivity (Beauchamp 2005).

Peak activations with supra-threshold and thresholdstimuli were compared within each ROI. Within theaudio ROIs, peak BOLD activation with supra-thresh-old stimuli was found to be signiWcantly greater thanwith threshold stimuli for A in the right (t = 16.40,P < 0.0001) and left (t = 15.31, P < 0.0001) hemi-spheres, and with AV in the right (t = 12.99,P < 0.0001) and left (t = 12.76, P < 0.0001) hemi-spheres, but not signiWcantly diVerent for V in eitherthe right (t = 0.13) or left (t = 0.59) hemispheres(Fig. 3a). Within the visual ROIs, peak BOLD activa-tion with supra-threshold stimuli was found to be sig-niWcantly greater than with threshold stimuli for V inthe right (t = 10.03, P < 0.0001) and left (t = 11.47,P < 0.0001) hemispheres, and for AV in the right(t = 14.05, P < 0.0001) and left (t = 11.85, P < 0.0001)hemispheres, but not signiWcantly diVerent with A in

Fig. 1 Unisensory audio, unisensory visual, and multisensory au-dio-visual ROIs. An example subject’s (LS) three ROIs, auditory(A), visual (V), and multisensory (AV)

123

90 Exp Brain Res (2007) 179:85–95

either the right (t = 1.70) or left (t = 1.22) hemispheres(Fig. 3b). Within the multisensory ROIs, peak BOLDactivation with supra-threshold stimuli was found to besigniWcantly greater than to threshold stimuli with Ain the right (t = 3.82, P < 0.005) and left (t = 5.60,P < 0.0005), and with V in the right (t = 8.93,P < 0.0001) and left (t = 9.20, P < 0.001), but not signiW-cantly diVerent with AV in the right (t = 2.01) or left(t = 2.01) hemisphere (Fig. 3c).

In addition, BOLD activation for intact and scram-bled stimuli with A, V, and AV stimuli presented dur-ing the experimental runs (in which threshold stimuliwere presented) were extracted from the ROIs andcompared. Within the audio ROIs, peak BOLD activa-tion with intact stimuli was signiWcantly greater thanwith scrambled stimuli with A in the right (t = 2.97,

P < 0.03) and left (t = 2.55, P < 0.04) hemispheres, andwith AV in the right (t = 3.00, P < 0.03) and left(t = 3.01, P < 0.03) hemispheres, but not signiWcantly

Fig. 2 BOLD activations with unisensory and multisensory stim-uli within the right multisensory ROI across subjects. a Averagetime courses across participants (n = 8) of BOLD activation with-in the multisensory ROI, depicting activations with unisensoryauditory (blue), unisensory visual (green), and multisensory au-dio-visual (red) stimuli. b Peak BOLD activations with unisenso-ry auditory (blue), unisensory visual (green), and multisensoryaudio-visual (red) stimuli, as well as the summed peak of BOLDactivations with unisensory audio and visual (blue and greenstacked) stimuli for comparison. Error bars represent SEM

Fig. 3 Peak BOLD activations with threshold and supra-thresholdstimuli within each right ROI. Peak BOLD activations with unisen-sory audio (blue), unisensory visual (green), and multisensory audio-visual stimuli (red). Solid bars indicate activations with thresholdstimuli and hashed bars indicate activations with supra-thresholdstimuli. BOLD activations are shown within the auditory (a), visual(b), and multisensory (c) ROIs. Error bars represent SEM

123

Exp Brain Res (2007) 179:85–95 91

diVerent with V in either the right (t = 0.15) or left(t = 0.32) hemispheres (Fig. 4a). Within the visualROIs, peak BOLD activation with intact stimuli wasfound to be signiWcantly greater than with scrambled

stimuli with V in the right (t = 2.85, P < 0.03) and left(t = 2.41, P < 0.05), and with AV in the right (t = 3.00,P < 0.03) and left (t = 3.67, P < 0.01) hemispheres, butnot signiWcantly diVerent with A in either the right(t = 0.11) or left (t = 0.25) hemispheres (Fig. 4b).Within the multisensory ROIs, peak BOLD activationwith intact stimuli was not found to be signiWcantlygreater than with scrambled stimuli with A with A ineither the right (t = 0.57) or left (t = 0.81) hemispheres,nor with V in either the right (t = 1.98, P < 0.1) or left(t = 0.56) hemispheres, but was signiWcantly greaterwith AV in the right (t = 2.35, P = 0.04) but not the left(t = 0.92) hemisphere (Fig. 4c).

Discussion

Comparing unisensory and multisensory stimuli

Using threshold non-speech stimuli, we were able toshow the same superadditive increase in BOLD activa-tion in STS that has been previously reported forspeech stimuli (Calvert et al. 2000). To our knowledge,this is the Wrst experiment using non-speech objects toWnd superadditive BOLD activation in any brainregion. This pattern of superadditivity was found bilat-erally, in contrast to Wndings with speech stimuli andnonsense stimuli, which are lateralized to the left hemi-sphere (Calvert et al. 2000, 2001). Other neuroimagingstudies have claimed to Wnd integrative BOLD activa-tion in STS with non-speech objects, but these claimshave been based on more liberal criteria, such as themaximum rule (Beauchamp et al. 2004a) or the meanrule (Beauchamp et al. 2004b). Our Wndings show thatsuperadditivity, which is a more conservative criterion,can be used successfully in fMRI studies with otherclasses of audio-visual multisensory stimuli thanspeech.

Although we are only beginning to understand theneural mechanisms involved in integration, it is clearthat the maximum rule criterion has signiWcance whenapplied to measures of single-unit activity. When amultisensory stimulus causes a neuron to Wre morethan either of the unisensory inputs, information fromthe two inputs must be interacting (Meredith 2002).But, when studying integration in populations of neu-rons, such as with fMRI, diVerent assumptions must bemade. When two sensory inputs converge on one brainregion, they do not necessarily have to interact. Thishas been described elsewhere as areal convergence(Meredith 2002). DiVerent groups of neurons may bepresent in a brain region that may be isolated fromeach other, receiving separate inputs and sending

Fig. 4 Peak BOLD activations with intact and supra-thresholdstimuli within each right ROI. Peak BOLD activations with uni-sensory audio (blue), unisensory visual (green), and multisensoryaudio-visual stimuli (red). Solid bars indicate activations with in-tact stimuli and hashed bars indicate activations with scrambledstimuli. BOLD activations are shown within the auditory (a), vi-sual (b), and multisensory (c) ROIs. Error bars represent SEM

123

92 Exp Brain Res (2007) 179:85–95

separate outputs, without their signals ever interacting.When this idea is applied to the BOLD signal, whichcan only measure populations of neurons, the use ofmore liberal criteria, such as the maximum or meanrule, becomes tenuous. There is evidence that the STSshows multisensory enhancement with non-speechaudio-visual stimuli at a level above the maximum andmean rule criterion (Beauchamp et al. 2004a, 2004b;Beauchamp 2005). This level of activation, however,can be explained without any interaction between theauditory and visual inputs. If STS has one group ofaudio neurons that receive audio-only inputs, and sendoutputs to audio-only areas, and one group of visualneurons that receive visual-only inputs and send outputto visual-only areas, then the activation with audio-visual stimuli would exceed the maximum rule. Thus,the maximum rule criterion cannot distinguish betweenareal convergence and true integration.

The more conservative criterion of superadditivity,which surpasses the level of enhancement that can beexplained by two non-interactive information streams,can distinguish between areal convergence and integra-tion. In the case of areal convergence, the BOLD sig-nal would be the linear sum of the activations withunisensory stimuli. Superadditivity by deWnitionexceeds that sum; therefore, Wnding enhancement thatexceeds the superadditivity criterion implies integra-tion of the two sensory inputs.

Superadditivity has been clearly shown in single-unitrecordings of neurons in STS (Barraclough et al. 2005),leaving little doubt that STS is a site of integration.Superadditive BOLD activation, however, has beendiYcult to achieve. This could be due to vascular ceil-ing eVects in the BOLD signal, to the heterogeneity ofneurons making up the populations, or because BOLDactivation does not reXect neural spiking. By usingthreshold stimuli, we were able to show superadditiveBOLD activation with non-speech objects, suggestingthat BOLD imaging can be used to detect superaddi-tive activation. Furthermore, our Wndings show meanactivation with AV stimuli that was 118% greater thanthe sum of the individual mean activations with A andV, a result that is well within the range of enhance-ments found in individual neurons (Barraclough et al.2005). Together, these results suggest that the lack ofBOLD eVects in previous studies was not due to com-plete insensitivity of BOLD measurements to superad-ditive patterns of neural activity, but rather was due toasymptotic BOLD activation.

It should be noted here that the neural propertiesunderlying the BOLD signal are still not completelyclear. Of particular interest here is the discovery thatthe spiking output of neurons is not the most predictive

measure of BOLD activation (for review, see Heegeret al. (2000); Logothetis et al. (2001); Atwell and Iade-cola (2002); Logothetis (2002, 2003); Logothetis andWandell (2004)). In fact, it appears that the total syn-aptic activity of a neuron is more predictive of BOLDactivation. Thus, care should be taken when attemptingto make predictions about BOLD activation based onsingle-unit recordings and vise versa. The ambiguity ofthe relationship between neural activity and BOLDresponse poses another problem for research on multi-sensory integration: the use of superadditivity as a cri-terion for multisensory integration in single-units isquantitatively sound, because spike counts are mea-sured on a ratio scale, that is, they have a deWned zeropoint. Zero BOLD activation, on the other hand, is notnecessarily related to zero neural activity or even tospontaneous Wring at rest, nor is a resting conditionnecessarily relateable to zero BOLD activity (Starkand Squire 2001; Binder et al. 1999). Thus, until therelationship between neural activity and BOLD activa-tion has been quantiWed, particularly during the restingstate, reports of superadditivity should be interpretedwith caution. To facilitate interpretation of our results,we have presented our data as percent signal changevalues, a scale that is fairly universal for neuroimagingdata. Furthermore, to evince enough statistical powerfrom our design, we used a rapid event-related designwith a distribution of ISIs between 2 and 14 s, exponen-tially distributed (see Methods). This type of distribu-tion of ISIs has been shown in simulations to provideexcellent estimates of BOLD activation (Birn et al.2002; Serences 2004), whether using event-relatedaveraging or deconvolution analysis techniques. For allof the results presented here, we used the averagingtechnique, but using deconvolution produced the samepattern in terms of statistical signiWcance.

Comparing supra-threshold and threshold stimuli

We compared BOLD activations with intact thresholdstimuli to those with the intact supra-threshold stimuli.Decreasing the contrast or intensity of the stimulidecreased the BOLD activation in regions that weresensitive to that sensory modality. For instance, in theauditory ROI, the activation with threshold A stimuliwas less than with supra-threshold A stimuli, whereasthere was no change in activation with V stimuli. In theauditory ROI, the AV stimuli followed the same pat-tern as the A stimuli, presumably because the activa-tion in this ROI was driven mainly by the auditorysignal. Likewise, in the visual ROI, there was a signiW-cant decrease in activation with the threshold stimuliwith V and AV, but no change in activation with A.

123

Exp Brain Res (2007) 179:85–95 93

The multisensory ROI showed a diVerent pattern.BOLD activations were signiWcantly reduced withthreshold A, V, and V stimuli, but were reduced lesswith threshold AV stimuli, compared with supra-threshold counterparts.

It is likely that our ability to detect the superadditiveBOLD activation enhancement with non-speechobjects is related to the decreased activation producedwith threshold stimuli. The reduction in BOLD activa-tion with unisensory stimuli in the multisensory ROIlessened the chance that BOLD activation with multi-sensory stimuli would reach its vascular ceiling. Thesuperadditivity criterion value for threshold stimuli(found by summing the individual activations with theunisensory stimuli) was 0.07% signal change, whichwas substantially lower than 0.24%, the level at whichBOLD activation asymptoted with our supra-thresholdstimuli. These Wndings suggest that the reason for otherfailures to Wnd superaddivity may be related to the vas-cular ceiling eVect in the BOLD signal.

Overcoming the population distribution obstaclerequires a quieting of unisensory activation while notsigniWcantly diminishing multisensory activation. Usingthreshold stimuli reduced activation with unisensorystimuli to a much greater extent than activation withAV stimuli within the multisensory ROI. What causesthis larger activation reduction with speciWcally unisen-sory stimuli when using threshold stimuli is currentlyunknown. A number of studies have shown that multi-sensory neurons respond with a greater superadditiveresponse enhancement with threshold stimuli than withsupra-threshold stimuli due to inverse eVectiveness(Meredith and Stein 1983, 1986; Perrault et al. 2005;Stanford et al. 2005). Therefore, using thresholdinstead of supra-threshold stimuli may have increasedthe relative contribution of multisensory neurons toBOLD activation. Further investigation of the functiondescribing the interaction of stimuli contrast andBOLD activation in STS to both unisensory and multi-sensory stimuli would be a logical step to take in under-standing the multisensory interaction within STS.

Researchers have also suggested that the inability toWnd superadditivity in BOLD activation with non-speech objects may have been due to attenuation neu-rons. Attenuation neurons respond strongly to onespeciWc modality when presented in isolation, but onlyweakly to multisensory stimuli and other unisensorymodalities (Barraclough 2005). As such, researchershave previously theorized that these neuronal responseproperties essentially cancel out the activation associ-ated with superadditive neurons (Beauchamp 2005).However, in our data, there is a greater reduction ofBOLD activation from supra-threshold to threshold

with unisensory stimuli in the right (A = 84%,V = 83%) and left (A = 91%, V = 93%) hemispheresthan with multisensory stimuli in both the right(AV = 32%) and left (AV = 39%) hemispheres(Fig. 3c). This pattern of BOLD activation, while notdirectly measuring neuronal output, suggests that theactivity of attenuation neurons may experience a XooreVect with threshold stimuli, which would lessen theireVect on BOLD activation. Thus, using threshold stim-uli may increase the relative contribution of activityfrom superadditive neurons to BOLD activation, andat the same time diminish the relative contribution ofactivity from attenuation neurons.

Comparing intact and scrambled stimuli

The comparison of intact to scrambled stimuli wasmade to determine if activation in STS is sensitive torecognizable objects, as has been previously suggested(Calvert et al. 2001; Beauchamp 2004b; Amedi et al.2005). DiVerences in BOLD activation of scrambledstimuli and intact stimuli within audio and visual ROIsshowed that activations with scrambled stimuli weresigniWcantly less than those to intact stimuli, but onlywhen the stimuli contained the area’s preferred modal-ity. In the auditory ROI, there was a signiWcantdecrease in activation with the scrambled stimuli withA and AV stimuli, but no change in activation with Vstimuli. Likewise, within the visual ROI, there was asigniWcant decrease in activation with the scrambledstimuli with V and AV stimuli, but no change in activa-tion with A stimuli. Within the multisensory ROI,there was a general overarching pattern of signalreduction with scrambled stimuli. This trend of reduc-tion in STS with scrambled stimuli as compared tointact stimuli is further evidence that STS is moresensitive to identiWable objects (Beauchamp 2004b;Amedi et al. 2005).

While there is evidence for a general reduction inBOLD activation to scrambled stimuli, the superaddi-tive pattern relating activations with unisensory andmultisensory stimuli remained unchanged. The stimuliwere scrambled in such a way that while object identiW-cation was aVected, the temporal and spatial propertiesremained intact and congruent, stimulus propertiesthat have been shown to be deWnitive factors in super-additive neural responses of individual cells (Hershen-son 1962; Morrell 1968; Meredith and Stein 1986, 1996;Meredith et al. 1987) and BOLD activations (Calvertet al. 2000) with multisensory stimuli. The consistencyin the pattern of BOLD activation to unisensory andmultisensory stimuli, even when there is not an identiW-able object, provides evidence that this pattern is not

123

94 Exp Brain Res (2007) 179:85–95

due to object identiWcation, but may be in response tospatially and temporally congruent AV stimuli. Theretention of this pattern of superadditivity in responseto scrambled stimuli further supports the idea thatthese Wndings can be generalized to other stimuli.

Conclusion

Our Wndings demonstrate that STS is a site of multisen-sory integration for non-speech objects. Furthermore,our results demonstrate that superadditivity may notbe an overly stringent criterion for use with fMRI andcan be elicited in known integrative regions such asSTS.

Acknowledgments This research was supported in part by theIndiana METACyt Initiative of Indiana University, funded inpart through a major grant from the Lilly Endowment, Inc.Thanks to Karin James and Laurel Stevenson, as well as JamesTownsend, Ami Eidels, and the Indiana University Neuroimag-ing Group for their insights on this work and manuscript.

References

Amedi A, von Kriegstein K, van Atteveldt NM, Beauchamp MS,Naumer MJ (2005) Functional imaging of human crossmodalidentiWcation and object recognition. Exp Brain Res166:559–571

Attwell D, Iadecola C (2002) The neural basis of functional brainimaging signals. Trends Neurosci 25:621–625

Barraclough NE, Xiao D, Baker CI, Oram MW, Perret DI (2005)Integration of visual and auditory information by superiortemporal sulcus neurons responsive to the sight of actions.J Cogn Neurosci 17:377–391

Beauchamp MS (2005) Statistical criteria in fMRI studies odmultisensory integration. Neuroinformatics 3:93–113

Beauchamp MS, Argall BD, Bordurka J, Duyn JH, Martin A(2004a) Unraveling multisenory integration: patchy organi-zation within human STS multisensory cortex. Nat Neurosci7:1190–1192

Beauchamp MS, Lee KE, Argall BD, Martin A (2004b) Integra-tion of auditory and visual information about objects in supe-rior temporal sulcus. Neuron 41:809–823

Benevento LA, Fallon J, Davis BJ, Rezak M (1977) Auditory-vi-sual interaction in single cells in the cortex of the superiortemporal sulcus and the orbital frontal cortex of the macaquemonkey. Exp Neurol 57:849–872

Binder JR, Frost JA, Hammeke TA, Bellgowan PSF, Rao SM,Cox RW (1999) Conceptual processing during the consciousresting state: a functional fMRI study. J Cogn Neurosci11:80–93

Birn RM, Cox RW, Bandettini PA (2002) Detection versus esti-mation in event-related fMRI: choosing the optimal stimulustiming. Neuroimage 15:252–264

Boynton GM, Engel SA, Glover GH, Heeger DJ (1996) Linearsystems analysis of functional magnetic resonance imaging inhuman V1. J Neurosci 16:4207–4221

Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436

Bruce C, Desimone R, Gross CG (1981) Visual properties of neu-rons in a polysensory area in superior temporal sulcus of themacaque. J Neurophysiol 26:369–384

Calvet GA (2001) Crossmodal processing in the human brain: in-sights from functional neuroimaging studies. Cereb Cortex11:1110–1123

Calvert GA, Campbell R, Brammer MJ (2000) Evidence fromfunctional magnetic resonance imaging of crossmodal bind-ing in the human heteromodal cortex. Curr Biol 10:649–657

Calvert GA, Hansen PC, Iversen SD, Brammer MJ (2001) Detec-tion of audio-visual integration sites in humans by applica-tion of electrophysiological criteria to the BOLD eVect.Neuroimage 14:427–438

Heeger DJ, Huk AC, Geisler WS, Albrecht AG (2000) Spikesversus BOLD: what does neuroimaging tell us about neuro-nal activity? Nat Neurosci 3:631–633

Hershenson M (1962) Reaction time as a measure of intersensoryfacilitation. J Exp Psychol 63:289–293

Hikosaka K, Iwai E, Saito H, Tanaka K (1988) Polysensory prop-erties of neurons in the anterior bank of the caudal superiortemporal sulcus of the macaque monkey. J Neurophysiol60:1615–1637

Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE(2005) On the use of superadditivity as a metric for charac-terizing multisensory integration in functional neuroimagingstudies. Exp Brain Res 166:289–297

Logothetis NK (2002) The neural basis of the blood-oxygen-lev-el-dependent functional magnetic resonance imaging signal.Philos Trans R Soc Lond B Biol Sci 357:1003–1037

Logothetis NK (2003) The underpinnings of the BOLD func-tional magnetic resonance inaging signal. J Neurosci23:3963–3971

Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A(2001) Neurophysiological investigation of the basis of thefMRI signal. Nature 412:150–157

Logothetis NK, Wandell BA (2004) Interpreting the BOLD sig-nal. Annu Rev Physiol 66:735–769

Malach R, Reppas JB, Benson RR, Kwong KK, Jiang H, Ken-nedy WA, Ledden PJ, Brady TJ, Rosen BR, Tootell RB(1995) Object-related activity revealed by functional mag-netic resonance imaging in human occipital cortex. Proc NatlAcad Sci USA 92:8135–8139

MateeV S, Hohnsbein J, Noack T (1985) Dynamic visual capture:apparent auditory motion induced by a moving visual target.Perception 14:721–727

McGurk H, MacDonald J (1976) Hearing lips and seeing voices.Nature 264:746–748

Meredith MA (2002) On the neuronal basis for multisensory con-vergence: a brief overview. Cogn Brain Res 14:31–40

Meredith MA, Nemitz JW, Stein BE (1987) Determinants of mul-tisensory integration in the cat superior colliculus neurons I:temporal factors. J Neurosci 7:3215–3229

Meredith MA, Stein BE (1983) Interactions among convergingsensory inputs in the superior colliculus. Science 221:389–391

Meredith MA, Stein BE (1986) Spatial factors determine theactivity of multisensory neurons in cat superior colliculus.Brain Res 365:350–354

Meredith MA, Stein BE (1996) Spatial determinates of multisen-sory integration in cat superior colliculus. J Neurophysiol75:1843–1857

Morrell LK (1968) Temporal characteristics of sensory interac-tion in choice reaction times. J Exp Psychol 77:14–18

Narain C, Scott SK, Wise RJ, Rosen S, LeV A, Iversen SD,Mathews PM (2003) DeWning a left-lateralized responsespeciWc to intelligible speech using fMRI. Cereb Cortex13:1362–1368

123

Exp Brain Res (2007) 179:85–95 95

Pelli DG (1997) The video toolbox software for visual psycho-physics: transforming numbers into movies. Spat Vis 10:437–442

Perrault TJ Jr, Vaughn JW, Stein BE, Wallace MT (2005) Superiorcolliculus neurons use distinct operational modes in the inte-gration of multisensory stimuli. J Neurophysiol 93:2575–2586

Saxe R, Brett M, Kenwisher N (2006) Divide and conquer: a de-fense of functional localizers. Neuroimage 30:1088–1096

Seltzer B, Pandya DN (1978) AVerent cortical connections andarchitectonics of the superior temporal sulcus and surround-ing cortex. Brain Res 149:1–24

Semple MN, Scott SK (2003) Cortical mechanisms in hearing.Curr Opin Neurobiol 13:167–173

Serences JT (2004) A comparison of methods for characterizingthe event-related BOLD timeseries in rapid fMRI. Neuroim-age 21:1690–1700

Stanford TR, Quessy S, Stein BE (2005) Evaluating the opera-tions underlying multisensory integration in the cat superiorcolliculus. J Neurosci 25:6499–6508

Stark CE, Squire LR (2001) When zero is not zero: the problemof ambiguous baseline conditions in fMRI. Proc Natl AcadSci USA 98:12760–12766

Stein BE, Huneycutt WS, Meredith MA (1988) Neurons andbehavior: the same rules of multisensory integration apply.Brain Res 448:355–358

Talairach J, Tournoux P (1988) Co-planar stereotaxic atlas of thehuman brain. Thieme Medical Publishers, New York

Ungeleider LG, Desimone R (1986) Projections to the superiortemporal sulcus from the central and peripheral Weld repre-sentations of V1 and V2. J Comp Neurol 248:147–163

123