Neural basis of multisensory looming signals

NeuroImage 65 (2013) 13–22

Contents lists available at SciVerse ScienceDirect

NeuroImage

j ourna l homepage: www.e lsev ie r .com/ locate /yn img

Neural basis of multisensory looming signals

Sascha Tyll a,b,⁎, Björn Bonath a,f, Mircea Ariel Schoenfeld c,d, Hans-Jochen Heinze c,Frank W. Ohl d,e, Tömme Noesselt a,e

a Institute of Biological Psychology, Otto-von-Guericke University, 39120 Magdeburg, Germanyb Institute of Cognitive Neurology and Dementia Research, Otto-von-Guericke-University, Leipziger Str. 44, 39120, Magdeburg, Germanyc Department of Neurology, Otto-von-Guericke University, 39120 Magdeburg, Germanyd Leibniz Institute for Neurobiology, 39118 Magdeburg, Germanye Center for Behavioral and Brain Sciences, Magdeburg, Germanyf Department of Child and Adolescent Psychiatry, Psychotherapy and Psychosomatics, Otto-von-Guericke University, 39120, Magdeburg, Germany

⁎ Corresponding author at: Institute of Cognitive NeurOtto-von-Guericke-University, Leipziger Str. 44, 39120, M3916117522.

E-mail address: [email protected] (S. Tyll).

1053-8119/$ – see front matter © 2012 Elsevier Inc. Allhttp://dx.doi.org/10.1016/j.neuroimage.2012.09.056

a b s t r a c t

Article history:Accepted 20 September 2012Available online 29 September 2012

Keywords:Multisensory integrationHumanAudiovisual loomingfMRIConnectivity

Approaching or looming signals are often related to extremely relevant environmental events (e.g. threats or col-lisions)making these signals critical for survival. However, the neural network underlyingmultisensory loomingprocessing is not yet fully understood. Using functional magnetic resonance imaging (fMRI) we identified theneural correlates of audiovisual looming processing in humans: audiovisual looming (vs. receding) signals en-hance fMRI-responses in low-level visual and auditory areas plusmultisensory cortex (superior temporal sulcus;plus parietal and frontal structures). When characterizing the fMRI-response profiles for multisensory loomingstimuli, we found significant enhancements relative to the mean and maximum of unisensory responses inlooming-sensitive visual and auditory cortex plus STS. Superadditive enhancements were observed in visualcortex. Subject-specific region-of-interest analyses further revealed superadditive response profiles within allsensory-specific looming-sensitive structures plus bilateral STS for audiovisual looming vs. summed unisensorylooming conditions. Finally, we observed enhanced connectivity of bilateral STSwith low-level visual areas in thecontext of looming processing. This enhanced coupling of STSwith unisensory regionsmight potentially serve toenhance the salience of unisensory stimulus features and is accompanied by superadditive fMRI-responses. Wesuggest that this preference in neural signaling for looming stimuli effectively informs animals to avoid potentialthreats or collisions.

© 2012 Elsevier Inc. All rights reserved.

Introduction

The ability to discriminate between approaching (looming) or re-ceding events is critical for survival in the environment. From an evolu-tionary perspective it seems to be highly advantageous for animals(including humans) to differentiate between suddenly approaching po-tential threats and spatially disengaging harmless events. A successfuldetection of potential threats gives the organism time to prepare a de-fense or motor response (Bach et al., 2009; Grassi, 2010) and thus in-creases the likelihood of survival. Even today's humans, as members ofthe common road traffic, are exposed to potential threats of suddenlyapproaching objects, to which they should react immediately (Wannet al., 2011).

Concordant with these theoretical considerations, several behavioralstudies reported that the perception of auditory and visual looming andreceding signals differs despite identical overall stimulus energy (auditory

ology and Dementia Research,agdeburg, Germany. Fax: +49

rights reserved.

modality: Gray, 2011; Hall andMoore, 2003; Maier and Ghazanfar, 2007;Schlauch et al., 2001; visualmodality: Franconeri and Simons, 2003; Schiffet al., 1962; Takeuchi, 1997). The only discrepancy between those signalsis their temporal profile: looming signals increase in amplitude or sizeover time whereas for receding signals the temporal pattern is reversed.Thus, these particular looming stimulus dynamics seem to selectivelyenhance perceptual sensitivity and/or a cognitive bias in comparison toreceding signals if presented unimodally (Ghazanfar et al., 2002; Parkerand Alais, 2007). In accord, neural processing of looming stimuli is relatedwith enhanced single cell firing (Liu et al., 2011; Maier and Ghazanfar,2007; Peron and Gabbiani, 2009) and increased fMRI-signals in corticalregions (Seifritz et al., 2002; Wittmann et al., 2010).

However, most everyday events provide information that is pickedup by more than one sense; e.g. for the perception of road traffic werely on both visual and auditory cues and combine them into a unifiedmultimodal percept (Alais et al., 2010; Driver and Noesselt, 2008;Ghazanfar and Schroeder, 2006; Stein and Stanford, 2008). In the caseof multisensory looming signals several behavioral studies reported abehavioral benefit over receding signals, supplementing the resultsof unimodal stimulation. Preferential looking tests demonstrated thatmonkeys (Maier et al., 2004) and young infants (Lewkowicz, 2008;

Fig. 1. Experimental design. Top: schematic illustration of visual and auditory stimuli,shown for looming and receding conditions. During the multisensory conditions visualand auditory stimuli were presented synchronously. Bottom: schematic illustration ofthe visual and auditory target trials. Here the movement streams contained deviants inthemiddle of themovement, which served as targets. Duringmultisensory target presen-tation the onset and offset of the deviants were synchronized between the modalities.

14 S. Tyll et al. / NeuroImage 65 (2013) 13–22

Walker-Andrews and Lennon, 1985) are able to discern betweenmulti-sensory looming and receding signals, thereby pointing to a potentialphylogenetic basis for this effect. Moreover, auditory looming signalshave been related to a behavioral benefit in visual orientation sensitivityin humans (compared to receding or static signals, Leo et al., 2011).Finally, Cappe et al. (2009) reported a selective integration effect formultisensory looming signals in humans with speeded reaction times,enhanced target detection and increased subjective ratings of move-ment. Notably, significantly enhanced detection rates were also foundformultisensory looming stimuli compared to static multisensory stim-uli in that study. Together, these results suggest a selective behavioralgain for multisensory looming signals in humans.

The neuroanatomical and neurophysiological basis of this multisen-sory looming effect is not yet fully understood in mammals. To ourknowledge, only three studies examined the neural underpinnings ofaudio-visual looming processing so far (Cappe et al., 2012; Maier etal., 2008; Romei et al., 2009). However, they only investigated the ef-fects in macaque A1 and STS with intracranial recordings (Maier et al.,2008), or they studied the effect bymeans of TMS-induced phosphenesin humans (Romei et al., 2009), or they investigated the temporal dy-namics of this effect using event-related potentials (ERPs, Cappe et al.,2012) which lack the spatial resolution for a precise identification ofthe neural underpinnings. Thus, the exact network of brain regions re-lated to multisensory looming processing has yet to be discovered. Inthe present study we set out to identify the neural basis of audio-visual looming signals in the human brain using event-related function-al magnetic resonance imaging (fMRI).

In addition, we sought to further characterize the underlying neu-rophysiological response profile within looming-sensitive brain areasby comparing uni- with multisensory looming and receding stimuli,because the exact shape of multisensory response profiles is still amatter of an ongoing debate (e.g. Cappe et al., 2010): previous singlecell studies have reported that the integration of different modalitiescan lead to a non-linear modulation of neural activity in response tomultisensory stimulation relative to the summed unisensory re-sponses (e.g. Stein and Meredith, 1993). However, for population re-sponses (as measured with fMRI) nonlinear response enhancementwith multisensory responses exceeding the sum of the unisensory re-sponses have not always been observed (Beauchamp, 2005b), andsome authors suggested that superadditive responses might only befound with near-threshold stimulation (e.g. Stevenson et al., 2007).Therefore, several alternative analysis approaches for identifyingmultisensory integration have been proposed. These include the com-parison of multisensory responses to the mean of (mean criterion), orthe maximum of (max criterion) unisensory responses (Beauchamp,2005b; Calvert et al., 2001; Love et al., 2011) instead of summedunisensory responses (super-additive criterion). Here, we directlycompared all three criteria for classification of multisensory responseprofiles in response to audio-visual looming stimuli. In addition to thewhole brain voxel-wise group-analysis we complemented our analysisby a subject-specific regions of interest analysis (ROIs) within unisen-sory and putatively multisensory brain regions, because some previousstudies suggested that voxel-wise group-analyses may be relatively in-sensitive for classification of particular response profiles, especiallysuper-additive effects (Beauchamp, 2005b; Stevenson and James, 2009).

Finally and in addition to the identification of looming-sensitive re-gions and characterization of intraregional response profiles we alsotested for the functional network selectively engaged in the processingof audiovisual looming stimuli.

Materials and methods

Participants

Twenty-one (8 female, aged 19–33, mean: 23.9 years, SD: 3.87)right-handed, subjects participated in the fMRI study. Participants

were healthy, had no preexisting neurological or psychiatric disor-ders, reported normal hearing and normal or corrected-to-normal vi-sion and were paid for participation. The study was approved by localethics, and the subjects gave written informed consent. Three sub-jects were excluded from analysis due to scanner malfunctioning orlarge head movements (exceeding 1° and/or 5 mm in abrupt move-ments in more than half of the runs).

Stimuli and procedure

Auditory stimuli consisted of a 3 kHz sinus tone with a rising(looming, 35 dB to 90 dB SPL, abbreviation: AL) or falling (receding,90 dB to 35 dB SPL, abbreviation: AR) amplitude to generate the per-ception of movement (see Fig. 1 top; Audacity was used for stimulusgeneration; http://audacity.sourceforge.net/). Tones (duration:500 ms with 10 ms on/offset ramps to avoid clicks) were presentedvia piezo-electric speakers attached to the top of the scanner boreto maximize audiovisual alignment. Visual stimuli consisted of cen-trally presented black disks, which expanded (looming, abbreviation:VL) or contracted (receding, abbreviation: VR) on a white backgroundover a period of 500 ms (range: 1.2°–12.8°, see Fig. 1 top). Stimuli wereeither presented unimodally or audiovisually (Presentation 9.13—Neurobehavioral Systems) resulting in eight stimulus conditions (2visual, 2 auditory and 4 audiovisual). Other behavioral studies usedalso a static sound condition (see e.g. Cappe et al., 2009). However, forfMRI-research it is essential to keep the stimulation across conditionsas similar as possible. Therefore we chose to only use looming and

15S. Tyll et al. / NeuroImage 65 (2013) 13–22

receding stimuli, which share the identical temporal properties, in ourexperiment.

Audio-visual stimuli were onset-synchronized and were movingcongruently (ALVL or ARVR) or incongruently (VLAR or VRAL; butnote that the results from incongruent trials is beyond the scope ofthis manuscript and will be reported elsewhere). The subjects' taskwas to attend to both visual and auditory modality and to detect infre-quent target stimuli (~12% of all trials) in either modality via buttonpress. Subjects were asked to respond as accurately as possible; re-sponse speed was not emphasized. Target stimuli could appear in anyof the eight conditions (unimodally or audiovisually). These targetsalways contained a deviant in the middle of the movement streams(i.e. 250 ms after stimulus onset). For auditory stimuli a brief 1 kHztone (duration: 16 ms, amplitude: 80 dB) in themiddle of rising/fallingtone served as target (Fig. 1 bottom). For visual stimuli a red circle brief-ly flashed in the middle of the expanding/shrinking visual stimulus(duration: 16 ms, size: 12°) served as target (Fig. 1 bottom). For audio-visual stimuli deviants occurred synchronously in both modalities.

The neural basis of audiovisual looming/receding stimulus process-ing was analyzed using only audiovisual non-target stimuli which re-quired attention to both modalities but was not contaminated by anymotor response. Each non-target condition was presented 16 timesper run in random order. Each trial began with the fixation on a centraldot equivalent in size to the start (looming conditions) or the end(receding conditions) size of the visual stimulus. Visual stimuli wereback-projected on a screen positioned behind the 8-channel head coil,while subjects viewed the stimuli via a mirror attached to the coilreflecting the images displayed on the screen. In order to control fixation,eye movements were monitored online throughout all runs using acustom-built MR-compatible eye-tracking device (Kanowski et al.,2007). Stimuliwere presented in silent interscan periods (800 ms) in be-tween volume acquisitions to avoid interference of scanner noise withauditory stimulus presentation (see e.g. Bonath et al., 2007; Noesselt etal., 2007 for similar experimental designs) with a mean inter-stimulusinterval of 3000 ms (2100–6400 ms, poisson-distributed).

fMRI data acquisition

fMRI-data were acquired on a 3-Tesla MR scanner (SiemensMagnetom Trio, Erlangen, Germany) using an 8-channel phased arrayhead coil (215 volumes in a T2*-weighted echoplanar imaging sequencefield of view, 200×200 mm; repetition time/echo time/flip angle,2000 ms/30 ms/80°; 30 slices; spatial resolution, 3.5×3.5×3.5 mm).Data from five experimental sessions were collected.

Data analysis

Behavioral data analysisWe calculated accuracies for each participant and for each target

condition (see above) to analyzewhether participants properly focusedtheir attention on both modalities. Pairwise t-tests (Bonferroni-corrected, SPSS 13.0, SPSS Inc., Chicago, USA) were used to test for dif-ferences between the target conditions. For completeness, we also ana-lyzed reaction times, but note that we used an unspeeded task andinstructed subjects explicitly to respond as accurately as possibly withno emphasis on response speed. Thus, the accuracy results should bemore informative than reaction times results in our study.

fMRI data analysisPreprocessing and statistical analysis of the functional data were

performed using SPM5 (Wellcome Trust Centre for Neuroimaging,University College London, UK; http://www.fil.ion.ucl.ac.uk/spm). Thefirst four volumes in each session of each participant were discardedto account for T1 equilibrium effects. Data preprocessing consisted ofslice-time-acquisition correction, realignment, spatial normalizationto MNI standard stereotactic reference space (2 mm3, MNI, Montreal

Neurological Institute) and smoothing with a 6 mm full-width athalf-maximum isotropic Gaussian kernel. Condition-related changesin regional brain activity were then modeled with a standard hemody-namic response function (HRF) in the framework of the general linearmodel (GLM, Friston et al., 1995). Per session, nine regressors weremodeled (8 non-target conditions plus one condition for all target trialspooled over conditions). For the second-level group analysis, a repeated-measurement of variance (ANOVA) with all eight non-target conditionsper subject was computed. Group-data were then analyzed using amultistep procedure following two goals: 1. The identification of voxelswhich yielded a stronger activation for audiovisual congruent loomingnon-targets in comparison to congruent receding non-targets (VLAL>VRAR). 2. To further characterize the multisensory response profileswithin these looming sensitive areas we then calculated which voxelsalso fulfilled the following different statistical criteria (Beauchamp,2005b; Gentile et al., 2011; Love et al., 2011; Werner and Noppeney,2010):

• super-additive-criterion (multisensory response is larger than thesum of the unisensory responses): VLAL>(VL+AL)

• max-criterion (multisensory response is larger than the maximumof the unisensory responses): VLAL>VL; if VL>AL or VLAL>AL, ifAL>VL

• mean-criterion (multisensory response is larger than the mean ofunisensory responses): VLAL>(VL*0.5+AL*0.5)

These contrasts were estimated for each subject at the first level andwere entered into independent second-level one-sample t-tests.Resulting t-mapswere overlaid on t-maps for the comparison of and au-diovisual looming minus receding conditions (i.e. looming-sensitiveareas). For all three comparisons results are reportedwith a significancethreshold of pb0.01 and voxel-extent threshold of k=20. Coordinatesof significant peak voxels are reported in a standard stereotactic refer-ence space (MNI, Montreal Neurological Institute). Further in-depth an-atomical identification of activated clusters was obtained using theAnatomy toolbox implemented in SPM5 (Eickhoff et al., 2005).

In addition to the group voxel-wise SPM analysis we also ran cor-roborating additional analyses on subject-specific regions of interest(ROI) within areas expressing a looming-sensitive response patternusing the SPM toolbox RFX-plot (Glascher, 2009). These more sensi-tive individual ROI analyses (Poldrack, 2007) were conducted to fur-ther validate the underlying response profiles obtained with thevoxel-wise group analysis. Regions of interest were limited to areashypothesized to be involved in multisensory integration (Calvert etal., 2001; Noesselt et al., 2008). This included activations in the visualcortex (e.g. Beauchamp, 2005a; Lewis and Noppeney, 2010; Noesseltet al., 2007), the auditory cortex (e.g. Doehrmann et al., 2010;Ghazanfar et al., 2005, 2008) and the STS (e.g. Beauchamp et al.,2004; Marchant et al., 2012; Noesselt et al., 2010). The ROIs for eacharea were selected by identifying the subject-specific peak for thecontrast “VLAL vs. VRAR” within a spherical search volume with a ra-dius of 6 mm centered on the group maxima of the contrast “VLALvs. VRAR” (see Table 1). The individual peak for each subject servedas center of a subject-specific 6 mm sphere. Within these individualROIs themean percent signal change of the fMRI-response amplitudeswere extracted from all voxels within the subject-specific sphere forthe relevant non-target conditions (AL, VL, AR, VR, ALVL, ARVR). Wefirst questioned our data about the most stringent criterion of multi-sensory integration (superadditivity) (Beauchamp, 2005b; Calvert etal., 2001; Stevenson et al., 2007). For this reason we compared theextracted percent signal change of the summed auditory and visualsignals with the audio-visual signals in a repeated measurementANOVA (super-additive-criterion). We tested for effects of the factorsanatomical structure (areas identified within the GLM contrast seeabove), movement direction (looming vs. receding) and non-linearity(summation of auditory and visual signals and audiovisual). To confirm

Table 1The table provides local maxima for the comparison of congruent audiovisual looming signals minus congruent audiovisual receding signals (ALVL>ARVR) threshold at pb0.01,k>20. Anatomical regions were labeled using the Anatomy Toolbox for SPM (Eickhoff et al., 2005). The rightmost column indicates which multisensory statistical criteria weremet by voxels within this region (see Fig. 2B–D).

Brain region Peak T p-Value Cluster size MNI coordinates Statistical criterion

X Y Z

Right calcarine gyrus/cuneus 3.73 0.001 97 22 −74 18 Mean, maxLeft calcarine gyrus/cuneus 3.51 0.001 88 −18 −74 14 Mean, max, super-additiveRight sup. temporal gyrus/planum temporale(including TE 1.1—probability 40%)

3.30 0.001 92 42 −30 10 Mean, max

Left Heschl gyrus/planum temporale(including TE 1.1—probability 70%)

3.45 0.001 132 −34 −32 14 Mean, max

Right sup. temporal sulcus 3.11 0.001 26 62 −28 8 MeanLeft sup. temporal sulcus 3.17 0.001 43 −62 −36 6 Mean, maxLeft sup. parietal lobe(5Ci—probability 50%)

3.25 0.001 34 −18 −32 40 Mean, superadditive

Left cingulate gyrus 3.03 0.001 28 −18 34 12

16 S. Tyll et al. / NeuroImage 65 (2013) 13–22

the criterion of super- or sub-additivity post-hoc paired t-tests wereused to test for differences between the sum of audiovisual looming(VL+AL) and receding (VR+AR) signal with their multisensory coun-terparts (VLAL, VRAR).

However, the exclusive use of the super-additive criterion for theidentification of multisensory integration processes can sometimesbe problematic (Beauchamp, 2005b; Laurienti et al., 2005). Significantsuper-additive response profiles could result from the summation of anegative unisensory response in one brain region (e.g. visual responsein auditory areas) and a second unisensory response that is identicalto the multisensory response (Wright et al., 2003). To rule out thistype of response pattern, we also compared the largest unisensoryresponse of the unisensory looming and receding conditions (VLmax

or ALmax, VRmax or ARmax) against their multisensory counterparts(VLAL, VRAR) using another repeated measurement ANOVA. In thiscase the factors anatomical structure, movement direction (loomingvs. receding) andmax-criterion (maximum of auditory and visual sig-nals vs. audiovisual) were used.

Moreover, to confirm previous observations of enhanced fMRI-signals in auditory cortex in response to auditory looming signals(Seifritz et al., 2002) and to extend this mechanism to visual cortexfor visual looming stimuli we analyzed the percent signal change forthe unisensory looming conditions using a repeated measurementANOVA with the factors anatomical structure (visual and auditorycortex), stimulation/brain area congruence (congruent: visual stimu-lation in visual cortex and auditory stimulation in auditory cortex in-congruent: visual stimulation in auditory cortex and vice versa) andmovement direction (looming vs. receding).

Finally, effective connectivity was assessed using the method ofpsychophysiological interactions (PPI; Friston et al., 1997). The PPIanalysis can be described as the change in contribution of one brainarea (seed point) to another due to a change in experimental manip-ulation or psychological context (Friston et al., 1997). Here all datafrom subject-specific ROIs served as seed points for PPI analyseswith the contrast VLAL minus VRAR as psychological context. There-fore, with these psychophysiological interactions we are able to iden-tify which brain areas (on a voxel-by-voxel basis) show an enhancedcoupling with our ROIs in the context of congruent audiovisuallooming signals compared to congruent audiovisual receding signals.For analysis, new single-subject SPMmodels with all nine experimen-tal conditions (see above) plus two additional regressors for the phys-iological response from the subject-specific defined seed regions andthe products of this physiological response with the psychological in-teraction contrast (psychophysiological interaction) were estimated(Macaluso et al., 2000; Noesselt et al., 2007, 2010). Finally, for eachseed region, individual PPI contrast images were entered into a one-sample t-test at the second level (pb0.001 and voxel-extent thresh-old of k=20).

Results

Task performance

Subjects' accuracy was consistently high (mean accuracy acrossconditions 95%, SD 4%—see Supplemental Table S1). We comparedthe response accuracies for all task conditions to ensure that theparticipants focus their attention on both modalities. Pairwise com-parisons revealed no significant difference between target conditions(all p's>0.05, Bonferroni-corrected), indicating similar deploymentof attentional resources across conditions. False alarm rate was al-ways low (2%) and did not differ across conditions. Additionally, weanalyzed reaction times: here, we found a general effect of stimula-tion modality with audiovisual responses being significantly fasterthan visual (t(17)=9.87; pb0.05, Bonferroni corrected) and auditoryresponses (t(17)=6.74; pb0.05, Bonferroni-corrected, mean visualRT: 775 ms±17 ms; mean auditory RT: 773±20 ms; mean audiovi-sual RT: 696 ms±21 ms). Moreover, we observed a significant effectfor the visual condition with looming stimuli (715±79 ms) yieldingsignificantly faster responses than did the receding stimuli (834 ms±72 ms; t(17)=12,371; pb0.05, Bonferroni corrected), in accord withprevious studies (e.g. Cappe et al., 2009). Comparing auditory and au-diovisual looming with receding stimuli did not result in significantmodulations of reaction times (p's>.20; Bonferroni-corrected). Notehowever, that we emphasized accuracy over response speed in ourtask instruction, which might explain the absence of significant re-sponses in the audiovisual and auditory conditions.

Whole brain voxel-wise group-analysis

We then compared our fMRI-signals for congruent audiovisuallooming stimulation minus congruent audiovisual receding stimula-tion (ALVL>ARVR). Voxel-wise group-analysis revealed enhancedfMRI-signals in low-level visual (bilateral calcarine gyrus, CG) andauditory (including Area TE 1.1, Morosan et al., 2001) brain areas. Inaddition to modulations in sensory-specific regions the bilateralsuperior temporal sulcus (STS), the left superior parietal lobe (SPL,Scheperjans et al., 2008) and the left cingulate gyrus expressed en-hanced fMRI-signals for audiovisual looming stimuli (see Table 1,Fig. 2A). The inverse contrast (ARVR>ALVL) yielded no significantfMRI-modulations in sensory-specific areas nor the STS.

Multisensory response profiles within looming-sensitive regions

The second goal of the current study was to characterize the re-sponse profiles within looming-sensitive areas. Therefore, we inquiredhow the response for unisensory relative to multisensory stimulidiffered in these looming sensitive regions in our voxel-wise group

Fig. 2. A. Significant BOLD-signal increases for audiovisual looming signals relative toaudiovisual receding signals (ALVL>ARVR, top row). Below B–D B: Voxels in yellowshow significant increase for ALVL relative to the mean of AL and VL overlaid on audio-visual looming-sensitive regions in red. C: Voxels in green show significant increase forALVL relative to the maximum of AL and VL overlaid on audiovisual looming-sensitiveregions in red. D: Voxels in purple show significant increase for ALVL relative to thesum of AL and VL overlaid on audiovisual looming-sensitive regions in red.

17S. Tyll et al. / NeuroImage 65 (2013) 13–22

analysis. Significant modulations fulfilling the mean-criterion werefound in almost all looming sensitive areas including parts of the leftand right calcarine gyrus, the left Heschl gyrus (TE 1.1) and the rightsuperior temporal gyrus (including TE 1.1) and left parietal cortex,only the cingulate gyrus did not fulfill the mean criterion (Fig. 2B,pb0.01; svc-corrected; with looming-sensitive regions being the smallvolume). For the max-criterion we observed significant modulationswithin parts of the left STS, in parts of the bilateral auditory regionsand in parts of the bilateral occipital cluster. Here the left and theright calcarine sulcus extending into superior occipital gyrus were acti-vated (Fig. 2C, pb0.01 uncorrected, but see below for a confirmatoryROI-analysis of this exploratory voxel-based analysis). Finally, super-additive responses were found in left calcarine gyrus and the leftsuperior parietal lobe (5Ci). No subadditive voxels were observed inlooming-sensitive regions (Fig. 2D, again pb0.01 uncorrected, but seebelow for a confirmatory ROI-analysis of this exploratory voxel-basedanalysis).

Regions of interest—analysis

Finally, subject-specific ROI analyses were conducted to confirm theobserved group-level voxel-based response patterns, especially since

the results from our (exploratory) voxel-basedmax- and superadditivecomparisons were not corrected for multiple comparisons. Data wereextracted from the following looming-sensitive regions: left visual cor-tex (mean coordinates over subjects:−18,−78, 16), right visual cortex(22,−74, 18), left auditory cortex (−34,−32, 16), right auditory cor-tex (42,−34, 10), left STS (−62,−36, 6), right STS (62,−28, 6). Fron-tal and parietal structures were not included in this analysis, since theanterior cingulate did not fulfill any of the criteria of multisensory inte-gration (see Table 1, Fig. 2), whereas the parietal region did fulfill meanand superadditive criterion but not the max criterion, pointing at apotential confound there. Extracted percent signal change of theremaining six regions (see Fig. 3, bar graphs) was analyzed using athree factorial repeatedmeasurement ANOVAwith the factors anatom-ical structure (left/right calcarine gyrus, left Heschl gyrus, right superiortemporal gyrus, right/left STS),movement direction (looming vs. reced-ing) and non-linearity (sum of auditory and visual vs. audiovisualBOLD-signals). This analysis revealed significant main effects for thefactors structure: F(5,85)=2.81, p=0.02 and movement direction:F(1,17)=85.83, pb0.001. Furthermore, significant interactions were ob-served for the factors structure×movement direction (F(5,85)=3.35,p=0.008) and most importantly for the factors non-linearity×movement direction (F(1,17)=8.03, p=0.001). Please note that theinteraction between the factors non-linearity×movement direction×structure had no significant impact (F(1,17)=0.79, p>0.52), so thedata from the different structures could be pooled for post-hoc testing.Post hoc paired t-tests confirmed a super-additive response pattern formultisensory looming stimuli collapsed across all structures (t(17)=2.76; p=0.01), whereas no effect was found for receding stimuli(p=0.36).

An additional ANOVA, testing the multisensory looming responseagainst the larger of the two unisensory responses revealed virtuallyidentical results to the ANOVA testing for non-linearity (main effects:structure: F(5,85)=3.59, p=0.05; movement direction: F(5,17)=6.64,pb0.001; interaction effects: structure×movement direction: F(5,85)=3.01, p=0.015; max criterion×movement direction: F(1,17)=3.94,p=0.004). Again, the interaction between the factors max-criterion×movement direction×structure had no significant impact (F(1,17)=0.49, p>0.69), so the data from the different structures could be pooledfor post-hoc testing. Post-hoc t-tests confirmed that this effect was dueto higher fMRI-signals in the multisensory condition relative to themaximal unisensory condition (t(17)=2.59, p=0.01) for the loomingconditions. This pattern of results demonstrates that the multisensoryfMRI-signal enhancement exceeds the maximal unisensory signals.

Finally, we tested, whether fMRI-responses to unisensory visualand auditory looming signals exceed unisensory receding signals invisual and auditory brain regions, respectively, in accord with previ-ous studies. A repeated measurement ANOVA revealed a significanteffect of the factor structure (F(3,51)=2.91, p=0.05) and stimulationcongruence (F(1,17)=40.34, pb0.001). Importantly, the factors stimu-lus congruence and movement direction interacted (F(1,17)=9.55,p=0.007), with post-hoc t-test indicating that visual looming signalssignificantly enhance fMRI-signals in visual cortex, whereas auditorylooming signals significantly increase fMRI-responses in auditory cor-tex, in comparison to receding stimuli in the respective unisensorycortices (t(17)=2.15, pb0.05, see Fig. 3A blue bars and Fig. 3B redbars).

Interregional coupling

As a final step, we employed psychophysiological interaction (PPI)analyses to identify changes in interregional effective coupling in thecontext of multisensory looming vs. receding signals (VLAL>VRAR).The subject-specific local maxima of the ROI analyses (see above)served as seed points. For the left and right STS-seed the PPI analysisrevealed enhanced connectivity with bilateral calcarine and lingualgyrus (Fig. 4, Table 2) in the context of audiovisual looming relative

Fig. 3. Signal changes extracted from subject-specific ROIs in left and right unisensory visual (A), auditory (B) and multisensory (C) regions. The bar graphs below brain sectionsdepict the percent signal change for the looming and receding conditions for visual stimulation (VL and VR, dark blue), auditory stimulation (AL and AR, red) and multisensorystimulation (light blue); see text for statistical analysis.

18 S. Tyll et al. / NeuroImage 65 (2013) 13–22

to receding stimulation. Except from the PPI analysis seeded inthe left Heschl gyrus (see Table 2), none of the additional analyses(seeded in the right/left calcarine gyrus and left/right sup. temporalgyrus) showed a significant covariation with other brain regions inthe context of audiovisual looming.

Fig. 4. Results of the PPI analyses with the left and right STS as seed region. Top: schematic selingual gyrus expressing enhanced connectivity with the seed regions in the context of aud

Discussion

The goal of the present study was to identify the neural underpin-nings of audiovisual looming processing in humans with event-relatedfMRI. We found that the audiovisual looming signals enhanced neural

ed region in the left and right STS. Bottom: significant clusters in bilateral calcarine andiovisual looming vs. receding stimulation (see Table 2).

Table 2Psychophysiological interaction (PPI): List of peak voxels for the PPI connectivity results seeded in either right STS, left STS or left Heschl gyrus in the context of audiovisual loomingminus audiovisual receding conditions.

Seed region Remote brain region Peak T pb Cluster size MNI coordinates

X Y Z

Right STS Right lingual gyrus 7.50 0.001 47 10 −60 2Right calcarine gyrus 4.28 0.001 21 12 −76 −4Left lingual gyrus 5.40 0.001 120 −6 −70 0Left calcarine gyrus 4.44 0.001 45 −12 −84 0

Left STS Left calcarine gyrus 6.32 0.001 967 −2 −88 −2Right calcarine gyrus 6.09 0.001 4 −84 4Paracentral lobule 5.41 0.001 118 2 −30 60

Left Heschl gyrus (TE 1.1) Right middle frontal gyrus 5.70 0.001 34 40 48 6

19S. Tyll et al. / NeuroImage 65 (2013) 13–22

processing within a network of low-level visual and auditory areas plusmultisensory convergence zones.With conventional voxel-based groupwe observed increased BOLD-signals in bilateral calcarine sulci, bilateralauditory belt regions (including TE 1.1 bilaterally), and bilateral STSs,plus left-hemispheric parietal and cingulate cortex for the comparisonof audiovisual looming minus receding conditions.

Our results extend previous findings on multisensory looming sig-nals, which were restricted to auditory cortex plus STS (Maier et al.,2008) or visual cortex (Romei et al., 2009) or by electrophysiologicalmethods invoking sourcemodeling (Cappe et al., 2012). Our convention-al voxel-based group analyses further revealed that BOLD-modulationswithin looming-sensitive regions were significantly enhanced relativeto themean andmaximum of unisensory BOLD-signals in visual and au-ditory cortex plus the STS. Moreover, audiovisual looming signals alsoexceeded the sum of BOLD signals evoked by unisensory stimuli in leftvisual and parietal cortex. The parietal modulation, however, failedto fulfill the max-criterion thereby suggesting a different underlyingintegration mechanism within this region. Methodologically, ourvoxel-based results from the three different analysis criteria show thatboth the mean and the max-criterion resemble the modulations ob-served in the direct comparison of looming and receding stimulation,whereas superadditive effects were only observed in a subset oflooming-sensitive regions, and may thus be a very conservative methodwith a considerable risk of false negative results. A more sensitivesubject-specific ROI-analysis extended our findings and revealed super-additive responses in bilateral auditory, visual and STS ROIs, which alsosatisfied the max-criterion, suggesting that ROI-analyses should beused when testing for superadditive effects. Finally, effective connectiv-ity analyses (i.e., interregional functional coupling) revealed that thelooming-sensitive STS expressed enhanced interregional coupling withlow-level visual areas whenever fast approaching multisensory signalswere presented.

The superior temporal sulcus, which showed enhanced connectiv-ity with visual regions in our study, is known to be an important sitefor multisensory integration (Beauchamp et al., 2010; Noesselt et al.,2010; Werner and Noppeney, 2010). It receives input from visualand auditory regions (Padberg et al., 2003; Seltzer and Pandya, 1994),contains multisensory neurons (Barraclough et al., 2005; Beauchampet al., 2004; Benevento et al., 1977) and also seems to be crucially in-volved in the perceptual binding of audiovisual stimulus features(Noesselt et al., 2010; Werner and Noppeney, 2010). Furthermore, theSTS plays an important role in the processing of audiovisual motionstimuli in general (Baumann and Greenlee, 2007; Sadaghiani et al.,2009; Werner and Noppeney, 2011) and also for auditory loomingstimuli (Bach et al., 2009; Seifritz et al., 2002). Here we show, thatfMRI-signals in STS are selectively enhanced by audiovisual loomingstimuli, concordant with results from one previous invasive study inmacaques (Maier et al., 2004).

In addition, we also observed enhanced BOLD activity for multi-sensory looming signals within low-level auditory cortex (includingTE 1.1). Looming signals are well studied for the auditory stimulation

in which humans and animals show a perceptual bias for rising soundintensities (Hall and Moore, 2003). Some experiments pointed outthat low-level auditory regions are involved in the processing ofrising sounds (Seifritz et al., 2002) and this modulation has beenlinked to auditory space perceptionwith a representation in auditorybelt regions (Bonath et al., 2007; Hall and Moore, 2003; Krumbholzet al., 2005; Warren and Griffiths, 2003). Moreover, Maier andGhazanfar (2007) investigated the neural representation of these dy-namic signals and discovered that macaque auditory cortex showed abias for auditory looming versus receding signals. The authors arguedthat auditory cortex seems to be an important node in the neural net-work correlated with looming perception. Using audiovisual stimuli,Maier et al. (2008) also reported that congruent looming signalsgenerated increased gamma-band coherence between macaque audi-tory cortex and the STS and concluded that this enhanced neural con-nectivity between these structures may act as a mechanism for bettercoordinated response to looming events. While we found enhancedfMRI-responses for looming stimuli in STS and low-level auditory cortexin our study, we did not observe enhanced coupling between low-levelauditory cortex and the STS. Differences in measures (BOLD vs. multi-unit activity) and analysis approaches may account for these differentresults.

In addition to looming-induced signal-enhancements in bilateralSTS and low-level auditory cortex ourwhole brain GLManalysis also re-vealed enhancements in bilateral low-level visual cortices (includingthe calcarine sulcus, i.e. primary visual cortex). In accord, a recentEEG-study (Cappe et al., 2012) localized the sources of audiovisuallooming signals in low-level visual (and auditory) cortex and a recentTMS study investigated the selective integration of auditory loomingsignals within the visual cortex and suggested that this particular cate-gory of sounds can enhance the excitability (i.e. phosphene induction)of low-level (V1/V2) visual areas (Romei et al., 2009). The authorshinted at the possibility that this enhancement may originate from astructure that is able to discriminate between audiovisual loomingand receding signals. The results from our connectivity analysis suggesta potential neural basis for this proposed mechanism: we observed anenhanced interregional functional coupling between STS and low-level visual cortex when processing multisensory looming signals.This indicates that the STS may be modulating the effects withinlow-level visual areas possibly by feedback connections (e.g. Falchieret al., 2002; Noesselt et al., 2007). Our results also provide a potentialmechanistic explanation of the results by Leo et al. (2011)who reportedthat structured auditory looming signals enhanced the visual sensitivityin Gabor orientation discrimination. This task is relatedwith neural pro-cessing in low-level visual areas (Hubel and Wiesel, 1968; Scobey andGabor, 1989) and points to the functional relevance of the observedmodulations in sensory-specific cortex: By enhancing neural represen-tations in sensory-specific cortex the salience of the represented stimu-lus features could in turn be increased.

Alternatively, this effect could in principle arise fromdifferent atten-tional demands in uni- and multisensory conditions. Several recent

20 S. Tyll et al. / NeuroImage 65 (2013) 13–22

studies have also highlighted in the influence of attention on multisen-sory processing (for review see e.g. Talsma et al., 2010). Importantly, at-tention to both modalities may be necessary when investigatingmultisensory effects (see e.g. Talsma et al., 2007), aswas used in our ex-periment. Another potential issue arises from the different deploymentof the attentional resources, which are focused on one modality inunisensory trials but divided inmultisensory trials. However, in a divid-ed attention context reaction times are usually increased (relative to afocused attention context, e.g. Nebel et al., 2005). Accordingly, wewould expect increased RTs for the (divided) multisensory loomingcondition relative to the (focused) unisensory looming condition. Yet,we found the opposite pattern. Thus, it seems that a divided attentionalfocus plays less of a role here.

In addition to the identification of looming sensitive-structures wealso characterized the response profile within these regions. As aproof of principle, subject-specific ROI analysis revealed enhanced sig-nals for auditory looming stimuli in auditory cortex and thus confirmedprevious reports (Seifritz et al., 2002). Our results also extend previousobservations in auditory cortex and provided evidence for enhancedfMRI-signals in visual cortex for visual looming stimuli. Importantly,the ROI-analysis also revealed that sensory-specific looming-sensitiveareas and the STS show a non-linear response profile with super-additive BOLD response for multisensory signals relative to the sumunisensory responses. Multisensory looming responses also exceededthe maximal unisensory response within low-level visual and auditoryregions plus the STS, whereas parietal and frontal structures did notproperly fulfill our criteria of multisensory integration and may there-fore not be involved in multisensory integration per se but rather intemporal processing or cognitive control (Carter and van Veen, 2007;Coull and Nobre, 1998; Fuster, 1995).

The use of nonlinear response characteristics was a standardmethod for the identification of multisensory interactions in earlysingle cell studies (e.g. Stein and Meredith, 1993), showing that theconvergence of different sensory inputs evokes a product that differssignificantly from either of the unisensory response. However, boththeoretical arguments (Ma and Pouget, 2008) and psychophysicalevidence (Alais and Burr, 2004) have been provided that a linearweighting of different sensory inputs might in fact best serve cuecombination (for discussion see Angelaki et al., 2009; see also Allmanet al., 2008 for neurophysiological results different from classicalnon-linear theories). Moreover, a certain emphasis in demonstratingsuperadditive multisensory interaction in many early studies couldreflect a bias induced by the choice of near-threshold stimulus intensi-ties, as superadditivity could be demonstrated to cease with increasedstimulus intensity (Stanford and Stein, 2007).Moreover, several studieshave also revealed that different interaction patterns (subadditive,additive and superadditive) can coexist in a single neural populationand that the manner how these populations respond to a multisensorystimulus depends upon spatial, temporal and low-level stimulus fea-tures (Stanford et al., 2005; Stein and Meredith, 1993; Werner andNoppeney, 2011).While especially superadditivity has been recognizedas a special case rather than a hallmark of functional multisensoryintegration, several brain imaging studies successfully adapted thisnonlinear criterion to identify brain areas which are involved in multi-sensory integration (Beauchamp, 2005b; Calvert et al., 2001; Gentileet al., 2011; Laurienti et al., 2005; Love et al., 2011; Stevenson andJames, 2009; Stevenson et al., 2007; Werner and Noppeney, 2011).Some studies failed to observe nonlinear response modulations:Beauchamp (2005b) failed to observe superadditive responses for au-diovisual objects in human STS and Love et al. (2011) suggested thatfor audiovisual speech stimuli the max-criterion might be more appro-priate. Others however, observed superadditive fMRI-responses forspeech and object-related stimuli in human STS when using degradedstimuli (Stevenson et al., 2007), in accord with the notion of inverse ef-fectiveness (Stein et al., 2009, though see Holmes, 2007); and one studyeven linked this superadditive neural response enhancement in STS to

the subject-specific behavioral benefit (Werner and Noppeney, 2010).These observations fit well with reports from the animal literature,which described correlations of superadditive neural responses withorientation behavior in cats (Stein, 1998; Wilkinson et al., 1996) andwith stimulus localization (Stein et al., 1988).

In our voxel-based analysis, however, we found that the resultsfrom the mean-criterion analysis most closely resembled the resultsfrom the looming vs. receding comparison (LRC), whereas resultsobtained by applying the max- and superadditivity-criterion only re-vealed a subset of voxels found with LRC. Especially, a superadditivecriterion appears to be quite conservative for voxel-based analyses.On the other hand, the mean-criterion might also identify voxels,which have in fact a higher response in one unisensory compared tothe multisensory condition. If a negative BOLD-response occurs inthe second unisensory condition these voxels may fulfill the mean cri-terion, even in the absence of multisensory signal enhancements (seee.g. Noppeney, 2012 for further discussion). Based on our results wetherefore suggest to use subject-specific ROIs, in addition to a voxel-based comparison, and to aim for the max-criterion, though wenote that these results may only identify a subset of the voxels com-pared to the direct looming-minus-receding-contrast, at least on thegroup level. However, the choice of criterion may be also linked tothe exact research question, but also the stimuli employed.

Notably, the present study employed salient, undegraded stimuliand still observed superadditive response profiles in multisensory andunisensory cortex. Our results are in accord with a recent ERP-study(Cappe et al., 2012), which reported superadditive ERP-modulationsover occipital and central electrodes. Thus, within the identifiedlooming-sensitive structures there seems to be a preference for process-ing this class of stimuli, which is reflected by the superadditive re-sponses. Our results fit well with known behavioral advantages forlooming stimuli, e.g. looming signals are perceived longer, with morephysical intensity and with a greater change for this intensity (Grassiand Darwin, 2006; Neuhoff, 1998; Schlauch et al., 2001) and partici-pants underestimate the time to contact of looming signals (Ball andTronick, 1971; Schiff et al., 1962; Seifritz et al., 2002). This bias forlooming signals is evolutionarily advantageous because a potentialprey has the ability to initialize preparatory behaviors (e.g. motor reac-tion) to an approaching event like a predator (Grassi, 2010) and ourbrain seems to bewell equipped to selectively process this type of stim-uli via superadditve response enhancements.

Conclusion

In conclusion the fMRI-results from our study indicate that theneural substrate of audiovisual looming processing is localized inhuman STS and low-level auditory and visual cortex. Within theselooming-sensitive regions all three criteria of multisensory integrationare fulfilled. The enhanced effective connectivity between the STS andthe visual cortex in the context of audiovisual looming processingmight potentially serve to enhance the salience of unisensory stimulusfeatures and is accompanied by superadditive fMRI-responses.We sug-gest that this selective enhancement of neural signaling for a particularstimulus class (looming) effectively informs animals about sudden, po-tentially dangerous changes in the environment to avoid potentialthreats or collisions.

Supplementary data to this article can be found online at http://dx.doi.org/10.1016/j.neuroimage.2012.09.056.

Funding

S.T., H.J.H. and T.N. were funded by the DeutscheForschungsgemeinschaft – Sonderforschungsbereich – TransRegio31/TP A8; F.O., by Deutsche Forschungsgemeinschaft –

Sonderforschungsbereich – TransRegio 31/TP A3; M.A.S., theDeutsche Forschungsgemeinschaft—Scho-1217/1-2.

21S. Tyll et al. / NeuroImage 65 (2013) 13–22

Notes

S.T. and T.N. designed the study. S.T. analyzed the data. All authorswrote/edited the article. We thank Denise Scheermann and RenateKoerbs for help with data acquisition.

References

Alais, D., Burr, D., 2004. No direction-specific bimodal facilitation for audiovisual motiondetection. Brain Res. Cogn. Brain Res. 19, 185–194.

Alais, D., Newell, F.N., Mamassian, P., 2010. Multisensory processing in review: fromphysiology to behaviour. Seeing Perceiving 23, 3–38.

Allman, B.L., Bittencourt-Navarrete, R.E., Keniston, L.P., Medina, A.E., Wang, M.Y.,Meredith, M.A., 2008. Do cross-modal projections always result in multisensoryintegration? Cereb. Cortex 18, 2066–2076.

Angelaki, D.E., Gu, Y., DeAngelis, G.C., 2009. Multisensory integration: psychophysics,neurophysiology, and computation. Curr. Opin. Neurobiol. 19, 452–458.

Bach, D.R., Neuhoff, J.G., Perrig, W., Seifritz, E., 2009. Looming sounds as warning signals:the function of motion cues. Int. J. Psychophysiol. 74, 28–33.

Ball, W., Tronick, E., 1971. Infant responses to impending collision: optical and real. Science171, 818–820.

Barraclough, N.E., Xiao, D., Baker, C.I., Oram, M.W., Perrett, D.I., 2005. Integration of vi-sual and auditory information by superior temporal sulcus neurons responsive tothe sight of actions. J. Cogn. Neurosci. 17, 377–391.

Baumann, O., Greenlee, M.W., 2007. Neural correlates of coherent audiovisual motionperception. Cereb. Cortex 17, 1433–1443.

Beauchamp, M.S., 2005a. See me, hear me, touch me: multisensory integration in lateraloccipital-temporal cortex. Curr. Opin. Neurobiol. 15, 145–153.

Beauchamp, M.S., 2005b. Statistical criteria in FMRI studies of multisensory integration.Neuroinformatics 3, 93–113.

Beauchamp, M.S., Argall, B.D., Bodurka, J., Duyn, J.H., Martin, A., 2004. Unraveling multi-sensory integration: patchy organization within human STS multisensory cortex.Nat. Neurosci. 7, 1190–1192.

Beauchamp, M.S., Nath, A.R., Pasalar, S., 2010. fMRI-guided transcranial magnetic stim-ulation reveals that the superior temporal sulcus is a cortical locus of the McGurkeffect. J. Neurosci. 30, 2414–2417.

Benevento, L.A., Fallon, J., Davis, B.J., Rezak, M., 1977. Auditory–visual interaction in sin-gle cells in the cortex of the superior temporal sulcus and the orbital frontal cortexof the macaque monkey. Exp. Neurol. 57, 849–872.

Bonath, B., Noesselt, T., Martinez, A., Mishra, J., Schwiecker, K., Heinze, H.J., Hillyard,S.A., 2007. Neural basis of the ventriloquist illusion. Curr. Biol. 17, 1697–1703.

Calvert, G.A., Hansen, P.C., Iversen, S.D., Brammer, M.J., 2001. Detection of audio-visualintegration sites in humans by application of electrophysiological criteria to theBOLD effect. Neuroimage 14, 427–438.

Cappe, C., Thut, G., Romei, V., Murray, M.M., 2009. Selective integration of auditory-visual looming cues by humans. Neuropsychologia 47, 1045–1052.

Cappe, C., Thut, G., Romei, V., Murray, M.M., 2010. Auditory-visual multisensory inter-actions in humans: timing, topography, directionality, and sources. J. Neurosci. 30,12572–12580.

Cappe, C., Thelen, A., Romei, V., Thut, G., Murray, M.M., 2012. Looming signals revealsynergistic principles of multisensory integration. J. Neurosci. 32, 1171–1182.

Carter, C.S., van Veen, V., 2007. Anterior cingulate cortex and conflict detection: an updateof theory and data. Cogn. Affect. Behav. Neurosci. 7, 367–379.

Coull, J.T., Nobre, A.C., 1998. Where and when to pay attention: the neural systems fordirecting attention to spatial locations and to time intervals as revealed by bothPET and fMRI. J. Neurosci. 18, 7426–7435.

Doehrmann, O., Weigelt, S., Altmann, C.F., Kaiser, J., Naumer, M.J., 2010. Audiovisualfunctional magnetic resonance imaging adaptation reveals multisensory integra-tion effects in object-related sensory cortices. J. Neurosci. 30, 3370–3379.

Driver, J., Noesselt, T., 2008. Multisensory interplay reveals crossmodal influences on‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57, 11–23.

Eickhoff, S.B., Stephan, K.E., Mohlberg, H., Grefkes, C., Fink, G.R., Amunts, K., Zilles, K.,2005. A new SPM toolbox for combining probabilistic cytoarchitectonic maps andfunctional imaging data. Neuroimage 25, 1325–1335.

Falchier, A., Clavagnier, S., Barone, P., Kennedy, H., 2002. Anatomical evidence of mul-timodal integration in primate striate cortex. J. Neurosci. 22, 5749–5759.

Franconeri, S.L., Simons, D.J., 2003. Moving and looming stimuli capture attention. Percept.Psychophys. 65, 999–1010.

Friston, K.J., Holmes, A.P., Poline, J.B., Grasby, P.J., Williams, S.C., Frackowiak, R.S.,Turner, R., 1995. Analysis of fMRI time-series revisited. Neuroimage 2, 45–53.

Friston, K.J., Buechel, C., Fink, G.R., Morris, J., Rolls, E., Dolan, R.J., 1997. Psychophysio-logical and modulatory interactions in neuroimaging. Neuroimage 6, 218–229.

Fuster, J.M., 1995. Temporal processing. Ann. N. Y. Acad. Sci. 769, 173–181.Gentile, G., Petkova, V.I., Ehrsson, H.H., 2011. Integration of visual and tactile signals

from the hand in the human brain: an FMRI study. J. Neurophysiol. 105, 910–922.Ghazanfar, A.A., Schroeder, C.E., 2006. Is neocortex essentially multisensory? Trends

Cogn. Sci. 10, 278–285.Ghazanfar, A.A., Neuhoff, J.G., Logothetis, N.K., 2002. Auditory looming perception in

rhesus monkeys. Proc. Natl. Acad. Sci. U. S. A. 99, 15755–15757.Ghazanfar, A.A., Maier, J.X., Hoffman, K.L., Logothetis, N.K., 2005. Multisensory integra-

tion of dynamic faces and voices in rhesus monkey auditory cortex. J. Neurosci. 25,5004–5012.

Ghazanfar, A.A., Chandrasekaran, C., Logothetis, N.K., 2008. Interactions between thesuperior temporal sulcus and auditory cortex mediate dynamic face/voice integra-tion in rhesus monkeys. J. Neurosci. 28, 4457–4469.

Glascher, J., 2009. Visualization of group inference data in functional neuroimaging.Neuroinformatics 7, 73–82.

Grassi, M., 2010. Sex difference in subjective duration of looming and receding sounds.Perception 39, 1424–1426.

Grassi, M., Darwin, C.J., 2006. The subjective duration of ramped and damped sounds.Percept. Psychophys. 68, 1382–1392.

Gray, R., 2011. Looming auditory collision warnings for driving. Hum. Factors 53, 63–74.Hall, D.A., Moore, D.R., 2003. Auditory neuroscience: the salience of looming sounds.

Curr. Biol. 13, R91–R93.Holmes, N.P., 2007. The law of inverse effectiveness in neurons and behaviour: multi-

sensory integration versus normal variability. Neuropsychologia 45, 3340–3345.Hubel, D.H., Wiesel, T.N., 1968. Receptive fields and functional architecture of monkey

striate cortex. J. Physiol. 195, 215–243.Kanowski, M., Rieger, J.W., Noesselt, T., Tempelmann, C., Hinrichs, H., 2007. Endoscopic

eye tracking system for fMRI. J. Neurosci. Methods 160, 10–15.Krumbholz, K., Schonwiesner, M., von Cramon, D.Y., Rubsamen, R., Shah, N.J., Zilles, K.,

Fink, G.R., 2005. Representation of interaural temporal information from left andright auditory space in the human planum temporale and inferior parietal lobe.Cereb. Cortex 15, 317–324.

Laurienti, P.J., Perrault, T.J., Stanford, T.R., Wallace, M.T., Stein, B.E., 2005. On the use ofsuperadditivity as a metric for characterizing multisensory integration in function-al neuroimaging studies. Exp. Brain Res. 166, 289–297.

Leo, F., Romei, V., Freeman, E., Ladavas, E., Driver, J., 2011. Looming sounds enhance orien-tation sensitivity for visual stimuli on the same side as such sounds. Exp. Brain Res.213, 193–201.

Lewis, R., Noppeney, U., 2010. Audiovisual synchrony improves motion discriminationvia enhanced connectivity between early visual and auditory areas. J. Neurosci. 30,12329–12339.

Lewkowicz, D.J., 2008. Perception of dynamic and static audiovisual sequences in 3- and4-month-old infants. Child Dev. 79, 1538–1554.

Liu, Y.J., Wang, Q., Li, B., 2011. Neuronal responses to looming objects in the superiorcolliculus of the cat. Brain Behav. Evol. 77, 193–205.

Love, S.A., Pollick, F.E., Latinus, M., 2011. Cerebral correlates and statistical criteria ofcross-modal face and voice integration. Seeing Perceiving 24, 351–367.

Ma, W.J., Pouget, A., 2008. Linking neurons to behavior in multisensory perception: acomputational review. Brain Res. 1242, 4–12.

Macaluso, E., Frith, C.D., Driver, J., 2000.Modulation of human visual cortex by crossmodalspatial attention. Science 289, 1206–1208.

Maier, J.X., Ghazanfar, A.A., 2007. Looming biases in monkey auditory cortex. J. Neurosci.27, 4093–4100.

Maier, J.X., Neuhoff, J.G., Logothetis, N.K., Ghazanfar, A.A., 2004. Multisensory integrationof looming signals by rhesus monkeys. Neuron 43, 177–181.

Maier, J.X., Chandrasekaran, C., Ghazanfar, A.A., 2008. Integration of bimodal loomingsignals through neuronal coherence in the temporal lobe. Curr. Biol. 18, 963–968.

Marchant, J.L., Ruff, C.C., Driver, J., 2012. Audiovisual synchrony enhances BOLD responsesin a brain network including multisensory STS while also enhancing target-detectionperformance for both modalities. Hum. Brain Mapp. 33, 1212–1224.

Morosan, P., Rademacher, J., Schleicher, A., Amunts, K., Schormann, T., Zilles, K., 2001.Human primary auditory cortex: cytoarchitectonic subdivisions and mapping into aspatial reference system. Neuroimage 13, 684–701.

Nebel, K., Wiese, H., Stude, P., de Greiff, A., Diener, H.C., Keidel, M., 2005. On the neuralbasis of focused and divided attention. Brain Res. Cogn. Brain Res. 25, 760–776.

Neuhoff, J.G., 1998. Perceptual bias for rising tones. Nature 395, 123–124.Noesselt, T., Rieger, J.W., Schoenfeld, M.A., Kanowski, M., Hinrichs, H., Heinze, H.J., Driver, J.,

2007. Audiovisual temporal correspondence modulates human multisensory superiortemporal sulcus plus primary sensory cortices. J. Neurosci. 27, 11431–11441.

Noesselt, T., Bonath, B., Boehler, C.N., Schoenfeld, M.A., Heinze, H.J., 2008. On perceivedsynchrony-neural dynamics of audiovisual illusions and suppressions. Brain Res.Cogn. Brain Res. 1220, 132–141.

Noesselt, T., Tyll, S., Boehler, C.N., Budinger, E., Heinze, H.J., Driver, J., 2010. Sound-induced enhancement of low-intensity vision: multisensory influences on humansensory-specific cortices and thalamic bodies relate to perceptual enhancementof visual detection sensitivity. J. Neurosci. 30, 13609–13623.

Noppeney, U., 2012. Characterization of multisensory integration with fMRI: experimentaldesign, statistical analysis, and interpretation. In: Murray, M.M., Wallace, M.T. (Eds.),The Neural Bases of Multisensory Processes. CRC Press, Boca Raton (FL). Chapter 13.

Padberg, J., Seltzer, B., Cusick, C.G., 2003. Architectonics and cortical connections of theupper bank of the superior temporal sulcus in the rhesus monkey: an analysis inthe tangential plane. J. Comp. Neurol. 467, 418–434.

Parker, A., Alais, D., 2007. A bias for looming stimuli to predominate in binocular rivalry.Vis. Res. 47, 2661–2674.

Peron, S., Gabbiani, F., 2009. Spike frequency adaptation mediates looming stimulusselectivity in a collision-detecting neuron. Nat. Neurosci. 12, 318–326.

Poldrack, R.A., 2007. Region of interest analysis for fMRI. Soc. Cogn. Affect. Neurosci. 2, 67–70.Romei, V., Murray, M.M., Cappe, C., Thut, G., 2009. Preperceptual and stimulus-selective

enhancement of low-level human visual cortex excitability by sounds. Curr. Biol.19, 1799–1805.

Sadaghiani, S., Maier, J.X., Noppeney, U., 2009. Natural, metaphoric, and linguistic audito-ry direction signals have distinct influences on visual motion processing. J. Neurosci.29, 6490–6499.

Scheperjans, F., Eickhoff, S.B., Homke, L., Mohlberg, H., Hermann, K., Amunts, K., Zilles,K., 2008. Probabilistic maps, morphometry, and variability of cytoarchitectonicareas in the human superior parietal cortex. Cereb. Cortex 18, 2141–2157.

22 S. Tyll et al. / NeuroImage 65 (2013) 13–22

Schiff, W., Caviness, J.A., Gibson, J.J., 1962. Persistent fear responses in rhesus monkeysto the optical stimulus of “looming”. Science 136, 982–983.

Schlauch, R.S., Ries, D.T., DiGiovanni, J.J., 2001. Duration discrimination and subjectiveduration for ramped and damped sounds. J. Acoust. Soc. Am. 109, 2880–2887.

Scobey, R.P., Gabor, A.J., 1989. Orientation discrimination sensitivity of single units incat primary visual cortex. Exp. Brain Res. 77, 398–406.

Seifritz, E., Neuhoff, J.G., Bilecen, D., Scheffler, K., Mustovic, H., Schachinger, H., Elefante,R., Di Salle, F., 2002. Neural processing of auditory looming in the human brain.Curr. Biol. 12, 2147–2151.

Seltzer, B., Pandya, D.N., 1994. Parietal, temporal, and occipital projections to cortex ofthe superior temporal sulcus in the rhesusmonkey: a retrograde tracer study. J. Comp.Neurol. 343, 445–463.

Stanford, T.R., Stein, B.E., 2007. Superadditivity in multisensory integration: putting thecomputation in context. Neuroreport 28, 787–792.

Stanford, T.R., Quessy, S., Stein, B.E., 2005. Evaluating the operations underlyingmultisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499–6508.

Stein, B.E., 1998. Neural mechanisms for synthesizing sensory information and produc-ing adaptive behaviors. Exp. Brain Res. 123, 124–135.

Stein, B., Meredith, M., 1993. The Merging of the Senses. The MIT Press, Cambridge, MA.Stein, B.E., Stanford, T.R., 2008. Multisensory integration: current issues from the perspec-

tive of the single neuron. Nat. Rev. Neurosci. 9, 255–266.Stein, B.E., Huneycutt, W.S., Meredith, M.A., 1988. Neurons and behavior: the same

rules of multisensory integration apply. Brain Res. 448, 355–358.Stein, B.E., Stanford, T.R., Ramachandran, R., Perrault Jr., T.J., Rowland, B.A., 2009. Chal-

lenges in quantifying multisensory integration: alternative criteria, models, andinverse effectiveness. Exp. Brain Res. 198, 113–126.

Stevenson, R.A., James, T.W., 2009. Audiovisual integration in human superior temporalsulcus: inverse effectiveness and the neural processing of speech and object recog-nition. Neuroimage 44, 1210–1223.

Stevenson, R.A., Geoghegan, M.L., James, T.W., 2007. Superadditive BOLD activation insuperior temporal sulcus with threshold non-speech objects. Exp. Brain Res. 179,85–95.

Takeuchi, T., 1997. Visual search of expansion and contraction. Vis. Res. 37, 2083–2090.Talsma, D., Doty, T.J., Woldorff, M.G., 2007. Selective attention and audiovisual integration:

is attending to both modalities a prerequisite for early integration? Cereb. Cortex2007 (17), 679–690.

Talsma, D., Senkowski, D., Soto-Faraco, S., Woldorff, M.G., 2010. The multifacetedinterplay between attention and multisensory integration. Trends Cogn. Sci. 14,400–410.

Walker-Andrews, A.S., Lennon, E.M., 1985. Auditory-visual perception of changingdistance by human infants. Child Dev. 56, 544–548.

Wann, J.P., Poulter, D.R., Purcell, C., 2011. Reduced sensitivity to visual looming inflatesthe risk posed by speeding vehicles when children try to cross the road. Psychol.Sci. 22, 429–434.

Warren, J.D., Griffiths, T.D., 2003. Distinct mechanisms for processing spatial sequencesand pitch sequences in the human auditory brain. J. Neurosci. 23, 5799–5804.

Werner, S., Noppeney, U., 2010. Superadditive responses in superior temporal sul-cus predict audiovisual benefits in object categorization. Cereb. Cortex 20,1829–1842.

Werner, S., Noppeney, U., 2011. The contributions of transient and sustained responsecodes to audiovisual integration. Cereb. Cortex 21, 920–931.

Wilkinson, L.K., Meredith, M.A., Stein, B.E., 1996. The role of anterior ectosylvian cortexin cross-modality orientation and approach behavior. Exp. Brain Res. 112, 1–10.

Wittmann, M., van Wassenhove, V., Craig, A.D., Paulus, M.P., 2010. The neural sub-strates of subjective time dilation. Front. Hum. Neurosci. 4, 2.

Wright, T.M., Pelphrey, K.A., Allison, T., McKeown, M.J., McCarthy, G., 2003. Polysensoryinteractions along lateral temporal regions evoked by audiovisual speech. Cereb.Cortex 13, 1034–1043.