Working Memory and Visual Perception Compete for …people.brandeis.edu/~sekuler/papers/Attention.pdfWorking Memory and Visual Perception Compete for Attention: ... Several groups

Working Memory and Visual Perception Compete for Attention:An ERP/EEG StudyYigal Agam & Robert Sekuler

Volen Center for Complex Systems, Brandeis University

How do working memory, perception and attention interact with each other? Recent theoriesof working memory suggest that they are closely linked, and in fact share certain brainmechanisms. Here we use a sequential motion imitation task for a direct, online examinationof memory load’s influence on the processing of visual stimuli. Based on event-relatedpotentials (ERPs) and frequency analysis of the electroencephalogram (EEG), we show that theprocessing of successive incoming visual stimuli becomes less and less effective as additionalitems have to be held in working memory. Using a task in which subjects tried to reproducerandom motion sequences from memory, we found a systematic decrease in ERP amplitudewith each additional motion segment that was viewed and memorized for later imitation.High-frequency (>20 Hz) oscillatory activity exhibited a similar position-dependent decrease.When trials were sorted according to the accuracy of subsequent imitation, the amplitude ofthe ERPs correlated with behavioral performance: The larger the amplitude, the more accuratethe imitation. As both ERP amplitude and high-frequency oscillations are known to reflectattention-driven visual processing, our results suggest that visual working memory is tightlylinked to attention and interferes with visual perception in a load-dependent manner. Sucha tradeoff between memory and perception may underlie fundamental properties of workingmemory, such as limited capacity and primacy effects.

Introduction

The interplay between working memory, visualperception, and attention, has been the subject ofconsiderable debate. Even though working memoryhas received much interest from researchers in psychologyand cognitive neuroscience, remarkably little is knownabout the mechanisms on which it depends, or about thefactors that set its limits. At the theoretical level, twogeneral classes of models posit different roles for attentionin working memory. One approach conceptualizes workingmemory as a set of specialized buffers for the storage ofinformation (Baddeley, 2003), which are controlled byan attention-based structure, the central executive. Themulti-component, modular model emphasizes limitedamounts of activation and temporal decay in the buffersas the sources of capacity limitations in working memory(Baddeley & Logie, 1999). Physiological demonstrations ofactivity in the prefrontal cortex (PFC) during the memoryretention period have lead to the identification of the PFCas the neural substrate for the proposed storage buffers(Goldman-Rakic, 1987; Postle, 2006). An alternativeview treats working memory not as a separate module,but as an emergent property, which harnesses existingneural mechanisms specialized for sensory perception andlong-term memory representations (Cowan, 1999, 2000;

Supported by NSF grant SBE-0354378 and NIH grantR01MH068404. E-mail: [email protected]

Jonides, Lacey, & Nee, 2005; Pasternak & Greenlee, 2005;Postle, 2006). By this account, after the visual stimulushas disappeared from sight, visuospatial working memoryis achieved by prolonged activations, via attention, of thesame occipital and parietal regions that are thought tomediate visual perception (Druzgal & D’Esposito, 2001;Postle, Druzgal, & D’Esposito, 2003; Todd & Marois, 2004;Vogel & Machizawa, 2004). In the “embedded-process”, or“emergent property”, framework, the prefrontal cortex doesnot provide the actual substrate for memory storage, butrather mediates attentional control of the sensory reactivationprocess (Curtis & D’Esposito, 2003; Lebedev, Messinger,Kralik, & Wise, 2004; Postle, 2005). Consequently, thisapproach emphasizes attentional control as a limiting factorin working memory capacity (Cowan, 1999, 2000).

Attempts to choose between these theoretical accountshave produced mixed results. The key issues that distinguishthe two accounts are (i) the degree to which short-termstorage of visual information overlaps with early stages ofvisual processing, and (ii) whether memory’s capacity limitis dictated by some limit on attentional selection of visualinput. Several groups used a visual search paradigm inconjunction with a working memory task, measuring thedegree to which the content of working memory affectssearch efficiency. Woodman, Vogel, and Luck (2001)found no detrimental effect on visual search when subjectshad to concurrently remember a visual object. Morerecently, however, they found that when the information tobe remembered was a location in space, the slope of thefunction relating reaction time to search array size increased,indicating reduced search efficiency (Woodman & Luck,

2 AGAM & SEKULER

2004). A similar result was reported by Oh and Kim (2004).Olivers, Meijer, and Theeuwes (2006) explored the relationbetween the content of working memory and the items in thesearch array, and reported that loading working memory withinformation increased interference from singleton distractors(see also Lavie & de Fockert, 2005), especially when thosedistractors were identical to or shared features with theremembered material. This is inconsistent with studiesby Downing and Dodds (2004) and Woodman and Luck(In press), which found no such interference effects. Asecond line of inquiry examined how perceptual processingis affected by a concurrent working memory task. Awhand colleagues, using functional imaging (Awh et al., 1999),ERPs (Awh, Anllo-Vento, & Hillyard, 2000), and behavioralmeasures (Awh, Jonides, & Reuter-Lorenz, 1998), showedthat remembering spatial locations has the same perceptualconsequences as spatial attention, i.e., increased sensitivityto visual stimuli at those locations. Theeuwes, Olivers, andChizk (2005) demonstrated the same principle using eyemovements: Remembering a location caused the subjects’gaze to deviate, just as if they were directing their attentiontowards that location. Downing (2000) showed that whensubjects remembered a face at one of two locations,perceptual judgements about stimuli subsequently shownin that location were faster than in the other location.In the verbal domain, Shulman and Greenberg (1971)observed a perceptual deficit in recognizing a digit whileremembering a list of consonants, a deficit that depended onlist length. Somewhat in contradiction with these studies,Fougnie and Marois (2006) used a dual-task study thatjoined multiple object tracking and working memory, andfound that although the two tasks interfered with each other,working memory capacity cannot be explained solely bylimits on attention, and working memory likely involves itsown distinct, capacity-limited subprocesses.

We believe the confusion to date arises, in part, from theuse of a dual task paradigm, whose outcome likely reflectsthe added cost of handling the two task at the same time,regardless of the exact nature of the tasks. Of course, suchcosts are difficult to quantify and may lead to unwarrantedconclusions about the relationship between the two tasks thatwere performed simultaneously (Olivers et al., 2006; Lavie,Hirst, de Fockert, & Viding, 2004). Given the decidedlymixed results from the dual task paradigm, we sought a moredirect approach to the question. We asked whether it mightbe possible to examine both visual processing and workingmemory using a single task, with no additional task demandsor irrelevant stimuli. We reasoned this could be achieved bymeans of a procedure in which items of visual informationare presented sequentially, so that subjects have to keepearly items in working memory while continuing to encodeeach subsequently-presented item. Under these conditions,perceptual responses can be gauged in the presence of asteadily increasing load on working memory. By recordingneural responses to the presentation of each additionalitem, we could measure the consequences of the growth inthe amount of stored information. Scalp EEG recordingsprovide a good basis for such an analysis: First, they

provide excellent temporal resolution, which is essential forevaluating differences between responses to closely-spacedstimuli. Second, ERP and EEG markers provide valuableinformation about visual perception: ERP amplitude isknown to correlate with attentive visual processing (Hillyard& Munte, 1984; Hillyard, Vogel, & Luck, 1998; Luck,Woodman, & Vogel, 2000; Awh et al., 2000), as doesactivity in the high-frequency (beta and gamma) bands of theEEG (Gruber, Muller, Keil, & Elbert, 1999; Muller, Gruber,& Keil, 2000; Tallon-Baudry, Bertrand, Henaff, Isnard, &Fischer, 2005). Thus, we used those electrophysiologicalmarkers to track changes in subjects’ processing of incomingvisual motion information while they were attempting to holdpreviously seen motion in working memory.

We recorded scalp EEG from human adults whoperformed a sequential imitation task (Agam, Bullock, &Sekuler, 2005). Fig. 1a shows a schematic diagram ofthe experimental paradigm. On each trial, subjects vieweda moving disc whose trajectory comprised five randomlyoriented, connected linear segments. Then, several secondslater, subjects used a stylus and a graphic tablet to reproducethe trajectory from memory (see also supplementary videoclips). We focused on the period during which subjectswere viewing the moving disc, the idea being that as thedisc progresses, there is more that the subject has to hold inmemory of what he or she has already seen, so we would bemeasuring responses to the disc’s motion under conditions ofvarying load in working memory.

Methods

Subjects and Procedure

Seventeen right-handed subjects (8 male, 9 female, agerange 18-26) participated after providing written informedconsent. Each of the observers performed between 200 and240 trials of the imitation task (memory condition). As thistask has been described in detail previously (Agam et al.,2005; Agam, Galperin, Gold, & Sekuler, In press), onlya brief outline is given here: Each motion stimulus wasgenerated by the steady movement of a yellow disc (1 degvisual angle in diameter) against a black background on acomputer screen, which subjects viewed from a distanceof 57 cm. Each model comprised a novel set of fivedirected motion segments, each 1.5 deg visual angle long,whose orientations varied quasi-randomly. To ensure properperceptual processing of the individual motion segments,the angular difference between the orientations of eachtwo adjacent segments was between 30 and 150 degrees.The moving disc took 525 ms to traverse each segment;Successive segments were separated by a 225 ms pause,in which the disc remained stationary. Subjects had toknit together the directed components in their mind’s eye,and hold the trajectory in memory for 3.75 seconds. Theythen tried to reproduce it with a stylus on a graphic tablet(Wacom, Vancouver, WA). The accuracy of the imitationwas assessed by an automatic segmentation algorithm (Agamet al., 2005), which used temporal and spatial criteria to

COMPETITION FOR ATTENTION 3

Figure 1. Stimuli and results. A: Example of a memory trial. Note that while viewing and reproducing the stimulus, the disc did notleave a trace, so subjects only saw its instantaneous position; the dashed lines are for illustration purposes. B: Behavioral results. Segmentorientation error is plotted against segment serial position. Error bars are within-subject SEM (Loftus & Masson, 1994). C: ERPs at themidline electrode locations for the memory condition, time-locked to the onset of the disc’s appearance on the display. Vertical dashed linesindicate the onset of individual motion segments. The 4.75 seconds for which ERPs are displayed correspond to the period during whichsubjects viewed the stimulus models.

decompose the imitation into individual segments. Aftersegmentation, the orientation of each reproduced segmentwas compared to the orientation of the correspondingsegment in the stimulus model. The absolute differencebetween orientations provided the principal measure of errorin imitation (see Agam et al., 2005, In press, for discussionsof alternative measures).

All seventeen subjects also performed a control task,meant to help us distinguish between effects of memoryload and vision-related effects, such as habituation. In thecontrol condition, stimuli were visually identical to those inthe imitation task. For nine subjects, on one third of 240trials (which were subsequently excluded from the analysis),

the speed of the disc through one of the five motion segmentschanged relative to the normal speed by up to 25%. Subjectshad to indicate whether such a speed change happened ona given trial. The eight remaining subjects performed adifferent control task: At the onset of every one of 100 trials,an arrow pointed in a random direction. Subjects were askedto count on how many, out of 15 segments, the direction ofmotion matched the direction of the arrow. Only the first fivesegments, which were visually identical to the stimuli in thememory condition, were included in the analysis.

Since there were more trials in the memory conditionthan in the control condition, memory trials were randomlyexcluded from analysis, until the number of trials was equal

4 AGAM & SEKULER

in both conditions for each subject. Although this proceduredecreased the signal-to-noise ratio in the memory data, itensured that any effects seen in the memory, but not in thecontrol, condition did not result from to a difference in noiselevels. For the analysis in Fig. 4, which pertains to thememory condition only, no trials were excluded.

Electrophysiological Recordings and Analysis

We recorded from 129 electrode sites at 250 Hz usingan Electrical Geodesics (Eugene, OR) system. Data werecleaned of bad channels, re-referenced to the grand averageand reduced to a standard 27-electrode montage using BESA(MEGIS Software GmbH, Munich), then averaged andanalyzed using MATLAB (The Mathworks, Natick, MA).Blink artifacts were eliminated by rejecting epochs in whichthe difference between the maximum and minimum voltageat any electrode exceeded 70 µV. Data were notch-filteredat 60 Hz and high-pass filtered at 1 Hz. For ERP analysis,an additional low-pass filter was applied at 30 Hz. Tocalculate the energy of the ERP signal, the ERP trace at eachelectrode was squared, summed across the entire 750 msduration of the segment and divided by the sample rate, thenlog-transformed (natural log) to correct for non-normalitydue to lower bounding at zero. For EEG analysis, energyspectra were computed using a 512-point (Hann-windowed)Fourier transform on each segment. Oscillatory energy wassummed across evenly-spaced points within each frequencyband (0.1 octave frequency steps), divided by the number ofFFT points and by the sample rate, and log-transformed.

Correcting for Multiple Comparisons

For each measure (ERP and EEG), we performed 108comparisons: 27 electrode locations x 4 time bins orfrequency bands. To correct for the resulting inflated riskof false statistical inferences due to multiple comparisons(Ludbrook, 1998), we used a random permutation procedure,which provides an empirical estimate, given existingcorrelations in the data, of the probability of a Type Ierror, i.e., false rejection of the null hypothesis (Hwanget al., 2005; Sederberg et al., 2006). To determine theappropriate significance threshold for the ANOVA tests,we randomized the order of segments (2 to 5) withineach trial in every condition, so that correlations betweennearby electrodes and time points within a segment remainedintact, but the effect of serial position was abolished. Thisprocess was repeated 1,000 times, to produce a set of 1,000random permutations. We then averaged the trials in eachpermutation and performed the same statistical tests as wewould perform on the real, unshuffled data. The numberof significant differences seen for a particular significancethreshold, divided by 1,000, provided the probability of aType I error. For example, if half the permutations produceda significant difference, then the global probability of atype I error would be 0.5, much higher than the desiredvalue of 0.05, meaning that the threshold for each individualcomparison needs to be reduced. Finally, we chose thesignificance threshold for individual comparisons, so that the

number of permutations that produced any false positives didnot exceed 50 out of the 1,000 permutations, i.e., a global0.05 probability of a Type I error.

Results

As explained earlier, to score the accuracy of subjects’reproductions in the memory condition, we defined the errorfor each segment as the absolute difference in orientationbetween the reproduced segment and the correspondingsegment in the stimulus. Behaviorally, the resultsdemonstrated a pronounced primacy effect and a modest,one-item recency effect (Fig. 1b), confirming previousfindings with this paradigm (Agam et al., 2005).

Fig. 1c shows event-related potentials (ERPs) atfive midline electrode locations, time-locked to stimulusdisc’s appearance, and encompassing the entire 4.75-secondperiod in which subjects viewed the stimulus disc duringthe memory task. As is clear from the figure, everymotion segment elicits a distinct electrical signature at afixed delay from its onset. To examine changes acrosssuccessive segments, we superimposed traces correspondingto individual segments, and tested the averaged ERPs andenergy spectra for differences between segments in thememory and control conditions. As can be appreciated fromFig. 1c, the ERP associated with the first segment is quitedifferent from the other segments’ ERPs, making a directcomparison difficult, particularly when selecting specificcomponents of the ERP for comparison. This may be dueto the onset of the disc’s motion, which not only entailsan abrupt change in the visual scene, but also a change incontext and in the subject’s arousal level. Such attentionaltransients are also known to be accompanied by a strongdecrease in oscillatory activity, particularly in the alpha band(Klimesch, 1996). Indeed, both conditions produced highlysignificant differences between segments in both time andfrequency, in which the first segment was clearly an outlier.To obtain more meaningful observations related to the effectsof memory load per se, we excluded the first segment fromfurther analysis, and focused on segments 2 through 5.

Analysis of segment-by-segment ERPs focused on epochsof interest identified by visual inspection of the data. Wechose four temporal epochs that encompassed the mostnotable features of the ERPs (Fig. 1c): The early, briefoccipital peak at around 200 ms after the onset of eachsegment, the large frontal peak at 300 ms, the smallerpeak following it at the central electrodes, and the slower,final phase (450-750 ms), most notable at frontal andoccipital locations. Such subjective choices of time bins mayseem arbitrary, but note how different the ERPs to motionsegments (Fig. 1C) are from the “standard” ERPs that aretypically elicited by a brief flash of a high-contrast stimulus(As Fig. 1C shows, the ERP to the appearance of the staticdisc at time 0 actually resembles a typical visual ERP.) Thenon-standard form of the ERPs to the moving stimulus meansthat we could not rely on well-characterized componentssuch as N1, P1, etc., found with more common visual stimuli(see Luck, 2005, for a review of ERP components), and had

COMPETITION FOR ATTENTION 5

Figure 2. Modulation of ERP amplitude. A: Left and middle panels show ERPs time-locked to the onset of segments 2 to 5 in the memoryand control conditions, respectively. Each color corresponds to a different serial position, and the vertical dashed lines represent averagingepoch boundaries. Colored stripes above each panel denote significance levels (ANOVA) for each time epoch (p values above 0.05 areunmarked). A star indicates that a p value exceeded the significance threshold after correction for multiple comparisons. The right panelsshow the energy of the ERPs from the left and middle panels; The solid and dashed lines represent the energy in the memory and controlconditions, respectively. The color within the solid and dashed squares represents significance levels (ANOVA) for the respective energyplots. B: Mean energy of the ERPs across all 27 electrode locations, as a function of segment serial position. Error bars are within-subjectSEM for each condition. In all energy plots, the solid and dashed circles indicate the energy for the first segment, which is only given forreference; It was not included in any statistical analysis.

6 AGAM & SEKULER

to resort to our own choice of time epochs. To provide, alongwith the ERP traces, a measure that reflects changes in theERP amplitude but does not depend on a subjective choiceof time bins, we calculated the sum of the squared samplesin each segment’s ERP. This is a measure that is akin to theenergy of the ERP trace, and reflects how much the ERPdeviates from zero, or from a “resting” state. In using thismeasure, we sacrifice some temporal resolution for a morecompact representation of the ERP.

The ERPs within each chosen epoch were averaged andsubjected to a repeated measures ANOVA at each of 27electrode sites. Fig. 2 demonstrates the segment-by-segmentdifferences between ERPs in the memory and controlconditions at representative electrode locations. The memorycondition produced widely-distributed differences betweensuccessive segments of the trajectory. Specifically, as eachsuccessive segment was seen, ERP amplitude decreased.We found very little difference between segments in theperceptual control condition, indicating that differencesamong segments in the memory condition are not causedby visual processes alone, but arise from the requirementto encode and maintain successive components of theseen trajectory in working memory. The decreasing ERPamplitude suggests that processing of the disc’s motionbecomes less and less effective as more of the trajectory’ssegments have to be held in working memory; It seemsthat early motion segments benefit from increased attentioncompared to later ones, whose processing is shared with therehearsal of early segments.

Although it is immediately clear from Fig. 2 that thetwo conditions produce substantially different results, we didperform a large number (27 electrodes × 4 time bins = 108)of comparisons, so some of the differences that appear tobe significant may in fact be due to chance. To reduce thelikelihood of false statistical inferences, we used a randompermutation procedure to correct for multiple comparisonsand adjust the significance threshold for each individualtest (see Methods). We chose a p-value so that the globalprobability of a Type-I error would be 0.05. This is aconservative approach, which would undoubtedly result indiscarding some perfectly valid, significant result, but itensures the statistical “purity” of the conclusions. In Fig.2 we indicate which statistical comparisons exceeded thecorrected significance threshold. Out of all 108 comparisons,23 “survived” the correction in the memory condition, butonly one in the control condition. Note that our failure toreliably reject the null hypothesis in the control conditiondoes not, by any means, imply that nothing is happening inthat condition (Loftus, 1996); In fact, the data show a certaindownward trend in ERP amplitude. But the weak controlresult, coupled with the clear significance in the memorycondition, does show that memory load has a much strongereffect on the ERPs than merely the repetition of the motionor the passage of time.

The decrease in ERP amplitude can be appreciatedmore easily from the energy plots for each electrode. If,indeed, attention gives rise to an amplification of the overallstimulus-driven response, without changing the time-course

of its components (see Fig. 1 in Luck et al., 2000, for asummary of this point), then energy should be a legitimatemetric by which to quantify such effects. In particular, themean energy of the averaged ERP signal across the entirescalp (Fig. 2B) provides a useful, compact index of themagnitude of the attentional effects. Even though positionwas a significant factor in the control condition (p < 0.03,one-way ANOVA), energy did not decrease consistently, andin fact increased at the fifth segment. Conversely, in thememory condition, the ERPs decreased monotonically andhighly significantly (p < 0.0001) in energy. A two-wayANOVA confirmed an interaction between factors conditionand serial position (p < 0.05).

Although more speculative than ERP measures, there isgrowing evidence that high-frequency (>20 Hz) oscillatoryactivity is correlated with attentive visual processing, asshown by EEG (Gruber et al., 1999; Muller et al.,2000), intracranial EEG (Tallon-Baudry et al., 2005), MEG(Sokolov et al., 1999) and monkey single-unit (Womelsdorf,Fries, Mitra, & Desimone, 2006) experiments. Tocomplement the ERP analysis and obtain more informationabout the differential processing of motion segments in ourimitation task, we examined changes in oscillatory activityacross time, while subjects were viewing the stimulusmodels. To that end, we divided the frequency spectrum intofour standard frequency bands: theta, 4-8 Hz; alpha, 8-13Hz; beta, 13-30 Hz; and gamma, 30-58 Hz (Hwang et al.,2005; Sederberg et al., 2006), and calculated the total energywithin each of the frequency bands (see Methods). Changesin oscillatory energy during the presentation of segments 2-5were assessed using a repeated-measures ANOVA at everyelectrode location and for each frequency band. A similarprocedure was used to correct for multiple comparisons.

High-frequency oscillations in the EEG demonstratedserial-position dependent effects analogous to those seenwith ERPs. Significant differences between segments werewidespread across the scalp, and were observed almostexclusively in the memory condition (twelve comparisonsexceeded the corrected threshold in the memory condition,and none in the control condition). As Fig. 3A shows,segmentwise differences in the EEG reveal a loss of energyin the high frequencies (beta and gamma bands) as a functionof serial position. In addition to comparing oscillatoryactivity in individual electrodes, we calculated the meanenergy across the entire scalp in each of the frequencybands. Fig. 3B shows the mean energy in each band asa function of segment serial position. A one-way ANOVAyielded a significant result only in the beta and gamma bands(p < 0.0001, p < 0.02, respectively) in the memory (butnot the control) condition. A two-way ANOVA indicatedan interaction between condition and serial position thatapproached significance in the beta band (p < 0.08). OurEEG results, therefore, support the idea that the amountof attention directed towards visual processing declines asworking memory demands increase throughout the stimuluspresentation.

Do our findings have any bearing on subjects’ actualperformance in the memory task? To find out, we sorted

COMPETITION FOR ATTENTION 7

Figure 3. Modulation of high-frequency oscillations. A: Energy in four standard frequency bands in the memory and control conditionsat representative electrode locations. Each group of bars corresponds to a different frequency band, and shows the energy in segments 3, 4and 5 (left, middle and right bars, respectively) relative to the energy in segment 2. B: Mean energy across all electrodes in each band as afunction of serial position, in the memory (solid line) and control (dashed line) conditions. The solid and dashed circles indicate the energyin the first segment. All significance markers are similar to Fig. 2. Error bars are within-subject SEM for each condition.

8 AGAM & SEKULER

trials in the memory condition according to the accuracy ofthe imitation on each trial. Orientation errors were averagedacross all five segments, and trials were binned into fourquartiles according to the size of the mean error. Therefore,for each subject we had four equally populous groupsof trials, representing four different levels of performance(Fig. 4A, p < 0.01, ANOVA). To examine whether theERP generated during encoding can predict the quality ofsubsequent imitation, we compared the ERPs in each of thefour groups. Fig. 4B shows the energy of the ERPs acrossthe entire scalp, for individual segments (left) and averagedacross segments 2 to 5 (right)in each group. The amplitude ofthe ERPs associated with viewing of the stimulus segmentswas positively correlated with subsequent performance: Thelarger the ERP amplitude, the more accurately subjectsthen reproduce the motion stimulus. When consideringthe dynamics of the ERPs across serial positions withineach group (Fig. 4B, left), an interesting pattern can beseen: In all four sets, ERP amplitude decreased significantly(p < 0.01, ANOVA); Later segments were less effective ineliciting ERPs than earlier ones. This leads to a possibleconclusion, that if a subject is able to maintain a sufficientlyhigh attention level early on in the sequence, as seen forthe two high-performance bins, the drop in attention wouldnot be as consequential and the probability of good recallwould increase. We did not observe a similar correspondencewith behavioral performance in the frequency data, possiblybecause of the higher noise levels in each group of trials.

Discussion

The results reported here provide evidence that workingmemory exerts a detrimental effect on the perception of newstimuli, which we interpret as competition for attentionalresources. As we have shown, the amplitude of theERP time-locked to the onset of each motion segmentdecreased with the segment’s serial position, as did thehigh-frequency oscillatory content of the accompanyingEEG signal. Furthermore, we showed that ERP amplitudeduring presentation of a stimulus model is a good predictorof the accuracy of subsequent reproduction. This predictiverelationship strongly supports a functional link betweenattention and working memory, and emphasizes real-timevisual processing of the stimulus components as an influenceupon the success of recall. Using a perceptual control taskwith identical stimuli, we demonstrated that the presumeddrop in the efficacy of visual processing is indeed due to therequirement to remember the motion sequences.

The competition for processing resources we propose hassome interesting implications: Without resorting to temporaldecay or storage buffer size, competition provides a simpleexplanation of why working memory is so severely limitedin capacity. Working memory capacity limitations, at leastin the nonverbal domain, could arise from a tradeoff betweenthe processing of incoming visual information and the needto keep remembered visual material active. Furthermore,the serial ordering observed in our data suggests that theprimacy effect, i.e., the better recall of early items (Fig.

Figure 4. Correlation between ERP amplitude and behavioralperformance. A: Imitation accuracy. Left panel shows orientationerror as a function of segment serial position for each quarter ofthe trials in the memory condition. Right panel shows mean erroracross all five segments plotted against quartile. B: Mean ERPenergy across all electrode locations. Left panel shows the energyof the ERPs time-locked to the onset of segments 2 to 5 in eachgroup. Circles denote energy in segment 1. Right panel showsthe energy of the averaged ERPs of all four segments (2 to 5) asa function of imitation accuracy. Note that the energy values in theright panel are smaller, since the averaging of all segments reducesnoise levels. All error bars are within-subjects SEM for each curveindependently.

1b), results from the uneven distribution of attentionalresources available to encode successive motion segments,so that earlier segments benefit from more robust encoding(Sederberg et al., 2006). Some models for the representationof serial order in memory (Page & Norris, 1998) assume a“primacy gradient”, in which the internal representation ofthe first item is most strongly active, and activation strengthdecreases with each additional item, thereby providing astrength-based code for serial order. We have previouslyproposed (Agam et al., 2005) that such a process underlies

COMPETITION FOR ATTENTION 9

behavioral performance in the imitation task described here.The results of the present study raise the possibility that sucha gradient of activation does not constitute a mechanismper se, but could be an implicit consequence of limitationson processing serial information with equivalent reliabilityacross successive items. Alternatively, there may be anexplicit attentional mechanism at play, as proposed by Arbib,Bonaiuto, and Rosta (2006), which determines the relativestrength of encoding of each sequence component.

The idea that working memory capacity is a derivative of amore general limit on attention has been proposed previouslyin various forms (Cowan, 1999, 2000; Kane, Conway,Bleckley, & Engle, 2001; Postle, 2006; Jonides et al.,2005). Although the results of the present study certainlyagree with such a hypothesis, they are not dispositive.The rival multi-component model could, in principle,accommodate our data, by assuming that working memorydoes indeed comprise of “stand-alone”, specialized buffers,but controlling the content of those buffers pulls attentionaway from perceptual processing. However, if that is thecase, then the multi-component model could benefit fromthe inclusion of an explicit connection between the so-called“central executive” — the main attentional construct —and the “crystallized systems” — sensory representationsand long-term memory stores (Baddeley, 2003). In otherwords, the “central executive” should also describe what iscalled “selective attention” in other contexts. That way, themulti-component model could more successfully account forinteractions between attention-driven sensory processing andworking memory maintenance. As Baddeley has noted, ”Thecentral executive is the most important but least understoodcomponent of working memory” (Baddeley, 2003). An exactdefinition of the role of attention will be essential for a betterunderstanding of the mechanisms that give rise to workingmemory. This study may be a step in that direction.

References

Agam, Y., Bullock, D., & Sekuler, R. (2005). Imitatingunfamiliar sequences of connected linear motions. Journal ofNeurophysiology, 94(4), 2832-2843.

Agam, Y., Galperin, H., Gold, B. J., & Sekuler, R. (In press).Learning to imitate novel motion sequences. Journal of Vision.

Arbib, M. A., Bonaiuto, J., & Rosta, E. (2006). The mirror systemhypothesis: From a macaque-like mirror system to imitation. InProceedings of the 6th international conference on the evolutionof language (p. 3-10).

Awh, E., Anllo-Vento, L., & Hillyard, S. A. (2000). The roleof spatial selective attention in working memory for locations:evidence from event-related potentials. Journal of CognitiveNeuroscience, 12(5), 840-847.

Awh, E., Jonides, J., & Reuter-Lorenz, P. A. (1998). Rehearsal inspatial working memory. Journal of Experimental Psychology:Human Perception & Performance, 24(3), 780-790.

Awh, E., Jonides, J., Smith, E. E., Buxton, R. B., Frank, L. R., Love,T., Wong, E. C., & Gmeindl, L. (1999). Rehearsal in spatialworking memory: evidence from neuroimaging. PsychologicalScience, 10(5), 433-437.

Baddeley, A. (2003). Working memory: looking back and lookingforward. Nature Reviews Neuroscience, 4(10), 829-839.

Baddeley, A. D., & Logie, R. H. (1999). Working memory: Themultiple component model. In A. Miyake & P. Shah (Eds.),Models of working memory: Mechanisms of active maintenanceand executive control (p. 28-61). Cambridge: CambridgeUniversity Press.

Cowan, N. (1999). An embedded-process model of workingmemory. In A. Miyake & P. Shah (Eds.), Models of workingmemory: Mechanisms of active maintenance and executivecontrol (p. 62-101). Cambridge: Cambridge University Press.

Cowan, N. (2000). The magical number 4 in short-term memory: Areconsideration of mental storage capacity. Behavioral & BrainSciences, 24, 87-185.

Curtis, C. E., & D’Esposito, M. (2003). Persistent activity in theprefrontal cortex during working memory. Trends in CognitiveSciences, 7(9), 415-423.

Downing, P. E. (2000). Interactions between visual workingmemory and selective attention. Psychological Science, 11(6),467-473.

Downing, P. E., & Dodds, C. M. (2004). Competition in visualworking memory for control of search. Visual Cognition, 11(6),689-703.

Druzgal, T. J., & D’Esposito, M. (2001). Activity in fusiformface area modulated as a function of working memory load.Cognitive Brain Research, 10(3), 355-364.

Fougnie, D., & Marois, R. (2006). Distinct capacity limits forattention and working memory: evidence from attentive trackingand visual working memory paradigms. Psychological Science,17(6), 526-534.

Goldman-Rakic, P. S. (1987). Cellular and circuit basis of workingmemory in prefrontal cortex of nonhuman primates. In H. B. M.Uylings, C. G. Van Eden, J. P. C. De Bruin, M. A. Corner,& M. G. P. Feenstra (Eds.), Progress in brain research (p.325-336). Amsterdam: Elsevier.

Gruber, T., Muller, M. M., Keil, A., & Elbert, T. (1999).Selective visual-spatial attention alters induced gamma bandresponses in the human EEG. Clinical Neurophysiology,110(12), 2074-2085.

Hillyard, S. A., & Munte, T. F. (1984). Selective attention tocolor and locational cues: an analysis with event-related brainpotentials. Perception & Psychophysics, 36, 185-198.

Hillyard, S. A., Vogel, E. K., & Luck, S. J. (1998). Sensory gaincontrol (amplification) as a mechanism of selective attention:electrophysiological and neuroimaging evidence. PhilosophicalTransactions of the Royal Society, London, Series B, 353(1373),1257-1270.

Hwang, G., Jacobs, J., Geller, A., Danker, J., Sekuler, R., & Kahana,M. J. (2005). EEG correlates of verbal and nonverbal workingmemory. Behavioral & Brain Functions, 1(20).

Jonides, J., Lacey, S. C., & Nee, D. E. (2005). Processes of workingmemory in mind and brain. Current Directions in PsychologicalScience, 14(1), 2-5.

Kane, M. J., Conway, A. R. A., Bleckley, M. K., & Engle,R. W. (2001). A controlled-attention view of working memory.Journal of Experimental Psychology: General, 130(2), 169-183.

Klimesch, W. (1996). Memory processes, brain oscillationsand EEG synchronization. International Journal ofPsychophysiology, 24(1-2), 61-100.

10 AGAM & SEKULER

Lavie, N., & de Fockert, J. (2005). The role of working memoryin attentional capture. Psychonomic Bulletin & Review, 12(4),669-674.

Lavie, N., Hirst, A., de Fockert, J., & Viding, E. (2004). Loadtheory of selective attention and cognitive control. Journal ofExperimental Psychology: General, 133(3), 339-354.

Lebedev, M. A., Messinger, A., Kralik, J., & Wise, S. P. (2004).Representation of attended versus remembered locations inprefrontal cortex. PLOS Biology, 2(11), e365.

Loftus, G. R. (1996). Psychology will be a much better sciencewhen we change the way we analyze data. Current Directionsin Psychological Science, 5(6), 161-171.

Loftus, G. R., & Masson, M. E. J. (1994). Using confidenceintervals in within-subject designs. Psychonomic Bulletin &Review, 1(4), 476-90.

Luck, S. J. (2005). An introduction to the event-related potentialtechnique. Cambridge, MA: MIT Press.

Luck, S. J., Woodman, G. F., & Vogel, E. K. (2000). Event-relatedpotential studies of attention. Trends in Cognitive Sciences,4(11), 432-440.

Ludbrook, J. (1998). Multiple comparison procedures updated.Clinical & Experimental Pharmacology & Physiology, 25(12),1032-1037.

Muller, M. M., Gruber, T., & Keil, A. (2000). Modulation ofinduced gamma band activity in the human EEG by attentionand visual information processing. International Journal ofPsychophysiology, 38(3), 283-299.

Oh, S. H., & Kim, M. S. (2004). The role of spatial workingmemory in visual search efficiency. Psychonomic Bulletin &Review, 11(2), 275-281.

Olivers, N. L., Meijer, F., & Theeuwes, J. (2006). Feature-basedmemory-driven attentional capture: Visual working memorycontent affects visual attention. Journal of ExperimentalPsychology: Human Perception & Performance, 32(5),1243-1265.

Page, M. P. A., & Norris, D. G. (1998). The primacy model: a newmodel of immediate serial recall. Psychological Review, 105(4),761-781.

Pasternak, T., & Greenlee, M. W. (2005). Working memory inprimate sensory systems. Nature Reviews Neuroscience, 6(2),97-107.

Postle, B. R. (2005). Delay-period activity in the prefrontalcortex: One function is sensory gating. Journal of CognitiveNeuroscience, 17(11), 1679-1690.

Postle, B. R. (2006). Working memory as an emergent property ofthe mind and brain. Neuroscience, 139(1), 23-38.

Postle, B. R., Druzgal, T. J., & D’Esposito, M. (2003). Seekingthe neural substrates of visual working memory storage. Cortex,39(4-5), 927-946.

Sederberg, P. B., Gauthier, L. V., Terushkin, V., Miller, J. F.,Barnathan, J. A., & Kahana, M. J. (2006). Oscillatory correlatesof the primacy effect in episodic memory. Neuroimage, 32(3),1422-1431.

Shulman, H. G., & Greenberg, S. N. (1971). Perceptual deficitdue to division of attention between memory and perception.Journal of Experimental Psychology, 88(2), 171-176.

Sokolov, A., Lutzenberger, W., Pavlova, M., Preissl, G., Braun,C., & Birbaumer, N. (1999). Gamma-band MEG activity tocoherent motion depends on task-driven attention. Neuroreport,10(10), 1997-2000.

Tallon-Baudry, C., Bertrand, O., Henaff, M. A., Isnard, J.,& Fischer, C. (2005). Attention modulates gamma-bandoscillations differently in the human lateral occipital cortex andfusiform gyrus. Cerebral Cortex, 15(5), 654-662.

Theeuwes, J., Olivers, C. N. L., & Chizk, C. L. (2005).Remembering a location makes the eyes curve away.Psychological Science, 16, 196-199.

Todd, J. J., & Marois, R. (2004). Capacity limit of visualshort-term memory in human posterior parietal cortex. Nature,428, 751-754.

Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predictsindividual differences in visual working memory capacity.Nature, 428(6984), 748-751.

Womelsdorf, T., Fries, P., Mitra, P. P., & Desimone, R. (2006).Gamma-band synchronization in visual cortex predicts speed ofchange detection. Nature, 439(7077), 733-736.

Woodman, G. F., & Luck, S. J. (2004). Visual search is slowedwhen visuospatial working memory is occupied. PsychonomicBulletin & Review, 11(2), 269-274.

Woodman, G. F., & Luck, S. J. (In press). Do the contents of visualworking memory automatically influence attentional selectionduring visual search? Journal of Experimental Psychology:Human Perception & Performance.

Woodman, G. F., Vogel, E. K., & Luck, S. J. (2001). Visualsearch remains efficient when visual working memory is full.Psychological Science, 12(3), 219-224.