The influence of color during continuity cuts in edited movies: an eye-tracking … · 2015-09-04 · The influence of color during continuity cuts in edited movies: an eye-tracking

The influence of color during continuity cuts in editedmovies: an eye-tracking study

Christian Valuch1,2 & Ulrich Ansorge2

Received: 28 February 2015 /Revised: 24 April 2015 /Accepted: 1 July 2015# Springer Science+Business Media New York 2015

Abstract Professionally edited videos entail frequent editorial cuts – that is, abrupt imagechanges from one frame to another. The impact of these cuts on human eye movements iscurrently not well understood. In the present eye-tracking study, we experimentally gauged thedegree to which color and visual continuity contributed to viewers’ eye movements followingcinematic cuts. In our experiment, viewers were presented with two edited action sportsmovies on the same screen but they were instructed to watch and keep their gaze on onlyone of these movies. Crucially, the movies were frequently interrupted and continued after ashort break either at the same or at switched locations. Hence, viewers needed to rapidlyrecognize the continuation of the relevant movie and re-orient their gaze toward it. Propertiesof saccadic eye movements following each interruption probed the recognition of the relevantmovie after a cut. Two key findings were that (i) memory co-determines attention after cuts inedited videos, resulting in faster re-orientation toward scene continuations when visualcontinuity across the interruption is high than when it is low, and (ii) color contributes to theguidance of attention after cuts, but its benefit largely rests upon enhanced discrimination ofrelevant from irrelevant visual information rather than memory. Results are discussed withregard to previous research on eye movements in movies and recognition processes. Possiblefuture directions of research are outlined.

Keywords Edited videos . Continuity . Color . Attention . Eyemovements .Memory

1 Introduction

Multimedia applications in educational, professional, and entertainment contexts often includeedited videos, such as movies, newscasts, instructional or entertainment video clips, and

Multimed Tools ApplDOI 10.1007/s11042-015-2806-z

* Christian [email protected]

1 Cognitive Science Research Platform, University of Vienna, Wien, Austria2 Faculty of Psychology, University of Vienna, Liebiggasse 5, A-1010 Wien, Austria

advertisements. Edited videos are sequences of spatio-temporally continuous video images –the distinct camera takes – which are juxtaposed with cuts – the abrupt transitions from onetake to another [10, 22, 34]. Video editing is a major principle of professional video contentproduction. Therefore, much thought has been invested in understanding the most efficientediting techniques [3, 24, 35].

In this context, continuity editing aims at rendering a movie’s transitions across the cuts assmooth as possible by limiting the viewer’s awareness of cuts, and maximizing the movie’snarrative flow. Such editing conventions are based on film makers’ intuitions and sometimesalso seem to require extensive experience from the viewers [3, 28, 31, 33]. What is also clear isthat memory plays a decisive role for continuity editing [17]: After each cut, viewers need torapidly and accurately decide whether and how the present video image relates to what hasbeen seen just previously [1, 17]. However, the influence of memory across cuts on theviewer’s attention is not well understood, because only little systematic research has beendevoted to this question.

Here, we investigated this question based on a previous study [39] which recorded eyemovements to explore how memory works across editorial cuts. In this experiment viewerssaw two edited movies on the same computer screen but were asked to always watch and keeptheir gaze on only one of these movies. Importantly, the authors had re-edited the movies suchthat cuts occurred simultaneously in both movies, and the movies randomly switched or kepttheir locations with every cut. For example, viewers were presented with a ‘surfing’ and a‘skiing’movie but were asked to always look at the skiing movie and ignore the surfing movie.At half of the cuts, unforeseeable to the participants, the movies switched their locations on thescreen. As soon as participants recognized that the movies had switched their locations, theymade a saccadic eye movement toward the new location of the task-relevant movie. The timebetween the cut and the initiation of this saccadic response informed about the recognition ofthe continuation of the movie at the alternative peripheral location. Specifically, the switchingmanipulation allowed the authors to test how quickly the participants could make a saccade tothe new location of the relevant movie.

Using one known principle of continuity editing, these authors compared two conditions. Inconditions of high visual continuity, a Bwithin-scene cut^ was realized. This means that themajor agent and situation within one video was identical in the pre-cut and in the post-cutvideo image. For example, a person in a red skiing suit would have been shown in total view ina pre-cut image and the same person would have been seen from a closer distance in the post-cut image. In conditions of lower visual continuity, a Bbetween-scenes cut^ was realized. Inthese conditions, the semantics of the scenes connected the pre- and post-cut images so that itwas also always possible for the viewer to recognize that the images across the cut belonged tothe same relevant movie. However, in between-scenes cuts, the major agent and/or thebackground could change. For instance, a surfer in a red suit in the image before the cutcould be juxtaposed to an image of a different surfer in a blue suit after the cut. Using thisprocedure, the authors found that the viewer’s deployment of attention toward movie contin-uations (as measured by saccades) was faster following within-scene cuts (or continuity cuts)than between-scene cuts (or more discontinuous cuts). Also, the authors reported that within-scene cuts implied a higher color similarity between pre- and post-cut images. The authorsconcluded that color might have been a crucial factor that supported memory and recognitionacross editorial cuts. However, since in this previous research the videos were always shown incolor, visual continuity and color similarity across two successive camera takes were generallyconfounded. An experimental dissociation of the potential contributions of color similarities

Multimed Tools Appl

versus other factors (e.g., shape similarities) could only be provided by independently manip-ulating the presence versus absence of color before and after an editorial cut. Conducting suchan investigation was the purpose of the present study.

2 Outline of the present study

The aim of the present experiment was to scrutinize the contribution of color informationacross the cut and determine how recognition of post-cut images in continuity cuts benefitsfrom color. In general, memory is a three-stage process of encoding, representation (orBstorage^), and retrieval. Here, we aimed at dissociating these sub-processes by adapting thegeneral procedure of the previous study [39]. However, with every cut we arbitrarily switchedcolor information on and off (see also Fig. 1c), and played the movies either in color (C), or inblack and white (B). Doing so allowed us to implement the standard conditions of movieviewing where color is present before and after the cut (CC). We were also able to testconditions in which color was absent before and following the cut by presenting bothsuccessive camera takes in black and white (BB). In addition, we could test conditions inwhich a previous black and white camera take was replaced by a post-cut color take (BC), or inwhich color was left out in the post-cut take following a colored pre-cut take (CB). Thesedifferent color conditions were realized orthogonally to three stages of visual continuity acrosscuts (see Fig. 1a), so that we were able to study whether and how color contributed to thecontinuity advantage.

To start with, color is known to be one of the most powerful features to guide attention [11,27, 42], and has been demonstrated to be a particularly helpful visual property for thesegmentation of images and the recognition of objects and scenes [13, 15, 16, 38, 41].However, the potential benefits of color for recognition in edited videos might go beyondthese general color-based discrimination advantages (see Fig. 1c). The present experimentaldesign allowed clear-cut comparisons between the four color conditions. It also allowedspecific conclusions about the sub-processes of memory – encoding or retrieval – at whichcolor could contribute to the deployment of attention, as well as a comparison of these benefitswith memory-free discrimination advantages driven by color.

First, color could enhance the strength of a memory representation during encoding intomemory [15]. Evidence for such a benefit would be given when the participants’ performanceis significantly better under conditions in which color is present before and following a cut(CC) than in conditions in which color is absent before the cut (and can therefore not beencoded) and only present following the cut (BC). An encoding advantage of color wouldtherefore be reflected in better performance for CC than BC, or in a positive CC–BC differencescore. Second, it is possible that color benefits recognition at the level of retrieval, for example,due to an enhanced overlap of memory representations of the pre-cut image with the currentlyavailable visual information [18]. Evidence for such an advantage would be given whenperformance is better where color is present before and following a cut, compared to acondition in which color is only present before the cut (CC-CB). Third, it could also be thatcolor enhances image segmentation – a benefit that is not necessarily based on memory. Such abeneficial influence of color on the discriminability between relevant and irrelevant movieswould be evident if recognition is better in a condition in which movies are first shown inblack and white and, following the cut, in color as compared to a condition in which moviesare shown in black and white both before and following the cut (BC-BB). Finally, color could

Multimed Tools Appl

also provide combined benefits via memory retrieval, and discriminability. Such combinedbenefits would become apparent when comparing conditions under which color is presentbefore and following the cut with conditions in which color is absent before and following thecut (CC-BB).

Importantly, visual continuity effects on recognition of post-cut images could also cruciallydepend on the beneficial influence of color. For example, in a previous study [39] the authorswent so far as to speculate that their continuity cut effects could have reflected color priming[20, 25]. This means that recognition advantages – measured by saccades – followingcontinuity cuts (as compared to more discontinuous cuts) could be restricted to conditions inwhich color is used both before and following the cut. This is possible to test within the current

Fig. 1 a The three levels of the Continuity factor and example stills from the corresponding movie stimuli. Eachpair of images shows the last pre-cut frame and the first post-cut frame. In addition to the continuity manipu-lation, the movies’ color rendition was manipulated, and movies could be either shown in black and white or incolor. All four combinations were realized across the cut/interruption (see below). b Schematic depiction of theprocedure in one experimental block. In each of 20 blocks, two movies were shown side-by-side on the samescreen. The starting location of the task-relevant movie was announced with a green square. Viewers watched andkept their gaze on the task-relevant target movie ‘T’ and ignored the irrelevant distractor movie ‘D’. Moviesalways stopped with the last frame before a cut/interruption. A filler task was presented in which participantsmanually reported the most recent location of the task-relevant target movie, but the key mapping changedrandomly on a trial-by-trial basis, depending on which of two symbols was shown. After the filler task, themovies’ playback continued with the first frames following the cut/interruption, either at their previous or atswitched locations. Participants had to quickly recognize the location of the task-relevant movie and re-orienttheir gaze toward it. c The four conditions of the Color manipulation and the four Benefit comparisons that weremade during data analysis. With each cut/interruption, both movies could switch from color to black and white,or vice versa; or they could stay in color, or in black and white, as they were before the cut/interruption. Fourdifferent comparisons between the Color conditions were made to obtain information about memory-related ormemory-unrelated benefits of color on attention in edited movies

Multimed Tools Appl

study because continuity versus discontinuity editing was realized orthogonally to the use ofcolor in the pre- and post-cut takes. In detail, to test these different possibilities, we re-edited aset of action sports movies that was already employed in previous research [39]. These sportsvideos were well-suited for the purpose of the present study because they allowed manipulat-ing the visual continuity of the videos without drastically violating the overall visual dynamicsor narrative content.

Using these videos, we manipulated visual continuity along three conditions (see Fig. 1a).In general, every time a cut occurred, playback of the movies was interrupted at the last framebefore the cut. The movies were taken off the screen and a short filler task was presented atscreen center before playback resumed with the first video frame after the cut (see Fig. 1b). In abaseline condition, we merely interrupted the videos during an ongoing camera take andcontinued with the next frame of the same take after the filler task. The baseline conditionhence implemented maximal visual continuity across the cut. In a second condition, weimplemented one type of continuity cut, by cutting between two different views of the sameagent and scene. Finally, we implemented a discontinuity condition, in which the generalsemantic topic of the movie stayed the same (i.e., movies continued with the same type ofsports before and after the cut), but the cut occurred between two different scenes.

3 Method

3.1 Participants

Sixty-four (46 female) observers with a mean age of 23 years (range 18–35 years, SD=3.4)participated individually in an eye-tracking experiment. None of these had participated in theprevious study [39]. They were recruited ad-hoc from the student population at the Faculty ofPsychology of the University of Vienna and were granted partial course credit in exchange fortheir participation. Only participants with intact color vision and visual acuity took part in theexperiment. (This was verified by screening tests at the beginning of each session). Informedconsent was obtained from all participants.

3.2 Apparatus

To record viewers’ gaze position, an EyeLink 1000 (SR Research Ltd.) infrared video-basedeye-tracking system, operating at a sampling rate of 1000 Hz, was used. Using a standard 5-point calibration, the eye-tracker was calibrated on the viewer’s dominant eye. To re-check thesystem’s calibration, a drift check routine was executed at the start of each experimental blockas well as every 10th trial. If during a drift check the measured gaze position was outside aradius of 1° around a circle fixation target presented at the center of the screen, the fullcalibration sequence was repeated.

The experiment was programmed in MATLAB (MathWorks, Natick, MA, USA) using thePsychophysics [5, 30] and Eyelink [9] toolboxes. The experiment was run on a consumer-standard personal computer running Microsoft Windows XP (SP3) with an Athlon Dual CoreCPU (AMD Inc.), a GeForce GT220 (Nvidia, Inc.) graphics card, and 2,048 Megabytes ofRAM. All stimuli were presented on a Multiscan G400 (Sony, Inc.) 19-in. color CRT monitor,set to a resolution of 1,280×1,024 pixels at 60 Hz. Manual responses for the filler tasks (seeProcedure) were recorded via a standard PS/2 keyboard.

Multimed Tools Appl

3.3 Stimuli

As in a previous study [39], movie stimuli were preselected from a range of sports DVDsavailable at Vienna’s public library, and additional footage was collected from online videoplatforms. Videos were re-edited using Premiere Pro (Adobe Inc.) to match the requirements ofthe current experiment. Ten pairs of two videos were prepared. Each pair of videos met threecentral criteria for the present experimental manipulations: (i) Each of the two videos showed adistinct sport from start to end (e.g., one video showed ‘skiing’, whereas the other videoshowed ‘surfing’); (ii) both videos were completely re-edited such that in the resulting pair allof the cuts occurred simultaneously in both videos (e.g., whenever there was a cut in the skiingvideo, the surfing video contained a cut as well); and (iii) the inserted cuts were classified intotwo distinct categories, namely cuts that occur within scenes (henceforth referred to ascontinuity cuts), or cuts that occur between scenes (henceforth referred to as discontinuitycuts). In addition to these two types of actual cinematic cuts, we also included a baselinecondition. For that, we defined time-points at which we would simply pause the movies duringan ongoing take.

Including this baseline condition was possible, because different to the previous study [39]the current experimental procedure scheduled brief interruptions of the movies at each cut.During the interruptions the movies were turned off, and a filler-task display was presented(see Procedure). The baseline condition implemented maximal visual continuity across theinterruption because it was realized simply by interrupting a movie during an ongoing take,and continuing with the next regular frame of the same take after the interruption. Importantly,the movies were not only interrupted in baseline conditions but also at all continuity ordiscontinuity cuts. At each of them, the movies’ playback stopped at the last frame beforethe cut and continued with the first frame after the cut, following the interruption. Researchwith better controlled displays showed that repeating colors of task-relevant objects after blankscreen intervals between consecutive trials results in strong facilitative effects on searchperformance [20, 21]. Therefore, we assumed that interrupting the movies at the cuts andblanking the screen for a central fixation (during the filler task) should not drastically diminishany color and/or continuity benefits that were previously observed in continuous viewing [39].The mean duration between two interruptions amounted to 4 s (SD=1.6). Movies variedslightly in the number of cuts/interruptions but were on average 2 min 22 s long (SD=14.1 s).All re-edited movies were encoded without audio tracks to MPEG H.264 videos at a resolutionof 400×300 pixels and 25 frames per second. Because the viewing distance during theexperiment was held constant at 64 cm using chin and forehead rests, all participants sawthe movies at a size of 9.7×7.1° on a black background. The movies were offset from thescreen center by 7.2° toward the left or right direction and vertically centered within the full-screen monitor area of 31×24.2°.

3.4 Experimental design

The study featured a 4 (Color)×3 (Continuity) factorial design. Conditions were fully crossedand measured within participants. The four-staged factor Color (see Fig. 1c) manipulatedwhether both movies were shown – before and/or after the interruption, respectively – in coloror in black and white (BW). The four conditions were as follows: (i) Color present before andafter the cut (CC), (ii) BW before and color after the cut (BC), (iii) Color before and BWafterthe cut (CB), and (iv) BW before and BW after the cut (BB). The three-step factor Continuity

Multimed Tools Appl

manipulated the visual continuity across the two movie segments separated by an interruption(see Fig. 1a).

In the (i) baseline condition, the visual continuity was maximal, because movies wereinterrupted (or ‘cut’) during an ongoing take and continued with the next frame of the sametake after the interruption. In the (ii) continuity condition, the interruption (cut) was placedbetween two shots that showed different perspectives on the same scene. In these cases, themain actors, actions, or central objects as well as some background information was repeatedafter the interruption, although the scene was presented from a different viewpoint. The (iii)discontinuity condition, finally, presented the viewer with a new scene, in which the semanticcontent but not necessarily the visual characteristics connected the scenes across the cut: As inthe other two conditions, the movies here continued with the same overall type of sports, butthe central agents, objects, and surroundings changed across the cut. Even in this condition,however, some visual similarity across the cut was present in most cases (e.g., in a surfingmovie even different scenes have a similar gist, due to colors diagnostic of water and sky; or, ina Formula 1 racing movie, borders and edges of the race track would be fairly similar betweendifferent scenes, unlike the cars visible at center). The factors Continuity and Color werecrossed, such that each level of Continuity occurred equally frequent in each of the four Colorconditions. To enhance the difficulty of the participant’s task of attending to one movie andignoring the other, the movies could either keep or switch their locations after each interruption(or cut).

Overall, location switches occurred at half of all interruptions, balanced across experimentalconditions. Location switches occurred in a pseudo-randomized sequence, such that partici-pants could not foresee whether their task-relevant movie would re-appear at the old locationor switch to the alternative location. Hence, participants had to recognize the continuation ofthe task-relevant movie anew after each interruption. Throughout the experiment, participantswere confronted with 456 interruptions. As experimental trials, we here consider the timeperiods following each interruption (or cut) in which the participants’ gaze behavior wasrecorded and analyzed with regard to the above mentioned manipulations.

3.5 Procedure

After completing initial screening tests for visual deficiencies and determination of eyedominance, participants were seated in front of the experimental PC where they could adjustthe height of the seat such that they felt comfortable when leaning toward the chin andforehead rests. Initial instructions were provided on screen. The instructions briefly explainedthe eye-tracking procedure and mentioned that the experiment investigated visual attentionprocesses during the perception of edited videos. The explanations also outlined the generalprocedure of the experiment: Participants were told that the experiment consisted of 20 blocksin total, and that there would always be two movies at the same time on screen. They were toldthat their task was to always follow only one of these movies and ignore the other. They werealso informed that the movies would frequently be interrupted and that they would resumeafter a short interruption either at the same or at switched locations.

The main task of the participants was to watch and keep their gaze as much as possible onthe task-relevant movie and ignore the other movie. Each block of two movies started with thepresentation of a red and a green filled rectangle occupying the left and right movie locationson screen. These rectangles indicated the starting locations of the task-relevant movie (green),and the irrelevant movie (red), respectively (see Fig. 1b). Over the course of the experiment,

Multimed Tools Appl

participants saw each pair of movies two times, such that each movie once served as the task-relevant movie and once as the irrelevant movie. All task-relevant movies from the first half ofthe experiment served as irrelevant movies in the second half of the experiment and vice versa.The order of assignments of each movie as task-relevant or irrelevant was counter-balancedacross participants. Participants were instructed to look at the task-relevant movie as quickly aspossible whenever it re-appeared after an interruption (or cut). Hence, each continuation of themovies essentially presented the participants with a two-alternative forced choice implicitrecognition task [19, 32].

During the interruptions, participants were presented with a filler task to ensure that theystayed vigilant throughout the experiment, and attentively followed the task-relevant moviewhen it was on screen, but also returned their gaze fixation to the center of the screen duringthe interruptions (as this was the starting position for the next trial). The filler task askedparticipants to manually report the location of the task-relevant movie – that is, participants hadto keep consciously track of the locations of task-relevant and irrelevant movie and of theirlocation switches, respectively. To make this task slightly more demanding, participants had toawait a response display during the interruption period in which a symbol informed them aboutthe actually applying location-to-key mapping. This was done to avoid that any manual buttonpresses (or their mental preparations) would interfere with the oculomotor behavior. Inaddition, we chose horizontally centered response keys (#8 and #2 on the numerical keypad)to minimize the possibility that the spatial compatibility of the response key location and therelevant movie’s location (left or right) could contribute to filler task performance or evenoculomotor behavior. The response display during the interruption showed either an outlinedgray square or an outlined gray diamond (the diamond was the square rotated by 45°; hence itwas fully equivalent in its physical features). For each participant one of the symbols (e.g., thesquare) indicated that the #8 had to be pressed for the right location of the relevant video andthe #2 for the left location, whereas the other symbol (e.g., the diamond) indicated the reversemapping. Both symbols (and hence the different mappings) were presented randomly andhence on average occurred equally often. The symbol-to-rule mapping was counterbalancedacross participants. Participants only received feedback on screen when they made an incorrectbutton press (‘WRONG’ displayed at screen center for 0.5 s). Otherwise, the symbol disap-peared right after the button press (manual response time was on average M=1.2 s, SD=0.36 s). This was followed up by a central fixation target replacing the symbol, and contin-uation of the playback as soon as participants fixated on the center. As soon as the playbackcontinued, participants quickly re-oriented their gaze toward the relevant movie (and avoidedthe irrelevant movie).

The playback-interruption-playback-interruption sequence continued until the movies with-in this particular pair finished. Then, the next block was announced by red and green squaresindicating the starting locations of task-relevant and irrelevant movies. Overall, the wholeexperiment (including initial screening tests and equipment calibration, potential re-calibrations of the eye-tracker during the experiment, short self-paced breaks of the participantsin between blocks, and final debriefing) lasted about 90 min per participant.

3.6 Data treatment

Eye movement data was recorded from a total of 29,184 trials (456 trials for each ofthe 64 participants). Of central interest for the present study were the properties ofattentional orienting right after each interruption or cut, depending on how the

Multimed Tools Appl

continuation related in terms of Color and Continuity to the content of the task-relevant movie before the interruption. Because saccades are tightly coupled toattentional selection processes, we analyzed the properties of the saccadic eye move-ments following each interruption.

We focused on two main dependent variables: (i) the percentage of trials in which the firstsaccade landed directly on the task-relevant movie (while no previous saccade targeted theirrelevant movie), and (ii) the saccadic reaction time (SRT) – that is, the time between the onsetof the movies after the interruption and the initiation of the first saccade that was directedtoward the task-relevant movie. The raw horizontal and vertical gaze position data (vectors of1,000 position samples per second) was processed using the SR Research event detectionalgorithm, which parses gaze position data into sequences of saccades, fixations, and blinks.For saccade detection, the algorithm used established thresholds of gaze displacement (>0.1°),velocity (>30 °/s), and acceleration (>8,000 °/s2). All recorded eye movement data were pre-processed in MATLAB and statistically analyzed using R [8]. All statistical tests were run onparticipant cell means. In cases in which the sphericity assumption of the calculated GLMswas violated (as indicated by Mauchly’s test), the reported p-values were obtained afterapplying a Greenhouse-Geisser correction to the F-tests’ degrees of freedom. In general weset α at 0.05 for statistically significant effects.

4 Results

4.1 Dependent variables

After determining oculomotor events (saccades, fixations, and blinks) for each record-ed trial, we determined the viewer’s first saccades with landing positions on the task-relevant or irrelevant movie, respectively. Saccades toward the movies were detectedwithin invisible areas of interest (AOIs) slightly extending over the borders of the twomovies, allowing some tolerance for measurement inaccuracies. Figure 2a shows a2D-gaussian kernel density histogram of the landing positions of the first saccadestoward the relevant movies (which could also be conceived of as the starting locationsof the first fixations on the relevant movies after each interruption). We analyzed thepercentages of trials in which the task-relevant movie attracted the first saccade.Histograms of this proportional measure, split for the experimental conditions, aredepicted in the left panels of Fig. 2b. Complementary to that, we also analyzed thesaccadic reaction time of the first saccade that landed on the task relevant movie(irrespective of whether there were any intervening saccades and fixations on theirrelevant movie). Histograms of this dependent measure are depicted in the rightpanels of Fig. 2b. While the proportion of first saccades toward the relevant moviecould be conceived of as the first rapid decision for any of two locations, the overalldistribution of this measure also shows that in a substantial proportion of trials thefirst saccades did not directly target the task-relevant movie. For a more generalassessment of the performance with the participant’s task-relevant movies we hencealso based our conclusions on the saccadic reaction time, which encompasses potentialdecision and rejection processes with respect to the irrelevant movie. Mean values ofthese variables were aggregated over participants and submitted to 4×3 repeatedmeasures ANOVAs, assuming a between-participant random effect.

Multimed Tools Appl

4.2 First saccades toward task-relevant movie

Statistical analysis yielded significant main effects of Color, F(3,189)=17.4, p<0.001, as wellas Continuity, F(2,126)=75.5, p<0.001. Also there was a significant interaction effect ofColor×Continuity, F(6,378)=15.6, p<0.001. Figure 3a shows that the percentage of firstsaccades going directly toward the relevant movie was generally lower and close to chancelevel in the discontinuity condition, and also the presence or absence of color did not clearlyinfluence the participants’ performance. In contrast, performance in the continuity conditionwas better and even reached baseline level, but only when color was present after the cut (inthe CC and BC conditions), and not when it was absent after the cut (CB and BB). In thebaseline condition, performance was overall the highest, and the presence or absence of colordid not modulate it as drastically as in the continuity cuts. To further scrutinize this interactionand determine the conditions under which color benefits attentional orienting toward thecontinuation of the task-relevant movie, we contrasted the participants’ performance in thefour Color conditions with respect to four possible mechanisms: (i) Encoding benefits of colorwhich were determined by subtracting each participant’s mean performance in the CCcondition from mean performance in the BC condition; (ii) Retrieval benefits of color whichwere determined by subtracting mean performance in the CC condition from mean perfor-mance in the CB condition; (iii) Discrimination benefits of color which were determined bysubtracting the mean performance in the BC condition from the mean performance in the BBcondition, and finally; (iv) Retrieval and Discrimination benefits of color, which weredetermined by subtracting mean performance in the CC condition from mean performancein the BB condition.

We ran another 4×3 ANOVA with the within-participant factors Benefit (encodingvs. retrieval vs. discrimination vs. retrieval+discrimination) and Continuity (baselinevs. continuity vs. discontinuity). Results showed that the values differed significantlyacross the four Benefit levels, F(3,189)=4.3, p=0.018. In addition, Continuity also

Fig. 2 a 2D-Histogram of landing locations of saccades that underwent statistical analysis in the present study.The small filled rectangles depict the borders of the movies. The plot combines trials in which the task-relevantmovie continued at the left, or at the right location. The slightly larger dashed rectangles depict the AOIs used todetermine these saccades during the pre-processing of the recorded eye movement data. b Histograms of thedistributions of the main dependent variables which underwent statistical analysis as a function of the experi-mental factors of Color and Continuity. The left panels depict the proportion of trials in which the first saccadelanded directly on the task-relevant movie and not on the irrelevant movie (histogram binwidth set to 5 %). Theright panels depict the saccadic reaction times – that is, the time between the onset of the movie’s continuationand the initiation of the first saccade that landed on the relevant movie (histogram binwidth set to 25 ms)

Multimed Tools Appl

had a significant main effect on the obtained differences, F(2,126)=16.6, p<0.001. Finally, therewas a significant interaction of Benefit×Continuity, F(6,378)=9.9, p<0.001. To scrutinize howbenefits interacted with continuity, we ran a series of t-tests comparing each difference valueagainst zero. Differences were assumed to be significant at anα-level of 0.05 and are in that casesmarked with asterisks in Fig. 3b. In each of these comparisons, positive difference scores indicatea beneficial influence of color on the respective processing stage. Interestingly, although the dataindicates stable color benefits on the levels of memory-based retrieval (CC-CB) andmemory-freediscrimination (BC-BB), their combined effect (CC-BB) does not outweigh either of them.Hence, color can facilitate saccades to the relevant movie already via enhanced discriminability,without relying on any color-based memory trace from the pre-cut scene.

4.3 Saccadic reaction time

For the analysis of SRTs we excluded all trials in which no saccade toward the relevant moviewas registered until the next interruption (or cut). Also, we excluded trials in which measuredSRT was below 50 ms (these were most likely invalid anticipations prior to actual attentionalselection), and trials in which SRT was outside a range of±2.5 SD around the participant’sindividual mean SRT for that condition. A total of 1,124 trials (3.9 % of the dataset) weredisregarded due to these criteria. All valid trials were again aggregated over participants andmean values were submitted to repeated measures ANOVAs for significance testing (seeFig. 4a). ANOVA results followed a similar pattern as for the proportions of first saccadestoward the task-relevant movie. The analysis yielded significant main effects of Color,F(3,189)=16.4, p<0.001, as well as Continuity, F(2,126)=107.3, p<0.001. There also wasa significant interaction of Color×Continuity, F(6,378)=12.6, p<0.001. Figure 4a shows thatSRT was generally shortest in the baseline condition, and longest in the discontinuity condi-tion. Similar as in the analysis of first saccade proportions, the continuity condition was mostsensitive to the presence or absence of color: SRTs were at the level of the baseline conditionwhen color was present after the cut (CC and BC) and declined in the direction of the

Fig. 3 a Proportions of trials in which the first saccade landed directly on the task-relevant movie, as a functionof the three-step factor Continuity (baseline, continuity, discontinuity), and the four-step factor Color (CB, BC,CB, BB). The dashed line marks chance level. Error bars depict ±1 SEM. b Post-hoc comparisons ofparticipants’ mean performance difference between two respective conditions. Four specific comparisons weremade which carried information about a particular potential benefit of color to the recognition of moviecontinuations. Positive values denote benefits, negative values denote costs. Asterisks mark comparisons inwhich differences between conditions were significant. Error bars depict 95 % CIs

Multimed Tools Appl

discontinuity condition when color was absent after the cut (CB and BB). Again, wescrutinized this interaction by focusing on specific differences between the conditions thatare indicative of particular color-driven benefits, depending on the movie’s continuity acrossthe cut/interruption.

Differences between participants’ condition means were submitted to a 4×3 repeated measuresANOVA, with the two within-participant factors Benefit and Continuity. Results showed SRTdifferences varied significantly across the four Benefit levels, F(3,189)=9.0, p<0.001. Again,Continuity also had a significant main effect on SRT differences, F(2,126)=13.8, p<0.001, andthere also was a significant interaction of Benefit×Continuity, F(6,378)=8.5, p<0.001. Testing ofdifferences against zero was again done using a series of t-tests, and significant results are markedwith asterisks in Fig. 4b. In Fig. 4b, any significant negative deviations indicate facilitative effects ofcolor (reflected in shortened SRTs). The results indicated the strongest color benefits at the retrievallevel, although – at least in continuity cuts – there were discrimination benefits of about the samesize. Again, the combined benefit of retrieval and discrimination did not outweigh the benefits at thetwo independent levels. This indicates that both mechanisms might contribute to overall facilitationeffects of color, but that memory is not a necessary precondition for color to facilitate saccadestoward the relevant movie’s continuation.

5 Discussion

The present study aimed at a better understanding of the factors that drive recognition and eyemovements during the perception of edited videos. In an eye-tracking experiment, we disso-ciated the influences of color and visual continuity on the implicit recognition of scenecontinuations and measured the effectiveness of directing the eyes toward a movie’s contin-uation after a cut.

The key findings of this experiment were twofold. First, the efficiency with which viewers’directed their gaze to a movie following a cut was determined by visual continuity [1, 39].

Fig. 4 aMean saccadic reaction time (SRT) for the first saccades that landed on the relevant movie, as a functionof the three-step factor Continuity (baseline, continuity, discontinuity), and the four-step factor Color (CB, BC,CB, BB). Error bars depict ±1 SEM. b Post-hoc comparisons of participants’mean performance between the twoconditions. Four specific comparisons were made which carried information about a particular potential benefitof color to the recognition of movie continuations. Here, negative values denote benefits, and positive valuesdenote costs. Asterisks mark comparisons in which differences between conditions were significant. Error barsshow 95 % CIs

Multimed Tools Appl

Second, color also benefited saccades following a cut. However, these color benefits could belargely explained by increased discriminability of relevant from irrelevant videos in the post-cut takes because color in the post-cut takes facilitated saccades to the relevant videos,regardless of whether color was also present in the pre-cut takes. That is, in continuity cutssaccadic benefits were more strongly dependent on the presence of color following the cut.Hence, color seems to enable viewers to recognize the continuation of the attended moviemore efficiently due to an increased discriminability of relevant from irrelevant information[11, 13]. Because this benefit had about the same size, irrespective of whether the movies werepresented in color or in black and white before the cut, memory traces of the just perceivedcolor content did not considerably contribute to the guidance of attention. Also of interest, inthe baseline condition, relatively constant continuity benefits were observed, irrespective ofwhether color was present before and/or following the cut. Thus the general color benefit wasmostly restricted to the continuity cuts.

In addition, the present data yielded some evidence for color benefits at encodingand retrieval stages. For example, in the baseline condition one can see a slightperformance advantage for each color condition as compared to the BB condition.However, the combination of retrieval and discrimination benefits did not outweigh adiscrimination benefit alone. Based on the present data, this conclusion is moststrongly warranted for continuity cuts. It is important to emphasize, however, thatthis role of color for memory across the cuts could not explain the full continuityadvantage. For example, even in the BB conditions, there was an SRT advantage incontinuity as compared to discontinuity cuts (see Fig. 4).

This is particularly interesting, since in a previous study the authors speculated that thecontinuity advantage could have reflected color priming [39]. Color priming denotes fastersaccades (or attention shifts) to previously attended-to color stimuli and is usually observed inlaboratory experiments which use simplified stimulus displays and visual search tasks [25].However, it is likely that humans flexibly adapt to the demands of the current task and thatshort-term color priming is restricted to situations where color features provide the strongestadvantage as compared to shape similarities or similarities of the overall orientations withinimages. Shape and orientation similarities, however, are valuable sources of continuity editingin videos in which far more complex objects are used than in highly-controlled laboratoryexperiments with their simplified stimuli. The possibility that more or less color priming isfound depending on the task demands and stimuli would be well in line with the assumed top-down contingency of a color priming effect [21].

The presently reported influence of memory across cuts on saccades in edited videos alsosubstantially extends prior research which has mostly focused on how visual feature distribu-tions during an ongoing camera take contribute to gaze guidance in video viewing [2, 6, 7, 12].We have recently suggested a model of attention and gaze behavior during movie and videoviewing that explains the interaction of memory and saccades [1] by assuming that (i) takes(i.e., spatio-temporally coherent phases of dynamic displays) and (ii) cuts (i.e., juxtapositionsof takes) require very different eye-movement strategies: Within a take, novel visual informa-tion should be fixated with the eyes and encoded into memory, but after a cut, the task becomesthat of recognizing the new take as a continuation or discontinuation of the pre-cut take, andmemory traces or memory representations of the pre-cut take are used as templates to guide thegaze in the search for visual similarities between the currently seen image and the previouslyseen images. This relevance and usefulness of memory for the guidance of eye movements hasbeen previously described in the context of recognition of static images [14, 40]. Together with

Multimed Tools Appl

our previous results [39], the present experiment shows that the principle of similarity-basedrecognition and gaze guidance could also generalize to viewing of edited movies, and, thus,informs a range of multimedia applications, such as perception inspired video processing, butalso the optimization of medical imaging devices, and graphical user interfaces in general [26].

In the end, it should be noted that the conclusions offered by the present data are limited inseveral respects. First, in our experimental design participants constantly discriminated betweena relevant and an irrelevantmovie. Although this allowed a precisemeasurement of the viewer’ssaccadic latency and saccade precision following the cuts, the procedure is clearly differentfrom the more ecological viewing situation, in which participants watch only one movie at atime. In addition, previous research on gaze behavior during movie viewing reported that taskinstructions modulated the viewer’s attentional processes [36]. Our forced-choice localizationtask might have therefore pushed the presently observed continuity or memory effects, andmore research is required to ultimately decide if a preference for video content that matchesrecently seen images (or memory traces) after abrupt image changes [1] is indeed the standardbehavior of the human visual system when viewing edited movies freely. In connection withthis, future research could also target the generalizability of this continuity or memory effectsacross different situations in which viewers are confronted with abruptly changing images.Possible applications are, for example, graphical user interfaces, browsing of webpages, or theperception of virtual dynamic 3D environments, such as in video gaming. Finally, after thepresent research the further fate of a memory representation generated while watching an editedmovie beyond the following post-cut image remained unexplored. In other words, it isunknown whether or for what duration, such memory representations persist across unrelated,intervening information, as it would be the case in cross-cutting back and forth betweendifferent scenes. One possibility is that memory capacity falls within the limits of visualworking memory [23, 29] but it is also possible that, similar to picture recognition, memoryfor takes is less limited [4, 37]. All these questions should be clarified by future experiments.

Acknowledgments The authors thank two anonymous reviewers for their excellent and helpful feedback on aprevious version of this manuscript, as well as Blerim Zeqiri and Stefan Kandioller for assistance with datacollection. This research was funded by a grant from the Wiener Wissenschafts-, Forschungs- undTechnologiefonds (WWTF, Vienna Science and Technology Fund), no. CS 11–009 to Ulrich Ansorge, ShelleyBuchinger, and Otmar Scherzer.

Compliance with Ethical Standards The authors declare that the research was conducted in the absence of anycommercial or financial relationships that could be construed as a potential conflict of interest. All researchprotocols complied with the Declaration of Helsinki and APA ethical standards.

References

1. Ansorge U, Buchinger S, Valuch C, Patrone AR, Scherzer O (2014) Visual attention in edited dynamicalimages. SIGMAP: Proceedings of the 11th International Conference on Signal Processing and MultimediaApplications 2014:198–205. doi: 10.5220/0005101901980205

2. Böhme M, Dorr M, Krause C, Martinetz T, Barth E (2006) Eye movement predictions on natural videos.Neurocomputing 69(16):1996–2004

3. Bordwell D, Thompson K (2001) Film art: an introduction. McGraw-Hill, New York4. Brady TF, Konkle T, Alvarez GA, Oliva A (2008) Visual long-term memory has a massive storage capacity

for object details. Proc Natl Acad Sci 105(38):14325–143295. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–4366. Carmi R, Itti L (2006) The role ofmemory in guiding attention during natural vision. JVis 6(9):4. doi:10.1167/6.9.4

Multimed Tools Appl

7. Carmi R, Itti L (2006) Visual causes versus correlates of attentional selection in dynamic scenes. Vis Res46(26):4333–4345

8. R Core Team (2014) R: a language and environment for statistical computing. R Foundation for StatisticalComputing, Vienna, Austria. http://www.R-project.org/

9. Cornelissen FW, Peters EM, Palmer J (2002) The Eyelink toolbox: eye tracking with MATLAB and thepsychophysics toolbox. Behav Res Methods Instrum Comput 34(4):613–617

10. Cutting JE, Brunick KL, Candan A (2012) Perceiving event dynamics and parsing Hollywood films. J ExpPsychol Hum Percept Perform 38(6):1476–1490

11. D’Zmura M (1991) Color in visual search. Vis Res 31(6):951–96612. Dorr M, Martinetz T, Gegenfurtner KR, Barth E (2010) Variability of eye movements when viewing

dynamic natural scenes. J Vis 10(10):2813. Duncan J, Humphreys GW (1989) Visual search and stimulus similarity. Psychol Rev 96(3):43314. Foulsham T, Kingstone A (2013) Fixation-dependent memory for natural scenes: an experimental test of

scanpath theory. J Exp Psychol Gen 142(1):4115. Gegenfurtner KR, Rieger J (2000) Sensory and cognitive contributions of color to the recognition of natural

scenes. Curr Biol 10(13):805–80816. Hansen T, Gegenfurtner KR (2009) Independence of color and luminance edges in natural scenes. Vis

Neurosci 26(1):35–4917. Hochberg J, Brooks V (1996) The perception of motion pictures. In: Carterette EC, Friedman MP (Eds)

Handbook of perception and cognition, pp. 205–29218. Huang L, Holcombe AO, Pashler H (2004) Repetition priming in visual search: episodic retrieval, not feature

priming. Mem Cogn 32(1):12–2019. Kirchner H, Thorpe SJ (2006) Ultra-rapid object detection with saccadic eye movements: visual processing

speed revisited. Vis Res 46(11):1762–177620. Kristjánsson Á, Campana G (2010) Where perception meets memory: a review of repletion priming in visual

search tasks. Atten Percept Psychophys 72:5–1821. LamyDF,KristjánssonÁ (2013) Is goal-directed attentional guidance just intertrial priming?A review. JVis 13(3):1422. Levin DT, Simons DJ (2000) Perceiving stability in a changing world: combining shots and integrating

views in motion pictures and the real world. Media Psychol 2(4):357–38023. Luck SJ, Vogel EK (1997) The capacity of visual working memory for features and conjunctions. Nature

390(6657):279–28124. Magliano JP, Zacks JM (2011) The impact of continuity editing in narrative film on event segmentation.

Cogn Sci 35(8):1489–151725. Maljkovic V, Nakayama K (1994) Priming of pop-out: I. Role of features. Mem Cogn 22(6):657–67226. MayJ,DeanMP,BarnardPJ(2003)Usingfilmcuttingtechniquesininterfacedesign.HumComputInteract18:325–37227. Mollon JD (1989) BTho’she kneel’d in that place where they grew…^ The uses and origins of primate colour

vision. J Exp Biol 146(1):21–3828. MurchW,CoppolaFF (1995) In theblinkof an eye: a perspective on filmediting. Silman-JamesPress,BeverlyHills29. Olivers CNL, Meijer F, Theeuwes J (2006) Feature-based memory-driven attentional capture: visual working

memory content affects visual attention. J Exp Psychol Hum Percept Perform 32(5):124330. Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers into movies.

Spat Vis 10(4):437–44231. Reisz K, Millar G (1953) The technique of film editing. Focal Press, London32. Rousselet GA, Fabre-Thorpe M, Thorpe SJ (2002) Parallel processing in high-level categorization of natural

images. Nat Neurosci 5(7):629–63033. Smith TJ (2012) The attentional theory of cinematic continuity. Projections 6(1):1–2734. Smith TJ, Henderson JM (2008) Edit blindness: the relationship between attention and global change

blindness in dynamic scenes. J Eye Mov Res 2(2):1–1735. Smith TJ, Levin D, Cutting JE (2012) A window on reality: perceiving edited moving images. Curr Dir

Psychol Sci 21(2):107–11336. Smith TJ, Mital PK (2013) Attentional synchrony and the influence of viewing task on gaze behavior in

static and dynamic scenes. J Vis 13(8):1637. Standing L (1973) Learning 10000 pictures. Q J Exp Psychol 25(2):207–22238. Swain MJ, Ballard DH (1991) Color indexing. Int J Comput Vis 7(1):11–3239. Valuch C, Ansorge U, Buchinger S, Patrone AR, Scherzer O (2014) The effect of cinematic cuts on human

attention. TVX ‘14: Proceedings of the ACM Interantional Conference on Interactive Experiences for TVand Online Video 2014:119–122. doi: 10.1145/2602299.2602307

40. Valuch C, Becker SI, Ansorge U (2013) Priming of fixations during recognition of natural scenes. J Vis 13(3):341. Wichmann FA, Sharpe LT, Gegenfurtner KR (2002) The contributions of color to recognition memory for

natural scenes. J Exp Psychol Learn Mem Cogn 28(3):509

Multimed Tools Appl

42. Wolfe JM, Horowitz TS (2004) What attributes guide the deployment of visual attention and how do they doit? Nat Rev Neurosci 5(6):495–501

Christian Valuch studied Psychology at the University of Vienna where he graduated in 2011. Since then, he hasbeen working at the Cognitive Science Research Platform of the University of Vienna as a research associate onthe project BModeling Visual Attention as Key Factor in Visual Recognition and Quality of Experience^. In hisresearch, he studies how visual attention is allocated within naturalistic scenes. Among other projects, he uses eyetracking and psychophysical experiments to study the impact of visual memory on attention in edited videos.

Ulrich Ansorge studied Psychology at the University of Bielefeld where he also obtained his PhD in 2000. In2009 he was appointed full professor for Cognitive and Experimental Psychology at the Faculty of Psychology,University of Vienna. He is an expert on top-down control of attention and eye movements. In 2015 he organizesthe 18th European Conference on Eye Movements, the world’s largest and most renowned interdisciplinarymeeting on eye movement research.

Multimed Tools Appl