An Evaluation of Lineup Presentation, Weapon Presence, and a Distinctive Feature using ROC Analysis

This article appeared in a journal published by Elsevier. The attachedcopy is furnished to the author for internal non-commercial researchand education use, including for instruction at the authors institution

and sharing with colleagues.

Other uses, including reproduction and distribution, or selling orlicensing copies, or posting to personal, institutional or third party

websites are prohibited.

In most cases authors are permitted to post their version of thearticle (e.g. in Word or Tex form) to their personal website orinstitutional repository. Authors requiring further information

regarding Elsevier’s archiving and manuscript policies areencouraged to visit:

http://www.elsevier.com/authorsrights

http://www.elsevier.com/authorsrights

Author's personal copy

Journal of Applied Research in Memory and Cognition 3 (2014) 45–53

Contents lists available at ScienceDirect

Journal of Applied Research in Memory and Cognition

jo ur nal homepage: www.elsev ier .com/ locate / ja rmac

An evaluation of lineup presentation, weapon presence, and adistinctive feature using ROC analysis�

Curt A. Carlson ∗, Maria A. Carlson1

Department of Psychology, Counseling, & Special Education, Texas A&M University – Commerce, Commerce, TX 75429, United States

a r t i c l e i n f o

Article history:Received 23 November 2013Received in revised form 24 March 2014Accepted 25 March 2014Available online 12 April 2014

Keywords:Eyewitness identificationReceiver Operating Characteristic analysisSimultaneous and sequential lineupsWeapon effectDistinctive feature

a b s t r a c t

We conducted an experiment (N = 2675) including both laboratory and online participants to testhypotheses regarding important system and estimator variables for eyewitness identification. Simulta-neous lineups were compared to sequential lineups with the suspect presented early versus late becausethere is evidence that suspect position could be an important factor determining a simultaneous versussequential advantage in guilty-innocent suspect discriminability. We also manipulated whether or notthe perpetrator held a weapon or had a distinctive feature on his face, to re-evaluate recent evidence thatthese factors interact. Overall, the simultaneous lineup yielded higher discriminability than the sequentiallineup, and there was no effect of sequential position. Discriminability was higher when the perpetratorhad no weapon, but only when no distinctive feature was present. We conclude with a discussion of theimportance of exploring interactions between system and estimator variables using Receiver OperatingCharacteristic (ROC) analysis.

© 2014 Society for Applied Research in Memory and Cognition. Published by Elsevier Inc. All rightsreserved.

1. Introduction

The number of DNA exonerations continues to increase, nowexceeding 300 (www.innocenceproject.org). A figure that hasremained relatively constant is 75%, the proportion of these casesinvolving mistaken eyewitness identification (Garrett, 2011). Inresponse to this problem, researchers have emphasized system overestimator variables (Wells, 1978) because the former are those overwhich the criminal justice system has control (e.g., lineup presenta-tion method). In contrast, estimator variables can only be estimatedafter the crime has taken place. However, this should not pre-clude research into estimator variables, as they can interact withkey system variables. The present research focused on two esti-mator variables, the presence of a weapon and the presence of adistinctive feature on the perpetrator’s face, that recent researchhas shown can affect eyewitness identification and could interactwith a system variable of considerable interest: lineup presentationmethod.

� This work was supported by a grant from Division 41 of the American Psycho-logical Association to Curt Carlson.

∗ Corresponding author. Tel.: +1 903 468 8723; fax: +1 903 886 5510.E-mail address: [email protected] (C.A. Carlson).

1 Tel.: +1 903 468 8723; fax: +1 903 886 5510.

Until very recently, all research comparing simultaneous (alllineup members presented at once) to sequential (members pre-sented individually) lineup presentation has utilized probativevalue measures conflating two elements of eyewitness decisions:discriminability (between guilty and innocent suspects) and will-ingness to choose. As explained below, this can lead to misleadingconclusions about the purported discriminability benefit of thesequential lineup over the simultaneous lineup because these mea-sures can signal a discriminability difference that is, in fact, drivenby a conservative criterion shift. The present research utilizes atechnique, Receiver Operating Characteristic (ROC) analysis, usedby the medical community to clearly establish discriminabilitydifferences (e.g., between two radiological methods, e.g., Lusted,1971). This approach now needs to be applied in the eyewitnessidentification domain (e.g., Gronlund, Carlson, et al., 2012; Mickes,Flowe, & Wixted, 2012; for a review see Gronlund, Wixted, &Mickes, 2014).

1.1. Important estimator variables: weapon presence and adistinctive facial feature

Although the majority of eyewitness identification research hasfocused on system variables (for a review, see Gronlund & Carlson,2013), significant research has been conducted on some estima-tor variables such as a difference in race between perpetrator

http://dx.doi.org/10.1016/j.jarmac.2014.03.0042211-3681/© 2014 Society for Applied Research in Memory and Cognition. Published by Elsevier Inc. All rights reserved.


46 C.A. Carlson, M.A. Carlson / Journal of Applied Research in Memory and Cognition 3 (2014) 45–53

and eyewitness, stress, and duration of exposure to a perpetra-tor (Wells, Memon, & Penrod, 2006). The present study involvedanother estimator variable that has received some attention: thepresence of a weapon during a crime, which can produce a WeaponFocus Effect (e.g., Cutler & Penrod, 1988; Loftus, Loftus, & Messo,1987; Pickel, 1998). When a perpetrator’s weapon is visible duringa crime, eyewitnesses can focus on it, to the detriment of attentiondirected toward either the perpetrator’s face or other aspects of thecrime. This reduced attention leads to a robust finding of poor mem-ory for visual elements of the crime scene other than the weapon(e.g., Saunders, 2009), and a less consistent finding of less accu-rate eyewitness identification from a lineup (see meta-analyses byFawcett, Russell, Peace, & Christie, 2013, and Steblay, 1992). Dueto the importance of mistaken eyewitness identification revealedby DNA exoneration cases, the present study involves recognitiondecisions from lineups rather than recall of crime details.

Recent research has connected the Weapon Focus Effect withanother potentially important estimator variable: a distinctive fea-ture on the perpetrator’s face (Carlson & Carlson, 2012). Unlikeweapon presence, there has been little research on perpetratordistinctiveness (Carlson, 2011; Carlson & Gronlund, 2011; Zarkadi,Wade, & Stewart, 2009). With the largest experiment (N = 600) onthe Weapon Focus Effect conducted to that point, Carlson and Carl-son replicated the weapon presence decrement by finding lowerprobative value of a suspect identification when the perpetratorheld a gun compared to no weapon. Interestingly, however, thispattern occurred only when the perpetrator did not have a dis-tinctive feature (a large sports sticker) on his face. One goal ofthe present study is to re-evaluate this finding using ROC analysis(described below), which unconfounds discriminability from will-ingness to choose, and which has never (to our knowledge) beenapplied to weapon manipulations or any research evaluating bothsystem and estimator variables.

1.2. Simultaneous versus sequential lineups

We also extended the investigation of this novel interactionbetween a distinctive facial feature and weapon presence tosequential lineups; Carlson and Carlson (2012) only used simul-taneous lineups. In so doing, we continue in the same vein as tworecent studies investigating perpetrator distinctiveness and simul-taneous/sequential lineup presentation: (a) Carlson and Gronlund(2011) found a sequential lineup advantage only for perpetratorspreviously rated as holistically distinctive, and (b) Carlson (2011)extended this effect to perpetrators with a distinctive facial feature(black eye, scar, or mole).

The ability of sequential versus simultaneous lineups to enhanceeyewitness discriminability between guilty and innocent suspectshas become controversial. Historically, there has been evidence ofa sequential superiority effect, such that the sequential lineup low-ers both correct and false identification rates, but reduces the latterto a greater extent (e.g., Lindsay & Wells, 1985; see meta-analysesby Steblay, Dysart, Fulero, & Lindsay, 2001, and Steblay, Dysart, &Wells, 2011). However, recent research has found that these twomethods result either in equivalent discriminability (e.g., Andersen,Carlson, Carlson, & Gronlund, 2014; Clark, 2012; Gronlund, Carlson,Dailey, & Goodsell, 2009; Gronlund, Carlson, et al., 2012; Palmer &Brewer, 2012; see review by Gronlund, Andersen, & Perry, 2012)or that the simultaneous lineup actually produces better discrim-inability (Dobolyi & Dodson, 2013; Mickes et al., 2012).

Wixted and Mickes (in press) presented a Diagnostic Feature-Detection Hypothesis to explain the nature of the simultaneouslineup advantage. Essentially it states that the simultaneous lineupshould yield higher discriminability because the eyewitness isable to compare the lineup members’ faces to each other, gath-ering important information about distinctive features from their

memory of the perpetrator that might set the perpetrator apartfrom all other common features shared by lineup members (e.g.,all Caucasian males in their 20s with dark hair, but guilty suspect isonly one with a particular face shape). Such diagnostic informationis not available to aid recognition during a sequential lineup, as themembers cannot be compared simultaneously. We did not deriveany predictions for the present experiment based on this theory,but we will use it to guide interpretation of some of our resultingpatterns later in this paper.

In addition to comparing simultaneous and sequential lineups,we also manipulated the position of the suspect in the sequentiallineup (position 2 versus 5) because this factor has recently beenshown to interact with simultaneous versus sequential perfor-mance. Using ROC analysis, Gronlund, Carlson, et al. (2012) founda simultaneous lineup advantage only compared to the sequentiallineup with early suspect position (position 2 of 6). They found nodiscriminability difference between simultaneous and sequentiallineups with later suspect position (position 5 of 6). They identi-fied this pattern in data from Gronlund et al. (2009) that collapsedover lineups at various levels of bias toward the suspect (biased,intermediate, and fair). We sought to replicate the pattern with anentirely new data set and with only fair lineups.

1.3. ROC analysis

ROC analysis, commonly used in the memory literature (seereview by Yonelinas & Parks, 2007; see also Swets, Dawes, &Monahan, 2000), is changing the way psychological scientists inter-pret eyewitness accuracy (Carlson, 2013). Probative value measuresrely on various combinations of correct identification (ID) rateand false ID rate. The two most common examples are correctID rate/false ID rate and correct ID rate/(correct ID rate + false IDrate). These both can provide misleading estimates of discrim-inability, as both increase simply as a function of choosing rate. Sixrecent papers have illustrated this problem within the frameworkof simultaneous versus sequential lineups (Andersen et al., 2014;Dobolyi & Dodson, 2013; Gronlund, Carlson, et al., 2012; Gronlundet al., 2014; Mickes et al., 2012; Wixted & Mickes, 2012). Wetherefore constructed ROC curves as a means of identifying whichlineup method yields the best discriminability between innocentand guilty suspects.

Moreover, it is important to use ROC analysis to determinewhether the presence of a weapon reduces discriminability, orwhether it simply makes eyewitnesses less likely to choose from alineup. Either is possible because the majority of the Weapon FocusEffect literature includes correct, but not false identification rate.Both are required, in combination with confidence data, to produceROC curves. Also, this analysis will help delineate the influence of adistinctive feature (on the perpetrator’s face) on weapon presenceand simultaneous versus sequential lineup performance.

1.4. Predictions

The present study tested three primary hypotheses: (a) in repli-cation of recent studies, the simultaneous lineup will producehigher discriminability than will the sequential lineup, (b) thepresence of a weapon will reduce discriminability, and (c) addinga distinctive feature to the perpetrator’s face will eliminate thisweapon effect.

2. Method

2.1. Participants

We collected data from 720 undergraduates, either online(n = 220) or in a laboratory setting (n = 500) across three


C.A. Carlson, M.A. Carlson / Journal of Applied Research in Memory and Cognition 3 (2014) 45–53 47

Table 1Demographic information for university and nationwide samples.

Demographic University Nationwide

GenderMale 202 1107Female 518 848

Age18–24 612 31325–30 65 43031–40 29 48941–50 14 35251–60 0 215Over 60 0 156

EthnicityAfrican-American 115 183Caucasian/White 403 990Hispanic/Latino 101 223Asian 58 220Other 29 192No Response 14 147

720 1955

Midwestern universities, and 2269 additional adult online par-ticipants from across the U.S. with SurveyMonkey.2 Participantsfrom each of these sources were represented in each of ourconditions (i.e., no condition was confounded with location of par-ticipant). We dropped 314 SurveyMonkey participants (range of9–17 dropped across the 24 conditions; SD = 2.76) due to eithertechnical issues watching the mock crime video or an inability toanswer a manipulation check question correctly (described below).Resulting patterns in the data did not differ significantly betweenour laboratory and online samples,3 and so we combined all data(N = 2675) for analyses below. See Table 1 for a demographic break-down of both undergraduate and SurveyMonkey samples.

2.2. Materials and stimuli

We utilized the mock crime videos and lineups from Carlsonand Carlson (2012). The videos were recorded so that participantscould view an assault from a first-person point-of-view. They fea-ture a Caucasian male perpetrator in his early 30s who approachesthe point-of-view of participants and appears to either assault themwith his fists or point a shotgun at them. In half of the conditions,the perpetrator had a large black letter “N” sticker (representingthe first author’s alma mater, the University of Nebraska-Lincoln)on one cheek to serve as the distinctive feature. Following thevideo, participants spent several minutes solving anagrams of U.S.states.

Our participants then viewed either a perpetrator-present (PP)or–absent (PA) six-person lineup presented simultaneously orsequentially. If the perpetrator in the video had the sticker on hisface, then all lineup members did as well (following Zarkadi et al.,2009). The PP and PA lineup contained different foils, all of whommatched the perpetrator’s description and were selected from theFlorida Department of Corrections Inmate Database (see Carlson &Carlson, 2012). To assess lineup fairness for both PP and PA lineup,we presented the perpetrator’s description to a separate group of80 undergraduates, and asked them to select the individual from

2 SurveyMonkey hosted the experiment both for our online undergraduate sam-ple and our large U.S. adult sample. In both cases, participants were not allowed toview the experiment on a mobile device, in order to better assure attention paid tothe experiment and to reduce errors in data transfer.

3 For example, there was no difference in SIM-SEQ effect size (D) between labo-ratory (2.05) and online (1.94) samples.

each lineup (presented in randomized order for each participant)who best matched that description. A common measure of lineupfairness is Tredoux’s E′ (Tredoux, 1998), which is a number from 1 tothe nominal lineup size (6 for our lineups), the higher the number,the fairer the lineup. This analysis revealed that both PP (E′ = 4.98,95% CI 4.29–5.93) and PA lineup (E′ = 4.88, 95% CI 4.23–5.77) werefair.

Carlson and Carlson (2012) collected pilot data to choose adesignated innocent suspect from the foils in the PA lineup. Itis important to designate such an individual so that false iden-tifications of this person can be compared directly with correctidentifications of the perpetrator. Their participants (N = 20) viewedthe mock crime, worked on a filler task, and then chose from thePA lineup the individual they thought might be the perpetrator.The one who was chosen the most became our designated inno-cent suspect. In the present study, the location of both perpetratorand innocent suspect was held constant in position 4 of the simul-taneous lineup. Those viewing a sequential lineup were randomlyassigned to see the perpetrator/innocent suspect in position 2 or5.

2.3. Procedure

Both laboratory and online participants undertook an identicalprocedure. After informed consent, participants were quasi-randomly assigned to one of the conditions based on answers theyprovided to questions (e.g., In what month were you born? Inwhat month was your best friend born?). They then were directedto a page warning them that a video would start playing auto-matically on the following page, and that they were to pay closeattention to it. After viewing the video, participants answered anopen-ended question about any problems there might have beenwith the video (e.g., choppiness due to buffering). After spendingbetween 5 and 10 minutes on the filler task (solving anagrams), asingle six-person lineup was presented (simultaneously or sequen-tially) with instructions that the perpetrator from the video may ormay not be present.

The simultaneous lineup consisted of a single 2 × 3 array of pho-tos requiring a single decision (choose a lineup member or rejectthe lineup). The sequential lineup involved photos presented oneat a time, with a decision made for each (choose/don’t choose), andparticipants were not allowed to return to a photo for which theyhad made a decision. If a sequential or simultaneous lineup mem-ber was identified, the lineup immediately ended. Otherwise, allsix photos were presented in the sequential lineup, though partic-ipants did not know how many lineup members there would be.After providing their identification decision, participants enteredtheir level of confidence on a 1–7 scale, with 1 designated “not atall confident” and 7 labeled “very confident”. Finally, participantsanswered a manipulation check question about the video (whatcolor was the perpetrator’s shirt?) as well as several demographicquestions, including age, gender, race, and level of education. A briefpilot study conducted in advance of the experiment showed that100% of participants who paid attention to the video answered themanipulation check question correctly.

2.4. Design

The fully between-subjects factorial design featured four inde-pendent variables: (a) weapon presence or absence, (b) distinctivefeature (large sticker) on perpetrator’s face or not, (c) perpetrator-present or–absent lineup, and (d) simultaneous lineup with suspectin position 4 or sequential lineup with suspect in position 2 versus5. The number of participants per cell of the design ranged from100 to 130 (M = 111.46, SD = 10.81).



Table 2Identification and rejection rates across conditions.

Lineup Weapon presence Feature presence Perpetrator-present lineups Perpetrator-absent lineups

Correct ID rate Foil ID rate Rejection rate False ID rate Foil ID rate Rejection rate

Simultaneous No Weapon No Feature .367 (40/109) .431 (47/109) .202 (22/109) .036 (4/112) .696 (78/112) .268 (30/112)Feature .231 (27/117) .538 (63/117) .231 (27/117) .092 (12/130) .615 (80/130) .293 (38/130)

Weapon No Feature .462 (48/104) .327 (34/104) .211 (22/104) .080 (8/100) .580 (58/100) .340 (34/100)Feature .250 (28/112) .446 (50/112) .304 (34/112) .047 (6/129) .651 (84/129) .302 (39/129)

Sequential 2 No Weapon No Feature .337 (34/101) .515 (52/101) .148 (15/101) .083 (10/120) .617 (74/120) .300 (36/120)Feature .216 (22/102) .569 (58/102) .216 (22/102) .071 (9/126) .690 (87/126) .239 (30/126)


Sequential 5 No Weapon No Feature .246 (28/114) .561 (64/114) .193 (22/114) .032 (4/124) .710 (88/124) .258 (32/124)Feature .115 (12/104) .731 (76/104) .154 (16/104) .053 (6/114) .711 (81/114) .236 (27/114)


Notes: ID = identification; raw proportions based on number of participants per condition in parentheses.

3. Results and discussion

Following ROC analysis, we will present typical measures of eye-witness identification (correct vs. false identifications) in order to:(a) elucidate patterns shown by ROC analysis, and (b) present thereader with analyses comparable to most of the eyewitness identi-fication literature. See Table 2 for the raw counts and proportionsfor correct identifications (of the perpetrator), false identifications(of the designated innocent suspect), foil identifications, and lineuprejections.

3.1. ROC analysis

In this section we can directly address our three primary pre-dictions: (a) discriminability between guilty and innocent suspectwill be higher for the simultaneous lineup, (b) the presence of a

weapon will reduce discriminability, and (c) adding the distinctivefeature to the perpetrator’s face will eliminate this weapon effect.An ROC curve for each condition is constructed from correct ID rates(for the perpetrator) and false ID rates (for the innocent suspect)across levels of confidence. To illustrate, in Fig. 1, the lower-leftpoint of each curve represents the participants who chose the sus-pect (guilty or innocent) and indicated the highest confidence level.The next point to the right of this point on each curve represents allparticipants who selected the highest level of confidence combinedwith all participants who selected the second-highest level of con-fidence, and so forth until the point farthest to the upper-right ofeach curve represents all participants in that condition. In essence,each curve depicts a cumulative record of discriminability acrossconfidence levels.

The basis for comparing conditions in ROC space is the partialarea under each condition’s ROC curve (pAUC), a non-parametric

Fig. 1. Receiver Operating Characteristic curves for simultaneous (SIM) and sequential (SEQ) lineups.



Table 3Partial area under the curve, with 95% confidence intervals, for each condition.

Lineup Weapon Feature pAUC Confidence limits

SIM .032 .026–.038SEQ2 .026 .021–.032SEQ5 .024 .019–.029Weapon .028 .023–.032No Weapon .026 .022–.030Feature .023 .018–.027No Feature .031 .027–.036

Collapsed over lineupWeapon Feature .030 .023–.036

No Feature .022 .019–.025No Weapon Feature .017 .012–.022

No Feature .036 .030–.042

Collapsed over WeaponSIM Feature .021 .014–.029

No Feature .040 .030–.050SEQ2 Feature .024 .017–.032

No Feature .028 .021–.037SEQ5 Feature .024 .017–.032

No Feature .024 .018–.030

Collapsed over featureSIM Weapon .033 .024–.042

No Weapon .030 .023–.038SEQ2 Weapon .026 .020–.035

No Weapon .026 .019–.034SEQ5 Weapon .027 .020–.034

No Weapon .021 .015–.028

No collapsingSIM Weapon Feature .025 .015–.036


No Feature .044 .034–.056SEQ2 Weapon Feature .030 .020–.041


No Feature .033 .023–.047SEQ5 Weapon Feature .037 .024–.050


No Feature .030 .022–.040

Notes: SIM = simultaneous lineup; SEQ2 = sequential lineup with suspect in position2; SEQ5 = sequential lineup with suspect in position 5. All pAUCs calculated basedon sensitivity = .88.

measure of discriminability. The closer each curve is to the upper-left corner of ROC space, the better that condition performs, becausethere is more space between each curve and the diagonal line rep-resenting chance performance (correct = false ID rate). We do notcompare entire AUCs because lineup data featuring false alarmsof only the innocent suspect (excluding foil identifications) pro-duce false alarm rates below 1.0. Rather, we capped the x-axis ofROC space at just above the highest false ID rate in the conditionsbeing compared (see also Mickes et al., 2012). Table 3 contains thepAUC (along with 95% CIs) for each condition collapsed across theothers (e.g., simultaneous lineup overall, including all feature andweapon manipulations), as well as each condition fully broken-down across all other manipulations (e.g., simultaneous lineuppresented after viewing crime with perpetrator having feature andweapon). Table 4 presents the inferential analyses, featuring thedifference (D, a measure of effect size4) between each pair of con-ditions relevant to our hypotheses5 along with associated p-value.

4 D = (pAUC1 − pAUC2)/s, where s is the standard error of the difference betweenthe two pAUCs estimated with bootstrapping.

5 Note that Table 4 shows no difference between Weapon and No Weapon condi-tions, which appears contrary to our prediction that the No Weapon condition wouldyield higher discriminability than the Weapon condition. There is no conflict here,as we predicted that this pattern would occur only when there is no distinctive

Table 4Results of inferential analysis of pAUC pairs.

Comparison D p

SIM vs. SEQ 1.97 .048SEQ2 vs. SEQ5 0.66 .508Weapon vs. NoWeapon −0.59 .552Feature vs. NoFeature −2.64 .008

Notes: SIM = simultaneous lineup; SEQ2 = sequential lineup with suspect in posi-tion 2; SEQ5 = sequential lineup with suspect in position 5. Negative effect sizesrepresent an advantage for the condition on the right (e.g., NoFeature). All compar-isons based on pAUCs calculated with sensitivity = .88. Alpha = .05 for these plannedcomparisons.

We used the R package from pROC to conduct these analyses (Robinet al., 2011).

3.1.1. Lineup comparisonsIn support of our first hypothesis, the simultaneous lineup (SIM)

yielded greater discriminability than the sequential lineup (SEQ)(see Fig. 1); there was no difference between sequential position 2(SEQ2) and 5 (SEQ5). This SIM advantage places our findings largelyin agreement with those from recent studies featuring ROC analy-sis of simultaneous and sequential lineups (e.g., Dobolyi & Dodson,2013; Mickes et al., 2012). However, the lack of a sequential positioneffect surprised us, as it does not replicate findings by Gronlund,Carlson, et al. (2012), who found that discriminability improvedwith later suspect position (see also Carlson, Gronlund, & Clark,2008). We can only speculate, but it is possible that our null resultwas driven by our low false alarm rates,6 particularly for SEQ5 (seeTable 2). Alternatively, a post hoc power analysis revealed that weonly had a 10% chance of detecting this effect if it is real. Due to theabsence of a SEQ position effect, we collapsed across this variablefor the remainder of our SIM-SEQ comparisons. Of the 16 individualSIM-SEQ comparisons across our weapon and distinctive featuremanipulations, there are two SIM advantages, zero SEQ advantages,and 14 nonsignificant differences ( ̨ = .003 based on Bonferroni cor-rection). Both SIM advantages arose when there was no weapon ordistinctive feature during the mock crime versus: (a) a SEQ lineupafter the mock crime with the feature but no weapon (D = 4.38,p < .001), and (b) a SEQ lineup after the mock crime with the weaponbut not the feature (D = 3.84, p < .001). We will investigate these SIMadvantages with a breakdown of correct versus false identificationsbelow.

3.1.2. Influence of weapon and distinctive featureAfter we collapsed over lineup type, the condition that yielded

the highest discriminability contained no weapon or feature (theNo Weapon/No Feature condition had the largest pAUC; see Fig. 2).It was significantly greater than the Weapon/No Feature condition(D = 2.20, p = .028; alpha set at .05 for these planned compar-isons), which supports our second hypothesis. And in support ofour third hypothesis (and Carlson & Carlson, 2012), this harmto discriminability driven by weapon presence during the mockcrime disappeared when the perpetrator had the distinctive fea-ture on his face (no difference between Weapon/Feature and NoWeapon/Feature conditions, D = 1.32, p = .186).

The No Weapon/No Feature pAUC also was greater than theNo Weapon/Feature pAUC (D = 4.56, p < .001), indicating that thepresence of our particular distinctive feature (large sticker on

feature present. The No Weapon versus Weapon comparison in Table 4 includesboth No Feature and Feature conditions, and the inclusion of the Feature conditioneliminates the weapon effect found when there is No Feature (see Fig. 2).

6 Though our false alarm rates and pAUCs are lower on average than thosereported by Mickes et al. (2012) and Andersen et al. (2014), they are comparable tothose reported by Gronlund, Carlson, et al. (2012) and Dobolyi and Dodson (2013).



Fig. 2. Receiver Operating Characteristic (ROC) curves illustrating the accuracy of eyewitness identification depending on presence of a weapon and a perpetrator’s distinctivefeature, collapsed across simultaneous and sequential lineups.

cheek) also lowered discriminability by itself. We next investigatedwhether the feature actually harmed discriminability more thanthe weapon. There are two ways of determining this. First, one cansee whether the ROC from the condition containing the feature butnot the weapon is closer to the chance diagonal than the ROC fromthe condition with the weapon but not the feature. This is confirmedwith a visual inspection of Fig. 2, and the pAUCs are significantlydifferent, D = 2.00, p = .045. Second, one can determine whether thesize of the weapon effect (2.20) is significantly smaller than the sizeof the distinctive feature effect (4.56). It is (z = 2.36, p = .009). We didnot predict this novel finding, and will discuss it further below.

3.2. Analysis of correct versus false identifications

Based on our hypotheses, there are three results that requirefurther investigation to shed some light on factors that could bedriving the discriminability differences revealed by ROC analysis.First, of the 16 SIM-SEQ comparisons, there were only two signif-icant differences, and they both were SIM advantages. Were theydriven by correct identifications, false identifications, or both? Sec-ond, we found that discriminability was lower after a mock crimewith a visible weapon compared to a similar crime with no weapon.The majority of the Weapon Focus Effect literature has found that,when this effect is occasionally found for eyewitness identification,it tends to be driven by a reduction in correct identifications. How-ever, Carlson and Carlson (2012) found a weapon effect driven byfalse identifications. Which do our data support? Third, the weaponeffect on discriminability was eliminated when the distinctive fea-ture was present. Is the locus of this effect correct identifications,false identifications, or both?

In order to address these questions, we applied logistic regres-sion and individual chi-square analysis to these binary data (e.g.,correct identification of perpetrator or not; false identification ofinnocent suspect or not) in order to identify patterns for correctIDs of the perpetrator and false IDs of the innocent suspect.

See Table 2 for the raw counts and proportions. As we addressthese questions, we provide some speculation as to underlyingmechanisms whenever possible based on the nature of our data.

3.2.1. Why might the two SIM advantages have occurred?Both SIM advantages arose when there was no feature or

weapon prior to presenting the SIM lineup. This created an advan-tage for this condition over the SEQ lineup after the weapon or thefeature was present, but not both. Overall, the best discriminability(collapsed across lineup type) came about when there was no fea-ture or weapon (see Fig. 2), and so it is understandable why the twoSIM conditions both included no feature or weapon. There was noweapon to reduce discriminability, and there was no feature addedto the perpetrator’s face to make for a difficult lineup scenario (allmembers sharing the salient distinctive feature from the video).

The feature in particular could have interfered with a processcontributing to a SIM advantage described by Wixted and Mickes(in press). According to their Diagnostic-Feature Hypothesis, beingable to compare the features among all lineup members simulta-neously allows for discriminability among specific characteristicsdistinguishing the guilty suspect from the foils. In the present study,all lineup members shared the salient distinctive feature (largeblack sticker) if it was present during the mock crime (Zarkadiet al., 2009). If the high saliency of the sticker prevents encodingof the perpetrator’s facial features, then the eyewitness will nothave adequate memory for the perpetrator’s actual features withwhich to compare to the foils’ features. To make matters worse,all lineup members had the salient feature, which could serve todistract the eyewitness from comparing the potentially diagnosticfeatures.

This is one possible explanation for the good discriminabilityfrom the SIM condition with no feature or weapon during the mockcrime, but why did this condition significantly improve discrim-inability over just two particular SEQ conditions (either weapon orfeature, but not both)? This could be due to the fact that each SEQ



condition included a factor that reduced discriminability: the fea-ture or the weapon. Why then would the SEQ condition containingboth feature and weapon not also be worse than this SIM condition?We can only speculate because we did not predict such a specificpattern, but we suggest that perhaps the feature maintained someattention on the perpetrator’s face rather than being drawn to theweapon. Spending more time on the perpetrator’s face would likelyestablish a memory trace that, in turn, should prevent the loss ofcorrect IDs. We do not have the eye tracker data to support thisspeculation, so we will move on to investigate the correct and falseidentifications involved in these two SIM advantages.

Logistic regression revealed a feature x weapon x lineup inter-action (�2(2) = 6.79, p = .03) for correct IDs, though there were notwo-way interactions. There also was a three-way interaction forfalse IDs (�2(2) = 8.67, p = .01), as well as all two-way interactions:weapon and feature (�2(1) = 7.25, p < .001), weapon and lineup(�2(2) = 34.41, p < .001), and feature and lineup (�2(2) = 23.60,p < .001). These interactions are difficult to interpret, and due toour interest in the two SIM advantages, we submitted both SIM-SEQ comparisons to a chi-square test for both correct and false IDs.We found that there were more correct IDs for the SIM lineup (noweapon or feature during mock crime) compared to both SEQ withfeature but no weapon during mock crime (�2(1) = 16.17, p < .001,� = .23) and SEQ with weapon but no feature during mock crime(�2(1) = 6.21, p = .01, � = .14). In contrast, there was no difference infalse IDs between SIM (no weapon or feature) and SEQ with fea-ture and no weapon, and marginally fewer false IDs for SIM (noweapon or feature) compared to SEQ with weapon and no feature(�2(1) = 4.13, p = .05, � = .12).

Evidently there is a complex relationship among lineup method,weapon presence, and distinctive feature presence. However, forthe two SIM advantages the story is a little clearer because bothwere driven by correct IDs more so than false IDs. When there is noweapon or artificial distinctive feature during a crime to impede aneyewitness from encoding the perpetrator’s face, the eyewitnessapparently is able to take advantage of this unimpeded memory torecognize the perpetrator when he or she is presented in a SIMlineup, increasing correct IDs (perhaps due to an evaluation ofdiagnostic features across members; Wixted & Mickes, in press).However, when either a weapon or such a salient feature is presentduring the crime, later discriminability is harmed, which is par-ticularly pronounced when a SEQ lineup is presented. This poorperformance is driven by some unknown weighted combination ofencoding and retrieval factors, but our data cannot speak to thesemechanisms. Finally, it is possible that the largely absent influenceof false IDs could be due to a floor effect, as false ID rate was lowacross conditions (see Table 2).

3.2.2. What created the weapon effect?The mechanism assumed to create the Weapon Focus Effect

is a kind of distraction during encoding of a crime, triggered bythe unexpected nature of a weapon brandished by a perpetrator(e.g., Loftus et al., 1987; Pickel, 1998). The unusual nature of theweapon is thought to draw eyewitnesses’ attention away from theperpetrator’s face, therefore leading to poor discriminability dur-ing a subsequent lineup. This type of encoding effect should reducecorrect IDs because of poorer memory for the perpetrator. How-ever, though the majority of the Weapon Focus Effect literaturepresented only perpetrator-present lineups to test this hypothesis,the effect was often not supported (see meta-analyses by Fawcettet al., 2013, and Steblay, 1992).

Carlson and Carlson (2012) included in their design bothperpetrator-present and–absent lineups (see also Erickson,Lampinen, & Leding, 2014), finding that correct IDs were notreduced, but false IDs increased after seeing a weapon. They didnot speculate as to why this might have occurred, but we note that

the unusualness hypothesis (e.g., Pickel, 1998) would not predictthis pattern. We found that discriminability is lower after viewing aweapon. Moreover, we replicated the general pattern identified byCarlson and Carlson (though theirs was more of a crossover interac-tion). Logistic regression revealed no effect of weapon presence oncorrect IDs, but false IDs did increase significantly after a weaponwas present (�2(1) = 43.90, p < .001). And due to the interaction withdistinctive feature presence, this false IDs pattern did not hold whenthe feature was present; rather, the weapon affected false IDs onlywhen no feature was present, �2(1) = 4.52, p = .034, � = .08. Alas, ourdata cannot speak to the mechanism(s) underlying this false IDs-driven weapon effect, but we strongly encourage future researchto investigate this unexpected pattern.

General discussion

We conducted a large experiment with both laboratory andonline participants in order to conduct ROC analysis on systemand estimator variables influencing eyewitness identification. Weincluded one system variable that has received a great deal ofresearch attention (lineup presentation), one estimator variablethat has yielded a moderate number of published studies (weaponpresence), and one estimator variable that has received scant atten-tion (perpetrator distinctiveness). There are three key findings: (a)the simultaneous lineup yielded higher discriminability than didthe sequential lineup, (b) the presence of a weapon during the mockcrime reduced discriminability, and (c) the presence of an artifi-cial, particularly salient, distinctive feature on the perpetrator’s face(and on all lineup members) eliminated this weapon effect.

A completely novel finding is that the particular distinctive facialfeature we used (large sticker on cheek) harmed eyewitness identi-fication more than the presence of a weapon. It is not surprising thatthis feature reduced the ability of participant-eyewitnesses to accu-rately make lineup decisions, especially because the sticker wasreplicated across lineup members (Zarkadi et al., 2009). In essence,if all that a participant remembered about the perpetrator was thedistinctive feature, and this information was not diagnostic of whothe perpetrator is in the lineup (because all lineup members sharethe feature), then discriminability should suffer. However, we cau-tion the reader before generalizing our distinctive feature effect toofar beyond this experiment, as we did use an extreme distinctivefeature manipulation. It is likely that this kind of feature would dif-fer in many ways, including saliency, compared to more commondistinctive features (e.g., scar, mole, or black eye; Carlson, 2011).

Of course, the vast majority of the Weapon Focus Effect literatureinvolves perpetrators without a salient facial feature. This researchhas found a robust effect for recall of visual crime details, but morelimited evidence for an effect on eyewitness identification (Fawcettet al., 2013; Steblay, 1992). Carlson and Carlson (2012) arguedthat one reason for this limited evidence is that many WeaponFocus Effect studies did not include perpetrator-absent lineups,thereby presenting only a partial view of eyewitness discriminabil-ity. To support this assertion, the weapon effect that they foundwas driven by false identification rate (from perpetrator-absentlineups) and not by correct identification rate (from perpetrator-present lineups). We also found a weapon effect driven entirely byfalse identification rate, and we encourage future research into thisunexpected pattern.

We caution the reader that this weapon effect could be drivenby particular qualities of the mock crime video we used. It standsout from the majority of videos utilized by eyewitness researchersbecause it presents a victim point-of-view in which the perpe-trator points a shotgun (in the weapon condition) directly at thevictim/witness. It is possible that our results (as well as those byCarlson & Carlson, 2012, who utilized the same video and lineups)



will not generalize to other crime situations with a weapon (e.g.,a robbery in which a perpetrator does not brandish a weapon soblatantly). We look forward to future research that applies ROCanalysis to weapon presence as well as other important factorsinfluencing eyewitness identification (e.g., cross-race effect andother estimator variables, interactions between lineup administra-tors and eyewitnesses).

This brings us to the final purpose of the present study: to pro-vide several comparisons of simultaneous and sequential lineupsacross a range of factor combinations. The fact that we found no sig-nificant sequential lineup advantage places our findings in contrastwith two meta-analyses of the literature comparing these lineuptypes that concluded in favor of a sequential lineup advantage(Steblay et al., 2001, 2011). However, all studies included in thesemeta-analyses used measures of probative value (e.g., diagnosticityratios such as correct/false IDs), which are ambiguous in determin-ing differences in discriminability versus choosing criterion.

Our findings are largely in agreement with two more recentmeta-analyses utilizing d’ to show no sequential advantage (Clark,2012; Palmer & Brewer, 2012) as well as other studies applyingROC analysis (e.g., Dobolyi & Dodson, 2013; Mickes et al., 2012).These ROC studies also found a simultaneous lineup advantage indiscriminability. However, the issue of sequential position effectsremains ambiguous. We found no position effect (see also Dobolyi& Dodson, 2013), but Gronlund, Carlson, et al. (2012) found that dis-criminability improved with later positioning of the suspect in thesequential lineup (see also Carlson et al., 2008). Based on post hocpower analysis, our study evidently did not have sufficient powerto detect a sequential position effect, so we await future researchon this potentially important issue.

4. Practical applications

Driven by an extensive literature comparing simultaneous tosequential lineups using measures of probative value that con-flate criterion and discriminability, several states and jurisdictionshave transitioned to presenting sequential lineups to eyewitnesses(Jonsson, 2007). Our findings combined with those by Gronlund,Carlson, et al. (2012) and Mickes et al. (2012) should raise questionsabout this recommendation, as all three of these studies utilizedan analytical technique known to yield a clear picture of discrim-inability across criterion levels (see also Andersen et al., 2014,and Dobolyi & Dodson, 2013). In addition, the present study alongwith that by Gronlund et al. and Mickes et al. featured very largedatasets compared to the extant literature. To illustrate, Steblayet al. (2011) meta-analyzed data from 13,143 participants, whereasthese three studies alone contributed data from 6989 participantsto the literature (53% of the number of participants represented inthe meta-analysis). The evidence against a sequential advantage indiscriminability is mounting.

Finally, we want to step away from the simultaneous-sequentiallineup debate to spotlight a broader issue. There is a paucity ofempirical research investigating potential interactions among esti-mator and system variables. Although there is a sizable amount ofarchival research on estimator variables (e.g., Valentine, Pickering,& Darling, 2003), the results are inconsistent (Horry, Halford,Brewer, Milne, & Bull, 2013). Ever since the call to emphasize sys-tem variable research (Wells, 1978), estimator variables have takena back seat, but they deserve more empirical research attention(preferably with ROC analysis) if they could help determine whichsystem-level procedures should be adopted.

Conflict of interest

The authors declare that they have no conflict of interest.

Acknowledgements

This project was funded by the Early Career Professional grantfrom Division 41 of the American Psychological Association to thefirst author. We would like to thank three anonymous reviewersand editor Ron Fisher for very insightful and helpful comments ona prior version of this manuscript.

References

Andersen, S. A., Carlson, C. A., Carlson, M. A., & Gronlund, S. D. (2014). Individ-ual differences predict eyewitness identification performance. Personality andIndividual Differences, 60, 36–40. http://dx.doi.org/10.1016/j.paid.2013.12.011

Carlson, C. A. (2013). How ROC analysis is changing our understanding ofeyewitness identification. American Psychology-Law Society Fall Newsletter,11–13.

Carlson, C. A. (2011). Influence of a perpetrator’s distinctive facial feature on eye-witness identification from simultaneous versus sequential lineups. AppliedPsychology in Criminal Justice, 7, 77–92.

Carlson, C. A., & Carlson, M. A. (2012). A distinctiveness-driven rever-sal of the weapon-focus effect. Applied Psychology in Criminal Justice, 8,36–53.

Carlson, C. A., & Gronlund, S. D. (2011). Searching for the sequentiallineup advantage: A distinctiveness explanation. Memory, 19, 916–929.http://dx.doi.org/10.1080/09658211.2011.613846

Carlson, C. A., Gronlund, S. D., & Clark, S. E. (2008). Lineup composition, suspectposition and the sequential lineup advantage. Journal of Experimental Psychology:Applied, 14, 118–128. http://dx.doi.org/10.1037/1076-898X.14.2.118

Clark, S. E. (2012). Costs and benefits of eyewitness identification reform: Psycholog-ical science and public policy. Perspectives on Psychological Science, 7, 238–259.http://dx.doi.org/10.1177/1745691612439584

Cutler, B. L., & Penrod, S. D. (1988). Improving the reliability of eyewitness identifi-cation: Lineup construction and presentation. Journal of Applied Psychology, 73,281–290. http://dx.doi.org/10.1037/0021-9010.73.2.281

Dobolyi, D. G., & Dodson, C. S. (2013). Eyewitness confidence in simultaneousand sequential lineups: A criterion shift account for sequential mistakenidentification overconfidence. Journal of Experimental Psychology: Applied,http://dx.doi.org/10.1037/a0034596 (Advance online publication)

Erickson, B. E., Lampinen, J. M., & Leding, J. K. (2014). The weapon focus effectin target-present and target-absent lineups: The roles of threat, novelty, andtiming. Applied Cognitive Psychology, http://dx.doi.org/10.1002/acp.3005 (Onlinefirst publication)

Fawcett, J. M., Russell, E. J., Peace, K. A., & Christie, J. (2013). Of guns and geese: Ameta- analytic review of the ‘weapon focus’ literature. Psychology, Crime & Law,19, 35–66. http://dx.doi.org/10.1080/1068316X.2011.599325

Garrett, B. L. (2011). Convicting the innocent: Where criminal prosecutions go wrong.Cambridge, MA: Harvard University Press.

Gronlund, S. D., Andersen, S. M., & Perry, C. (2012). Presentation methods. In B. Cutler(Ed.), Reform of eyewitness identification procedures. Washington, DC: APA.

Gronlund, S. D., & Carlson, C. A. (2013). System-based research on eyewitness iden-tification. In T. J. Perfect, & D. S. Lindsay (Eds.), The SAGE handbook of appliedmemory. London, UK: SAGE Publications.

Gronlund, S. D., Carlson, C. A., Dailey, S. B., & Goodsell, C. A. (2009). Robustness ofthe sequential lineup advantage. Journal of Experimental Psychology: Applied, 15,140–152. http://dx.doi.org/10.1037/a0015082

Gronlund, S. D., Carlson, C. A., Neuschatz, J. S., Goodsell, C. A., Wetmore, S.,Wooten, A., et al. (2012). Showups versus lineups: An evaluation using ROCanalysis. Journal of Applied Research in Memory and Cognition, 1, 221–228.http://dx.doi.org/10.1016/j.jarmac.2012.09.003

Gronlund, S., Wixted, J., & Mickes, L. (2014). Evaluating eyewitness identificationprocedures using ROC analysis. Current Directions in Psychological Science, 23,3–10.

Horry, R., Halford, P., Brewer, N., Milne, R., & Bull, R. (2013). Archival analyses ofeyewitness identification test outcomes: What can they tell us about eyewit-ness memory? Law and Human Behavior, http://dx.doi.org/10.1037/lhb0000060(Online first publication)

Jonsson, P. (2007, February 6). The police lineup is becomingsuspect practice. Christian Science Monitor,. Retrieved fromhttp://www.csmonitor.com/2007/0206/pp01s02-usju.htm

Lindsay, R. C. L., & Wells, G. L. (1985). Improving eyewitness identifications from line-ups: Simultaneous versus sequential lineups presentations. Journal of AppliedPsychology, 70, 556–564. http://dx.doi.org/10.1037/0021-9010.70.3.556

Loftus, E. F., Loftus, G. R., & Messo, J. (1987). Some facts about ‘weapon focus’. Lawand Human Behavior, 11, 55–62. http://dx.doi.org/10.1007/BF01044839

Lusted, L. B. (1971). Signal detectability and medical decision-making. Science, 171,1217–1219.

Mickes, L., Flowe, H. D., & Wixted, J. T. (2012). Receiver operating characteristicanalysis in the assessment of lineup-based eyewitness memory. Journal of Exper-imental Psychology: Applied, 18, 361–376. http://dx.doi.org/10.1037/a0030609

Palmer, M. A., & Brewer, N. (2012). Sequential presentation encourages less biasedcriterion setting but does not improve discriminability. Law and Human Behavior,36, 247–255. http://dx.doi.org/10.1037/h0093923



Pickel, K. L. (1998). Unusualness and threat as possible causes of ‘weapon focus’.Memory, 6, 277–295. http://dx.doi.org/10.1080/741942361

Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J. C., et al.(2011). pROC: An open-source package for R and S+ to analyze and compareROC curves. BMC Bioinformatics, 12, 77. http://dx.doi.org/10.1186/1471-2105-12-77

Saunders, J. (2009). Memory impairment in the weapon focus effect. Memory &Cognition, 37, 326–335. http://dx.doi.org/10.3758/MC.37.3.32

Steblay, N. M. (1992). A meta-analytic review of the weapon focus effect. Law andHuman Behavior, 16, 413–424.

Steblay, N. M., Dysart, J., Fulero, S., & Lindsay, R. C. L. (2001). Eyewitness accu-racy rates in sequential and simultaneous lineup presentations: A meta-analyticcomparison. Law and Human Behavior, 25, 459–474.

Steblay, N. M., Dysart, J. E., & Wells, G. L. (2011). Seventy-two tests of the sequen-tial lineup superiority effect: A meta-analysis and policy discussion. Psychology,Public Policy, and Law, 17, 99–139. http://dx.doi.org/10.1037/a0021650

Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science canimprove diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26.http://dx.doi.org/10.1111/1529-1006.001

Tredoux, C. G. (1998). Statistical inference on measures of lineup fairness. Law andHuman Behavior, 22, 217–237.

Valentine, T., Pickering, A., & Darling, S. (2003). Characteristics of eyewitness iden-tification that predict the outcome of real lineups. Applied Cognitive Psychology,17, 969–993. http://dx.doi.org/10.1002/acp.939

Wells, G. L. (1978). Applied eyewitness testimony research: System variables andestimator variables. Journal of Personality and Social Psychology, 36, 1546–1557.http://dx.doi.org/10.1037/0022-3514.36.12.1546

Wells, G. L., Memon, A., & Penrod, S. D. (2006). Eyewitness evidence: Improv-ing its probative value. Psychological Science in the Public Interest, 7, 45–75.http://dx.doi.org/10.1111/j.1529-1006.2006.00027.x

Wixted, J. T., & Mickes, L. (2012). The field of eyewitness memory shouldabandon “probative value” and embrace Receiver Operating Char-acteristic analysis. Perspectives on Psychological Science, 7, 275–278.http://dx.doi.org/10.1177/1745691612442906

Wixted, J. T., & Mickes, L. (2014). A signal-detection-based diagnostic-feature modelof eyewitness identification. Psychological Review (in press).

Yonelinas, A. P., & Parks, C. M. (2007). Receiver Operating Characteristics(ROCs) in recognition memory: A review. Psychological Bulletin, 133, 800–832.http://dx.doi.org/10.1037/0033-2909.133.5.800

Zarkadi, T., Wade, K. A., & Stewart, N. (2009). Creating fair lineups forsuspects with distinctive features. Psychological Science, 20, 1448–1453.http://dx.doi.org/10.1111/j.1467-9280.2009.02463.x

An Evaluation of Lineup Presentation, Weapon Presence, and a Distinctive Feature using ROC Analysis

Documents