-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
C H A P T E R 13
Protocol Analysis and Expert Thought:Concurrent Verbalizations
of Thinking
during Experts’ Performance onRepresentative Tasks
K. Anders Ericsson
The superior skills of experts, such as accom-plished musicians
and chess masters, can beamazing to most spectators. For
example,club-level chess players are often puzzled bythe chess
moves of grandmasters and worldchampions. Similarly, many
recreational ath-letes find it inconceivable that most otheradults
– regardless of the amount or type oftraining – have the potential
ever to reachthe performance levels of international com-petitors.
Especially puzzling to philosophersand scientists has been the
question of theextent to which expertise requires innategifts
versus specialized acquired skills andabilities.
One of the most widely used and simplestmethods of gathering
data on exceptionalperformance is to interview the
expertsthemselves. But are experts always capableof describing
their thoughts, their behaviors,and their strategies in a manner
that wouldallow less-skilled individuals to understandhow the
experts do what they do, and per-haps also understand how they
might reachexpert level through appropriate training?To date, there
has been considerable contro-versy over the extent to which experts
are
capable of explaining the nature and struc-ture of their
exceptional performance. Somepioneering scientists, such as Binet
(1893 /1966), questioned the validity of the experts’descriptions
when they found that someexperts gave reports inconsistent with
thoseof other experts. To make matters worse,in those rare cases
that allowed verifica-tion of the strategy by observing the
perfor-mance, discrepancies were found betweenthe reported
strategies and the observations(Watson, 1913). Some of these
discrepancieswere explained, in part, by the hypothe-sis that some
processes were not normallymediated by awareness/attention and
thatthe mere act of engaging in self-observation(introspection)
during performance changedthe content of ongoing thought
processes.These problems led most psychologists infirst half of the
20th century to reject alltypes of introspective verbal reports as
validscientific evidence, and they focused almostexclusively on
observable behavior (Boring,1950).
In response to the problems with thecareful introspective
analysis of images andperceptions, investigators such as John
B.
2 2 3
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
2 2 4 the cambridge handbook of expertise and expert
performance
Watson (1920) and Karl Duncker (1945)introduced a new type of
method to elicitverbal reports. The subjects were askedto “think
aloud” and give immediate ver-bal expression to their thoughts
while theywere engaged in problem solving. In themain body of this
chapter I will review evi-dence that this type of verbal
expressionof thoughts has not been shown to changethe underlying
structure of the thought pro-cesses and thus avoids the problem of
reac-tivity, namely, where the act of generat-ing the reports may
change the cognitiveprocesses that mediate the observed
per-formance. In particular, I will describe themethods of protocol
analysis where verbalreports are elicited, recorded, and encodedto
yield valid data on the underlying thoughtprocesses (Ericsson &
Simon 1980, 1984 ,1993).
Although protocol analysis is generallyaccepted as providing
valid verbalizationsof thought processes (Simon & Kaplan,1989),
these verbal descriptions of thoughtsequences frequently do not
contain suffi-cient detail about the mediating cognitiveprocesses
and the associated knowledge tosatisfy many scientists. For
example, thesereports may not contain the detailed proce-dures that
would allow cognitive scientiststo build complete computer models
that arecapable of regenerating the observed perfor-mance on the
studied tasks. Hence, inves-tigators have continued to search for
alter-native types of verbal reports that generatemore detailed
descriptions. Frequently sci-entists require participants to
explain theirmethods for solving tasks and to give
detaileddescriptions of various aspects. These alter-native
reporting methods elicit additionaland more detailed information
than is spon-taneously verbalized during “think aloud.”The desire
for increased amounts of reportedinformation is central to the
study of exper-tise, so I will briefly discuss whether it
ispossible to increase the amount reportedwithout inducing
reactivity and change ofperformance. The main sections of this
chap-ter describe the methods for eliciting andanalyzing concurrent
and retrospective ver-bal reports and how these methods have
been applied to a number of domains ofexpertise, such as memory
experts, chessmasters, and medical specialists. The chap-ter is
concluded with a broad overview ofthe issues of applying protocol
analysis tothe study of expert performance.
Historical Development of VerbalReports on Thought Processes
Introspection or “looking inside” to uncoverthe structure of
thinking and its mentalimages has a very long history in
philos-ophy. Drawing on the review by Ericssonand Crutcher (1991),
we see that Aristotleis generally given credit for the first
system-atic attempt to record and analyze the struc-ture of
sequences of thoughts. He recountedan example of series of thoughts
mediat-ing the recall of a specific piece of infor-mation from
memory. Aristotle argued thatthinking can be described as a
sequence ofthoughts, where the brief transition periodsbetween
consecutive thoughts do not con-tain any reportable information,
and this hasnever been seriously challenged. However,such a simple
description of thinking wasnot sufficiently detailed to answer the
ques-tions about the nature of thought raised byphilosophers in the
17th, 18th, and 19th cen-turies (Ericsson & Crutcher,
1991).
Most of the introspective analysis ofphilosophers had been based
on self-analysisof the individual philosophers’ own thought.In the
19th century Sir Francis Galton alongwith others introduced several
importantinnovations that set the groundwork forempirical studies
of thinking. For example,Galton (1879, see Crovitz, 1970)
noticedrepeatedly that when he took the same walkthrough a part of
London and looked ata given building on his path, this
eventtriggered frequently the same or similarthoughts in memory.
Galton recreated thisphenomenon by listing the names of themajor
buildings and sights from his walk oncards and then presented a
card at a timeto observe the thoughts that were triggered.From this
self-experiment Galton arguedthat thoughts reoccur with
considerable
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
protocol analysis and expert thought 2 2 5
frequency when the same stimulus isencountered.
Galton (1883) is particularly famous forthe innovation of
interviewing many peo-ple by sending out a list of questions
aboutmental imagery – said to be the first ques-tionnaire. He had
been intrigued by reportsof photographic memory and asked
ques-tions of the acuity of specific memories, suchas the clarity
and brightness of their mem-ory for specific things such as their
break-fast table. He found striking individual dif-ferences in the
clarity or vividness, but noclear superiority of the eminent
scientists;for example, Darwin reported having weakvisual images.
Now a hundred years later itis still unclear what these large
individualdifferences in reported vividness of memoryimages really
reflect. They seem almost com-pletely unrelated to the accuracy of
memoryimages and there is no reproducible evidencefor individuals
with photographic or eide-tic memory (McKelvie, 1995 ;
Richardson,1988).
In one of the first published studies onmemory and expertise
Binet (1893 /1966)reported a pioneering interview of chessplayers
and their ability to play “blind-folded” without seeing a chess
board. Basedon anecdotes and his interviews Binet con-cluded that
the ability required to main-tain chess position in memory during
blind-fold play did not appear to reflect a basicmemory capacity to
store complex visualimages, but a deeper understanding of
thestructure of chess. More troubling, Binetfound that the verbal
descriptions on thevisual images of the mental chess
positionsdiffered markedly among blindfold chessplayers. Some
claimed to see the boardas clearly as if it were shown
perceptu-ally with all the details and even shad-ows. Other chess
players reported seeingno visual images during blindfold play
andclaimed to rely on abstract characteristicsof the chess
position. Unfortunately, therewas no independent evidence to
support orquestion the validity of these diverse intro-spective
reports. Binet’s (1893 /1966) classicreport is a pioneering
analysis of blindfoldchess players’ opinions and
self-observations
and illustrates the problems and limits ofintrospection.
In a similar manner Bryan and Harter(1899) interviewed two
students of telegra-phy as they improved their skill and
foundevidence for an extended plateau for bothas they reached a
rate of around 12 wordsper minute. Both reported that this arrest
indevelopment was associated with attemptsto move away from
encoding the Morsecode into words and to encode the code
intophrases. Subsequent research (Keller, 1958)has found that this
plateau is not a necessarystep toward expert levels of
performanceand referred to it as the phantom plateau.
In parallel with the interviews and theinformal collection of
self-observationsof expertise in everyday life,
laboratoryscientists attempted to refine introspectivemethods to
examine the structure of think-ing. In the beginning of the 20th
century,psychologists at the University of Würzburgpresented
highly trained introspectiveobservers, with standardized questions
andasked them to respond as fast as possible.After reporting their
answers, the observersrecalled as much as possible about
thethoughts that they had while answering thequestions. They tried
to identify the mostbasic elements of their thoughts and imagesto
give as detailed reports as possible. Mostreported thoughts
consisted of visual andauditory images, but some
participantsclaimed to have experienced thoughts with-out any
corresponding imagery – imagelessthoughts. The principle
investigator, KarlBühler (1907), argued that the existence
ofimageless thoughts had far-reaching theoret-ical implications and
was inconsistent withthe basic assumption of Wilhelm Wundt(1897)
that all thoughts were associatedwith particular neural activity in
some partof the brain. Bühler’s (1907) paper led toa heated
exchange between Bühler’s intro-spective observers, who claimed to
haveobserved them, and Wundt (1907), whoargued that these reports
were artifacts ofinappropriate reporting methods and thetheoretical
bias of the observers. A devastat-ing methodological conclusion
arose fromthis controversy: the existence of imageless
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
2 2 6 the cambridge handbook of expertise and expert
performance
thoughts could not be resolved empiricallyby the introspective
method. This findingraised fundamental doubts about
analyticintrospection as a scientific method.
The resulting reaction to the crisis wasto avoid the problem of
having to trustthe participants’ verbal reports about inter-nal
events. Instead of asking individuals todescribe the structure of
their thoughts, par-ticipants were given objective tests of
theirmemory and other abilities. More gener-ally, experimental
psychologists developedstandardized tests with stimuli and
instruc-tions where the same pattern of performancecould be
replicated under controlled con-ditions. Furthermore, the focus of
researchmoved away from complex mental pro-cesses, such as experts’
thinking, and towardprocesses that were assumed to be unaf-fected
by prior experience and knowledge.For example, participants were
given well-defined simple tasks, such as memorizationof lists of
nonsense syllables, e.g., XOK,ZUT, where it is easy to measure
objectiveperformance. In addition, experimentersassumed that
nonsense syllables were com-mitted to memory without any
reportablemediating thoughts, and the interest in col-lecting
verbal reports from participants vir-tually disappeared until the
cognitive revo-lution in the late 1950s.
In one of the pioneering attempts toapply this approach to the
study of exper-tise, Djakow, Petrowski, and Rudik (1927)tested the
basic abilities of world-class chessplayers and compared their
abilities to otheradults. Contrary to the assumed importanceof
superior basic cognitive ability and mem-ory, the international
players were only supe-rior on a single test – a test involving
memoryfor stimuli from their own domain of exper-tise, namely,
chess positions. A few decadeslater de Groot (1946/1978d)
replicated chessplayers’ superior memory for chess positionsand
found that correct recall was closelyrelated to the level of chess
skill of the player.
Many investigators, including the famousbehaviorist and critic
of analytic introspec-tion, John Watson, are very critical of
theaccuracy of verbal descriptions of skilledactivities, such as
where one looks dur-ing a golf swing (Watson, 1913). He real-
ized that many types of complex cogni-tive processes, such as
problem solving, cor-responded to ongoing processes that
wereinherently complex and were mediated byreportable thoughts. In
fact, Watson (1920)was the first investigator to publish a
studywhere a participant was asked to “thinkaloud” while solving a
problem. Accord-ing to Watson, thinking was accompaniedby covert
neural activity of the speechapparatus that is frequently referred
to as“inner speech.” Hence, thinking aloud didnot require
observations by any hypotheticalintrospective capacity, and
thinking aloudmerely gives overt expression to these sub-vocal
verbalizations. Many other investiga-tors proposed similar types of
instructionsto give concurrent verbal expression of one’sthoughts
(see Ericsson & Simon, 1993 , for amore extended historical
review).
The emergence of computers in the 1950sand 1960s and the design
of computer pro-grams that could perform challenging cog-nitive
tasks brought renewed interest inhuman cognition and higher-level
cogni-tive processes. Investigators started study-ing how people
solve problems and makedecisions and attempted to describe andinfer
the thought processes that mediateperformance. They proposed
cognitive the-ories where strategies, concepts, and ruleswere
central to human learning and problemsolving (Miller, Galanter,
& Pribram, 1960).Information-processing theories (Newell
&Simon, 1972) sought computational modelsthat could regenerate
human performanceon well-defined tasks by the application
ofexplicit procedures. Much of the evidencefor these complex
mechanisms was derivedfrom the researchers’ own
self-observation,informal interviews, and systematic ques-tioning
of participants.
Some investigators raised concerns almostimmediately about the
validity of thesedata. For example, Robert Gagné and hiscolleagues
(Gagné & Smith, 1962) demon-strated that requiring
participants to ver-balize reasons for each move in the Towerof
Hanoi improved performance by reduc-ing the number of moves in the
solutionsand improving transfer to more difficultproblems as
compared to a silent control
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
protocol analysis and expert thought 2 2 7
Figure 13 .1. An illustration of the overt verbalizations of
most thoughtspassing through attention while a person thinks aloud
during theperformance of a task.
condition. Although improvements are wel-come to educators, the
requirement toexplain must have changed the sequencesof thoughts
from those normally gener-ated. Other investigators criticized the
valid-ity and accuracy of the retrospective ver-bal reports. For
instance, Verplanck (1962)argued that participants reported that
theyrelied on rules that were inconsistent withtheir observed
selection behavior. Nisbettand Wilson (1977) reported several
exam-ples of experiments in social psychology,where participants
gave explanations thatwere inconsistent with their observed
behav-ior. These findings initially led many inves-tigators to
conclude that all types of verbalreports were tainted by similar
methodolog-ical problems that had plagued introspec-tion and led to
its demise. Herb Simonand I showed in a review (Ericsson andSimon,
1980) that the methods and instruc-tions used to elicit the verbal
reports hada great influence on both the reactivity ofthe verbal
reporting and on the accuracy ofthe reported information. We
developed aparfticular type of methodology to instructparticipants
to elicit consistently valid non-reactive reports of their thoughts
that I willdescribe in the next section.
Protocol Analysis: A Methodologyfor Eliciting Valid Data on
Thinking
The central assumption of protocol analy-sis is that it is
possible to instruct subjects
to verbalize their thoughts in a manner thatdoes not alter the
sequence and contentof thoughts mediating the completion of atask
and therefore should reflect immedi-ately available information
during thinking.
Elicitation of Non-Reactive VerbalReports of Thinking
Based on their theoretical analysis, Ericssonand Simon (1993)
argued that the clos-est connection between actual thoughts
andverbal reports is found when people verbal-ize thoughts that are
spontaneously attendedduring task completion. In Figure 13 .1
weillustrate how most thoughts are given a ver-bal expression.
When people are asked to think aloud(see Ericsson and Simon,
1993 , for completeinstructions), some of their verbalizationsseem
to correspond to merely vocalizing“inner speech,” which would
otherwise haveremained inaudible. Nonverbal thoughts canalso be
often given verbal expression by brieflabels and referents.
Laboratory tasks stud-ied by early cognitive scientists focused
onhow individuals applied knowledge and pro-cedures to novel
problems, such as men-tal multiplication. When, for example,
oneparticipant was asked to think aloud whilementally multiplying
36 by 24 on two testoccasions one week apart, the following
pro-tocols were recorded:
OK, 36 times 2 4 , um, 4 times 6 is 2 4 , 4 ,carry the 2 , 4
times 3 is 12 , 14 , 144 , 0, 2times 6 is 12 , 2 , carry the 1, 2
times 3 is 6,
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
2 2 8 the cambridge handbook of expertise and expert
performance
7, 72 0, 72 0, 144 plus 72 0, so it would be4 , 6, 864 .
36 times 2 4 , 4 , carry the – no wait, 4 ,carry the 2 , 14 ,
144 , 0, 36 times 2 is, 12 , 6,72 , 72 0 plus 144 , 4 , uh, uh, 6,
8, uh, 864 .
In these two examples, the reportedthoughts are not analyzed
into their per-ceptual or imagery components as requiredby
Bühler’s (1907) rejected introspectionistprocedures, but are
merely vocalized innerspeech and verbal expressions of
intermedi-ate steps, such as “carry the 1,” “36,” and “144plus
720.” Furthermore, participants werenot asked to describe or
explain how theysolve these problems and do not generatesuch
descriptions or explanations. Instead,they are asked to stay
focused on generat-ing a solution to the problem and thus onlygive
verbal expression to those thoughts thatspontaneously emerge in
attention duringthe generation of the solution.
If the act of verbalizing participants’thought processes does
not change thesequence of thoughts, then participants’task
performance should not change as aresult of thinking aloud. In a
comprehen-sive review of dozens of studies, Ericsson andSimon
(1993) found no evidence that thesequences of thoughts (accuracy of
perfor-mance) changed when individuals thoughtaloud as they
completed the tasks, com-pared to other individuals who
completedthe same tasks silently. However, some stud-ies have shown
that participants who thinkaloud take somewhat longer to complete
thetasks – presumably due to the additionaltime required to produce
the overt verbal-ization of the thoughts.
The same theoretical framework can alsoexplain why other types
of verbal-reportingprocedures consistently change
cognitiveprocesses, like the findings of Gagné andSmith (1962).
For example, when partici-pants explain why they are selecting
actionsor carefully describe the structure anddetailed content of
their thoughts, they arenot able to merely verbalize each thought
asit emerges, they must engage in additionalcognitive processes to
generate the thoughtscorresponding to the required explanations
and descriptions. This additional cognitiveactivity required to
generate the reportschanges the sequence of generated thoughts(see
Chi, Chapter 10, for a discussion ofthe differences between
explanation andthinking aloud). Instructions to explain thereasons
for one’s problem solving and todescribe the content of thought are
reliablyassociated with changes in the accuracy ofobserved
performance (Ericsson and Simon,1993). Subsequent reviews have
shown thatthe more recent work on effects of ver-bal overshadowing
are consistent with reac-tive consequences of enforced generation
ofextensive verbal descriptions of brief experi-ences (Ericsson,
2002). Even instructions togenerate self-explanations have been
foundto change (actually, improve) participants’comprehension,
memory, and learning com-pared to merely thinking aloud during
theseactivities (Ericsson, 1988a, 2003a; Neuman& Schwarz,
1998).
In summary, adults must already pos-sess the necessary skills
for verbalizing theirthoughts concurrently, because they areable to
think aloud without any system-atic changes to their thought
process aftera brief instruction and familiarization in giv-ing
verbal reports (see Ericsson and Simon1993 , for detailed
instructions and associatedwarm-up tasks recommended for
laboratoryresearch).
Validity of Verbalized Informationwhile Thinking Aloud
The main purpose of instructing partici-pants to give verbal
reports on their think-ing is to gain new information beyond whatis
available with more traditional measuresof performance. If, on the
other hand, ver-bal reports are the only source for somespecific
information about thinking, howcan the accuracy of that information
bevalidated? The standard approach for evalu-ating methodology is
to apply the method insituations where other converging evidenceis
available and where the method’s data candistinguish alternative
models of task perfor-mance and disconfirm all but one
reasonablealternatives.
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
protocol analysis and expert thought 2 2 9
Theories of human cognition (Anderson,1983 ; Newell & Simon,
1972 ; Newell, 1990)proposed computational models that
couldreproduce the observable aspects of humanperformance on
well-defined tasks throughthe application of explicit procedures.
Oneof the principle methods applied by thesescientists is an
analysis of the cognitive task(see Chapter 11 by Schraagen for a
discussionof the methods referred to as cognitive taskanalysis),
and it serves a related purpose inthe analysis of verbal protocols.
Task analy-sis specifies the range of alternative proce-dures that
people could reasonably use, inthe light of their prior knowledge
of facts andprocedures, to generate correct answers to atask.
Moreover, task analysis can be appliedto the analysis of
think-aloud protocols; forexample, during a relatively skilled
activity,namely, mental multiplication, most adultshave only
limited mathematical knowledge.They know the multiplication tables
andonly the standard “pencil and paper” proce-dure taught in school
for solving multiplica-tion problems. Accordingly, one can
predictthat they will solve a specific problem suchas 36 · 24 by
first calculating 4 · 36 = 144 ,then adding 20 · 36 = 720. More
sophisti-cated adults may recognize that 24 · 36 canbe transformed
into (30+6)(30–6) and thatthe formula (a+b)(a−b) = a2−b2 can beused
to calculate 36 · 24 as 302 –62 = 900–36 = 864 .
When adults perform tasks while think-ing aloud the verbalized
information mustreflect information generated from the cog-nitive
processes normally executed duringthe task. By analyzing this
information, theverbalized sequences of thoughts can becompared to
the sequence of intermediateresults required to compute the answer
bydifferent strategies that are specified in atask analysis
(Ericsson & Simon, 1993). Thesequence of thoughts verbalized
while mul-tiplying 24 · 36 mentally (reproduced in theprotocol
examples above) agrees with thesequence of intermediate thoughts
specifiedby one, and only one, of the possible strate-gies for
calculating the answer.
However, the hypothesized sequence ofintermediate products
predicted from the
task analysis may not perfectly correspondto the verbalizations.
Inconsistencies mayresult from instances where, because ofacquired
skill, the original steps are eithernot generated or not attended
as distinctsteps. However, there is persuasive evidencefor the
validity of the thoughts that are ver-balized, that is, that the
verbalizations canreveal sequences of thoughts that matchthose
specified by the task analysis (Ericsson& Simon, 1993). Even if
a highly skilled par-ticipant’s think-aloud report in the
multipli-cation task only consisted of “144” and “720,”the reported
information would still be suf-ficient to reject many alternative
strategiesand skilled adaptations of them becausethese strategies
do not involve the generationof both of the reported intermediate
prod-ucts. The most compelling evidence for thevalidity of the
verbal reports comes from theuse of task analysis to predict a
priori a setof alternative sequences of concurrently ver-balized
thoughts that is associated with thegeneration of the correct
answer to the pre-sented problem.
Furthermore, verbal reports are only oneindicator of the thought
processes that occurduring problem solving. Other indicatorsinclude
reaction times (RTs), error rates, pat-terns of brain activation,
and sequences ofeye fixations. Given that each kind of empir-ical
indicator can be separately recordedand analyzed, it is possible to
examine theconvergent validity established by indepen-dent analyses
of different types of data.In their review, Ericsson and Simon
(1993)found that longer RTs were associated witha longer sequence
of intermediate reportedthoughts. In addition, analyses show a
closecorrespondence between participants’ ver-balized thoughts and
the information thatthey looked at in their environment
(seeEricsson & Simon, 1993 , for a review).
Finally, the validity of verbally reportedthought sequences
depends on the timeinterval between the occurrence of a thoughtand
its verbal report, where the highestvalidity is observed for
concurrent, think-aloud verbalizations. For tasks with
relativelyshort response latencies (less than 5 to 10 sec-onds),
people are typically able to recall their
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
2 30 the cambridge handbook of expertise and expert
performance
sequences of thoughts accurately immedi-ately after the
completion of the task, andthe validity of this type of
retrospectivereports remains very high. However, for cog-nitive
processes of longer duration (longerthan 10 to 30 seconds), recall
of past spe-cific thought sequences becomes more dif-ficult, and
people are increasingly temptedto infer what they must have
thought, thuscreating inferential biases in the
reportedinformation.
Other Types of Verbal Reportswith Serious Validity Problems
Protocol analysis, as proposed by Ericssonand Simon (1980, 1984
, 1993), specifies theconstrained conditions necessary for
valid,non-reactive verbalizations of thinking whileperforming a
well-defined task. Many ofthe problems with verbally reported
infor-mation obtained by other methods can beexplained as
violations of this recommendedprotocol-analysis methodology.
The first problem arises when the inves-tigators ask
participants to give more infor-mation beyond that which is
containedin their recalled thought sequences. Forexample, some
investigators ask participantswhy they responded in a certain
man-ner. Participants may have deliberated onalternative methods;
thus, their recalledthoughts during the solution will providea
sufficient answer, but typically the par-ticipants need to go
beyond any retriev-able memory of their processes to givean answer.
Because participants can accessonly the end-products of their
cognitiveprocesses during perception and memoryretrieval, and they
cannot report why onlyone of several logically possible
thoughtsentered their attention, they must makeinferences or
confabulate answers to suchquestions.
In support of this type of confabula-tion, Nisbett and Wilson
(1977) found thatparticipants’ responses to “why-questions”after
responding in a task were in manycircumstances as inaccurate as
those givenby other participants who merely observedthese
individuals’ performance and tried to
explain it without any memory or first-handexperience of the
processes involved. Moregenerally, Ericsson and Simon (1993)
rec-ommended that one should strive to under-stand these reactive,
albeit typically ben-eficial, effects of instructing students
toexplain their performance. A detailed anal-ysis of the different
verbalizations elicitedduring “think-aloud” and “explain”
instruc-tions should allow investigators to identifythose induced
cognitive processes that areassociated with changes (improvements)
intheir performance.
A very interesting development that cap-italizes on the reactive
effects of generatingexplanations involves instructing students
togenerate self-explanations while they readtext or work on
problems (Chi, de Leeuw,Chiu, & LaVancher, 1994 ; Renkl,
1997).Instructing participants to generate self-explanations has
been shown to increase per-formance beyond that obtained with
merelyhaving them “think aloud,” which did notdiffer from a control
condition (Neuman,Leibowitz, & Schwarz, 2000). The system-atic
experimental comparison of instructionsinvolving explanations or
“thinking aloud”during problem solving has provided furtherinsights
into the differences between mecha-nisms underlying the generation
of explana-tions that alter performance and those thatmerely give
expression to thoughts whilethinking aloud (Berardi-Coletta,
Buyer,Dominowski, & Rellinger, 1995).
The second problem is that scientistsare frequently primarily
interested in thegeneral strategies and methods participantsuse to
solve a broad class of problems ina domain, such as mathematics or
textcomprehension. They often ask participantsto describe their
general methods aftersolving a long series of different tasks,which
often leads to misleading summariesor after-the-fact
reconstructions of whatparticipants think they must have done.In
the rare cases when participants havedeliberately and consistently
applied a singlegeneral strategy to solving the problems,they can
answer such requests easily byrecalling their thought sequence from
any ofthe completed tasks. However, participants
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
protocol analysis and expert thought 2 31
typically employ multiple strategies, andtheir strategy choices
may change duringthe course of an experimental session.Under such
circumstances participantswould have great difficulty describing
asingle strategy that they used consistentlythroughout the
experiment, thus theirreports of such a strategy would be
poorlyrelated to their averaged performance.Hence, reviews of
general strategy descrip-tions show that these reports are usually
notvalid, even when immediate retrospectiveverbal reports after the
performance of eachtrial provide accounts of thought sequencesthat
are consistent with other indicators ofperformance on the same
trials (see Ericsson& Simon, 1993 , for a review).
Similar problems have been encoun-tered in interviews of experts
(Hoffman,1992). When experts are asked to describetheir general
methods in professional activ-ities, they sometimes have
difficulties, andthere is frequently poor correspondencebetween the
behavior of computer pro-grams (expert systems) implementing
theirdescribed methods and their observeddetailed behavior when
presented with thesame tasks and specific situations. Thisfinding
has led many scientists study-ing expertise (Ericsson, 1996a;
Ericsson &Lehmann, 1996; Ericsson & Smith, 1991;Starkes
& Ericsson, 2003) to identify a col-lection of specific tasks
that capture theessence of a given type of expertise. Thesetasks
can then be presented under stan-dardized conditions to experts and
less-skilled individuals, while their think-aloudverbalizations and
other process measuresare recorded.
In sum, to obtain the most valid and com-plete trace of thought
processes, scientistsshould strive to elicit laboratory
conditionswhere participants perform tasks that arerepresentative
of the studied phenomenonand where verbalizations directly reflect
theparticipants’ spontaneous thoughts gener-ated while completing
the task. In the nextsection I will describe how protocol
analysishas been applied to study experts’ superiorperformance on
tasks representative of theirrespective domain of expertise.
Protocol Analysis and theExpert-Performance Approach
The expert-performance approach to exper-tise (Ericsson, 1996a;
Ericsson & Smith,1991) examines the behavior of expertsto
identify situations with challenging taskdemands, where superior
performance inthese tasks captures the essence of exper-tise in the
associated domain. These natu-rally emerging situations can be
recreated aswell-defined tasks calling for immediateaction. The
tasks associated with these sit-uations can then be presented to
individualsat all levels of skill, ranging from novice
tointernational-level expert, under standard-ized conditions in
which participants areinstructed to give concurrent or
retrospec-tive reports.
In this section I will describe the expert-performance approach
and illustrate itsapplication of protocol analysis to studythe
structure of expert performance. First,de Groot’s (1946/1978)
pioneering work onthe study of expert performance in chesswill be
described, followed by more recentextensions in the domain of chess
as well assimilar findings in other domains of exper-tise. Second,
the issue of developing andvalidating theories of the mechanisms
ofindividual experts will be addressed and sev-eral experimental
analyses of expert perfor-mance will be described.
Capturing the Essence of Expertiseand Analyzing Expert
Performance
It is important to avoid the temptation tostudy differences in
performance betweenexperts and novices because there are
readilyavailable tasks to measure such differences.Researchers need
to identify those natu-rally occurring activities that correspondto
the essence of expertise in a domain(Ericsson, 2004 , Chapter 38).
For exam-ple, researchers need to study how chessplayers win
tournament games rather thanJust probing for superior knowledge of
chessand test memory for chess games. Similarly,researchers need to
study how doctors areable to treat patients with more
successful
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
2 32 the cambridge handbook of expertise and expert
performance
outcomes rather than test their knowledgefor medicine and memory
of encounteredpatients. It is, however, difficult to com-pare
different individuals’ levels of naturallyoccurring performance in
a domain becausedifferent individuals’ tasks will differ in
dif-ficulty and many other aspects. For exam-ple, for medical
doctors who primarily treatpatients with severe and complex
problemsbut with a relatively low frequency of fullrecovery, is
their performance better thanthe performance of doctors who
primarilytreat patients with milder forms of the samedisease with
uniform recovery? Unless alldoctors encounter patients with nearly
iden-tical conditions, it will be nearly impossibleto compare the
quality of their performance.The problem of comparing performers’
per-formance for comparable tasks is a generalchallenge for
measuring and capturing supe-rior performance in most domains.
For example, chess players rarely, if ever,encounter the same
chess positions duringthe middle part of chess games (Ericsson&
Smith, 1991). Hence, there are no nat-urally occurring cases where
many chessplayers select moves for the identical com-plex chess
position such that the qualityof their moves can be directly
compared.In a path-braking research effort, de Groot(1946/1978)
addressed this problem by iden-tifying challenging situations
(chess posi-tions) in representative games that requiredimmediate
action, namely, the selection ofthe next move. De Groot then
presentedthe same game situations to chess playersof different
skill levels and instructed themto think aloud while they selected
the nextchess move. Subsequent research has shownthat this method
of presenting representa-tive situations and requiring generation
ofappropriate actions provides the best avail-able measure of chess
skill that predicts per-formance in chess tournaments
(Ericsson,Patel, & Kintsch, 2000; van der Maas
&Wagenmakers, 2005).
the pioneering studies of chess expertise
In his pioneering research on chess expertise,de Groot
(1946/1978) picked out chess posi-
tions that he had analyzed for a long time andestablished an
informal task analysis. Basedon this analysis he could evaluate the
relativemerits of different moves and encode thethoughts verbalized
by chess players whilethey were selecting the best move for
thesepositions.
The verbal protocols of both world-class and skilled club-level
players showedthat both types of players first
familiarizedthemselves with the position and verballyreported
salient and distinctive aspects ofthe position along with potential
lines ofattack or defense. The players then exploredthe
consequences of longer move exchangesby planning alternatives and
evaluating theresulting positions. During these searchesthe players
would identify moves with thebest prospects in order to select the
singlebest move.
De Groot’s (1946/1978) analysis of theprotocols identified two
important differ-ences in cognitive processes that explainedthe
ability of world-class players to selectsuperior moves compared to
club play-ers. De Groot noticed that the less-skilledplayers didn’t
even verbally report think-ing about the best move during their
moveselection, implying that they did not, in fact,think about it.
Thus, their initial inferior rep-resentation of the position must
not haverevealed the value of lines of play startingwith that move.
In contrast, the world-classplayers reported many strong first
moveseven during their initial familiarization withthe chess
position. For example, they wouldnotice weaknesses in the
opponent’s defensethat suggested various lines of attack andthen
examine and systematically comparethe consequences of various
sequences ofmoves. During this second detailed phaseof analysis,
these world-class players wouldoften discover new moves that were
superiorto all the previously generated ones.
mechanisms mediating chess expertise
De Groot’s analysis revealed two differentmechanisms that
mediate the world-classplayers’ superiority in finding and
selectingmoves. The first difference concerns the best
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
protocol analysis and expert thought 2 33
players’ ability to rapidly perceive the rele-vant structure of
the presented chess posi-tion, thus allowing them to identify
weak-nesses and associated lines of attack that
theless-accomplished players never reportednoticing in their verbal
protocols. These pro-cesses involve rapid perception and encod-ing,
and thus only the end products of theseencoding processes are
verbalized. There hasbeen a great deal of research attempting
tostudy the perceptual encoding processes byrecording and analyzing
eye fixations dur-ing brief exposures to reveal the
cognitiveprocesses mediating perception and memoryof chess
positions (see Gobet & Charness,Chapter 30). However, most of
this researchhas not studied the task of selecting the bestmove but
has used alternative task instruc-tions, namely, to recall as many
chess piecesas possible from briefly presented positions,or to find
specific chess pieces in pre-sented postions. These changes in the
tasksappear to alter the mediating cognitive pro-cesses, and the
results cannot therefore bedirectly integrated into accounts of the
rep-resentative expert performance (Ericsson &Kintsch, 2000;
Ericsson & Lehmann, 1996;Ericsson et al., 2000).
The second mechanism that underlies thesuperior performance of
highly skilled play-ers concerns a superior ability to
generatepotential moves by planning. De Groot’sprotocols showed
that during this planningand evaluation process, the masters
oftendiscovered new moves that were better thanthose perceived
initially during the familiar-ization phase. In a more recent study
Char-ness (1981) collected think-aloud protocolson the planning
process during the selectionof a move for a chess position.
Examples ofan analysis of the protocols from a club-leveland an
expert-level chess player are givenin Figure 13 .2 . Consistent
with these exam-ples, Charness (1981) found that the depth
ofplanning increased with greater chess skill. Inaddition, there is
evidence that an increase inthe time available for planning
increases thequality of the moves selected, where moveselection
during regular chess is superior tothat of speed chess with its
limited time formaking the next move (Chabris & Hearst,
2003). Furthermore, highly skilled playershave been shown to be
superior in mentallyplanning out consequences of sequences ofchess
moves in experimental studies. In fact,chess masters, unlike
less-skilled players, areable to play blindfold, without a
visibleboard showing the current position, at a rel-atively high
level (Chabris & Hearst, 2003 ;Karpov, 1995 ; Koltanowski,
1985). Experi-ments show that chess masters are able tomentally
generate the chess positions associ-ated with multiple chess games
without anyexternal memory support when the experi-menter reads
sequences of moves from mul-tiple chess games (Saariluoma, 1991,
1995).
In sum, the analyses of the protocols alongwith experiments show
that expert chessplayers’ ability to generate better movescannot be
completely explained by theirmore extensive knowledge of chess
pat-terns. Recognition of patterns and retrievalof appropriate
moves that they have storedin memory during past experiences of
chessplaying is not sufficient to explain theobserved reasoning
abilities of highly skilledplayers. As their skill increases, they
becomeincreasingly able to encode and manipulateinternal
representations of chess positions toplan the consequences of chess
moves, dis-cover potential threats, and even developnew lines of
attack (Ericsson & Kintsch,1995 ; Saariluoma, 1992). (For a
discussionof the relation between the superior mem-ory for
presented chess positions and thememory demands integral to
selecting chessmoves, see Ericsson et al., 2000, and Gobet&
Charness, Chapter 30.)
medicine and other domains
The expert-performance approach has beenapplied to a wide range
of domains, whereskilled and less-skilled performers solve
rep-resentative problems while thinking aloud.When the review is
restricted to studiesin domains that show reproducibly supe-rior
performance of experts, the think-aloudprotocols reveal patterns of
reports that areconsistent with those observed in chess.For
example, when expert snooker play-ers are instructed to make a shot
for a
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
2 34 the cambridge handbook of expertise and expert
performance
Figure 13 .2 . A chess position presented to chess players with
the instruction to select the best nextmove by white (top panel).
The think-aloud protocols of a good club player (chess rating =
1657) anda chess expert (chess rating = 2004) collected by Charness
(1981) are shown in the bottom panel toillustrate differences in
evaluation and planning for one specific move, P-c5 (white pawn is
movedfrom c4 to c5), which is the best move for this position.
Reported considerations for other potentialmoves have been omitted.
The chess expert considers more alternative move sequences and some
ofthem to a greater depth than the club player does. (From
Ericsson, K. A., & Charness, N., 1994 ,Expert performance: Its
structure and acquisition. American Psychologist, 49(8), 725–747,
Figure 13 .2copyright American Psychological Association).
given configuration of pool balls, they ver-balize deeper plans
and more far-reachingexploration of consequences of their shotsthan
less-skilled players (Abernethy, Neal, &Koning, 1994).
Similarly, athletes at expertlevels given protocols from dynamic
situa-tions in baseball (French, Nevett, Spurgeon,Graham, Rink,
& McPherson, 1996) andsoccer (Ward, Hodges, Williams, &
Starkes,
2004) reveal a more complete and superiorrepresentation of the
current game situa-tion that allow them to prepare for
futureimmediate actions better than less-skilledplayers in the same
domains. In domainsinvolving perceptual diagnosis, such as in
theinterpretation of Electrocardiograms (ECG)(Simpson &
Gilhooly, 1997) and micro-scopic pathology (Crowley, Naus,
Stewart,
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
protocol analysis and expert thought 2 35
& Friedman, 2003), verbal protocols revealthat the experts
are able to encode essentialinformation more accurately and are
moreable to integrate the information into anaccurate
diagnosis.
Most of the research on medical diag-nosis has tried to minimize
the influenceof perceptual factors and has relied primar-ily on
verbal descriptions of scenarios andpatients. This research on
medical exper-tise has shown that the process of generat-ing a
diagnosis becomes more efficient asmedical students complete more
of theirmedical training. The increase in efficiencyis mediated by
higher levels of represen-tation that is acquired to support
clinicalreasoning (Boshuizen & Schmidt, 1992 ;Schmidt &
Boshuizen, 1993). When stud-ies present very challenging medical
prob-lems to specialists and medical students,the experts give more
accurate diagnoses(Ericsson, 2004 ; Norman, Trott, Brooks,
&Smith, 1994). The specialists are also moreable to give
complete and logically sup-ported diagnoses (Patel & Groen,
1991) thatappear to reflect higher-level representa-tions that they
have acquired to support rea-soning about clinical alternative
diagnoses(Ericsson & Kintsch, 1995 ; Ericsson et al.,2000;
Patel, Arocha, & Kaufmann, 1994).
There are also studies showing differ-ences in knowledge between
experts andless-accomplished individuals that mediatesuccessful
task performance in experimen-tal design of experiments in
psychology(Schraagen, 1993) and detection of fraudin financial
accounting (Johnson, Karim, &Berryman, 1991). The work on
account-ing fraud was later developed into a gen-eral theory of
fraud detection (Johnson,Grazioli, Jamal, & Berryman, 2001).
Inthis handbook there are discussions ofthe applications of verbal
report method-ology to study thinking in several differ-ent domains
of expertise, such as medicine(Norman, Eva, Brooks, & Hamstra,
Chap-ter 19), software design (Sonnentag, Niessen,& Volmer,
Chapter 21), professional writ-ing (Kellogg, Chapter 22), artistic
perfor-mance (Noice & Noice, Cahpter 28), chessplaying (Gobet
& Charness, Chapter 30),
exceptional memory (Wilding & Valentine,Chapter 31),
mathematical expertise (But-terworth, Chapter 32), and historical
exper-tise (Voss & Wiley, Chapter 33).
The evidence reviewed in this section hasbeen based primarily on
findings that arebased on averages across groups of experts.In the
next section we will search for evi-dence on the validity of
reported thoughts ofindividual experts as well as individual
dif-ferences between different experts.
Individual Differences and Validityof Verbal Reports from Expert
Performance
It is well established that to be successfulin competitions at
the international level,experts need to have engaged in at leastten
years of intensive training – a findingthat applies even to the
most “talented”individuals (Ericsson Krampe, & Tesch-Romer,
1993 ; Simon & Chase, 1973). Conse-quently, researchers have
not been surprisedthat verbal reports of experts and, thus,the
corresponding sequences of reportedthoughts, differ between expert
performers– at least at the level of detailed thoughts. Inthe
previous section I showed how protocolsuncover many higher-level
characteristics ofexpert performers’ mediating mechanisms,such as
skills supporting the expanded work-ing memory (Ericsson &
Kintsch, 1995). Inthis section I will discuss attempts to
exper-imentally validate the detailed structure ofthe reported
cognitive processes of individ-ual expert performers.
The complexity of the knowledge andacquired skills of expert
performers inmost domains, such as chess and medicine,makes it
virtually impossible to describethe complete structure of the
expertiseof an individual expert. For example,Allen Newell
(personal communication)described a project in which one of
hisgraduate students in the 1970s tried to elicitall the relevant
knowledge of a stamp col-lector. After some forty hours of
interviews,Newell and his student gave up, as therewas no sight of
the end of the knowledgethat the expert had acquired. As it may
bedifficult, perhaps impossible, to describe all
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
2 36 the cambridge handbook of expertise and expert
performance
the knowledge and skills of experts, scien-tists should follow
the recommendationsof the expert-performance approach.Namely, they
should focus on the repro-ducible structure of the experts’
mecha-nisms that mediate their superior perfor-mance on
representative tasks (Ericsson,1996b). Consequently, I will focus
onselected domains of expertise in whichregularities in the verbal
reports of differenttrials with representative tasks have
beenanalyzed.
In the early applications of protocol anal-ysis there were
several studies that col-lected protocols from experts solving
repre-sentative problems while thinking aloud. Forexample, Clarkson
and Metzler (1960) col-lected protocols from a professional
investorconstructing portfolios of investments. Sim-ilar detailed
analyses of individual expertsfrom different domains have been
brieflydescribed in Ericsson and Simon (1993) andHoffman (1992).
These analyses were not,however, formally evaluated, and the
pro-posed mechanisms were not demonstratedto account for
reproducibly superior perfor-mance on representative tasks.
The most extensive applications of theexpert-performance
approach using proto-col analysis to study individual experts
haveexamined people with exceptional memory(Ericsson & Lehmann,
1996). In the intro-duction of this chapter I mentioned
Binet’s(1894) pioneering work studying individu-als with
exceptional memory for numbers.Several subsequent studies
interviewed peo-ple with exceptional memory, such as Luria’s(1968)
Subject S and Hunt and Love’s (1972)VP (see Wilding and Valentine,
1997, for areview). However, the first study to trace
thedevelopment of exceptional memory fromaverage performance to the
best memoryperformance in the world (in some mem-ory tasks) was
conducted in a training studyby Chase and Ericsson (1981, 1982 ;
Erics-son, Chase, & Faloon, 1980). We studied acollege student
(SF) whose initial immedi-ate memory for rapidly presented digits
wasaround 7, in correspondence with the typ-ical average (Miller,
1956), but he eventu-ally acquired exceptional performance
forimmediate memory and after 200 hours of
practice was able to recall over 80 digitsin the digit-span
task. During this extendedtraining period SF gave retrospective
reportson his thought processes after most memorytrials. As his
memory performance startedto increase he reported segmenting the
pre-sented lists into 3 -digit groups and, when-ever possible,
encoding them as runningtimes for various races because SF wasan
avid cross-country runner. For example,SF would encode 358 as a
very fast miletime, 3 minutes and 58 seconds, just belowthe
4-minute mile. The central questionconcerning verbal reports is
whether wecan trust the validity of these reports andwhether the
ability to generate mnemonicrunning-time encodings influences
memory.
To address that issue Bill Chase and Idesigned an experiment to
test the effects ofmnemonic encodings and presented SF withspecial
types of lists of constrained digits. Inaddition to a list of
random digits we pre-sented other lists that were constructed
tocontain only 3 -digits groups that could notbe encoded as running
times, such as 364 asthree minutes and sixty four seconds, in a
list(364 895 481 . . . ). As predicted his perfor-mance decreased
reliably. In another exper-iment we designed digit sequences
whereall 3 -digit groups could be encoded as run-ning times (412
637 524 . . . ) with a reli-able increase in his associated
performance.In over a dozen specially designed experi-ments it was
possible to validate numerousaspects of SF’s acquired memory skill
(Chase& Ericsson, 1981, 1982 ; Ericsson, 1988b).Other
investigators, such as Wenger andPayne (1995), have also relied on
protocolanalysis and other process-tracing data toassess the
mechanisms of individuals whoincreased their memory performance
dra-matically with practice on a list-learningtask.
More generally, this method has beenextended to any individual
with exceptionalmemory performance. During the first step,the
exceptional individuals are given mem-ory tasks where they could
exhibit theirexceptional performance while giving con-current
and/or retrospective verbal reports.These reports are then analyzed
to iden-tify the mediating encoding and retrieval
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
protocol analysis and expert thought 2 37
mechanisms of each exceptional individ-ual. The validity of
these accounts is thenevaluated experimentally by presenting
eachindividual with specially designed memorytasks that would
predictably reduce thatindividuals’ memory performance in a
deci-sive manner (Ericsson, 1985 , 1988b; Wilding& Valentine,
1998). With this methodol-ogy, verbal reported mechanisms of
supe-rior performance have been validated withdesigned experiments
in a wide range ofdomains, such as a waiter with superiormemory for
dinner orders (Ericsson &Polson, 1988a, 1988b), mental
calculators(Chase & Ericsson, 1982) and other indi-viduals with
exceptional memory perfor-mance (Ericsson, 2003b; Ericsson,
Delaney,Weaver, & Mahadevan, 2004).
Exceptional memory performance fornumbers and other types of
“arbitrary” infor-mation appears to require that the
expertperformers sustain attention during the pre-sentation
(Ericsson, 2003b). The difficultyto automate memory skills for
encoding newstimuli makes this type of performance par-ticularly
amenable to examination with pro-tocol analysis. More generally,
when individ-uals change and improve their performancethey appear
able to verbalize their thoughtprocesses during learning (Ericsson
& Simon,1993). This has been seen to extend tolearning of
experts and their ability to altertheir performance through
deliberate prac-tice (Ericsson et al., 1993). There is nowan
emerging body of research that examinesthe microstructure of this
type of trainingand how additional specific deliberate prac-tice
improves particular aspects of the tar-get performance in music
(Chaffin & Imreh,1997; Nielsen, 1999) and in sports (Deakin
&Cobley, 2003 ; Ericsson, 2003c; Ward et al.,2004) – for a more
extended discussion seethe chapter by Ericsson (Chapter 38)
ondeliberate practice.
Conclusion
Protocol analysis of thoughts verbalized dur-ing the experts’
superior performance onrepresentative tasks offers an alternative
tothe problematic methods of directed ques-
tioning and introspection. The think-aloudmodel of verbalization
of thoughts has beenaccepted as a useful foundation for dealingwith
the problems of introspection (see theentry on “Psychology of
Introspection” inthe Routledge Encyclopedia of Philosophy byVon
Eckardt, 1998, and entries on “ProtocolAnalysis” in the Companion
to Cognitive Sci-ence [Ericsson, 1998] and the
InternationalEncyclopedia of the Social and Behavioral Sci-ences
[Ericsson, 2001]. This same theoreti-cal framework for collecting
verbal reportshas led to the accumulation of evidence thathas led
many behaviorists to accept dataon cognitive constructs, such as
memoryand rules (Austin & Delaney, 1998). Conse-quently, the
method of protocol analysis pro-vides a tool that allows
researchers to iden-tify information that pass through
expertperformers’ attention while they generatetheir behavior
without the need to embraceany controversial theoretical
assumptions.In support of this claim, protocol analysishas emerged
as a practical tool to diagnosethinking outside of traditional
cognitive psy-chology and cognitive science. For example,designers
of surveys (Sudman, Bradburn,& Schwarz, 1996), researchers on
second-language learning(Green, 1998) and textcomprehension
passages (Ericsson, 1988a;Pressley & Afflerbach, 1995), and
computersoftware developers (Henderson, Smith,Podd, &
Varela-Alvarez, 1995 ; Hughes &Parkes, 2003) regularly collect
verbal reportsand rely on protocol analysis.
The complexity and diversity of themechanisms mediating skilled
and expertperformance is intimidating. To meet thesechallenges it
is essential to develop meth-ods to allow investigators to
reproduce theexperts’ superior performance under con-trolled and
experimental conditions on tasksthat capture the essence of
expertise in agiven domain. Process tracing, in particu-lar
protocol analysis, will be required touncover detailed information
about most ofthe important mechanisms that are respon-sible for the
superiority of the experts’achievement. Only then will it be
possibleto discover their structure and study theirdevelopment and
refinement with trainingand deliberate practice.
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
2 38 the cambridge handbook of expertise and expert
performance
References
Abernethy, B., Neal, R. J., & Koning, P.
(1994).Visual-perceptual and cognitive differencesbetween expert,
intermediate, and novicesnooker players. Applied Cognitive
Psychology,18, 185–211.
Anderson, J. R. (1983). The architecture of cog-nition.
Cambridge, MA: Harvard UniversityPress.
Austin, J., & Delaney, P. F. (1998). Protocol anal-ysis as a
tool for behavior analysis. Analysis ofVerbal Behavior, 15 ,
41–56.
Berardi-Coletta, B., Buyer, L. S., Dominowski,R. L., &
Rellinger, E. R. (1995). Metacogni-tion and problem solving: A
process-orientedapproach. Journal of Experimental
Psychology:Learning, Memory, & Cognition, 2 1, 205–223 .
Binet, A. (1893 /1966). Mnemonic virtuosity:A study of chess
players. (Original paperappeared in 1893 and was translated by M.
L.Simmel & S. B. Barron). Genetic PsychologyMonographs, 74 ,
127–162 .
Binet, A. (1894). Psychologie des grands calcu-lateurs et
joueurs d’echecs [The psychologyof great calculators and chess
players]. Paris:Libraire Hachette.
Boring, E. B. (1950). A history of experimental psy-chology. New
York: Appleton-Century Crofts.
Boshuizen, H. P. A., & Schmidt, H. G. (1992).On the role of
biomedical knowledge in clin-ical reasoning by experts,
intermediates andnovices. Cognitive Science, 16, 153–184 .
Bryan, W. L., & Harter, N. (1899). Studies on thetelegraphic
language: The acquisition of a hier-archy of habits. Psychological
Review, 6, 345–375 .
Bühler, K. (1907). Tatsachen und Probleme zueiner Psychologie
der Denkvorgaenge: I. UeberGedanken [Facts and problems in a
psychol-ogy of thinking: I. On thoughts]. Archiv für diegesamte
Psychologie, 9, 297–365 .
Chabris, C. F., & Hearst, E. S. (2003). Visualiza-tion,
pattern recognition, and forward search:Effects of playing speed
and sight of the posi-tion on grandmaster chess errors. Cognitive
Sci-ence, 2 7. 637–648.
Chaffin, R., & Imreh, G. (1997). “Pulling teethand torture”:
Musical memory and problemsolving. Thinking and Reasoning, 3 ,
315–336.
Charness, N. (1981). Search in chess: Age and skilldifferences.
Journal of Experimental Psychology:
Human Perception and Performance, 7, 467–476.
Chase, W. G., & Ericsson, K. A. (1981). Skilledmemory. In J.
R. Anderson (Ed.), Cognitiveskills and their acquisition (pp.
141–189). Hills-dale, NJ: Erlbaum.
Chase, W. G., & Ericsson, K. A. (1982). Skill andworking
memory. In G. H. Bower (Ed.), Thepsychology of learning and
motivation, Vol. 16(pp. 1–58). New York: Academic Press.
Chi, M. T. H., de Leeuw, N., Chiu, M.-H., & LaVancher, C.
(1994). Eliciting self-explanations improves understanding.
Cogni-tive Science, 18, 439–477.
Clarkson, G. P., & Metzler, A. H. (1960). Portfo-lio
selection: A heuristic approach. Journal ofFinance, 15 ,
465–480.
Crovitz, H. F. (1970). Galton’s walk: Methods forthe analysis of
thinking, intelligence, and creativ-ity. New York: Harper & Row
Publishers.
Crowley, R. S., Naus, G. J., Stewart, J., &Friedman, C. P.
(2003). Development ofvisual diagnostic expertise in pathology:
Aninformation-processing study. Journal of theAmerican Medical
Informatics Association, 10,39–51.
Crutcher, R. J. (1994). Telling what we know: Theuse of verbal
report methodologies in psycho-logical research. Psychological
Science, 5 , 241–244 .
de Groot, A. (1978). Thought and choice and chess.The Hague:
Mouton. (Original work published1946).
Deakin, J. M., & Cobley, S. (2003). A searchfor deliberate
practice: An examination of thepractice environments in figure
skating and vol-leyball. In J. Starkes & K. A. Ericsson
(Eds.),Expert performance in sport: Recent advances inresearch on
sport expertise (pp. 115–135). Cham-paign, IL: Human Kinetics.
Djakow, J. N., Petrowski, N. W., & Rudik, P. A.(1927).
Psychologie des Schachspiels [The psy-chology of chess]. Berlin:
Walter de Gruyter.
Duncker, K. (1945). On problem solving. Psycho-logical
Monographs, 58(5 , Whole No. 270).
Ericsson, K. A. (1985). Memory skill. CanadianJournal of
Psychology, 39, 188–231.
Ericsson, K. A. (1988a). Concurrent verbalreports on reading and
text comprehension.Text, 8, 295–325 .
Ericsson, K. A. (1988b). Analysis of memory per-formance in
terms of memory skill. In R. J.Sternberg (Ed.), Advances in the
psychology of
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
protocol analysis and expert thought 2 39
human intelligence, Vol. 4 (pp. 137–179). Hills-dale, NJ:
Erlbaum.
Ericsson, K. A. (Ed.) (1996a). The road toexcellence: The
acquisition of expert performancein the arts and sciences, sports,
and games. Mah-wah, NJ: Erlbaum.
Ericsson, K. A. (1996b). The acquisition of expertperformance:
An introduction to some of theissues. In K. A. Ericsson (Ed.), The
road toexcellence: The acquisition of expert performancein the arts
and sciences, sports, and games (pp. 1–50). Mahwah, NJ:
Erlbaum.
Ericsson, K. A. (1998) Protocol analysis. In W.Bechtel & G.
Graham (Eds.), A companionto cognitive science (pp. 425–432).
Cambridge,MA: Basil Blackwell.
Ericsson, K. A. (2001). Protocol analysis in psy-chology. In N.
Smelser & P. Baltes (Eds.), Inter-national Encyclopedia of the
Social and Behav-ioral Sciences (pp. 12256–12262). Oxford,
UK:Elsevier.
Ericsson, K. A. (2002). Toward a procedure foreliciting verbal
expression of nonverbal expe-rience without reactivity:
Interpreting the ver-bal overshadowing effect within the
theoreticalframework for protocol analysis. Applied Cog-nitive
Psychology, 16, 981–987.
Ericsson, K. A. (2003a). Valid and non-reactiveverbalization of
thoughts during performanceof tasks: Toward a solution to the
central prob-lems of introspection as a source of scientificdata.
Journal of Consciousness Studies, 10, 1–18.
Ericsson, K. A. (2003b). Exceptional memori-zers: Made, not
born. Trends in Cognitive Sci-ences, 7, 233–235 .
Ericsson, K. A. (2003c). The development ofelite performance and
deliberate practice: Anupdate from the perspective of the
expert-performance approach. In J. Starkes & K. A.Ericsson
(Eds.), Expert performance in sport:Recent advances in research on
sport exper-tise (pp. 49–81). Champaign, IL: HumanKinetics.
Ericsson, K. A. (2004). Deliberate practice andthe acquisition
and maintenance of expert per-formance in medicine and related
domains.Academic Medicine, 10, S1–S12 .
Ericsson, K. A., Chase, W., & Faloon, S. (1980).Acquisition
of a memory skill. Science, 2 08,1181–1182 .
Ericsson, K. A., & Crutcher, R. J. (1991). Intro-spection
and verbal reports on cognitive pro-cesses – two approaches to the
study of thought
processes: A response to Howe. New Ideas inPsychology, 9,
57–71.
Ericsson, K. A., Delaney, P. F., Weaver, G., &Mahadevan, R.
(2004). Uncovering the struc-ture of a memorist’s superior “basic”
memorycapacity. Cognitive Psychology, 49, 191–237.
Ericsson, K. A., & Kintsch, W. (1995). Long-term working
memory. Psychological Review,102 , 211–245 .
Ericsson, K. A., & Kintsch, W. (2000). Shortcom-ings of
article retrieval structures with slots ofthe type that Gobet
(1993) proposed and mod-eled. British Journal of Psychology, 91,
571–588.
Ericsson, K. A., Krampe, R. Th., & Tesch-Römer,C. (1993).
The role of deliberate practice in theacquisition of expert
performance. Psychologi-cal Review, 100(3), 363–406.
Ericsson, K. A., & Lehmann, A. C. (1996).Expert and
exceptional performance: Evidenceon maximal adaptations on task
constraints.Annual Review of Psychology, 47, 273–305 .
Ericsson, K. A., Patel, V. L., & Kintsch, W.(2000). How
experts’ adaptations to represen-tative task demands account for
the expertiseeffect in memory recall: Comment on Vicenteand Wang
(1998). Psychological Review, 107,578–592 .
Ericsson, K. A., & Polson, P. G. (1988a). Memoryfor
restaurant orders. In M. Chi, R. Glaser, &M. Farr (Eds.), The
nature of expertise (pp. 23–70). Hillsdale, NJ: Erlbaum.
Ericsson, K. A., & Polson, P. G. (1988b). Anexperimental
analysis of a memory skill for din-ner orders. Journal of
Experimental Psychology:Learning, Memory, and Cognition, 14 ,
305–316.
Ericsson, K. A., & Simon, H. A. (1980). Verbalreports as
data. Psychological Review, 87, 215–251.
Ericsson, K. A., & Simon, H. A. (1984). Proto-col analysis:
Verbal reports as data. Cambridge,MA: Bradford books/MIT Press.
Ericsson, K. A., & Simon, H. A. (1993). Protocolanalysis:
Verbal reports as data (revised edition).Cambridge, MA: Bradford
books/MIT Press.
Ericsson, K. A., & Smith, J. (1991). Prospectsand limits in
the empirical study of expertise:An introduction. In K. A. Ericsson
& J. Smith(Eds.), Toward a general theory of
expertise:Prospects and limits (pp. 1–38). Cambridge:Cambridge
University Press.
French, K. E., Nevett, M. E., Spurgeon, J. H.,Graham, K. C.,
Rink, J. E., & McPherson,S. L. (1996). Knowledge representation
and
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
2 40 the cambridge handbook of expertise and expert
performance
problem solution in expert and novice youthbaseball players.
Research Quarterly for Exerciseand Sport, 67, 386–395 .
Gagné, R. H., & Smith, E. C. (1962). A study ofthe effects
of verbalization on problem solving.Journal of Experimental
Psychology, 63 , 12–18.
Galton, F. (1879). Psychometric experiments.Brain, 2 , 148–162
.
Galton, F. (1883). Inquiries into human faculty andits
development. New York: Dutton.
Green, A. J. F. (1998). Using verbal protocols in lan-guage
testing research: A handbook. Cambridge,UK: Cambridge University
Press.
Henderson, R. D., Smith, M. C., Podd, J., &Varela-Alvarez,
H. (1995). A comparison ofthe four prominent user-based methods
forevaluating the usability of computer software.Ergonomics, 39,
2030–2044 .
Hoffman, R. R. (Ed.) (1992). The psychology ofexpertise:
Cognitive research and empirical AI.New York: Springer-Verlag.
Hughes, J., & Parkes, S. (2003). Trends in the useof verbal
protocol analysis in software engineer-ing research. Behaviour
& Information Technol-ogy, 2 2 , 127–140.
Hunt, E., & Love, T. (1972). How good can mem-ory be? In A.
W. Melton & E. Martin (Eds.),Coding processes in human memory
(pp. 237–260). New York: Holt.
Johnson, P. E., Karim, J., & Berryman, R. G.(1991). Effects
of framing on auditor deci-sions. Organizational Behavior &
Human Deci-sion Processes, 50, 75–105
Johnson, P. E., Grazioli, S., Jamal, K., & Berry-man, R. G.
(2001). Detecting deception: Adver-sarial problem solving in a low
base-rate world.Cognitive Science, 2 5 , 355–392 .
Karpov, A. (1995). Grandmaster musings. ChessLife, November, pp.
32–33 .
Keller, F. S. (1958). The phantom plateau. Journalof the
Experimental Analysis of Behavior, 1, 1–13 .
Koltanowski, G. (1985). In the dark. Coraopolis,PA: Chess
Enterprises.
Luria, A. R. (1968). The mind of a mnemonist. NewYork: Avon.
McKelvie, S. J. (1995). The VVIQ and beyond:Vividness and its
measurement. Journal of Men-tal Imagery, 19, 197–252 .
Miller, G. A. (1956). The magical number seven,plus or minus
two: Some limits of our capac-ity for processing information.
PsychologicalReview, 63 , 81–97.
Miller, G. A., Galanter, E., & Pribram, K. H.(1960). Plans
and the structure of behavior. NewYork: Holt, Rinehart, and
Winston.
Neuman, Y., & Schwarz, B. (1998). Is self-explanation while
solving problems helpful?The case of analogical
problem-solving.British Journal of Educational Psychology, 68,15–24
.
Neuman, Y., Leibowitz, L., & Schwarz, B. (2000).Patterns of
verbal mediation during prob-lem solving: A sequential analysis of
self-explanation. Journal of Experimental Education,68, 197–213
.
Newell, A. (1990). Unified theories of cognition.Cambridge, MA:
Harvard University Press.
Newell, A., & Simon, H. A. (1972). Human prob-lem solving.
Englewood Cliffs, NJ: Prentice-Hall.
Nielsen, S. (1999). Regulation of learning strate-gies during
practice: A case study of a singlechurch organ student preparing a
particularwork for a concert performance. Psychology ofMusic, 2 7,
218–229.
Nisbett, R. E., & Wilson, T. D. (1977). Tellingmore than we
can know: Verbal reports onmental processes. Psychological Review,
84 ,231–259.
Norman, G. R., Trott, A. D., Brooks, L. R., &Smith, E. K. M.
(1994). Cognitive differencesin clinical reasoning related to
postgraduatetraining. Teaching and Learning in Medicine,
6,114–120.
Patel, V. L., Arocha, J. F., & Kaufmann, D. R.(1994).
Diagnostic reasoning and medicalexpertise. In D. Medin (Ed.), The
psychology oflearning and motivation, Vol. 30 (pp. 187–251).New
York: Academic Press.
Patel, V. L., & Groen, G. J. (1991). The generaland specific
nature of medical expertise: A crit-ical look. In K. A. Ericsson
& J. Smith (Eds.),Toward a general theory of expertise (pp.
93–125). Cambridge, MA: Cambridge UniversityPress.
Pressley, M., & Afflerbach, P. (1995). Verbal pro-tocols of
reading: The nature of constructivelyresponsive reading. Hillsdale,
NJ: Erlbaum.
Renkl, A. (1997). Learning from worked-outexamples: A study on
individual differences.Cognitive Science, 2 1, 1–29.
Richardson, J. T. E. (1988). Vividness andunvividness:
Reliability, consistency, and valid-ity of subjective imagery
ratings. Journal ofMental Imagery, 12 , 115–122 .
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
protocol analysis and expert thought 2 41
Saariluoma, P. (1991). Aspects of skilled imageryin blindfold
chess. Acta Psychologica, 77, 65–89.
Saariluoma, P. (1992). Error in chess:
Theapperception-restructuring view.
PsychologicalResearch/Psychologische Forschung, 54 , 17–26.
Saariluoma, P. (1995). Chess players’ thinking.London:
Routledge.
Schmidt, H. G., & Boshuizen, H. (1993). Onacquiring
expertise in medicine. EducationalPsychology Review, 5 ,
205–221.
Schraagen, J. M. (1993). How experts solve anovel problem in
experimental design. Cogni-tive Science, 17, 285–309.
Simon, H. A., & Chase, W. G. (1973). Skill inchess. American
Scientist, 61, 394–403 .
Simon, H. A., & Kaplan, C. A. (1989). Founda-tions of
cognitive science. In M. J. Posner (Ed.),Foundations of cognitive
science (pp. 1–47). Cam-bridge, MA: MIT Press.
Simpson, S. A., & Gilhooly, K. J. (1997). Diagnos-tic
thinking processes: Evidence from a con-structive interaction study
of electrocardio-gram (ECG) interpretation. Applied
CognitivePsychology, 11, 543–554 .
Starkes, J., & Ericsson, K. A. (Eds.) (2003).Expert
performance in sport: Recent advancesin research on sport
expertise. Champaign, IL:Human Kinetics.
Sudman, S., Bradburn, N. M., & Schwarz,N. (Eds.) (1996).
Thinking about answers:The application of cognitive processes to
surveymethodology. San Francisco, CA: Jossey-Bass.
van der Maas, H. L. J., & Wagenmakers, E. J.(2005). A
psychometric analysis of chess exper-tise. American Journal of
Psychology, 118, 29–60.
Verplanck, W. S. (1962). Unaware of where’sawareness: Some
verbal operants-notates,moments and notants. In C. W. Eriksen(Ed.),
Behavior and awareness – a symposiumof research and interpretations
(pp. 130–158).Durham, NC: Duke University Press.
Von Eckardt, B. (1998). Psychology of intro-spection. In E.
Craig (Ed.), Routledge encyclo-pedia of philosophy (pp. 842–846).
London:Routledge.
Ward. P., Hodges, N. J., Williams, A. M., &Starkes, J. L.
(2004). Deliberate practice andexpert performance: Defining the
path toexcellence. In A. M. Williams & N. J. Hodges(Eds.),
Skill acquisition in sport: Research, the-ory and practice (pp.
231–258). London, UK:Routledge.
Watson, J. B. (1913). Psychology as the behavioristviews it.
Psychological Review, 2 0, 158–77.
Watson, J. B. (1920). Is thinking merely the actionof language
mechanisms? British Journal of Psy-chology, 11, 87–104 .
Wenger, M. J., & Payne, D. G. (1995). On theacquisition of
mnemonic skill: Application ofskilled memory theory. Journal of
ExperimentalPsychology: Applied, 1, 194–215 .
Wilding, J., & Valentine, E. (1997). Superior mem-ory. Hove,
UK: Psychology Press.
Wundt, W. (1897). Outlines of psychology (Trans-lated by C. H.
Judd). Leipzig: WilhelmEngelmann.
Wundt, W. (1907). Über Ausfrageexperimenteund über die
Methoden zur Psychologie desDenkens [On interrogation experiments
andon the methods of the psychology of thinking].Philosophische
Studien, 3 , 301–360.
Author Notes
This article was prepared in part with sup-port from the
FSCW/Conradi Endowment Fundof Florida State University Foundation.
Theauthor wants to thank Robert Hoffman, KatyNandagopal, and Roy
Roring for their valu-able comments on an earlier draft of
thisChapter.
-
P1: JzG052184097Xc13 CB1040B/Ericsson 0 521 84087 X February 27,
2006 16:30
242