Lynn, *-" S. ■*»"' haw "2) J 3*_rton_ Unwurety L***nes CO* J£E4h^ T^t. .- Box I^M_-- Zxm/2ZS-B±2' Fnl 33 Fol. Title —
Lynn, *-" S.■*»"'
haw
"2) J 3*_rton_ Unwurety L***nes
CO* J£E4h^ T^t. .-Box I^M_-- Zxm/2ZS-B±2'Fnl 33 Fol. Title —
SD 70-576
DECISION MAKING: AN INDIVIDUALLYPARAMETERIZED DETERMINISTIC MODEL
By
Richard Shelly Lynn
fife Space DivisionNorthAmericartßockwell
.
SD 70-576
DECISION MAKING: AN INDIVIDUALLYPARAMETERIZED DETERMINISTIC MODEL
By
Richard Shelly Lynn
fife Space DivisionNorth American Rockwell
fife Space DivisionNorth American Rockwell
FOREWORD
This dissertation was presented by R.S. Lynn to thefaculty of the University of Southern CaliforniaGraduate School in partial fulfillment of the require-ments for the degree of Doctor of Philosophy, withmajor in Psychology. The format is that required bythe university.
The training and research leading to the dissertationwere supported, in part, by the Advanced Degree programof North American Rockwell Corporation.
The dissertation describes a novel collection oftechniques for the analysis and computer simulation ofmanifest aspects of decision making. The publication ofthis document indicates the readiness and capability toapply and extend these techniques to decision makingresearch of broader economic and scientific interest.
- 1HSD 70-576
fife Space DivisionNorthAmerican Rockwell
ACKNOWLEDGMENTS
I wish to acknowledge my appreciation for the advice
and criticism offered by Drs. Norman Cliff, Ronald Weitzman,
and William Michael, my dissertation committee. I am
thankful to Dr. Albert Marston for setting me back on the
correct track during a difficult period in the research. I
am indebted to colleagues Frank Dean and Maurice Braun for
their suggestions on evaluation of theory parameters. I
wish to acknowledge the influence of the papers of Nico
Frijda, Earl Hunt, and Lee Gregg and Herbert Simon.
My graduate training in psychological measurement
began and continued because of the kindness and encourage-
ment of Dr . J. P. Guilford. I shall always be grateful to
my wife and children, who kept faith during the long years
of study, research, and writing.
- v -SD 70-576
fife SpaceDivisionNorthAmerican Rockwell
TABLE OF CONTENTS
ACKNOWLEDGMENTS V
LIST OF TABLES
LIST OF ILLUSTRATIONS xi
Chapter
I. PURPOSE OF THE STUDY 1
11. BACKGROUND OF THE STUDY 9
Computer ModelsArtificial IntelligenceExisting Game-Playing and Problem-
Solving Machines
111. THE EXPERIMENT 15
The GameExperimental ProcedureSubjectsApparatusAnalysis of Verbal Behavior
IV. THE MODEL 23
Induction of Process RulesDevelopment of the Computer ProgramSimulation Program Structure and FunctionDerivation of Individual Parameter Values
- vii -SD 70-576
fife Space DivisionNorthAmerican Rockwell
Chapter
V. EVALUATION OF THE MODEL 48
Simulation of Individual SubjectsSensitivity of Model to Parametric
VariationCross-validation of the Model
VI. SUMMARY AND CONCLUSIONS 75
APPENDIXESAPPENDIXES
Appendix A: Game Instructions 84
Appendix B: Excerpts from All Protocols 87
Appendix C: Synopsis of Decisions for AllSubjects 113
Appendix D: The Simulation Program SourceCode 118
Appendix E: Example Simulations 126
REFERENCES 132
- viii -SD 70-576
fife SpaceDivisionNorthAmerican Rockwell
LIST OF TABLES
Table Page
1. Pre-Game Questionnaire Responses 19
2. Parameter Estimates 47
3. Summary Statistics 49
4. Meaning of Bits 59
5. Average Simulation Error Per Cent Associatedwith Each Model Variation 60
6. Partitioning of Average Error of Simulation . . 63
7. Error Per Cent Summary 70
8. Error Matrices for 16 Analyzed Hands 71
9. Error Matrices for Ik Holdout Hands, UsingParameters Estimated from the 16 AnalyzedHands 72
10. Error Matrices for 1^ Holdout Hands, UsingParameters Estimated from the HoldoutHands 7 3
- ix -SD 70-576
fife Space DivisionNorthAmerican Rockwell
LIST OF ILLUSTRATIONS
Figure
1. Flow Chart of Decision-Making Model 35
SD 70-576
fife Space DivisionNorth American Rockwell
CHAPTER I
PURPOSE OF THE STUDY
The purpose of the research was to study and develop
methodology applicable to computer simulation of manifest
behavior in simple decision making or game playing situa-
tions, and to the evaluation of the adequacy of the simula-
tion. It was not the goal of the research to learn anything
about problem solving, choice making, or any other kind of
cognitive behavior. The techniques to be described might,
however, be employed by researches which do have such goals.
The research was not experimental, in the usual sense of
this word. No hypotheses regarding natural laws were enter-
tained or tested. No assumptions were made about the popu-
lation of Ss sampled, and no generalizations beyond the
particular sample were made. The sample of Ss was too small
and too homogeneous, and the sample of stimuli too limited,
to permit such generalization. From its inception, the
research was not experimental, but methodological.
1 -SD 70-576
fife Space DivisionNorth American Rockwell
A simple experimental task requiring a sequence of
binary choices was presented in the form of a one-person
game . The particular game used was not important to the
purpose; the basic research approach would have been the
same for a wide variety of tasks. It is of course simpler
if the choices are binary, but this is not necessary. The
stimuli should be easily quantifiable to facilitate analy-
sis, simulation, and evaluation. The memory load imposed by
the task should be assumed to be negligible; i.e., all
stimuli should remain in view after presentation. Each S_
must receive the same stimuli in the same sequence. The
methodology is especially applicable in situations in which
the effects of each decision are propagated throughout the
remainder of the sequence. The methodology, in whole or in
part, has potential applications to the study of a variety
of natural decision making problems. Possible examples are:
bid or no bid on a request for proposal, acceptance or re-
jection of a proposal, securities portfolio selection, and
personnel decisions such as hiring, promoting, layoff, and
firing. However, no analysis of any natural decision making
was performed to support the suggestion of applicability.
Some of the techniques employed in this research
were suggested by the background literature. Other
2SD 70-576
fife SpaceDivisionNorthAmerican Rockwell
techniques were invented, only to be discovered later in
more extensive reading. The methodology represented by the
particular combination of techniques is offered as being
unique .The model is based upon formally stated processes
which are strictly deterministic. A formal process model
may be heavily dependent upon stochastic variables for sev-
eral reasons . A theorist may take the approach that random
variables may be used to simulate the effects of a number of
variables he chooses to ignore. Another theorist may admit
that he uses random variables to bridge the gaps in his
knowledge of the natural processes . Without explicit ref-
erence to the need to span areas of ignorance, a theory may
posit that the behavior being simulated, at some points in
an otherwise deterministic process, is fundamentally random
by nature. Stochastic elements were completely excluded
from the model in this research partly for esthetic reasons
and partly because the quantitative evaluation employed
demanded it.
The computer program mechanizing the model makes
decisions which are in effect predictions of what an in-
dividual S_ will do. When these predictions are in error,
the program is set back on the correct track. If an
- 3 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
incorrect move were allowed to stand, the program and S_
being simulated would diverge, and the situations faced by
the machine and the human would no longer be the same. The
bases from which the machine made its subsequent moves would
be different from those used by the S_. The setting-back-on-
the-track technique is fundamental and essential to the
methodology of the research. This technique is necessary
because of the conditional prediction effect, described by
Feldman (196 3) . The use of setting-back -on-the-track in
this research is due to Feldman.
The choice of a programming language in which to
express the model is definitely a methodological matter,
with significant implications for psychological theory.
Some of these implications have been described by Newell and
Simon (1963b) :
To what extent do we make implicit assumptions ofpsychological theory when we decide to write a simu-lation program in an information-processing language?
The mode of expression almost surely influences
that a list-processing language like IPL-V is a (weak)psychological theory. It is an assertion that theelementary information processes that will be dis-covered to underlie human behavior are easily con-structable from the instructions of the list-processing system, that its memory is organized interms of lists and list structures of associatedsymbols, and that it is capable of executing sequences
- k -SD 70-576
:he thought ... It is probable that psychological.ostulates enter the simulation by way of the struc:ure of the programming language. ... We conclude
fife Space DivisionNorthAmerican Rockwell
of behaviors organized as hierarchical list struc-tures, (pp. 422-425)
It must not be understood from the above quotation that pro-
gramming languages other than IPL (Information Processing
Language) could not fairly easily express the same struc-
tures or that IPL must necessarily mechanize only structures
of a certain kind. What is true is that the use of IPL
tends to facilitate characteristic approaches to the model-
ing of human behavior. This is one of the reasons for not
using IPL in the present research; it would have tended to
give direction to the analysis and simulation process when
what was wanted was a more naive approach. The programming
language chosen was a dialect of Fortran. Fortran is per-
haps more neutral with respect to psychological theory im-
plications than other higher level languages, and has the
considerable advantage of being available as a standard (if
not the only) language at nearly every computing installa-
tion.
As a matter of methodology, the analysis which led
to the model was intended to be free of hypotheses, no mat-
ter how plausible, about the processes the Ss brought to
bear upon the decision making task. No particular notions
of memory processes or cognitive concepts were assumed prior
- 5 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
to detailed protocol examination. It was intended that the
protocols speak for themselves, at least in early analysis.
The model predicts the behavior of individual Ss,
not general or ideal behavior. The idiographic approach was
selected because of an interest in predicting individual
variations in behavior. Several Ss were used, not just one
or two. The model was to be structurally homogeneous for
all Ss . To serve as a model for several different individ-
uals, parameterization was needed. Just as Weizenbaum* s
(1966) Eliza program works from different scripts to produce
different verbal behavior, the model in this research works
from profiles of different parameter values to behave as
different subjects.
Frijda (1967) asks,
When a program is constructed on the basis of a given
set of data, how can we make sure that it applies toother sets? With abstract tasks there is not too muchdifficulty. One varies the inputs for both subjects
and program. (p. 66)
This implies cross-validation. The model is individually
parameterized, using parameter estimates which are optimal
for the sample of behavior analyzed. The process of finding
optimal values for these parameters capitalizes upon chance.
Any model which is supposed to simulate human behavior, but
which is not cross -validated, is of unknown generality or
specificity. (However, a model which attempts to predict
- 6 -SD 70-576
fife Space DivisionNorth American Rockwell
the behavior of several Ss is necessarily of some general-
ity, in its structure at least.) The research employs
cross-validation as a means of evaluating the generality of
the estimated values of individual parameters.
Fundamental to the original purpose of the research
was the development of a quantified measure of goodness of
simulation, preferably a measure which could be directly
related to a classical distribution function. In contrast
with this approach would be the usual reliance upon subjec-
tive evaluation of model output, used with nearly all proc-
ess models. Stochastic theories, typified by statistical
learning theories, are also evaluated by essentially sub-
jective techniques, but even when highly quantified tech-
niques are used, they "must be interpreted as validations of
the laws of probability rather than of the psychological
assumptions of the theories, and . . . the classical teststhe classical tests
of statistical significance cannot properly be applied to
testing theories of this kind" (Gregg and Simon, 1967, p.
275). Because the model was not to include stochastic ele-
ments, there was the hope that classical tests of statisti-
cal significance could properly be applied.
It was the purpose of this research to study a
- 7 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
methodology composed of techniques and approaches which were
determined prior to the gathering of any experimental data.
In summary, the methodology included: (a) a strictly de-
terministic process model, (b) with predictions kept on the
correct track, (c) mechanized by a computer program written
in Fortran, (d) based upon decision making processes em-
pirically induced from verbatim protocols, (c) which pre-
dicted the behavior of several Ss , based upon individual
parameterization, (f) with cross-validation input data, and
(g) evaluated by quantitative measures of goodness of fit.
- 8 -SD 70-576
fife Space DivisionNorth American Rockwell
CHAPTER II
BACKGROUND OF THE STUDY
Computer Models
A theory to account for individual playing behavior
in a simple card game was defined pursuant to detailed
The theory is similar to ananalysis of verbatim protocols
information processing model. This class of models has been
described by Feigenbaum (1963) as follows:
a. These are models of mental processes, not
bra in 'hardware. They are psychological models ofmental function. No physiological or neurologicalassumptions are made, nor is any attempt made toexplain information processes in terms of more ele-mentary neural processes.
b. These models conceive of the brain as an
information processor with sense organs as inputchannels, effector organs as output devices, and
with internal programs for testing, comparing, ana-
lyzing, rearranging, and storing information.c. The central processing mechanism is assumed
to be "serial; i.e., capable of doing only one (or a
very few) things at a time .d. These models use as a basic unit the infor-
mation symbol; i.e., a pattern of bits which is
assumed to be the brain's internal representationof environmental data.
- 9 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
c. These models are essentially deterministic,not probabilistic. Random variables play no funda-mental role in them. (pp. 297-298)
Similarly, Simon and Newell (1966) remark:
Information-processing theories of human thinkingemploy unobserved entities —symbols —and unobservedprocesses —elementary information processes. Thetheories provide explanations of behavior that aremechanistic without being physiological. That theyare mechanistic—that they postulate only processescapable of being effected by mechanism—is guaranteedby simulating the behavior predicted on ordinary digi-tal computers. Simulation provides a basis for test-ing the predictions of the theories but does not implythat the protoplasm in the brain resembles the elec-tronic components of the computer. (p. 337)
Basically, the theory posits that the subject inputs
relevant features of the environment as information, proc-
esses this information through stored procedures, changes
the environment, and modifies the stored information. Dif-
ferential employment of specific processes would ideally be
sufficient to account for all observed individual differ-
ences in behavior. The theory is expressed as a process
model (Gregg and Simon, 1967) in the form of a computer
program. Computer simulation of psychological processes is
one facet of the research. Frijda (1967) describes func-
tions in which this simulation can be of value:
Computer programs can serve as unambiguous for-mulations of a theory. The program language is
- 10 -SD 70-576
\
II
fife Space DivisionNorth American Rockwell
*■,
precise; the meaning of a given process is fullydefined by what it does. . . . Computer simulation>is a means to demonstrate and test the consistencyand sufficiency of a theory. If the behavioral datawhich the theory wants to explain are in fact repro-duced by running the program, the theory has beenproved capable of explaining these facts .... Ex-tensive experimentation is possible by running dif-
:> ferent versions of the program; decreasing or in-creasing fit with behavioral data can indicate therole of various components and parameters. (p. 59)i>
i\
Artificial Intelligence
The research was centered on quantitative method-«.
ology in psychological theory construction and testing.i4
Because of its computer mechanized aspects, it has a secon-
1 dary relationship with artificial intelligence. The goal
. of artificial intelligence is the construction of machines
that exhibit behavior one would call intelligent if it wereV
observed in animals..
Slagle (1967) writes:.
The main purposes of Artificial Intelligence andtherefore heuristic programming are to understand(human) intelligence and to use machine intelligenceto acquire knowledge and solve intellectually diffi-cult problems . A researcher having the first purposeis a psychologist. He observes subjects thinkingaloud while trying to solve intellectually difficultproblems. He constructs a model of such problem-solving in the form of a computer program. He noteshow the performance of his program deviates from theperformance of his subjects. He observes the subjectssome more and constructs an improved model. The cycleis repeated over and over . There are two importantadvantages in embodying a model in a computer program.
.
.
.
.
.
.
1
- 11 -iSD 70-576
*
*r
fife Space DivisionNorth American Rockwell
The model is completely specified and consequencesof the model may be obtained by simply running theprogram on a computer. A researcher having thesecond purpose is interested in getting intelligentbehavior and does not care whether the computer usesmethods that people use. (pp. 3-4)
Slagle thus identifies both psychological and ma-
chine performance interests within the area of artificial
intelligence. In contrast are Feigenbaum and Feldman
(1963) . They draw a distinction between artificial intelli-
gence research and simulation of cognitive processes re-
search :
An artificial intelligence researcher interestedin programming a computer to play chess would behappy only if his program played good chess, prefer-ably better chess than the best human player. How-ever, the researcher interested in simulating thechess -playing behavior of a given individual wouldbe unhappy if his program played better (or worse)than that individual, for this researcher wants hisprogram to make the same moves as the human player,regardless of whether these moves are good, bad, orindifferent. (p. 269)
Existing Game-Playing and Problem-Solving Machines
Three categories of machines may be distinguished.
In the first and largest category are machines designed to
behave intelligently without simulating human techniques.
In the second are machines that are supposed to simulate
generalized or idealized human techniques and processes.
- 12 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
In the third and smallest category are machines that attempt
to simulate specific individuals. Category 1 machines may
employ search techniques that are virtually impossible for
humans to use, being based upon extreme depth of search and
superhuman speed and precision. Few of the game-playing
machines reported in the literature examined appear clearly
to be in categories 2 or 3, though the authors of the NSS
chess player indicate that their machine uses some human
techniques (Newell, Shaw, and Simon, 1958; see De Groot,
1965, p. 376). The General Problem Solver (GPS) seems to be
a category 2 problem-solving system (Newell and Simon,
1963a; Ernst and Newell, 1967). Category 3 programs are
Feldman's (1963) binary choice simulator, Clarkson's (1963)
trust investment selection program, and Johnson's (1964)
concept-formation model. The General Game Playing Program
(GGPP) of Thomas Williams (1965) can play many games, by
following rules and making legal moves. Thiele, Lemke, and
Fu (196 3) programmed a machine to play a modified game of
hearts. Balzer (1966) has written a machine program that
decides which cards to pass in the game of hearts. Berle-
kamp (196 3) has described a program for solving problems in
no-trump double-dummy bridge problems. Except for GPS and
the Johnson model, the aforementioned systems seem to be
- 13 -SD 70-576
fife SpaceDivisionNorth American Rockwell
moribund. Still very much alive is the checker player of
Samuel (1959) . No published references have been located on
the chess player of McCarthy at Stanford. Greenblatt (1967)
at MIT has described a successful chess player. Both chess
machines are in current research and development.
Feigenbaum's (1963) EPAM system demonstrates certain
phenomena usually associated with human behavior in paired
associates and serial anticipation learning of nonsense
syllables. Laughery and Gregg (1962) developed a program
that would follow a scheme to simulate the behavior of in-
dividual Ss in serial learning. Simon and Kotovsky (1963)
say they have modeled human techniques in serial pattern
learning. Findler (1966) presents an attempt to model be-
havior in a complex man-machine interactive problem.
Hunt has done a significant amount of research in
this area and presents a good review of computer simulation
studies (1968). He warns, "Teachers of general psychology
should be aware that, as of this date, no program has been
shown to simulate human problem solving, although there have
been several programs which solve problems" (p. 160). In an
overview of artificial intelligence studies, Solomonoff
(1966) reviews no work of much significance to psychological
theory.
ll+ -SD 70-576
fife Space DivisionNorthAmerican Rockwell
CHAPTER 111
THE EXPERIMENT
The Game
Instructing the subject to verbalize about his be-
havior is a good way to get information about aspects of
behavior that are difficult to observe. In a competitive
game, concurrent or post hoc verbalization would disclose
strategy, etc., to the opponents and disrupt the competitive
aspect of the play. Therefore, a one-person game was con-
sidered necessary. A simple card game of unknown origin was
selected. It is coincidental that one £ vaguely recalled a
similar two-handed game. In the game, a 24-card deck of
ordinary playing cards is used, ace through queen of one red
and one black suit. The top 10 cards, one at a time, are
exposed. The player must accept five of the 10 cards. The
goal of the game is to maximize the sum of the values of the
five selected cards, with ace worth one point and queen
worth 12 points. The reds and blacks are summed separately,
- 15 -SD 70-576
fife SpaceDivisionNorthAmerican Rockwell
and the larger sum is the recorded score. (See the game
instructions in Appendix A.)
The game was not analyzed prior to the experimental
sessions . It is possible that an optimum strategy does
exist, involving the expected value of the unexposed reds
and blacks. Computation of these expected values requires
fractions with denominators which quickly become too large
for unaided calculation in any reasonable time. Regardless
of the rules of thumb, heuristics, or calculations a S_ might
use, the game consists of a sequence of binary decisions,
each of which is necessarily influenced by the effects of
previous decisions in the same hand. When the first of the
10 possible cards to be offered is exposed, S_ must decide
to accept or reject it. S_ may reject the first five cards,
in which case the acceptance of the last five is forced. If
the offered card is accepted, it is held by S_; otherwise it
is left on the table face up. Within the first few cards,
S_ will accept one. Now color (suit) necessarily becomes of
some importance. Only the suit with the highest sum counts
for the score on a hand. A priori, a bias for accepting
cards of the same color as the first would seem reasonable.
If S_ has accepted what he considers a high card of one
color, he might thereafter accept only cards of that color.
- 16 -SD 70-576
fife Space DivisionNorth American Rockwell
If he has selected only one or two cards with a modest par-
tial sum, he may switch to the other color if a card of
sufficiently high magnitude is offered. Whether his handhas one or both suits, he should select cards which willtend to give him the highest final score. He might wish to
hold out for only high cards, but he knows or quickly learnsthat the unseen remaining cards might all be lower than the
ones he has passed up. As more cards are displayed, the
player has increasingly more information about the unexposed
cards, but he has fewer opportunities left to make selec-
tions. The minimum number of choices is five, which obtainsif S either accepts or rejects all of the first five offered
cards. The maximum number of choices is nine, consisting of
four rejections and five acceptances, in any order.
Experimental Procedure
The 12 Ss were recruited and tested by an independ-
ent experimenter as part of a separate research project.
Each S_ was administered the same sequence of decks, 30 in
each of two one-hour sessions spaced one week apart. During
the play, E displayed the top 10 cards, one at a time, call-
ing out the color and value. S was instructed to "think
aloud" as he decided whether to accept or reject each card.
17 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
After all hands had been played, E conducted an inquiry,
asking S what processes he employed. The verbalization wastape-recorded, and the verbatim transcription of the record-
ings constituted the experimental data.
Subjects
The 12 Ss, six male and six female, were enrolled
in introductory psychology classes at the University of
Southern California. They volunteered to participate in the
experiment in partial fulfillment of an obligation imposed
upon all such students. No monetary reward or other incen-
tive to participate was offered. No incentive was employed
to motivate high scores in the experimental task.
Subject responses to a pre-game questionnaire are
summarized in Table 1.
The grade point average is based upon A = 4 points .The last four columns of the table are based upon responses
to the following question and instructions. "Which of the
following games have you played during the past 12 months?
Please answer each of the following with a check mark." The
games specified were blackjack, bridge, canasta, hearts,
pinochle, poker, and rummy. The checks could be made under
"often" (average of once per week), "sometimes" (average of
18 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
19 -SD 70-576
tnQJErdO
S-lQJ> in ro r^ (N ro ro kO
OJ
<tf
r^
tnG
"H>.id
r-ia,
EO
rHQJ
CO
<-i <N ro o CM
CM
ro ro CM r-i in ro
4-1o
inOJE
"H-PQJEo
c/_
OJ
O rH o ro CM CN
CN CM
O o O>ivcI)
en0)s-ife
cQJ
4-14-1O
o o o o O O CN CN o O o Ow0)o.ao0.
cc;
CDSh
"H
CDC
ao"H-P
CQ
Q)
a?
0)or"
oS3inQJ O
2oS3
tna)
inQJ
inOJ1*
in ina;J*
tnat QJ
OJQJ tnrd vv 0)O >
inr-~(N
in
OJ
"roro
OJ
oro
O#
ro(N
ror^
t
CN
m
CM
I
rooro
inOJro
in
t
rooro
0)E
CO
Oi
0)*H
PL,
QJCOJ
rd tn+J >ig rfiaja
rdcrdMP
U
0.
EOJ
rHW
Otno<-i"H
C■HcTSrd
tn3
CQ
r-irdo
"HT_)
QJE0)>-i
rd rdO M
-H Q)tn -C>. -Pfia.
-H Q)rd OU C
"H QJ-P -H"H OrH 03o
«Hidv
"H
gQJVi0.
>itno
rHo"H
CQ
tnS-itncQJ
v1)2
rH cvrd vo c
-H d)-P -H"H UrH 03OOn
S-iO
■i—ird2
QJtn<
CTiiH
00I—l
O
CN
CT.i—l rH
CDrH
CTirH
OJfN rH
00rH
o(N
r^
tntnrdrHv
rC01QJS-lfe
"g,o
c/_
■g,o
COoCO
tnQJi-ife
rCtnOJvfe
■g,oU_
S-lO
"HcQJ
CO
tnOJ
fe
fitnQJS-lfe
S-lo"Hc»-3
fitnOJ
fe
X0)
COfe fe fe 2 2 2 fe 2 2 fe 2 fe
rOl3 S3CO
rH
CN
ro <tf m vo r^ CO (Ti oiH f-i
OliH
fife SpaceDivisionNorth American Rockwell
once per month), "seldom" (average of four times a year),
and "never."
Apparatus
Ordinary playing cards were used as stimulus mate-
rials. Because the decks consisted of only two suits red
or black and aces through queens—each standard 52-card deckyielded two 24 -card decks. A computer program was used to
shuffle the cards in a pseudorandom arrangement. Sixty in-
dependent shufflings were used to produce the 60 decks used
in the experiment. For each trial, the integers 1 through
24 were randomly permuted. With the cards arranged black
ace through black queen and red ace through red queen,
numerals representing the permuted integers were marked on
the backs with a felt-tip pen. Arranging the decks with the
numerals in monotonically increasing order produced the
desired random shuffle. By marking the trial number also
on the back of each card, it was simple to present each S
with the same sequence of stimuli by reordering the cards
between sessions. A tape recorder was used to record S_'s
response to each stimulus and his concurrent verbalization.
A room ordinarily used for research with human Ss was em-
ployed.
- 20 -SD 70-576
fife SpaceDivisionNorthAmerican Rockwell
Analysis of Verbal Behavior
S_s were instructed to "think aloud" while responding
to the offered cards. They were not, however, encouraged to
be especially productive verbally. To have reminded Ss
frequently to explain why they accepted or rejected each
card would have been disruptive. Furthermore, it may have
been threatening in instances where choices might have
seemed unwise upon second thought. Requiring a reason for
each choice would have tended to produce behavior that was
more than ordinarily analytic for most S_s . Whatever the
disadvantages of oversoliciting verbal behavior concurrent
with choice making, it appears in retrospect that the in-
structions should have strongly encouraged each S_ to give a
reason for each choice, even if in a highly abbreviated
manner. E could have reminded Ss to give reasons when ver-
balization became too sparse. Only a few Ss gave reasons
for most of their choices. This more or less meager verbal
component to the game -playing behavior dictated that the
analysis focus upon a single bit of information at each
choice point rather than upon stated reasons for making a
move. This facilitated automatic analysis of the behavior,
but it was not included in the preliminary research scheme.
After all 60 hands were played, each S was asked if
- 21 -SD 70-576
ii
t!
fife Space DivisionNorthAmerican Rockwell
he had developed any strategy or style of play. The reasonsgiven were not sufficient in themselves to permit develop-ment of a theory to account for the playing behavior. Ss
did not necessarily play the way they said they had. How-
ever, results of this inquiry, plus the verbalization during
play, were useful in suggesting a model and eliminating some
a priori expectations. For example, Ss were not quantita-
tive in approach. One S stated that he played as if both
colors were equally likely to be offered, although he had
the opportunity to notice from the exposed cards that the
cards not yet offered would be relatively richer in one
color or the other. No complex strategies were verbalized,
hence the model had to be correspondingly simple. Any
theory developed had to be consistent with the verbaliza-
tions during play or inquiry, taking into consideration,
however, the inconsistency in playing behavior itself.
- 22 -SD 70-576
fife Space DivisionNorth AmericanRockwell
CHAPTER IV
THE MODEL
Induction of Process Rules
The first session provided an opportunity for the
Ss to over learn the game and to develop stable playing be-
havior. The second session of 30 hands provided the experi-
mental data. The transcriptions were arranged with two
hands per page. To permit cross-validation, alternate pages
were set aside before detailed analysis was begun. Sixteen
hands thus constituted the corpus for derivation of a fea-
sible model and parameter values. Basic to the research was
the notion of a single model nomothetic in structure but
idiosyncratic in function, with the idiosyncracy due to
individual parameter values. No
structure dictated the analysis,
mation about the subjects except
tocols, the analyst did not know
a priori ideas about the
however. Having no infor-
that contained in the pro-
that of the 12 students
only one had an academic major interest related to
- 23 -SD 70-576
l!
fife Space DivisionNorthAmerican Rockwell
quantitative methods. Point-counting strategies for the
game of blackjack have been well publicized, so some simi-
larly quantitative approach was the object of a shallow
search through the protocols, especially the postplay sum-
mary statement by the S_s . No point -counting strategy was
apparent. One protocol was then selected for detailed
analysis, move by move. The protocol selected was that of
S_ll, a relatively verbal S_, who also had the highest average
score over the entire 60 hands .The recorded verbal content alone was not enough to
permit induction of a theory to account for the decisions
made during play. (See the protocol of S_ll in Appendix B.)
Serious protocol analysis was begun with an attempt to en-
code detailed behavior. It was hoped that some basis for
understanding might emerge once the essential elements were
exposed by stripping away the not -very-helpful verbal mate-
rial. Hand 31 was thus encoded; eight decisions. It became
apparent, even with this very small sample, that context was
involved and that no simple coding of each decision would be
adequate. What seemed better was encoding of the entire
hand. For S.ll on H3l, the very first hand analyzed, no
understanding emerged from the sequence of the successive
decisions, which were related, of course, to the offered
- 2k -SD 70-576
l!
fife Space DivisionNorthAmerican Rockwell
cards. Examining the eight decisions as a whole suggested
that a 7 was high enough to accept as the first card, a 12
was high enough to accept in the opposite color, and that a
4 was too low to accept any time. Two more hands were ana-
lyzed for these three parameters, after which it seemed
that three were insufficient. The insufficiency was indi-
cated in H35. After the first accepted card, no others wererejected. The notion of dominance was defined to facilitate
a description of taking cards of opposite colors. A color
was said to be dominant if the sum of the values of the
cards already accepted of that color was greater than the
corresponding sum for the other color. Before any cards
were taken, neither color was dominant, and tied sums are
not especially rare. The first four hands were scored for
five parameters: highest reject before acceptance, first
acceptance, lowest acceptance of nondominant card, highest
reject of dominant card, and highest reject of nondominant
card. Then, because more context information seemed im-
portant, the dominant sum at the time of the decision making
was linked to four of the previously mentioned parameters,
and lowest acceptance of a dominant card was added. That
made a total of 10 parameters. In addition, the number of
changes in dominance (which might be called color switching)
- 25 -SD 70-576
fife Space DivisionNorth American Rockwell
was recorded, as was the number of forced acceptances. Note
that forced acceptances were not counted as choices, and no
notice was taken of the color or magnitude of these cards inany analysis. The first seven hands were scored or rescored
for these 12 kinds of information. Because color switching
may occur four times in a hand, the number of entries for a
hand under this scoring scheme may exceed 12.
In the first seven hands analyzed, Sll both accepted
and rejected 6 in considering the first card to count toward
his score. In trying to put this into context, it was noted
that a 6 was accepted as the fifth or third offer, but not
as the first. This called for addition of another statistic
—the number of cards rejected before first acceptance. To
facilitate subsequent computation of goodness of simulationanother statistic was added to the list —number of choices .All 16 hands were scored for these 14 statistics . The ana-
lyst then attempted to organize all this information to de-
velop the first model, one that would describe the play of
Sll.
Three pieces of information were recorded about the
first acceptance: highest reject before acceptance, the
number of cards offered before acceptance, and the magnitude
of the first accepted card. On H5l, Sll accepted a 5 after
- 26 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
having already rejected a 5 and a 6. This, plus his accept
ing a 6 as the fifth- or third- but not the first-offeredcard led to this induction: the threshold for first accept-
ance changed as a function of the number of cards offered.
For Sll, it was true that he always accepted a 7 or higher
and accepted as low as 5 on the third or subsequent offer.
This kind of behavior —changing the threshold as play pro-
gressed—became a candidate for inclusion into the model.
The structure of the model related to the first acceptance
seemed satisfactory.
Attention next turned to acceptance of nondominant
color cards —color switching. For the 16 trials analyzed,
the highest reject of nondominant cards ranged from 2 to 12.
There were six rejects of 12. Yet S_ll four times accepted
nondominant cards such as 6, 8, and 9. Clearly, something
besides magnitude would be needed to describe his behavior
in accepting the nondominant color. That something was a
context factor —the dominant sum at the time of the deci-
sion. Thirteen times out of 16 when S_ accepted a nondomi-
nant color he shifted the dominance margin—the amount by
which the dominant sum exceeded the nondominant sum. Put
another way, his partial score at that point would be higher
if he took the nondominant color card. Twice he took as the
- 27 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
second card a card of the same magnitude but opposite colorBut twice he rejected the same kind of offer. He verbalizedabout "opening up options" by taking a card of opposite
color. The analyst formed the hypothesis that Sll's play
with respect to nondominant cards could be described by this
rule: if you have already accepted a card, accept the samevalue in the opposite color to open up options. After ac-
cepting two or more cards, you accept a card of nondominantcolor only if that color will become dominant if the card is
accepted. On H6O, Sll took an 11 red when his partial sum
in black was 19. This was his only instance of taking a
nondominant card, after already taking two cards, when it
did not shift the color dominance. It was assumed to be
anomalous behavior since it did not "open up options" and
led to a below-average score.
A consistent description of behavior with respect
to decisions about dominant-color cards was difficult. He
both accepted and rejected 2, 3, and 4 of dominant color.
No simple threshold for acceptance would be adequate. An
attempt was made to take context into account. Already re-
corded was the dominant total at the time of the decision.Perhaps the desirability of a card was relative to what he
already had in his hand. A detailed analysis of the partial
- 28 -SD 70-576
fife Space DivisionNorth American Rockwell
sums of the cards already accepted indicated that no con-
sistent rule could be formed by including partial sums.
Similarly, taking into account the number of choices
remaining did not yield consistency for S_ll. Magnitude
alone was tentatively selected as the only parameter of the
decision to take a card of dominant color.
The first model, based upon the protocol of Sll, was
expressed as five process rules .Considering first acceptance:
1. Accept first or second offer if greater than or
equal to 7.
2. Accept third or subsequent offers if greater
than or equal to 5.
Considering subsequent acceptances:
3. If only one card has been accepted, "open up
options" by accepting a card of equal value in other color
4. If more than one card has been accepted, "stick
to" dominant color by accepting only cards greater than or
equal to 6 in the dominant color.
5. However, accept card of nondominant color if it
gives a higher partial score.
These five process rules refer only to card-
magnitude values. A fundamental notion to be tested was
- 29 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
that other Ss would behave similarly, but with some varia-
tion in the magnitudes, which constituted thresholds. Im-
plicit in the five rules is another kind of threshold: the
point at which the magnitude threshold is changed. The
offer number on which the threshold for first acceptance
changes can be viewed as a threshold. It was predicted that
some Ss would require more than just a higher partial score
in accepting a nondominant card. They might require the
partial score to be changed by a minimum amount. Similarly,
it was predicted that other Ss might not accept opposite but
equal cards as the second acceptance. They might not want
to "open up options." The latter two predictions were a_
priori , as no other protocols had been analyzed. They were
hypotheses to be tested.
The five process rules were rewritten to permit
seven parameters. With slight wording change, the rules as
hypothesized after analysis of S_ll were used throughout the
remainder of the research.
1. Considering first acceptance, reject values less
than P , until offer P .2. Then first acceptance must be as large as P .3. Considering subsequent acceptances, if the num-
ber of cards already accepted is less than P , accept a card
30 -SD 70-576
1
1
fife Space DivisionNorth American Rockwell
of the nondominant color if it will result in a shift in the
dominance margin (gain) by as much as P .4. If the number of cards already accepted is equal
to or greater than P , then the gain must be as much as P .4 65. Accept a card of the dominant color, or of
either color if neither is dominant, if it is as much as P .7Closely related to these five process rules is a
description of the seven parameters.
P —lnitial threshold for acceptance of first card,
in terms of card magnitude.
P —Offer number on which initial threshold is
changed.
P —Secondary threshold for acceptance of first
card, coming into effect if S_ does not accept a
card before offer P^ .P —Number of cards accepted before changing gain
threshold (for acceptance of nondominant suit
cards). P. is inversely related to the number
of choices remaining.
p —Initial gain threshold (for accepting nondomi-
nant suit cards ) .p —Secondary gain threshold, coming into effect if6
S_ has accepted P cards .
- 31 -SD 70-576
I
1
I
1
1
fife SpaceDivisionNorth American Rockwell
P ?—Threshold for acceptance of cards of dominant
suit, in terms of card magnitude.
The parameter values for S_ll, based upon the origi-
nal five process rules, would be 7, 3, 5, 2, 0, 1, and 6.
Once the structure of the first model was erected,
the next step was to see how well it fit other Ss . The
process of analyzing the protocol of S_ll was tedious, taking
many hours of labor. To maximize the return from this kind
of labor, it was decided to select as the next subject to be
analyzed one whose behavior differed from Sll's.
Before the next protocol was selected for analysis,
the similarity of responses between Ss was computed by find-
ing the correlation between the Ss based upon the cards ac-
cepted. The maximum number of choices per hand was nine,
because of the forced acceptance rule, but each of the 160
cards that could enter into the scoring was included in the
correlation computation. The correlations with Sll were
ranked, and S_6 was selected. He had a low correlation with
Sll and was reasonably verbal.
S_6 was not difficult to analyze with respect to the
first three parameters, which refer to the first acceptance.
The lowest acceptance was 7, except for a 6 taken after four
consecutive rejects. The first three parameters were
- 32 -SD 70-576
I
l
fife Space DivisionNorthAmerican Rockwell
therefore 7, 5, and 6. S6 was less consistent in decisions
about nondominant cards . Seven times he took nondominant
cards that did not increase his partial score. Defining
gain to be the difference in partial score to be realized if
a nondominant card were taken, it was noted that only six of
the 13 acceptances of nondominant cards resulted in positive
gains, three yielded zero gains, and four resulted in nega-
tive gains. Perhaps the very magnitude of the nondominant
card accepted was the determining aspect of the decision.
However, the magnitudes were 9, 4, 12, and 11 for the in-
stances of negative gain; only two were "face" cards. For
the three instances of zero gain, all were second accept-
ances, equal in magnitude, but opposite in color, with mag-
nitudes of 7, 6, and 9. No simple hypothesis about accept-
ing nondominant on the basis of magnitude could be supported
by the recorded play.
Values for the three parameters related to accept-
ance of nondominant cards were estimated to be 2, 0, and 2,
meaning take nondominant cards for zero or greater gain as
the second acceptance, but raise this threshold to 2 for any
subsequent acceptances.
With respect to decisions about taking dominant
cards, S6 was much more consistent than Sll. While the
- 33 -SD 70-576
Il
f
fife Space DivisionNorthAmerican Rockwell
optimum threshold for acceptance of dominant cards yielded
eight instances of inconsistency for Sll, the corresponding
number for S6 was only two. S_6 both accepted 5 and rejected
6, but he accepted 5 only twice and rejected only one 6. It
was decided to set the parameter at 5, because, in his sum-
mary statement, S6 mentioned "4 or possibly 5 as a minimum"
in reference to taking what has been defined as dominant.
For S_6, the seven parameter values were estimated
to be 7, 4, 6, 2, 0, 2, and 5. Over-all, S6 seemed to fit
the model induced from the behavior of S_ll rather well. In
fact, there were fewer inconsistencies for S_6 than for Sll
(7 versus 10), given that the model was an acceptable de-
scription of the way Ss made decisions. Therefore, the
first model appeared to have some general validity, based
upon this tiny sample.
Development of the Computer Program
The process rules were transformed into a computer
program flow chart (Figure 1) . By ordinary programming
techniques, several procedures, each performing only a small
part in the administration and playing simulation, were
written in Fortran and checked out. It should be noted that
what was simulated were the decisions to accept or reject
- 3^ -SD 70-576
35SD 70-576
£j^ Space DivisionNorth American Rockwell
5I*
zUJs><
zoUJ_J
III
v—1ca__
i>
AlA/ \
/
\
\/
r , A ' \ SI / \ /_ \
i_vK^ii^ <2is§,>3*a I \<z
/
\-°o v/ sI f I
\
/ \
O
/ I
ir I oA r i r i oI \ | I Sal
/ * ! ° I 1 9
Q
5 I eh/3 .. cA v I < <__ § o
\ / s i y 5S »y U lt J g
i
I
fI
It
I.
t
.Ir— '
1 Al " S
O O
■t h iti A s."
UJ>
5-Zuj
_| co
s-
__vUJ__
- Xzi—
o
r—— 1l II °" °"I
i^i- i
ii nj o liI ?x-I "III
oz
f—l!_,..-_I
O
II |I ix-K-iIs| ;
ii \
■or\/
V
': is 1 - A\ -I I 3 I 2 __/_.._ 2 \" r. i z *\
_gj^ / (U. i y ' „ \ So / v
| I- i g V* / So! ! J O. U-l V £
111! c iI UJ
[T J?
'' \ \~ iv* / i !L—i
c
"_
L\->2
C
i
O
z
/
Q
< "a z\ x
___»
a_
5zv.u-o
" < °££_vl
UO
£
i!
IIi
i
■
i
'
t
i
>-a<UJ
Q-
U3is*
oz
___.
UUJ<___
XzI—
o
t_.
<vBz___:UJu-u.o
/UJ >__-<si So\< J V 1? .?_5 2
UJJ-
I:!H
II
fife Space DivisionNorthAmerican Rockwell
the offered cards . The cognitive behavior that accompanied
these decisions was not observable and, therefore, could not
be simulated. The programming language itself has conven-
tions that tend to obscure the psychological functions the
program procedures perform. The administrator functions are
mixed in with the subject functions in most of the proced-
ures. Still, the coding reflects the seven process rules
accurately, and it should be understandable to almost every-
one acquainted with Fortran. Most existing computer pro-
grams related to artificial intelligence or game playing
were written in IPL or LISP. The languages were designed
for these very applications, but are not readily available
everywhere .
Simulation Program Structureand Function
In its latest version, the computer program that
mechanizes the model, based upon analysis of Sll's protocol
and partly supported by analysis of S_6 's , consists of a main
procedure and nine entry points (subroutines). The follow-
ing description should be read in conjunction with the
source code in Appendix D and the flow chart (Figure 1).
MAIN calls SHUFFL, which reads in the already ar-
ranged stimulus cards and the responses of each S_ to each
- 36 -SD 70-576
111
I.ii
fife Space DivisionNorthAmerican Rockwell
card. MAIN then goes into the outermost loop, iterated once
per S_, by calling NXSUB, which reads in subject identifica-
tion and parameter values and prints them out. MAIN then
calls NXDECK, which gets the identifications of the next
deck, the cards for the trial, and S's responses to them.
NXDECK also heads a new page with information about
the hand, initializes several program variables, and estab-
lishes the initial thresholds for first acceptance (P_. ) and
for acceptance of nondominant cards (P. 5) . MAIN then calls
NXCARD, an administrator function that offers each card in
turn. If, by the rules of the game, S_ is forced to accept
the card, MAIN has S_ do so immediately after NXCARD offers
it. Beginning with offer P 2 , the threshold for first ac-
ceptance is changed from P^ to P_ . If the magnitude of the
card is less than the threshold, MAIN calls REJECT, which
compares the decision of the model with that recorded for
the human S_. REJECT helps keep count of the number of er-
rors and choices made. If the model is correct in rejecting
the offer, control is passed to statement 10, and another
card is offered. If the model has incorrectly rejected, an
error message is printed and control passes to MAIN'S call
on ACCEPT. The model is not allowed to diverge from S_'s
track. Along with REJECT, ACCEPT keeps count of errors and
- 37 -SD 70-576
II
,
fife Space DivisionNorthAmerican Rockwell
choices. ACCEPT compares the model's acceptance with S_'s
recorded decision. If they match, the card is added to S_'s
hand. If the hand is not complete, control is returned to
MAIN. Otherwise, ACCEPT calls COUNT, which first determines
if there has been any change in S_'s hand since it was last
called. This is to correspond with the assumption that
human Ss probably do not count up their hands again unless
there has been a change. They remember. The partial red
and black scores are counted, and it is determined which
color is dominant. The partial score for the hand (the
final score if five cards have been accepted) is the red sum
if red is dominant, the black sum if black is dominant, and
the color of the last card accepted if neither is dominant.
COUNT returns to ACCEPT, which then calls QUIT.
QUIT keeps count of the total numbers of errors and choices
over all hands, computes error percentage, and prints cer-
tain information at the end of each hand and at the end of
each subject's session. QUIT returns to ACCEPT, which re-
turns to MAIN at statement 1. However, if ACCEPT is called
after a model decision that does not match the corresponding
human S's decision, the card is not added to the hand, and
control is passed to statement 10 in MAIN, after recording
the error and printing a message. The model is kept on S_'s
- 38 -SD 70-576
1
1
n
i
fife Space DivisionNorthAmerican Rockwell
track. The first card acceptance loop in MAIN continues
until the human S_ accepts a card, either forced or volun-
tary. Forced acceptances are represented by the coding be-
ginning at statement 41 in MAIN. Program control remains in
a short loop until five cards are accepted, in the case of
forced acceptance. (Once S has rejected so many cards that
an acceptance is forced, all subsequent acceptances are also
forced, obviously.) If acceptance is not forced, then con-
trol in MAIN is passed to statement 5, which calls COUNT.
The next statement tests whether the card under considera-
tion is of the dominant or equal color. If it is, then if
its magnitude is greater than or equal to P? , the model de-
cides to accept it.
The error checking described for first acceptance is
performed similarly for acceptance/rejection of dominant and
nondominant cards. Whether the decision was to accept or
reject a dominant card, control is passed to statement 4,
where another card is offered. (Any time ACCEPT takes the
fifth and last card, control goes through COUNT and QUIT, as
described.) If the card under consideration is of the non-
dominant color, control in MAIN is passed to statement 7.
If as many as P4 cards have already been accepted, then the
threshold for acceptance of nondominant is changed from P_5
- 39 -SD 70-576
I;;
II;IIII
!li
:S
I
fife Space DivisionNorthAmerican Rockwell
to P^ . The threshold is for the minimum gain that would be
realized by accepting the nondominant card. MAIN calls
GAIN, which calculates this value. If the gain is as great
as the threshold, the card is accepted; otherwise it is re-
jected. Incorrect decisions regarding nondominant are cor-
rected and recorded, and a message is printed. Whether the
decision was to accept or reject, control is passed to
statement 4 in MAIN to look at the next offer. Note that
the program is never permitted to make a move that differs
from the recorded move of the S it is simulating.
Because of the many IF statements, a verbal descrip-
tion of the flow of control is awkward, although the program
itself is actually as simple as the process rules that con-
stitute the model.
Derivation of IndividualParameter Values
The analysis necessary to derive the seven parameter
value estimates for Ss 11 and 6 was lengthy and tedious.
Partly in anticipation of a more complicated model with more
parameters, many context statistics were determined for each
choice point. As many as 30 or more statistics were re-
corded for a single hand. The analyst attempted to discover
process rules that might account for divergences from the
- kO -SD 70-576
I
I
fife Space DivisionNorth American Rockwell
seven-parameter model. The tedious analysis was carried out
on the protocols of S_l and S_2 . It was thereafter decided
not to complicate the model further with more parameters,
even though this promised to improve upon the proportion of
identical choices between the human and simulated subjects.
The number of statistics for each hand was cut to 10, which
were needed to estimate the existing seven parameters. Any
one hand might have more or less than 10 because of inap-
plicability or multiple changes in dominant color. The
statistics were highest reject before first acceptance and
the associated offer number, the first acceptance and asso-
ciated offer number, the gain (plus or minus) taken in ac-
cepting nondominant and the number of cards already ac-
cepted before making the decision, the gain (zero or plus)
not taken in rejecting nondominant and the number of cards
already accepted, the highest reject of dominant, and the
lowest acceptance of dominant. (Dominant, in general, in-
cludes both colors if partial sums are equal.)
The simulation program was modified to play each
hand exactly as S_ had played it (the model made no deci-
sions) and to output information that would help define the
10 statistics. From this point on, the original protocols,
with their fairly meager verbalization, were not referenced.
- I+l -SD 70-576
fi:if:
|.
!8iii
fife Space DivisionNorth American Rockwell
The verbalization had helped in inducing the basic model,
and in estimating parameter values for Sll and S6 , but S_ll
and S6 were among the most verbal Ss . Only the decision
itself—accept or reject —was used in subsequent analysis.
This modified program played the 16 hands of the 12 Ss, and
the results were compared with the already completed analy-
ses of Ss 11, 6, 1, and 2. Analysis of S3, using the pro-
gram output, took only one hour, instead of the tens of
hours required for the earlier analyses. This was due in
part to the decision not to look for more than seven model
parameters, but was possible only because the output high-
lighted the information needed.
Because only the binary information at each choice
point was to be included in the analysis, the statistics for
any hand would be the same for all Ss playing it the same
way. The only rarely stated reasons for making a particular
decision might differ among various S_s , but this information
was not considered. To save analysis time, it was noted
which hands had been played the same way by two or more S s ,
and thereafter each particular play of a hand was analyzed
for the 10 statistic values only once. The last eight
analyses took only about five hours total, as compared with
tens of hours each for the first few. This great economy
- U2 -SD 70-576
i'l
ji;
If
[!;
i
ii[
1 1'
fife SpaceDivisionNorthAmerican Rockwell
was due to the restricted set of statistics recorded, the
assistance of a computer program that abstracted the sig-
nificant information, and the one-time analysis of hands
played in a given way by any number of S_s .Only the gathering of statistics was computer-aided.
Estimation of parameter values was a matter of judgment by
the analyst. For some of the parameters, this judgment was
easy to make. P 7 is the threshold for acceptance of domi-
nant. For all S_s, the highest reject was greater than the
lowest acceptance, over the 16 hands. Thus, no matter what
estimate of P 7 was chosen, there would be some errors of
simulation in accepting dominant. The estimate chosen was
the one that would yield the fewest errors. Py may be con
sidered as a one -dimensional parameter, but the other six
parameters belong to two sets that may be considered to be
three-dimensional: accept card value X until V cards have
been offered or accepted, then accept card value Z. This
made it more difficult to find the point of minimum error.
The search for reasonable parameter estimates was
systematic, involving the construction of several sets of
two-dimensional tables and careful search for points of
overlap or inconsistency between rejected and accepted
values . The elements in the tables represented mainly the
- I+3 -SD 70-576
i
j.
X
I.
itt
hi
i;
fife Space DivisionNorthAmerican Rockwell
extrema, e.g., highest reject and lowest acceptance.
After the parameter values were estimated, the simu-
lation program played the 16 hands by inputting the esti-
mates and behaving at the choice points accordingly. The
simulation results were generally good. Only one unusual S_
was simulated with less than 90 per cent accuracy. Because
the parameter estimates were made through a systematic, but
not completely deterministic technique, there remained the
possibility that the estimates were not optimal. Examining
only the extrema might be overlooking some useful informa-
tion. Early spot checks indicated that the critical region,
around the threshold, was similar whether all information
for each hand or just the extrema were included. Still,
there were differences that might affect the over-all accu-
racy of simulation. Consideration of all information was
out of the question for a manual effort, feasible only if it
could be computer-aided. Several of the modules of the
simulation program were used intact and several others were
drastically modified, and a program was developed that com-
puted the error of simulation associated with every possible
parameter value. The MAIN and NXSUB procedures were com-
pletely rewritten to record the errors associated with each
possible P7 estimate in a one -dimensional array, and the
- I+l+ -SD 70-576
li!iii
ifi|:si1
iii
is!';■■■[
fife Space DivisionNorthAmerican Rockwell
errors associated with the possible p i~p 2~P^ anc^ P 4 -P[. -P6
triples in three-dimensional tables. For each S_, 1,120
error -table entries were computed. To estimate optimum
parameter values, it was sufficient to select those associ-
ated with the fewest errors. In many cases, the minimum
error appeared at more than one point in a table, and sev-
eral rules were considered for resolving the ambiguity. The
rule finally chosen was the simplest: to use the lowest
parameter estimate if two or more yielded the same number of
errors. For example, if three errors would result whether
the P7 estimate were 4, 5, or 6 , the one selected would be
4.
By means of this exhaustive-enumeration-like analy-
sis, the 84 (7x12) parameter estimates were made in a matter
of minutes, once the computer output was available. To de-
sign, code, and check out the program to perform the ex-
haustive analysis took longer by an order of magnitude than
the manual but computer -output -aided analysis. Were several
dozens of Ss to be analyzed, the time advantage would be
reversed. Of the 84 parameter estimates, 29 were different
from those made earlier. However, the goodness of simula-
tion was improved for only three Ss . This is attributable
to the minima (least errors) appearing several times in some
- I+s -SD 70-576
fife Space DivisionNorth American Rockwell
tables. Several sets of estimates were optimal. Some dif-
ferences were due to errors in the manual analysis. The
averages presented in Table 2 omit the aberrant S_4 .In a sense, there are only three parameters in the
model, not seven. The parameters within the triples (P -P -P and P.-P -P^) are not independent, but in fact closely234 5 6 J
related. Excluding S4, P > P for nine out of 11 Ss . If
P = 1, then P must equal P , and the converse is also
true. P < Pr in all cases, although this is not necessary.5 — 6
If p, =1, then P. must equal Pr , and the converse is true4 5 6
also. P is unrelated to other parameters. There was most
variation in the first triple, less in the second, and least
in P , corresponding to nine, seven, and four different
sets, excluding S_4 .
- I+6 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
11
II
I
»;
I
f1
3
I
II
i!
Table 2. Parameter Estimates
- I+7 -SD 70-576
SubjectNumber
Derived viaManual AnalysisParameter Number
Derived via Exhaustive(Computer) Analysis
Parameter Number
1 2 3 4 5 6 7 1 2 3 4 5 6 7
1 6 1 6 1 0 0 7 6 4 7 1 0 0 6
2 8 2 7 4 0 5 6 8 1 8 1 0 0 7
3 7 3 6 4 0 6 4 7 3 6 4 0 6 4
4 4 4 3 3 10 0 2 4 2 12 1 10 10 2
5 7 2 6 2 0 6 5 7 2 6 2 0 6 5
6 7 5 6 2 0 2 5 7 4 2 2 0 1 5
7 7 2 6 3 2 3 5 7 2 6 1 1 1 5
8 7 4 6 2 0 3 4 7 4 2 1 0 0 4
9 5 2 6 2 2 2 4 5 2 6 1 8 8 4
10 6 1 6 3 1 3 5 6 1 6 1 1 1 5
11 7 3 5 2 0 3 6 7 3 5 1 0 0 6
12 7 2 7 2 6 6 4 7 1 7 1 6 6 4
Mean(rounded) 7 2 6 2 1 4 5 7 2 6 1 1 3 5
Mode 7 2 6 2 0 3 4, 7 1, 6 1 0 0 4.5, 2, 5
4.
rf
(l
[
fife Space DivisionNorth American Rockwell
CHAPTER V
EVALUATION OF THE MODEL
Simulation of Individual Subjects
The model was nomothetic in structure and idiosyn-
cratic in function: nomothetic because a single model was
used to describe the behavior of the 11 synoptic Ss and
idiosyncratic because the model predicted the peculiar be-
havior of these S_s if initialized with their peculiar
parameter estimates .A model can be evaluated by the amount of agreement
between predictions generated by the model and observations
of natural behavior. Because the computer program mechaniz-
ing the game-playing model was set back on the correct track
after each incorrect prediction, the measurement of agree-
ment was simple. The program played through the 16 analyzed
hands, computing certain statistics for each hand and sum-
mary statistics across the hands (Table 3) .The model shows the least agreement for S_4, which
- I+B -SD 70-576
¥
I+9SD 70-576 .
.-J.H
<NH
O
rH
en
oH
CD
HOJH
ro
r—
t
00
CN
H
ro
00-H
<N
CN
in
CNCTi
CO
roen
ooin
m
CN
41m Space DivisionNorthAmerican Rockwell
I
I 'i
i;'i
fn.'
■J..8&,■»:.Inill'1.1.ii1 '
Ife',
en oro
r—
iIN
r—
l
ro
Cn
en
O
i—lHroH
CNrH
Cn CO
<* vI3 enCN
oo
C^
CN
ro
<_>"H4-30.
"H-Pro4L>
r-CNH
ro (Tiin
!h
CO
aBCQ
<D
CT>
CN"H
00CN
roin
CTi
r^
"*
enH
O
(N
r—
lro
CN
CO
ooinH
t
1,
ion IfiH£)roEH
O
rH
CD
CN (N
CN
CN0>
CO
Is'
CNrH
CM(N
roro
CN
o
c^
oo
CT>
Iii
CD
inCN
oro
ro 143
Ij
(ii
!j.i1JIi
i■i
liij*I:t,F
,1
f
vd)
S-J.
ovvm4-1ov<ii
■i3
CO0)U
■Hov
o1-101
I"y
T3<D
O
Vj.O4-<
>-l
y
U<Dvvoo
catv0)a.
S-l
Ol-l140)
+ 1c01vvQi
0,
I
U
i
i
fife Space DivisionNorthAmerican Rockwell
was because S4 rejected virtually all offers of black cards,
regardless of magnitude. If the model had a parameter for
color bias, better agreement could have been achieved, but
at the expense of introducing an ad hoc modification for a
single S_. This would have been antithetical in a model that
should be descriptive of general behavior. No test for
color bias was made for any S_, but no such bias was manifest
except in S_4 .The model was based upon the behavior of S_ll. How-
ever, this did not lead to predictions for Sll that were
more nearly accurate than those for other Ss . The fit of
the model was better for seven Ss and worse for three (syn-
optic) Ss . The S_ whose behavior was best simulated by the
model was S7 . His optimal parameter estimates were the most
nearly typical. The typicalness of S_7 is further gauged by
the fact that he alone failed to play a single hand in a
unique way. That is, his pattern of choices was identical
to that of at least one other S_ on each hand, while S_4
played nine hands uniquely.
Among the synoptic Ss , the percentage of disagree-
ments between the model and observed behavior ranged from
3.17 to 9.84, with a mean of 6.57. The possible range of
error per cent was from 0 to 100. The obtained range, for
- 50 -SD 70-576
«"«
Pi
,!
I:
fife Space DivisionNorth American Rockwell
the 11 S_s on the 16 hands, was from 0 to 71 per cent.
Averaging across the Ss puts the range for the hands from 1
to 16 per cent. Some hands, apparently, were much easier to
simulate than others. When the decisions are easy for Ss to
make, because of extremes of card magnitudes, then of course
the decisions can be predicted with less error.
The only goodness-of-fit measure presented is error
per cent. Another measure considered was phi, the fourfold
point correlation coefficient. Phi is quite sensitive to
the direction of the error —whether model-accepted but S_-
rejected or vice versa—while error per cent is completely
insensitive to it. Phi is undefined for the several cases
of zero marginal frequencies. When phi is unity, proportion
correct is also unity, but the converse is not true because
of the undefined phi cases. Error per cent was selected as
it is well behaved and easily interpreted. Tests for the
statistical significance of proportions or of differences
between proportions are inappropriate because the decisions,
upon which the error per cent was based, were highly corre-
lated because of the conditional prediction effect. Unfor-
tunately, then, evaluation of the model as a description of
the behavior of this sample of Ss remains subjective. A
sometimes suggested test of computer program output
- 51 -SD 70-576
Ii
li
fife Space DivisionNorthAmerican Rockwell
agreement with recorded natural behavior is a kind of
Turing's test (Turing, 1963, p. 11). Turing called it the
"imitation game." Stylized protocols of human subjects and
computer program output, where the program did not use the
conditional prediction technique, could be presented to
naxve judges. The judges would then try to segregate the
machine and human protocols. However, this would not test
the model, but only whether machines can be programmed to
play the game involved, an obviously unnecessary demonstra-
tion .In discussing the model evaluation problem, Newell
(1966) notes that
in assessing the validity of the program to describeor explain the subject's behavior, two things aremissing to which psychologists have been accustomed.First, there is no acceptable way to quantify thedegree of correspondence between the trace of theprogram and the protocol. This is not a problem ofmaking the inference definite or public. Trace andprotocol can be laid side by side. . . . However,comparison still must be made between an elaborateoutput statement and a free linguistic utterance.Although a human can assess each instance qualita-tively, there are no available techniques for quan-tifying the comparison, or summarizing the resultsof a large set of comparisons .
Second, the program has been created partly withthe subject's protocol in view. Thus, somethinganalogous to the calculation of degrees of freedomused in fitting curves with free parameters to datais appropriate. But programs are not parameterizedin any simple way and no analytic framework yet ex-ists for allowing for degrees of freedom. (pp. 3-4)
- 52 -SD 70-576
fife Space DivisionNorth American Rockwell
The present research cannot claim to have solved the
problem of assessment, and indeed it has not, but error per
cent based upon conditional prediction is a reasonable quan-
tification of comparisons, and the model was cross -validated
with protocols that were not in view during program crea-
tion.
Sensitivity of Model toParametric Variation
The behavior of the computer program was of course
completely determined by the parameter values. The pre-
dictive ability of the computer program was determined by
the proportion of variance in natural behavior accounted for
in terms of the parameters . Information on the relative
importance of the parameters can be obtained by comparing
models with different parameterization. If a parameter were
added to the original model, differences between the pre-
dictive ability of the original and augmented models could
be attributed to the added parameter. By generating model
variations of different strengths, attributable to different
parameterizations , the predictive validity of the individu-
alizing parameters can be estimated.
i ir'i
i .
i
During the manual analysis of the protocols, some
parameters were considered in addition to the seven. For
ii- 53 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
example, after the simulations were performed, and the
optimal parameter estimates computed, it seemed that the
triplet for accepting nondominant was insufficient to pre-
dict the associated behavior. Although during the original
analysis of Sll, the notion of gain seemed important, in
several hands for each of the Ss the magnitude of the non-
dominant offerings apparently led to acceptance of a card
that could not hope to lead to an over-all higher score.
There seemed to be a "face card" effect, and there was some
mention of this in the summary statements. It was decided,
however, not to add a parameter for face card effect in
accepting nondominant offerings. This and any other pos-
sible parameters would have complicated the model. The goal
of the research was not to obtain the most accurate simula-
tion possible, but to explore and understand methodology.
It can safely be contended that additional parameters would
have resulted in a better average fit. It will be shown
that each of the seven parameters contributed some covari-
ance in the prediction process. It would be no surprise,
and no contribution to theory, to discover that adding
parameters (predictors, in effect) resulted in greater cor-
respondence between model and S_ behavior. As long as there
were non-zero correlation between the additional parameter
ii
Ij i
t
- 5^ -SD 70-576
fife Space DivisionNorthAmerican Rockwell
and human play and non-unity correlation with other param-
eters, an increase in correspondence is a mathematical in-
exorability. It was more interesting to make the model less
predictive by removing the effects of parameters in certain
ways. By using exactly the same MAIN program, it is pos-
sible to examine the power of weaker models.I.
Consider a model with no change in threshold for
first acceptance. Using the existing simulation program,
we get exactly this simpler model by setting P 9 = 1, and
P 3 = P 1 ; only P can have any variation in the first trip-
let. In effect, the model then has only five parameters.
Another simplification would permit no change in threshold
for gain in accepting nondominant. This is achieved by set-
ting P = 1, and P = P , leaving only one variable in the
second triplet. Were both of these simplifications made,
the model would have only three parameters . if p were set
to some fixed value, instead of being allowed to take on the
value optimal for each S_, then it would no longer function
as a parameter, but as a constant. The elements of the
triplets must be treated together, as they are closely con-
nected. The triplets and P could all be set constant, and
the result could no longer be termed a parametric model.
I
i
i
ii ■
mli
; ;
ft
If the sample had been very large, then a model
- 55 -SD 70-576
i
fi
I
f
t
1!
II
t
s
fife Space DivisionNorth American Rockwell
that had only average values as constants would still con-
tain information about human behavior. Such a model could
predict only average behavior. The errors of simulation
would vary for each S_, but there would be an average error.
Nullifying the effect of the various parameters by holding
them constant would lead to increases in average error, if
the parameters had any predictive power. For example, al-
lowing only P ? to vary, holding the triplets to modal values
would yield a virtually one-parameter model. The difference
in average error between such a model and one in which all
values were constant would measure the effect of P_ .There are 32 possible model variations of different
strengths mechanized by the same MAIN program. The first
acceptance and nondominant acceptance parameters may have
one or three elements each, and the parameter estimates used
in the simulation may be the idiosyncratic optimal values or
held constant to the average (modal) value. The 32 varia-
tions belong to four submodel groups, according to the num-
ber of elements involved in making the decisions for first
acceptance and for acceptance of nondominant suit cards .Because the several variations are frequently referred to
in the following discussions, it is convenient to label
these four submodel groups with more or less arbitrary
- 56 -SD 70-576
li<
1
.
fife Space DivisionNorth American Rockwell
numeric identifiers .Submodel o—no0 —no change in threshold for accepting
first card as more cards are offered and rejected. Only one
parameter involved in accepting first card. No change in
gain threshold (for accepting nondominant suit cards) as
more cards are added to hand. Only one parameter involved
in accepting nondominant suit cards .Submodel I—only1 —only one parameter involved in accepting
first card. Three parameters involved in accepting non-
dominant suit cards, due to change in gain threshold as more
cards are added to hand.
Submodel 2 —three parameters involved in accepting
first card, because of a change in threshold for acceptance.
Only one parameter involved in accepting nondominant suit.
Submodel 3 —three parameters involved in accepting
first card and three parameters involved in accepting non-
dominant suit cards .There are three distinct kinds of decisions: (1)
accepting cards of dominant suit, (2) accepting cards of
nondominant suit, and (3) accepting first card. The param-
eters involved in making these three kinds of decisions may
be optimal or modal. There are eight ways that the three
3kinds of decisions can be made (2 = 8) . Because parameters
- 57 -SD 70-576
I
I
il
I!
$
fife Space DivisionNorth American Rockwell
can be optimal or modal (two states), binary notation is
convenient. More or less arbitrarily, the low order three
bits of a five bit binary number can be used to convey in-
formation about these three kinds of decisions. Table 4
shows the meanings of all five bits. The bits are numbered
from low order to high order, from Ito 5 . The two high
order bits are used to identify which submodel group is
being considered. The five bits identify the four submodels
and the eight variations within each submodel. Octal nota-
tion is more compact than binary notation, and at some
points in the following discussion, variations are identi-
fied by an octal number. Variations 00-07 are in submodel
0, 10-17 are in submodel 1, etc., according to the first
octal character. The second octal character identifies the
variation within the submodel and represents the low order
three bits in the binary identifier.
As described earlier, there are three kinds of de-
cisions, and predictions of these decisions can be based
upon optimal or modal parameter estimates. In Table 5, the
variations within each submodel group are listed in order of
the number kinds of decisions based upon optimal parameter
estimates. There is one none-optimal variation, three one-
optimal variations, three two-optimal variations, and one
- 58 -SD 70-576
i
fife Space DivisionNorthAmerican Rockwell
Table k. Meaning of Bits
" '
' " -
■
"* """'" ' '—
5 Three first take param-eters
P 2 =1, P 3 = P 1; one firsttake parameter
4 Three nondominant takeparameters
p4 = 1, P, = P ; one non
dominant take parameter
3 First take parameter (s)optimal
First take parameter (s )modal
2 Nondominant take param-eter (s ) optimal
Nondominant take param-eter (s ) modal
1 Dominant take parameteroptimal
Dominant take parametermodal
True (=1) alse (=0)
fife Space DivisionNorthAmerican Rockwell
lP
- Go -SD 70-576
ooi-HQJ
TSOE
rQ
3c/_
CW -H
..-I
J*v v
*X>iH
"OrH
Oro
en
r^oo00
CN
CO
rHoCO
00ro
in
QJ QJ■-H rC"H 4-1fiS 4H
O
rHrH
T. <CGrd
CNi-HQJ
TSOE
■8CQ
c-H «
21 tnS o
c Jv v
in rHIA
xsy>"H
rH
OrH
oroen
en
OJ
en
CN
r^
oo
ro
00
00in00
r^
tnen
tn ts4-J C"H rd
rQ
Sh ojO
44 -,ro
inCQ
romm
IT_J0
-Pro■Hooenco<
tnTS tnC -P ""id -h c4-1,0 0tn -h
tn infi Q) ino > QJ"H -H Sh
d ftrH a.
Sh QJQJ TS QJfi 0 fiE E 4J3 ,0G 3 >i
tn ,0rHrd C Sh4-) -H 0O fi 440 4-)
■H TSC S QJrd 4-i
C Ctn 0 3
■H -H O4-1 U
Sh rd UQJ -H rd
"H Sh44 n_ flj"H > Sh4-> rdG U .QJ O QJ
T3 44 rH"H fi
Sh rdrH Q) 4-1
QJ -HTS 4H inO -H -Hc -v
rQ
C 4-)3 QJtn T> C
"H -H
CQ
ojCQ
CN
"I
mCQiH
i
CQen"l
OJ
mr^co
"I
rHCQ>X>
CO
I
r-in
"orH
4L>aQJo
rH
rHQJ
TSOE
■§CQ
? -H■H "5fe U
c .^4-) 21
§18 v
r^in
"or-i
rHr^
en
en
CN
en
00r^
■
en
Ol
CO
CN
enoo
oo
00
roiX)
a
Sh OO
-H
ft
4Js-i roW -h
ft
c roo >"Hy rtro qj
3 oc s"HCQ X
O
oj ro
CD
>
O
rHaj
TSOE
CQ
cH «fe
c '-
I s° 8
in
O-H
rH
en
o
en
cor^
"enCO
00
OJ
en00
rHenCO
inO
"00
Lf\
QJ rHQJ
TSOE
rQ
3c/_
S-iCD
rH 'H10 -H
° cv OJT.H
o rH
OJ
<* ro in >X> r-
C-HrC-P"H
oooiHOO
orHo
OOrH
rHO
rHOr—l
OrHrH
iHrH<-i
cO
"H4->rd
"HS-ird
S-lQJ
>i-HM 4-1rd -HC 4J
■H CCQ OJ
"OIH
QJ
H S4 QJrd -HC Sh"H 4->
rQ
CQJas
,c oj
4-i ro
;l
■I
I
1
fife Space DivisionNorthAmericanRockwell
three-optimal variations.
Variation 00 uses three distinct values (P , P , and
P), all of which are modal, while P, p p and P are2 3 4 6nullified in effect. Variation 37 uses seven values (the
seven parameters), all of which are optimal. The variations
have from zero to seven model elements that can take on
idiosyncratic optimal values. It can be shown that the num-
ber of idiosyncratic elements in a variation is given by the
expression B ! +B2 +B3+ 2B
2 B4 + 2B3
B5' where B -i represents
bit 1, etc. The products B^ and B B represent interac-
tions (joint effects). The predictive power of a variationis related to the number of idiosyncratic elements. The
expression should be compared with the expression for error
of simulation, to be described later.
For the sample of S_s examined, there is no change
in threshold for nondominant, on the average. That is, P
is modally 1, and the threshold for gain in this case is
always P_ . Therefore, when B is false, B. gives no addi--5 TL 4
tional information. Variations represented by B false have
the nondominant take parameters modal. The modal value of
P. is 1, which means that there is no change in threshold
and thus only one nondominant parameter; B false implies
B false, for this sample. Thus, some of the variations are
redundant. The pairs of identical variations are 00-10,
- 61 -SD 70-576
I
II;
Hi
ii
i!
i
fife Space DivisionNorth American Rockwell
01-11, 04-14, 05-15, 20-30, 21-31, 24-34, and 25-35. There
are only 24, not 32, distinct variations.
The simulation program played the 16 hands based
upon the 24 variations for each of the 12 Ss . The results
are shown in Table 5. Based upon the average errors of
simulation associated with each variation, it is possible to
isolate the contribution of each bit and of each interac-
tion .The interactions are between B and B , and between
B_ and B_ . If there are three parameter elements for ac--3 b
ceptance of nondominant, then the effect of allowing idio-
syncratic variation is greater than for the case of only one
parameter element, the basis for the B -B interaction.
Because, in this sample, the modal values are such that B
false implies B false, B can have no contribution of its
own. By examining the differences among the average errors
in Table 5, it is possible to write an expression that gives
these errors as a function of the bit values. The expres-
sion is exact. Thus the sensitivity of the model to its
constituent parameters can be estimated. Each error of
simulation yields about 0.8 per cent error, considering the
approximately 127 choices predicted. For the experimental
sample, the error of simulation for the several variations
- 62 -SD 70-576
I
p.
it
x
t!ti
f !'lf'.'
.
i
-.
Iliif
if
fife Space DivisionNorthAmerican Rockwell
is given by 10.57 - .868 - -878 - .798 - .OB - .418
.428 B4 - .658 B . To determine the error reduction asso-
ciated with each bit, the average error per cent for the
variation represented by the binary number with only that
bit true is subtracted from the average error per cent for
variation 00 (all bits false) . The difference in error is
the reduction associated with setting that bit true. The
results of the several subtractions are given in Table 6.
Table 6. Partitioning of Average Error of Simulation
- 63 -SD 70-576
BinaryIdentifier
Octal ErrorBit (s) Identifier Reduction
B, 00001 01 .86
B_ 00010 02 .87
B 00100 04 .79
B„ 01000 10 .004
B 10000 20 .41
2&4 01010 12 .42
3&5 10100 24 .65
t
it
f
I!
:i
fife Space DivisionNorth American Rockwell
The difference between 00 and 37 (all bits true) is 4.00,
but the sum of the first order effects associated with each
bit is only 2.93; there remains 1.07 to be accounted for by
interaction effects. If there were no obvious interaction
effects to look for, it would be necessary to make many sub-
tractions, according to a regular search scheme. In the
present case, the obvious interactions are between the num-
ber of parameter elements in a decision and whether these
elements are optimal or modal. The difference between 00
and 24 is 1.85, but this can be partitioned into a B_ ef-3
feet, a B n effect, and a B_B r
reduction is given by 1.85 -the difference between 00 and
interaction effect. The B B3 5
79 - .41 = .65. Similarly,
12 is 1.29, which is the sum
of the B and BB4 effects. (The B 4 effect is zero.) The
According to Table 4, setting any one bit true must
yield a non-negative contribution to the strength of the
model. This is obvious for bits 1, 2, and 3 from the fact
that they stand for optimal parameter values. Bits 4 and 5
each stand for the addition of two parameters to the model,
which implies greater predictive strength. The research did
not examine the theoretical foundations for the assumption
that the effects represented by the bits are necessarily
- Gh -SD 70-576
B^B^ interaction is 1.29 - .87 = .42.2 4
!
I
%
it;
i!
fife SpaceDivisionNorthAmerican Rockwell
additive in the manner shown. The calculation of inter-
action effects is dependent upon this assumption. It should
be mentioned parenthetically, however, that the partitioning
described is valid for two other error tables not included
in this report.
The coefficient for B4 is zero, as implied previ-
ously. It can be seen from the expression that B , B , and
B3 each contribute to error reduction by about one error,
while the B_B. interaction and B„ each account for about__ 4 _>
half an error, and the B B interaction accounts for about
three-quarters of an error.
The relative contributions of each of the model ele-
ments can be expressed in a slightly different way, using
the same error information:
Dominant acceptance optimal instead of modal . . . 0.86
Nondominant acceptance optimal instead of modal . 0.87
First acceptance optimal instead of modal .... 0.79
Three nondominant acceptance parameters insteadof one
If parameters optimal 0.42
If parameters modal 0.00
Three first-acceptance parameters instead of one
If parameters optimal 1.06
If parameters modal 0.41
- 65 -SD 70-576
li
i
jii!ti
I
1
fife SpaceDivisionNorthAmerican Rockwell
Table 5 indicates that the model is not very sensi-
tive to parametric variation. There are several reasons for
this. The conditional prediction effect tends to keep the
error per cent low, reducing the range of variation. The
subjects tended to play in much the same way, although the
individual variation was sufficient to be interesting. (See
Appendix C for a synopsis.) The dispersion of idiosyncratic
optimal parameter estimates about the mode is the source of
increased error of simulation when the model element is set
to the modal value. This dispersion was small for the ex-
perimental sample. Also, for many of the parameters, sev-
eral estimates were optimal for the particular Ss . When
more than one estimate was associated with optimal predic-
tion, the convention used was to select the numerically
smallest of the optimal estimates. The modal value was one
of the optimal values in many cases.
In spite of the fact that no one parameter accounted
for more than about one error on each hand, on the average,
the combined effect of the parameters was very significant.
The difference between the performances of models using
all idiosyncratically optimal parameter estimates and
models using constant parameter values (the modal values)
is 10.57 - 6.57 =4.00. Based upon correlated samples, and
- 66 -SD 70-576
f
1
fife Space DivisionNorth American Rockwell
using an appropriate normalizing transform, t_ = 6.62, df =
10, p < .001. The model predicts significantly better if it
incorporates individually optimal parameter estimates, even
though the effect of any single parameter is not very great.
Cross-Validation of the Model
The model was derived from analysis of behavior on
16 trials out of 60. The first 30 were considered as prac-
tice trials . Fourteen trials were set aside from the second
30 as cross-validation data. No information in these trials
was used, or even known, before the model had been derived
and mechanized. If the model could predict behavior only
on the analyzed hands, it would hardly be of interest. If,
on the other hand, the model could predict almost as well
the behavior on the cross-validation sample, it could pro-
vide the basis for further research and improvement, if the
decision-making behavior in this game were intrinsically
interesting .Both the model structure and the individual param-
eter estimates were empirically derived from the analyzed
protocols . The parameter estimates were especially subject
to capitalization upon chance. By playing the 14 holdout
hands using the parameter estimates that were optimal for
- 67 -SD 70-576
.
fife Space DivisionNorth American Rockwell
the analyzed hands, this capitalization is completely
eliminated. Table 7 records the results of this cross-
validation. The cross-validation shrinkage for the strong-
est variation (37) is from 6.57 per cent to 7.86 per cent,
a shrinkage of less than 20 per cent. This difference is
statistically significant, based upon a t-test of the dif-
ference between the correlated samples. Using the appro-
priate normalizing transform, Arcsine square root (p/100),
t_ = 1.91 with df = 10, P < .05. The choices of each subject
were determined jointly by the situation (the offered cards
as stimuli), individual propensity, and personal-situation
interaction. The parameter estimates were determined
strictly from the recorded choices, and therefore from these
three factors (plus "chance"). The cross-validation shrink-
age of 20 per cent reflects the elimination of the personal-
situation interaction factor. But this shrinkage is con-
founded with whatever differences in model structure might
have arisen from the situation (the particular 16 hands
analyzed) . The truth is that the structure of the model was
induced from the behavior of just one S_ and originally sup-
ported by analysis of just one other S_. Still, had the 14
hold-out hands been the behavior originally analyzed, the
model might have been structurally different. The
- 68 -SD 70-576
j
|i.
fife Space DivisionNorthAmerican Rockwell
independence of the model structure from the particular
stimuli in the analyzed 16 hands cannot be tested directly.
If the model used parameter estimates which were optimal for
the 14 cross-validation hands, and if the predictions for
these hands were significantly inferior to the corresponding
predictions for the 16 analyzed hands, also using optimal
parameter estimates, then it could be supposed that the
model was determined largely from the characteristics of the
16 hands. The results summarized in Table 7 show that the
model predicts better for the cross-validation hands than
for the analyzed hands, with an average error of only 5.65
per cent as compared with 6.57 per cent, for the strongest
variation. This supports the conclusion that the model
structure is valid for behavior which could not have in-
fluenced its derivation. That the difference is in favor
of the cross-validation hands can be attributed partly to
easier predictions, but no analysis for this ease of predic-
tion factor was carried out.
Table 7, Error Per Cent Summary, gives the error of
prediction for six combinations of variation and sample.
The information for each subject is averaged over the 16
hands of the analyzed sample and the 14 hands of the hold-
out sample. Tables 8, 9, and 10 give this information for
- 69 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
!
t
lij
I
!
| i{
- 70 -SD 70-576
01
C
IBX
oi —-v oV op0) _, c
V. PIB ii IB0. * -H
InrH IB
rt £
vO
enrt
rort
enort
rororo
rt
O
rt
rt
t*- vo(N
en inrt
r~rtrt
CN
rHrt
vortrt
in
ort 01
T3
C
IB
YDrt
arHOjcmw
01 —M r--0) ro« m cegoIB .5 H>t z p
IB Q. IB
rH IBrH >rt —
vo
CO
encor-
CO
inrt
1~~ VO
in
IN
ro oo voCN
VO
in 00in vo
co■HPIB
-O■HH
>I0101OIH
O
oN>i
C
<
0] —-n odi o4-1D.CI*Bro m H
to ii na. ** Hi-i
■H >
H
>.!HroBE3
01JJ
IN or~
o00
CD
CD
CN
CO
CN
VO CDO
rH
rt
en
CD
enrt
vO
omrH
enrooo 44oc
cQJ
01"acIB.C
ft
QJ
01■ac19
X
01 —I-i r-Ul roa> « cegoIB .5 HM Z 44
rH >< —
in
coft
rt
in
roin
roCN
IC
(N
1^
in
in
ro
VO
H
VO
o invoin
-a0)N>.rHIB
C
IB
5-1o
ft
H
0)r-Ha,Emw
vO
H
CN rH
ID
o
CD CDCD
CN
coi^-
o
CD VO
<NH
enort
orH
ort
rH
en
0)■cPco
"HPIB■a
■HrHIB>I0101oft
01
-~Sh O
4) oP01 * cc rH oIB IB -HIn T3 PIB 0 IBa_ 2 -h
MrH ,fl-H >
oJJ44o&0]0)
.cp"H
01 -~In r~0) roP *fl) rH
C
c fl ora c -hkH -H 4Jo. a-h
O
XrH (0
2 £
(N
O OCO
CN
rtCN
in
roo CN
ID
O
enCN
00
CO
envoen
O
rt
vo
CO
f-
rHIBTJ
O
cvo
rHIBE"rlPao«p
odl
■ro433
CD
rH r>j fi TT in
VO
r^
CD
0> Ort rH
(NH
tn01
C
ffi-HIB P CTkl -P ~ifl. H "I< §.
fife Space DivisionNorthAmerican Rockwell Ii
\'l
t!
m
- 71 - !SD 70-576
-j<a: 0~-»»pJ"»eo,fiA.£ecoj_rr-ai %d—«t7>r^C'~*m—
«
m p»- —» in> """"""""""""O o^ft©fi©r,-«CCJ*m —* — — OJ o
_J__J<tf C'J"-*!^^ -Cr-COdir<_Do>uj <t co ct _d -fi
m^-^"0
,s oji*'t*->*""■""""""""CJ3O4 r"ir>'*'irr*'<\oD*»co* m *
IT.
o -rmcf-ff us a* m or © r- co
Q£
..""""■"""
C en
o
U_lo'Os^"ar^*^*"^CO'CT"C_^Ci', UT>
(J^<ifl^.f\iJ*tfNJ©OOs^OJ^>*<_r*(_oNC4'irio'>O'<olf» r* "^< —»
acIA
c it'nj—c © oj © o — oj oj m-0
Mvj'-OONOO — oj oj Oj*lTi OJ
,-»
Cg IS. i-«
m
t
m*-4
o c~- — moojoc
«^
rg "-* if* ogOJOJ
ff 1
ff- c -« "-* CT -h O —
«
— .-* — —-«i/\
,■""""""""""
0-<-«oj — O—— — — ~< *-
,-*
_
,*
-, .^—
_*
— — — ©
p4
o0*
0-*©-s'.-«0.-00'-«-*
o
oj00 o
Ifl
oo
©IT* """"""<s.r\jr\..*o—«OCOj.-*oj©
ITOJ
CO
o
0_
aroWT_J0)N>_
rH
CD
a<vorH
Uoeh
CDQJo"H
Sh
4-3
£
O
mEh
CO<D
rHrQ
CD
E_H
O
c
sD
<
QO
X
£Ca
__J<o3xX
2CZ
-J<____or
i/iacv.
roozz
UJzo
*-*/lec_!_.
z
i/lCLLUCOX3Z___:
ujO
(—oUJ-9as3..A
t/NiT'CCCCi^CCCCOi^iTi """"""""""""ov O COC'N'OOOOO'VJ
** r\i O* — —*fN_mOmc.mOoOir»oo*riA
■
"""""""""*"
—< u>omfi
*i/>oif\Of»'Mf>^'
(Mm
,■«
rg — O — — i\j .© oj en oj
CD CO©.-«njC©— ~
COO
coc — njoc —
~*o©e
r- iroiroirooiMAOmor\j©r\_OojC©ojoj©ojO_,
,-«
iA — — — — «*
4- ©i\_OOCm©©ooo©"4- " " -? -Io(\ioiA©ojeoooco
<m
r- -*
r.rg r\jf\.OjH(\NfVJfV.Nf\J
O
l\i©.-«fliAr''l©©fl
f>'© pfc
injC —
firM-a-COfi
l^©*(V m — — fl " *■'
tr ooc-cooooooc.-*fi ,*"""""»""""
OCOmCOCO©©©-*in —'
* .oonj.fj.CO'JOlAU*'©'^©fi """"""""""""ccocgceeoCccojogmfio
r\j og i\i oj eg ,-. — oj **!
in
o©rvJ~fsJ>'
>pnp,0 *N'',n,sjOm """"""""""""eg im pg —fi oj pj f nj
rsjOfl©©o©oo©OOr~IT.
#""""""""""*
©■* OOOOOOOO^'0—> IM **— lACiAOjiAiAiAfirtAf^mnj(*.""""""""""""
f\j(NjrgI\JINjOJOJOjr~fIOJOJ— 0J — — <*
_»f\jflcriA*ofto «DC7'O«-t 'NJ
4"<C
r~-*c*a
r-o"
ng
fi
<*
*Ap*
fi
*oCO
ODfl
■&m__*>
OJmCD
f-
og
"oom
UJo<ACUJ>
(Acr-
©"a
00m
4-c-t
m
©mat
o
*<>ooo
C
r-
©
. (*-
CDOJ
.*■O
OJ
UJ I/)o4 toec 3uj Z> ~< X
m
l—
Q-
CXoa
t—
aoXszcz
t-ao
t—
inecIL
Xacz3Z
UJUJacx>-
acv.UJUJecXI—
acUJccX3Z__C(JUJo
oUJ->cc3
iTii>cCf'iCirCf , woc ©\T\ """""""■"■" "ojcc^-oo^.
o^-^rcoo
r\jO*>©oi©©c-*o©—*o IT
—i t-i rg —< -n
~*
come »^oj —t
y\ """"""■"""""OjirtC
cC CCo^O—
tog-^rg —. (M —
CCOCC-i— CC— — CO O«* OOC—
-HCC-**-t©o
c
r«-if*Cir>foir« ©ciriur\©if\ ©"""""""# """ "ojooj(*.ouo©o»;OJ©OJ©
~* «-"-
-*■*
Oo«-40_rt©o_iooooo o— oj —
«
— oc OOOv'—<rg©©ogfg ©—i rg — ~~ —
»t
—o—-« — — omo—ooo o4- """"#
-4- O — O O © ©
O'OOOO.COOOOOO —fl """"""""""""oocooooooooo -«pg —<
■^""""""""""« "iroco^-— o©o*-«o— o
oo© — — o o© — o — o
oj©f>©oo©o©f>oo oo-r
oooooo^roo
o— kTifll_n— 000-Jf OMT —m«. """"""""""rs. .*-— OOO— O^soj —f-ivmO^— f- — — —«
— r\jf^-*-in*_Sr-ac O»o— oj
Owf>
r-
mr-
r-CD
c
"f\flog
mo"4-
o
CO
OJCO
"If!
CO
"og
O*
o
m
iro
UJo«acUJ>
"in
"o-ui
"CO
flc"fl
o
oj
"o
cr
4-fl
"ino
fifi
"flo"fl
o
"og
OO
"
4-UJ _/.o< Mac 3UJ z> —< X
fife Space DivisionNorthAmerican Rockwell
I
'I
I i
■
iii
■li
itii
!72
SD 70-576
-J<acuj pg — o ao gr cc O .__) 0
s
p~ 0* O>«""" ♦"oo*
(M — -^ — —CO
o
4
©"
.J<ecujpgOOpgmOpgoojto^O>"""""""■""""Op-p-co — m r» j) ff 1 ooff1 o**- to
CO
b3 «j
>r-maoco4, P-p- on cd< rg — -* — — CO
IAo
oo Opgi*icoin^mo*coo, fiO'*or************>r*j)p-r> fig3ineop
fc
O'Ox —< — — mco
o
P-
tn
ft
0)4->(UBro
ft
roPh
eoooo-ooo©
00000!/>"""""""*****ooomooo 00000
tn 4"
r^tA.— m.omiAn_ in in m in c"""""rvj — rg en Pg oj rg f- pj oj oj O
_,_.*rg — — pgrn — — — in
..j-o—
r-oO—
Op-p-000
O-J. 00-« O *D O O O
r-
OJOsO
gr"of\
"*■
CO
m
000000 00000000tA " " " * "*""""■"
000-J- 00000000
mp-oc oo — ooomoIA """■
»"""""""
im— mo oo — mmoogO— — oj — pg Pg —g_r 00P-J3O —
OP*P*OOO
iA " * " . """"""""OO£OD o — o^gsooo« rg —
©
o
p»4"
"P*
JOm
g_»in
"ina"r->en to
C- row aro ojlU to
-P rHd roo ct. <rHo voW rH
-=|-
CIJ
rH fl-P
O
Seh
Oft
ra fx.QJ
"H QJv y>y> roro ES -h
4^
ft
mo (_qfn
ft
O
QJr4flroEh
o
o
«c
—I<ooIIoo
__I8zcz_J
<ooX.-i/iocv.
X3Ozoz
UJzo
t-l/>acv.
uuzo
I/)
acUJCOX3z
*oUJa
UUJcc3
trt
m — — — ,-— —O — — —iA ""000 — — 000 — — o — —oo—r-o©c — — op—m— — p-
—m— — — m
crooooooommooo""*"""" """""OOCOOOOOJogoO©
4" — —
,aOfi-
minmtn m m fi in mrgro — f-"t— f>— — —« m — —mooor-00000000
,£-"""*"""
"""""o o o -o o o o 00000
IM — — — — om— — — lA —» O— — oj— ir»—
,»
— — — — — — — — OJ— 0000000 00000,)■ """"""" """""0000000 00000
pg
<m
(DOOO^OONIMrIIOi^O1
m«**««*» """""OOOfIOOOJOjfIOfIOJfi oj oj fi ia fi g}*
p» No*"fiMiHir''.OHiro
im o — ■* — — oj gJ- o — oj O
g>*_nO_no_noo omooo»*."♦""""" """""oj o im o oj o o OOJOOO
mmop-p-omm rg © ia o o!*>""""""""""""
M — — — — (M —"Mogpngrm-oo- «.
<r
o — rg
p-OJ
pg
firg
f.
oj
»n
opn
oIM
oin
"mp-m
mm
CDP--0*
ojoo
I-
oCOCO
UJo<ecUJ>
mOJ
«rpg
0*
o-Pg
Pg
<*■i-ac
oorg—rgOD
"inco
m"o0*
m*n"gf
CDO
"CO
g>UJ i/1o< Wloc 3uj Z> —< X
P-m
i-___.O
?.O
t—
Ql
oX3©2cz
t—
a.o
acv.
XoazazUJUJCLXh-
t--4/1ecXL.uuUJacX
t—
*/>acUJ
___>
X3_2
__coUJO
oUJ~>tc3
m — — © oj — — — ojm " " " " """"""""— — O Pg — — — OO— — IM_-h rg — — — **_4(M
OOO— O" OOO— OO— 0*m " " " " """"""""oo— pj coo— oo— oj—— — «000- O 0000000-*
"* """" """"""""OO— OOOOOOOO —*Oi*ipgOinfl^OOfllAOJ" """"""""""""OWNiTiMfONO OWWO
ojftrgoj—fl— f.(^
mooofioooooooo
o o o f. 00000000
OjojojOO OtA — (M o o OJ ing* """" """"""*"pgpgOO opg — oj©inrgoj
ojPj #-^<rg ojim-
— oooooooomooo-jf """"
""""""«"
0000
00©0<MoOO
— IM
eoOoOPnOO — oj**liAfl©fl
■
" " """""*""""0001*100 — IMfIP-flOfl *-i im fl P"! f!
P^IMO- <0— —
lAflO
— tAONO>«(D->-iN^OmNOrg ,-, rg — — — — — —
j-moiAfiinooo'nooopg O oj 4- pjOOOOjOOO
<*l0— OO OiA»A>»^^«lA>^—O — OO OOJIM— — pg — —— OJ f! -J/ lA
«<_
P- CO O O — OJ
O
O
IMo*
(Ato—<oOp-
cop^
"rg
OinOJ
og
gj-OJ
"oj
rSIA
UJo<acUJ>
ocf
flc
IMC
"OJ
"tf.■G
c"o
fl
mo-»oj
om"OJ
m
"mlAgt
«g)-
CD
gfUJ (/)u>« l^ac 3UJ z> —« X
-J-
fife SpaceDivisionNorthAmerican Rockwell
It.
\
i
-iii
!- 73 -SD 70-576
-Jac
LtiNCOco-tr'fMro-eo-O
Qh-t-eoeo-»"cog_)Off, o*g_»in
oo
a*mCD
-J<acuj mm — -r miAfMfMmfi *"* O
O fifi^f-m-o
CDflg9
m
m
cqU0)
y>0)srov
COPL,
rsrgf<.OOogojo*incomeO(*.tf ,,. ...♦»»"""> r- _ocoeog)"eoinOcoe>in«©< pg — —
acooo*oOOOO©©coOOOiAOOOOOOOO
IA ■*
r__o — O^oo — ooconlr> , """"""""""♦
o — ceeoo—inooofi— IM ""* og fl
XCO
"0*
>oo<
o"
I*lOJCO
-tg_>
fl
IMfl
"
oOm^ff-o >j-o.<f> »H^o_Dog
>< "-"coOOOPgOOOOOOOOi/\ """" """"""""
000*MDOOOOOOO
rg IM
h-OGm<coo- inmcoom """"""""""""OOOJOD O O
■—
«M OJ O
OO
—fM — — —
co
<*>OJmm
fi4-
co4-
"mOJCO
OJgj-
W)a"H
CO
t>
CQ
CQ
Gt3 roc wroW 4^>
4^ O3 T)
O
rHrd
O
rH MoW cv
fly
-* yiH
Sv oo st
CQ Id0) <D
O
4->"H
CD
fi B+3 -H
CO
4-5S
CQ
W?H
O
*HW
orH
QJrHfl
CO
&4
o
o
■*
-I<acxxoo__<
_.
iXszcz—I
©oX
1/1ocv.
sozoz
UJzo
t-ac__L
UJzo
tAecli.CDX3z__coUJc
vUJ-1cc3l/l
,*
O — r-oo — op-p*c oom
,#"""""""*""
qh<cOO"o
*---:-- -- J* J z000 — — coo—— ©— —
OO'-'^-000-
- --'©''^^— lA -* —' """' "1
erooooooommooo.*"♦""""""""""ooooooo«n"moooJ. — ——
tninfiiAiAinfimtn
*OflrtNNrtNlMl>"« N N(Ml*.—4 f+\ —" »*1 — — — fl — —lAOOOP-00000000
,*.».""*""♦"""*
000000000000"o
py———-I©O — — — O — O— — — — ©ia — — — ia — m— — — — rg — — — im — oj— 000000000000J.
♦»""""*""""*
0000000 00000pg «*
CD— — — I*l— — — — IMIAPJJOfl «__»_!-_."
p*.^©— m — — infio — mo(<.»"""""""""""
pyO- -*■—— OJJJ-O-^OJOrg — — — — —— — —-JtiAOinOIAOOOIAOO©
fl * "N3MOIMOOONOOO
rn'iA O p* f* otnun OJOIAOOm 2*22pgO^^OOJOJOjOOJOO
— *m f* ** *n * **■ eecro—oj
OJo*0
p»OJOJ
f.OJfl
Og*rin
oI*l
oJM
»nin
fl
flfl
"flp-gj-
"CO—OJo
t
o
o---4-oCDCO
UJo«acUJ>
p-m<o
p--I*l
"OJ—g/IMtr
p-OJOJ
J*p~
"CD
o»o
COfl
OJCD
OJ
r--I*l
iningr
CDOCO
4-UJ wio41 l/>ac 3uj z> —« X
p"fl
gj-
t-a.o__Cl
_-a.orszc
a.at~tAecv.
X3O__zUlUlacXp-
p-fc/>
v.UJUJacX
i/.AcUJ__QX3
*UJo
vUJ-1CD3</»
O — P-'O
O^Of^r-000
m """"""""""""© — <CCOO— OJ) <0000— —(M — — —fiO-^Om— —» © o
«-^
©m """"""""""""o — om— — — oo— — o—m—— — — —omo —fiooo — oo— 0>
m """"""""""""pg © — <? oo©-— OO— oj— — —
_*
— 4-
e>oo — 000000000oo—ooooooooo
o o —rgom- moo — mm4-
«""""*"""*""
O — og O rg— rgoo — ojrgIM— OJ ,-_i-^— — — —
mooo — 00000000000 — 00000000
pg — — O — OO— ojOm — O>*■ """"""""""""— — o — oo — rg©oj — m— — — — rg — — pg— ooooooooomoo>*■ ""*" ""■"""""
ooo©
OoOoOrgOOfM —
cd ©oo — oo— — ojoojofl
"""""""""»"*
© O O -^ O O — — Pg iA pg O— — — rg rg im
r-rgo — fi— < —
m^O
— mom """*"""""■""rg O — g? — —oj^o— (MOfM ——— — —— — —£moinomoG omooom """"""""""""pgopgoojoooojooo
p*imooooom—— 000m**»« """"""""IMOOOOOPJ— — 000
«4Pufi<j>m4>p»cDO< 0- oj
o
CD
flflCD
CDmCD
fl
o
«gj*
o
fl
o—p~
OJ
m"to
ojo"o
p-—gj-
"fi
UJo4acUJ>
r-m"
o
"oo"CD—o
—oo
o---4-"0*
4-
cnpn
"cof.
mm
IMg}-
gj-UJ l/tcg| i/>ot 3uj Z> —4 X
T.
1
i
I
i
i
j
I
'i
:l
81d
fife Space DivisionNorth AmericanRockwell
each hand, and the same information averaged over Ss . Table
8 gives this information for the 16 analyzed hands, using
the parameter estimates which were optimal or modal for the
16 analyzed hands, according to the rules for forming the
variations of different strengths. Table 9 gives the errors
of prediction for the 14 holdout hands using the parameter
estimates which were optimal or modal for the 16 analyzed
hands. Table 10 shows the results of using the parameter
estimates which were optimal or modal for the 14 holdout
hands in the prediction of behavior on the 14 holdout hands .
- 7^ -SD 70-576
1:
I
I
It
I
M
fife Space DivisionNorthAmerican Rockwell
CHAPTER VI
SUMMARY AND CONCLUSIONS
The purpose of the research, as outlined previously,
was essentially methodological. The basic approach and most
of the techniques applied were established before any data
were collected, and in fact before the game was selected.
The outcomes of the research will be discussed with respect
to the stated purpose.
Five process rules constitute the model. Decisions
to accept offered cards are based solely upon the rules of
the game, the color and magnitude of the offered card, and
the previous decisions. The model is completely determinis-
tic .The computer program is successfully kept on the
correct track by very simple features in the source code
associated with the entry points ACCEPT and REJECT. After
a prediction which differs from the recorded human choice,
an error of simulation is scored and the program is forced
- 75 -SD 70-576
M
fife Space DivisionNorthAmerican Rockwell
to make the recorded choice. Thus, there are no problems
associated with the conditional prediction effect.
Evaluating the use of Fortran as the programming
language has two aspects. Certainly the model was simple
enough to express in Fortran, and only basic skills were
needed to develop and test the program. The main purpose,
however, for using Fortran was to make public all of the
model mechanization. The understandability of the source
code was not tested or measured in any way, but most sci-
entists do have some familiarity with Fortran or access to
others who do. The source code presented in Appendix D
does include some dialect features associated with a par-
ticular manufacturer's implementation of Fortran, but the
code could quite easily be rewritten in any other dialect.
The successful use of Fortran in this research is therefore
a reasonable but unproven conclusion.
The only evidence for the strictly empirical deri-
vation of the model is in the details of the analysis proc-
ess described in Chapter IV. The analysis was successfully
free of any conscious bias . No assumptions regarding cog-
nitive processes or the structure of the model were held in
advance of protocol analysis . The process rules induced by
the analyst (the author) are not the only ones possible, and
- 76 -SD 70-576
SD 70-576 11
M
fife Space DivisionNorth American Rockwell
other rules might emerge from the same protocol data for
other analysts. Had the first protocols analyzed been those
of Ss other than SjLl and S6 , somewhat different rules could
be expected. Had the sample Ss been different, the model
might have been very different. Nonexperimental anecdotal
support for this is found in the fact that a haphazard
sample of two scientific computer programmers, upon reading
the rules of the experimental game, at once proceeded to try
to discover optimal playing strategy based upon very complex
probability calculations. It would appear in retrospect
that to empirically derive a more nearly general model it
would be necessary to ensure a more diverse subject sample.
The model was reasonably successful in predicting
the behavior of 11 individual S_s , each of whom played the
game at least somewhat differently. A twelfth S_, unlike any
of the others, had a very strong color bias which was not
accounted for by the model. The model has seven numeric
parameters which were fitted to individual S_ behavior. Each
S had a unique set of parameter estimates. One of the pur-
poses of the research was to develop a model which was gen-
eral in structure but individual in function. The general-
ity of structure is indicated by the fact that a single
short main computer program predicted behavior for all
- 77 -
f
.
fife SpaceDivisionNorthAmerican Rockwell
subjects, with 93.4 per cent accuracy for 11 synoptic sub-
jects. The individuality of function was provided by the
incorporation of individually fitted parameter estimates.
The importance of these individualizing parameters is seen
from the fact that when their values are held constant in-
stead of being allowed to vary according to individual dif-
ferences, model performance is very significantly degraded.
The results may be interpreted to support the conclusion
that individuals can be characterized in terms of their
parameter estimates, and that the parameterized model can
account for individual differences in behavior.
The research purposed to develop a measure of the
goodness of simulation. The only measure presented was
error per cent. For the case of sequential decisions, the
conditional prediction effect obtains, so that error per
cent reflects the highly interdependent decisions made by
the model. It would be invalid to use error per cent in the
usual formulae for the significance of a proportion, the
significance of differences between proportions, etc. The
research did not develop a method to test for whether a
model predicted significantly better than chance. For the
cross-validation 14 hands, the average error per cent was
5.65 per cent. The question which should be asked and
- 78 -SD 70-576
r
/
t.
fife Space DivisionNorthAmerican Rockwell
answered is this : does this model account for all but an
insignificant proportion of the variance in the observed
natural behavior? For the author, this remains a challeng-
ing open question.
On the other hand, it may be that error per cent is
quite satisfactory for many purposes. In classical statis-
tical hypothesis testing, the researcher chooses, presumably
in advance, a type I error per cent he is willing to toler-
ate. For experimental decision-making situations like the
one used in this research, the model can incorporate in-
creasingly more parameters, and use up degrees of freedom,
until every single decision point is represented by a
parameter. As the number of parameters approached the num-
ber of choice points, the error per cent would approach
zero, inexorably. The researcher could establish in advance
an error per cent tolerable for his purposes, and add param-
eters to his model until this value was reached. Comparing
models in terms of average error per cent is valid. Error
per cent is directly based upon numbers of errors, which are
arrived at by counting. Ratio scale status for the error
per cent measure is not required for the arithmetic opera-
tions used in the t-tests, however. The theoretical distri-
bution of this error per cent measure is not known (to the
- 79 -SD 70-576
f:
fife Space DivisionNorth American Rockwell
author, at any rate). For each S_, the denominator used in
calculating the error per cent might be different, as it was
for 10 out of the 12 Ss in this study. The conditional pre-
diction effect upon the distribution is probably not amen-
able to any closed-form analysis. It is, however, reason-
able to assume that this measure may be manipulated as be-
longing to an interval scale, with tolerable inaccuracy.
The research developed a technique for studying the
effects of the parameters on the predictive power of the
model. The technique involves the generation of models of
varying strengths by selectively nullifying the effects of
the various parameters, and calculation of the error reduc-
tion associated with particular parameter combinations. The
technique permits the derivation of an exact expression for
the contribution of the parameters to the predictive
strength of the model. Arbitrary though it might seem, the
bit notation used facilitates the calculations of parameter
contributions, while the octal identifiers for the varia-
tions are much more convenient than mnemonic identifiers
composed of English words. The technique involving model
variation and error partition emerged from the search for a
means to test simulation adequacy in terms of classical
hypothesis testing. That search is not over, but the
- 80 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
methodology described in this report is offered as part of
a feasible approach to that goal.
i.
81 -SD 70-576
jm
fife Space DivisionNorth American Rockwell
APPENDIXES
- 82 -SD 70-576
fife SpaceDivisionNorthAmerican Rockwell
APPENDIX A
GAME INSTRUCTIONS
- 83 -SD 70-576
JO*
fife Space DivisionNorth American Rockwell
APPENDIX A
GAME INSTRUCTIONS
The card game you are about to play was developed as
part of a research program designed to define a methodology
for understanding observed behavior in a problem-solving
(game-playing) situation.
1. The card deck for this game has 24 cards (12 of
one red suit, ace through queen, and 12 of one black suit,
ace through queen) . Each deck has been randomly shuffled
using a list of random numbers, and the sequence of decks is
itself a random arrangement (there is no pattern in the pres-
entation of successive decks). Each person will play the
same hands in the same sequence.
2. I will show you the top 10 cards, one at a time.
As each card is exposed, I will call out the color and value
of the exposed card and you decide whether to accept or
reject it. "Think aloud" as you make your choice. Of the
top 10 cards, you must accept five.
- 8^ -SD 70-576
Jr
fife Space DivisionNorthAmerican Rockwell
3. The numbered cards have a value equal to the
number on the card; the ace is worth one point, the jack,
11 points, and the queen, 12 points. The goal of the game
is to get the maximum total score for the five cards you
select. The values of the red cards and the black cards
will be added separately, and the larger sum will be your
total score for that particular hand. Thirty hands of the
same game will be played during each one-hour session. You
should be able to improve your performance with practice.
4. You are asked to do all of your thinking aloud,
and a tape recording will be made during the course of play.
You may use pencil and paper if you wish, provided you con-
currently verbalize what you are doing.
- 85 -SD 70-576
fife SpaceDivisionNorth American Rockwell
APPENDIX B
EXCERPTS FROM ALL PROTOCOLS
- 86 -SD 70-576
jf
Jy
fife Space DivisionNorth American Rockwell
APPENDIX B
EXCERPTS FROM ALL PROTOCOLS
SI (after H3O)
I think it works best to try to get all cards of the
same color instead of mixing them.
(Summary comment, on inquiry)
I found that the best system that I worked out was
trying to get all of one color, instead of mixing the two
colors and having half red and half black, and get all reds
or blacks, which generally gave a higher score.
S2 (at start of H4 )
810 I accept it; I'm going to work on blacks and
high numbers
Rll I think I'll accept that too so I can work on
both.
R3 I reject it.
812 That settles it; I'm going to be working on
the blacks now.
- 87 -SD 70-576
j£
fife Space DivisionNorth American Rockwell
(At start of H26)
R9 I accept it and hope I don't see a high black.
810 Oh darn! Since the 10 is higher I'll take
that.
B6 I accept it and work on blacks .(After accepting three cards on H4O)
Rll Darn! Since the jack and 10 are higher than
the 7 and 9, I'll switch back to reds again and
accept it.
(After accepting three cards on H5O)
Rl2 It's too late to switch colors.
(Summary statement)
It's best to wait for a high card when you're first
starting and stick with the high cards in that one color.
If two high cards come up in a row, one in each color, then
you take them both and wait for the higher cards in which-
ever color shows up. But half the time, when the last cards
are coming around, you have to gamble, and that's what I do.
S3 (on H5, after accepting four cards)
B9 I'm holding 12 black in my hand, 26 red in my
hand. There are nine cards showing. I'll
leave it and gamble that the last card is red.
- 88 -SD 70-576
M
fife Space DivisionNorthAmerican Rockwell
(On H2l, after taking two cards)
B5 I'll take it, because I'm building blacks and
the other cards have been small, so there might
not be many high cards in this hand.
(On H2 3, after taking B8 and Bll)
R9 Six cards showing. I'll take the red 9, in
case the rest of the cards are red.
Rll I'll take it; high red which gives me a higher
score in the red than in the black.
(Summary statement, on inquiry)
My strategy would be that I wait for a 7 or better
in the first three cards before I accept a card and the
color of that card determines which color I will start col-
lecting, trying always to get five of one color. If a high
card shows up in each color in the first three cards, I
usually take them both and let the color of the next high
card decide which color I will build on.
S4 (after Hi)
Since I started collecting blacks, it doesn't seem
worthwhile to take high reds since in all probability that
section won't be counted anyway, and I don't know how the
blacks will be stacked because of the random distribution.
- 89 -SD 70-576
Jt
fife Space DivisionNorth American Rockwell
(After H4)
It seems as though in the previous hands, when I had
started accepting one color, high cards in the other color
would come up and I didn't choose those and it seems as
though the large numbers have been coming in series, so I
decided to collect high numbers in both colors.
(During H7 )
After you've passed up one or two high cards in a
color (red) because you've started with the other color
(black) you may as well continue with the blacks and reject
all reds .(During H2l)
the theory now being that now that I've
started with the black series I'll take any blacks over
five, and reject the rest.
(On H3l, the beginning of the second session)
R7 I accept, because it's red and on the way up
the stairs today I decided to go with the reds
on all hands today. On some hands I might have
less of a chance, but on the over-all 30 hands,
I thought it would be nice to have some kind of
a system.
90SD 70-576
fife Space DivisionNorth American Rockwell
(On H32, considering first card offered)
B9 Reject, black. it looks like a nice card and
I'm tempted to take it, but I'm going to go
ahead with the reds so that I can have the
security of knowing that I'm doing something.
(After H39)
Once again, I think it is just as well to go on with
the reds, because although it might not always be best on an
individual hand, since there is no way of knowing what is
coming up on the different hands or what their distribution
is, sticking to the reds seems to work well over-all. The
fact that so many blacks came up this time is just a special
case .(During H44 , after 812, 8 7 , B 6 )
I have the urge, when I see a high black card on the
first card, to accept, but I decide to be consistent with
the reds because it doesn't mean anything, it's just like a
good omen .(At start of H54 )
B9 Accept. I just want to see what happens if I
go with the blacks this time .(After H54)
My new philosophy is this: since it doesn't seem to
- 91 -SD 70-576
k.
fife SpaceDivisionNorth AmericanRockwell
make much difference whether I stick to the reds or change
to the blacks, I'm going to play each hand as I please for
the remainder of the session.
(During H56 )
As it turns out, I would have been better off to
change to blacks when the black 9 came up because the black
9 and 12 would have been higher than the red 11 and 8, but
it's too late now and I'll have to take the last three.
(At start of H57)
Accept, red, and I'm going back to my original sys-
tem for this session and accept only reds, because when you
have to play 30 hands, it's better to stick with one system.
(Summary, after H6O)
Since you're not gaining or losing anything, it
seems at first that it would be better to play each hand by
ear and kind of try a system for each hand, but then, be-
cause of my own temperament, I decided to go with the reds
for the second session. It would seem that it is better to
have some system rather than none at all, and over the long
haul it might be more beneficial to go with the reds, even
though I knew definitely that I'd miss out on a few hands.
- 92 -SD 70-576
__.
fife Space DivisionNorth American Rockwell
S_s (H_4 complete)
810 I'll take the black 10; high black.
Rll I'll take the red jack, too, so I can accept
high cards in either color.
R3 I'll let the red 3 ride; too low.
812 I'll take the black queen, high black, and now
I'm looking for blacks.
B6 I'll take the black 6. I'd rather take a
fairly high black early than be stuck with the
low cards later in the hand.
B8 I'll take the black 8, high black, and there
may not be a higher black left.
(At start of H8)
r6 I'll let the red 6 go; I'm looking for some
thing higher to start with
B7 I'll take the black 7. Sevens are high enough
to start with, and I have a feeling there will
be more high blacks.
(Hl7, after rejecting R5 and accepting R9)
r6 I'll take the red 6 because it looks like there
might be a string of reds .(Summary, after H6 0)
What I did, I examined the first few cards coming
- 93 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
out, and saw whether they were high cards or low cards, and
after I had accepted two relatively high cards in one color,
I stuck with that color and tried to get as many in that one
color as I could, because that's how the high point hands
came. That basically was my strategy. That is, to try to
get as many cards as possible in one suit and try to get
them high.
S6 (after Hi)
For the first time around, I wasn't exactly sure how
they would come up but I suspected that I wouldn't take any-
thing lower than a7or 8 . Other than that I had no real
strategy for the first time around.
(Start of H2O)
r3 I'll probably never take a 3, especially so
early in the hand. I'll pass it up.
B2 Same with the 2.
r8 I'll take that; I'd take that most any time.
(After taking R7 , RIO, 89, on H43)
812 I didn't want this, but I better take it be-
cause the 12 and 9 are higher than the 7 and 10.
(Summary statement, after H6O)
First of all a hunch that I had in the beginning
- 9h -SD 70-576
____.
fife Space DivisionNorthAmerican Rockwell
proved out: that it was never really worth taking lower
than a 4 and possible a 5. I think I learned a little later
in the game than I should have that I shouldn't have split
in many cases where I did, early in the game. That is, I'd
have two or three good cards in one color and a high card
would come up in the other color, and I was so impressed
that I grabbed it real quick, without realizing that my
chances were so slim of bettering my hand in this second
color, even though it might be a jack or queen. Lastly,
whenever I had three [cards] in one color, it was never
worth taking the opposite color; you'd never build up any-
thing on two cards .
S7 (summary statement, after H6O)
I'd wait for the first high card to accept, and if
it were red, for example, I'd pretty much play the reds.
Sometimes I'd have to wait through as many as five cards
before a card, high enough to take, would come up. I tried
to get them all of one color, even if I had to take a rather
low number like a 4 or a 3 , rather than to take a high card
of the other color because getting all five cards of one
color seemed to give more points than taking three of one
color and two of another. This, generally, seemed to work
- 95 -SD 70-576
__.
fife Space DivisionNorth American Rockwell
best and is what I tried to do.
S8 (at the start of H3l)
R7 I'll reject it; too low.
B_7 Reject it; too low.
812 Accept, high.
B4 Reject; too low.
Rl2 Accept; high and possibility of going either
way .(Summary statement, after H60)
If a card comes up which is a 7 or an 8 in either
the red or black suit, you take that as the first card; be-
low that you wait until you get one of those. Then you
build on the color of the first high card accepted, always
taking a 7 or above in that color. Then, toward the end,
when there are only three or four cards left and you still
need two cards to complete the hand, you take any card in
the suit you're building on no matter how low it is. Also,
if in the beginning, you have rejected three or four cards
for being too low, and two high cards come up, one in each
color, take both of them and see what color the next high
card is, and accept it, build on that color, and take any
cards of that color that come up after that.
- 96 -SD 70-576
_w
fife Space DivisionNorth American Rockwell
S9 (H24, complete)
B2 Pass; low card.
B3 Pass; still too low.
R9 Take it.
810 Take it.
Rl Pass .B_6 I will take it; semi-high.
R7 Will not take; stick to blacks.
R8 I will take it.
RIO I'll take that one.
(Summary statement)
Basically, I think how I did try to play the game
was if a card was a 5 or higher, I would try to take that
card- if it was lower, I would just pass. And whatever
color it would be, I would try to stick to that color in-
stead of trying to split colors. I would split points or
whatever and not come up so high. Usually, when we got past
four cards, I would take [pause] . You see, if the cards
were coming out quite low, usually I would try to take the
color that came out the worst high, which doesn't make
sense. But say if it was a split color and they were coming
out black and reds, and they were coming out quite low, I
would try to take that color that came up next. But,
- 97 -SD 70-576
____
fife SpaceDivisionNorthAmerican Rockwell
basically, what I did try to do, within the first three or
four cards , I would try to take the card that would be 5 or
higher and basically stick to that color, unless I saw that
the opposite color was coming up and quite high cards, I
would try to maybe take one of those cards and build up the
other color .SlO (HlB, complete)
R4 Reject it; too low.
B7 Accept it ; high .B_2 Reject; too low.
R2 Reject it; how.
B9 Accept it; high.
81l Accept it; it's
B6 Accept it; it's
high.
black and it's high
high.810 Accept it; it's
(Summary statement)
I guess it would be take the first high card and,
depending on what color it is, take mostly the cards that
are of that color. Or else, if you have a high black and
right away have a high red, if the next card is a high
black, take all the rest blacks, or if the next card is a
high red, take all the rest red.
- 98 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
810 Yes, I'll take that, too.
B6 No; too low.
R3 No.
R5 Not too low any more; I'll take the 5.
B5 No; I'm going for reds.
R6 I'll take the 6.
Rl2 Take it.
Sll, session 2 (H3l)
R7 Accept, high.
B7 Reject, black.
- 99 -SD 70-576
fife Space DivisionNorth American Rockwell
812 Accept, high.
B4 Reject, low.
Rl2 Accept, high.
R6 Accept, red.
R4 Reject, too low.
RIO Accept, high red.
B8
R2
Score 35R
(H32)
B9 Accept, high.
r4 Reject, low.
B3 Reject, low.
Rl Reject, low.
R2 Reject, low.
Rll Accept, high.
Bl Reject, lov.
B8 Accept.
Rl2 Accept.
B6 Accept .Score 23R, 238
(H35)
R3 No, low.
100 -SD 70-576
____.
_X
fife Space DivisionNorthAmerican Rockwell
R5 No, too low.
B_4 No, low.
Rl No , low .R6 Take it, high enough.
B6 Take it, high enough.
Rll Take it, high red.
R4 Take it, red.
810 Yes, I'll take it, high and I can work both
ways .B5 No , too low .Bl No, too low.
B2 I'll take it.
R9 Yes, high red.
B9 Yes, I'll take it.
812
Score 218
101 -SD 70-576
fife Space DivisionNorthAmerican Rockwell
(H39)
81l Take it, high.
B9 Yes, black.
B2 No, low.
B7 Yes, high.
R5 No, red.
R2 No, red.
Rl2 Nope, red.
RIO No.
B8 Accept .B4 Accept .Score 398
(H4O)
R6 Too low.
B_7 Yes, take it, high enough.
B2 No, too low.
RlO Yes, I'll take it, high.
R4 No , low .R2 No, low.
B9 Yes, high.
Rll Yes, high.
810 Yes, high black.
B4
Score 268
102 -SD 70-576
fife Space DivisionNorth American Rockwell
(H43)
R7 Yes, that's high enough
RIO Yes, high red.
B2 No, low
B_4 No, low black
B9 No, black.
B5 No, black
R4 Yes, red.
812 Nope .R9 Yes, high red [had to accept].
R5 Yes [had to accept] .Score 35R
(H44)
812 Yes, high
B_7 Yes, high enough.
B6 Take it .B8 Yes, high black.
R6 No, red.
R3 No, red.
R2 No, red.
B2 No, low.
Rl2 No, red.
R9 Accept .Score 338
103 -SD 70-576
I
fife SpaceDivisionNorthAmerican Rockwell
(H47)
B4 Too low.
Bl Too low.
81l Well, I hope enough black come up, I'll take
the jack.
R3 No, red.
812 Take it, high black.
B2 Yes, I'll take it, I don't think too many more
blacks will come, so I'll take the 2.
R6 No, red.
Rl No, red.
B3 Accept.
B8 Accept.
Score 368
(H4B)
R8 Yes, it's high enough.
83 . No, too low.
B6 Too low, black.
B2 No, low black.
R7 Take it, red and high enough.
Rl No, I think better reds will come up.
Rl2 Yes, high red.
81l Nope, black.
10U -SD 70-576
!
fife SpaceDivisionNorthAmerican Rockwell
Rll Accept.
812 Accept.
Score 38R
(H5l)
R5 No, low.
B6 No, too low.
B5 It's kind of hard to make a choice, they're all
mediocre; I'll take the 5.
Bl No, too low.
812 That, I'll take; high black.
B4 Yes, black.
RIO No, I don't want the 10, it's red.
B3 Yes, I'll take the 3, it's black.
810 I'll take it, high black.
R9
Score 348
(H52)
B5 No, low.
Rl Too low.
R3 Too low.
B9 That's good, I'll take it.
R9 I'll take the red 9 too, open up my options.
B8 I'll take the 8, high black.
105SD 70-576
I
i
(H55)
(H56)
fife Space DivisionNorthAmerican Rockwell
R2 No, too low.
Rll Eight cards, I'll take the jack, and take the
higher of the next two cards coming up.
Rl2 Can't be any higher than that, take it.
B2
Score 32R
B7 That's pretty good, I'll take it.
B4 That's too low.
Rl No, low red.
Bl Too low.
R4 No, it's red.
81l I'll take it, high black.
Rl2 It's the seventh, no it's red, I'll take the
last three.
R5 Accept.
R2 Accept .R9 Accept.
Score 188
Rll That's good, high red.
Rl Too low.
R3 I'll take it, red.
- 106 -SD 70-576
fife SpaceDivisionNorthAmerican Rockwell
R8 That too, high red.
B4 No, black.
B9 No, it's black.
812 No, it's black.
810 Black [reject] .B5 Accept .
(H59)
B8 That's high enough [accept]
81l Yes, high black.
)
Rl2 No, I don't want that, red.
R2 No, that's red.
B4 Yes, I'll take the 4, black.
812 Yes, high black.
B2 It's too low [reject]
Rll No, it's red.
Bl Accept.
Score 22R
fife Space DivisionNorthAmerican Rockwell
B3 No, too low.
Rll Take it, open up my options.
81l It'll be black [his option—accept].
B5 Too low.
B2 It's the ninth card; no, I'll take my chances
on the last card.
R8 Accept.
Score 308
(Summary statement)
Well, the strategy varies with the first three or
four cards laid down. Usually it depended on whether the
first card thrown down, as far as color was concerned, was
a 7 or higher, and if any other card came by of the same
color, 6 or higher, I would usually stay with that color.
But if a card of a very liigh magnitude, let's say a jack or
queen, came by of the opposite color, I would take it and
open up my options, in most cases. Then I would take the
next higher card that came along in either color, and then
play it out in that color. That worked the best, in most
cases. The hands I had the most trouble with most often
were the hands in which the mediocre cards were thrown down,
108 -SD 70-576
R7 It's red.
R4 Nope .
fife Space DivisionNorthAmerican Rockwell
let's say a4, 5, or a
6,
in sequence. In one hand, espe-
cially, the first three or four cards were like that and I
didn't know what to do. I tried to assume that about as
many black cards as red cards would be in the deck. That's
what it came to when I got to about seven or eight cards in
a deck and I held four of one color, I would assume that of
the remaining cards, half would come up black and half red,
and I didn't pay too much attention to the cards that had
already been thrown down, as far as the number of cards of
each color. That, in general, is the strategy that devel-
810 I'll take that; high number.
Rll No, it's red.
R3 No, red and not high enough.
812 Yes.
B_6 Take that .B8 Take that.
Rl No.
R6 No.
B9 Take that.
109 -SD 70-576
oped.
Sl2 (H4, complete)
fife Space DivisionNorth AmericanRockwell
(Summary statement)
I think the best approach was to pick the color of
the first high card that came up and then continue to choose
that color of card. I took a lot of chances because I found
out that usually I got better results when I took chances
than when I played it safe and just picked the first five.
Number of Times (Hands) S Played Unique!
110 -SD 70-576
1 2
2 4
3 4
4 9
5 3
6 5
7 0
8 1
9 1
.0 3
.1 2
.2 8
fife Space DivisionNorthAmerican Rockwell
For the 16 Hands, the Sets of SsResponding Identically
Number of WaysHand Played
9
4
6
7
3
7
3
4
5
3
8
5
5
4
5
5
The average number of ways each hand was played by
the 12 Ss was 5.2; excluding S_4, the average was 4.6.
11l -SD 70-576
fife Space DivisionNorth AmericanRockwell
APPENDIX C
SYNOPSIS OF DECISIONS FOR ALL SUBJECTS
112SD 70-576
L
OECK31
DECK32
DECK35
DECK36
fife Space DivisionNorthAmerican Rockwell
SUBJFCT
I23<»56789
101112
SUBJECT
123
A
if56789
101112
SUBJECT
1234 A56789
101112
SUBJECT
<r A56789
101112
- 113 -SD 70-576
-7 7 12 <. -12 -6 -<. -10 8 . -2
AAA A AA A AAA
AAA A AA A A A AAAA A AAAA A AA A A A A
A A AAAA A A A AA A A A AA AAA. AA A A A A
9-4 3-1 -2 -11 1 8-12 6
A AAAAAA
A AAAA A A A AA AAAA AAAA AAAA AAAA AAAA AAAA A A A
A A A A
AAA
AAAAAAAA
-3 -5 <► -1-6 6 -11 -A. -10 -7
A A A A AA A A A A
A A A A AA AAAAAA A AAAA A AA A A A AA A A A AA A A A AA A A A AAA. A A A
A A A A A
-4 -2 -6 10 5 1 2 -9 9 12
AAAAAA
A A
123
A AA A
AAAAA A
A AA A
A AA A
AAAA A
A A
AAAA..... A .AAAAAAA
A A AA A A
A AAAA
AAA
_1_
fife SpaceDivisionNorthAmerican Rockwell
HECK SUBJECT39 11 9 2 7 -5 -2 -12 -10 8 A
1 A A A A A2 AAA A A3 AAA A A4 A A A A A
.__
5 . A_ A. _A._ A A6 AAA A A7 AAA A A8 AAA A A9 AAA A A
10 AAA A All._. ... A. . ...A . A ...A A12 A A A A A
DECK SUBJECT.40 -6 7 2 . -10. . -A -2 9 -11. 10 4
I A A A A A23
AA
A AAAA AAA
4 A A A A A5 A
_.
A __.A A A6 A A AAA7 A
AA AAA
8 A AAA9 A A A A A
10 A A A A A11 -A A A ..... A A12 A A A A A
DECK SUBJECT_ -7 10 2. _ A
__.
5 _ -443 12__ -9 -5
1 A AAAA
._
A_
AA
A A2 AAA3 A A
AA A
4 A A A__5 -A
6 A
.-
A A. AA A A A
7 A A AAA8 A A A A A9 A A AAA
10 A A AAA11 A A A_ A A12 A A AAA
DECK SUBJECT44 _12 7 .6 a. -6-_-..-___ ... -2 2_ -12 -9
1 AAA
AAA
A AA
A AAAAA
. AA
A23
A A A45 A
AAAAAAA
6789
101112
- Hi. -SD 70-576
AAAA AAAA
A A AA AAAAAAAAAAAAAA A AAAA
DECK47
DECKA8
DECK51
DECK52
.____
fife SpaceDivisionNorth American Rockwell
SUBJECT
123456789
101112
SUBJECT
123456789
101112
SUBJECT
1234
___.-
-6789
1011-12
SUBJECT
123456789
101112
4 111-3 12 ...2 -6 -1 3 8 _A AAA
AA
AAA
AAAA AA
AAAA AA .. AA . A A
AAAAAA
A AA A
A AA A AA AAA A .
AAAA AA AA. A A
AAAA A
11 -11 121 -123 .6 2 -7-8
A A A AA A A AA A A AAAA AA .... A
._
AAA A A AA A A AAAA AAAA AA A A AA A . . _ A AA A A A
3 _10 -94 -10
A AAAA
A AA AA
AAAAAAA
A AA.AAA
AAAAA AA AA A
A AAAA AA
AA
A_ AAA A
11 -12 . 28 -2
A A AAAAAA A AA A AAA
A A AAAA
-_
A AAAA A AAAA A AAAA A AA A A AAAA A AAAA A AA A A A
115SD 70-576
AAAAAAAAAAAA
-5
A
A
5
A
A
6_ 5
._
1 12
A A
A ___..._
A A
A AA
A A
-1 -3 9 -9
A A
fife Space DivisionNorth American Rockwell
DECK-55..
SUBJECT7 _ 4 _rj J -4 11.-12 -.5 -2 -J
1 A234 A
-.5 A
A A A AA A A A AA A A A AA AAAAAA
._A__ —A A A.67
AA
A AA
A AAAA
8 A
...__.
A A A A9 A A AAA
10 A A A A A11 A A A A A12 A A A A A
DECK SUBJECT-56 -11 rl ___-3 -B <t_ 9 12 IQ A..... 1.
1 A A AAA2 A A AAA3 AAA A A4 A A AAA5 A A A A _A_6 A A AAA7 A A AAA8 _ A
__._
_ . A A A A9 AAA A A
10 A A AAA11 A A__..._.A _A A12 A A AAA
DECK_59 12 ____-!_ 9 3
1 A A A A A2 AAA A A3 A A A A A
A A4 A A AAA5. -A A A A6 A A A A A7 A A A A A8 A A A A A
A A9 A A A A A10 AAA A A
. 11 A A_. A .A AA12 A A A A
DECK SUBJECTS 10 3_. rll .11 5 .-7 -A 260
1 A A A A A2 A A A A A3 A A A A A4 A A A A A5 . --A... _A .._ A ......A. . . . A_A A6 A A A A AA A7 A A A A A8 A A A A A9 A A A A A
A A10 A A A A A11 ... A . A A . A . _ A12 A A A A A
n6 -SD 70-576
SUBJECTa u__=i2 =2 a :
fife Space DivisionNorthAmerican Rockwell
THE SIMULATION PROGRAM SOURCE CODE
fife Space DivisionNorthAmerican Rockwell
118 -I SD 70-576
"o
UJr-<a
i
z<acfr-O-
CD
ii.
o-o
<«.
■vI/.
O
r~a.11lIS)
COrH
-IUJ>LU_J
o»
QLUOza.<x " "o " »z » »" * »
Q » ♦< * "
O
* »H " »" * - «
1_ » . "v " r- '■£ »vi » o. i *Q " < »O "
►
X *z * <o o *►*a a «>- * "v. #
» »
"— * m e_ "-j »p ♦ a. uu i-
of
"o » v
Q-
< »z * "o_ x ♦» » *" o " o "o » a. v. -o o »— ♦ a. "
O
»
»
"
«
»o»»rnl-o» —a_»*o. — m ai » ouj * * oc a. v * -H""»"»»Z ac » J
uj » ♦ rg « 3 » —
o**o.»-»-ll*
t-a: " * a a * orj ♦ ♦ «Ac » " lv
o*»-h<»x»
->_o»»aoror-# uj
" » »
Q.
o_ » DC
O
* * » » Z *J- » » OC IN » » _J»» » 2IN "" P
>-»»ai/>a.O*
<»N «Z ♦ Ac ac «» to a. vLU » U^HO* l/l -HMz » a. » x "* —— » " » " Z -H_JX*»-H»IN» X -H
-< « l_I_I» P OLfIX
HOI« U^ZVIO
P t-3_LOoo»iuacoac» < ax=_MO»OlO_I» U U»-Iy- CC * " ►"-<!►-» I—O. (X ♦ o " — —□ » » * — " ~"Z* .-«» rH (Nt-O
Z— » * X ♦ -OQ.H-""_r < "
>_
U i/J * "U« X ■* lil
aC
» " "" "x » v
ox*
o a _» — -i
" Z ■— I- » LU _J_.i__.u_»__. " . ..a_ —H#<».
_ < »-vi *x * P __ O a_ a_ o
Q.
X**
IIniUtCZZ-ul
<»">. » lIOUKQDWOz » ♦ o_ " nuiauoc'reou
* « oi
#xoxx< «.<.«_
|»»Z 13
»</»
— ZZtJO —* » O UJ *</> " * X t- * P — P P PZ*»IZ " P PP Jo » »
O
— »<U.«»<U-U.U-<L: » # v »o— o. o~ — — vt- » " "
o*
» *_
*" " o(* " " » -1 —OJ » "pu v ■ v"■_
CLXoo
rg <*|
O O OOOOOrH^H.^
o o 00000000o o 00000000
z Z ZZZZZZZZi/l IA IS)lAlS)ISllS)lS)iS).lS)
o
in
o_-__o
Tl
LUp
Xo_ J-Z * <J
I o -.UJ u3
ec U —z oc ►-o o o a.a_ oc v. uj«tv < uUU II vX < *z — a
LUP—U P op «Jl-< v.
O
<
O
v — v. o
O
rH
"O f» 0" O h-h *h »h rg rg0000000000z zzz zIS) IS) IS) IS) IS)
J-
uLU-.
r- lv__
O
PI- po <o v
O
f-a..
-I h- —" p-»"
UJ "CC —
rt
<QJX 0- —o va
►-
#
— LUZ
O
i^ v__; 0_ a_ va < "< < "»v o —p 1 op -ll-< v. a. <
O
v—— v o
mv
in r^ m corg rg rg n N0000000000zzz z z
I IS) IS) <S) IS) IS)
"*vUJ->LUoc
p-0 Pa. <vn
IN —IlA (NCC XX LI►- oc
X
■* — "v.
Q.
I- -►- J -H. JJ " _JLU Z —U I-
" — ar- UJ
X Z lvv—— zooc «t <i *z u>— _J -H- -J
O
p Jl-u- < v. «t
C
—v— v o
f-
u
ff HMr\j r*. r*\ ro r*^0000000000zzz z z
I LH l/. LO '/. UT)
v
I
ozLU
-orooozIS)
o*IIzvLUzp
oII
aoz<X
IIUJX<z
"vUJli.v.UJ
z
zo►-
o
v.UJacXozo
I—
aLUoz
Q.
<tXaza<op
-_
vUJoozt-l/l
pozvova.LU
_"
UJvCC
oIS)
*vUJv.v.LU
z</)zoo.oJ_
fife Space DivisionNorth American Rockwell
~i o
t
—
119SD 70-576
3
II
<t3
XZ<eci—ac
O
x
o.0rr\*vl/lO
ff-0
►-a.UJi/.
CO
aUJoza.<x ♦o »z «" "o »< *a »p »
* "v r- 13 «uj a. Ac *a < "o"x » - »z -O
O
v. z »" a o *: v; »
I- U Q »IS) »
►
UJ I- »—
«\3 >OJ
lo *p a. lv r- z ac p »□ v a. < "Z - CC « X " *
»Nfo»za
at *o a. u_ o ■* a Z » ac-. o.
O
* » ocq»»i-»u » uj
U-OX-uOOLU » I—to a. — in „j oj o * vuj ec cl o *» Z - at * »aj in » ac
O
lo ♦-» »
O
a- * >*■ Z v. Ac *cc a »" * a * *-. » oc v " oe »jj 3
O
h < .vi X ac ♦o t-_o
Q.
O ro
O
— UJ » — Uo. oc »
O
a.o
«"
»z - *x.»■ 0C rg » — UJ »
O
"II Z X "N <N » O * sv Qv- a is) a. " a — ■»-» lvzacacpacOO * o0«*X»« Xlv ♦ <S> OC_uu*-rH>-uuac * ec c_Z Q.O k- «s "
O
U.— »- - ♦ Ac_l .-4.rg—
»O»0C
" t ac ► I in* " * vi—. 0"/)ZCv^O — v o» UJo uKca'jjiczo^ »»i vii on-KKZJ en» —
►_
__,>_<i_i-i«_a_ » * aa. o Ii- v»
uiO
Xo ... >^ » v to. .*_. oc»— uz _umi^ o a: » a a. » ac_
x. u<looc uj*x uj lo z«» uj j_ OO M-«ui3sei!ex i- v o co i o ac a»— z a o o
_» __t_>-LOuj jkt-o<j*Ji3 — X ac n *
r
; ■/) a o
«-
im 30»0-''Nr»\._
;
zs
>_.
>-x*-> oo.roz
__.
ac U * II f ff O ff.- uj acc»n ff
" -_)__■
O
Z UJI-» * I- ►- '—'a o lv o -J
»*acz
z >- z„y ,_ X OJ<»
OU-
-"Z —zfflX Z x
--*>->-ac*-3C
£55 »_ c z<»uuQ.i__3ca.2^u v -a* a. a a.v. *
Q.
U_ Lj <*_ '__ <-_ *a
'
:* :111p <->a.Xo
pj^ in .0 *- co ff OHigfl
OO O
O
OO OOrH-1-.rH
OO O
O
OO 00000000 o o
OO
000000
oLUvac 1
O
'v.Qz<LOacoac□cUJ
LOUJvoXoX00
UOU
<L-<
O
ac> aca: lv< fc-X — vx a.a —ist 3 »
IV- O *z a oXI- l-ac Ua. a.
Q_ Ulv a ac acX to e
O
LO UJ Ul li.—" ac *- az a 1--_ r_ H- "ojinoii —O OC OJ Z I o—
C
U LO ii * X0 ac oc — a
«"" O
v1 oc o t- 0 "- a
OUJU-H-LOIO
r-u p v a. "♦ ♦ ♦ uj 3 ac—> " *~ I ac
_j _ uioaj lv
O
ac ac a z — o t-
XluOlo
" — " OU t- LL * O t—
O O
a lo
aC
O O ""►—►—"— — z 0 -" -000x *: -" in *11 11 11 tv 11 o o 11 11 11UJ II ff ff— 3CUU.Q OC
"-OCO
O ac ac — _c oc h »— a ac acxujO — aujzzxxia01-U. h-f-i-r — UH-U.oca u-uoocncoao*->-)— — a. a. a. &. >- >-
1 v '
.a
sffOHNrt>lin
W~H-H -i^hINIMNNIM000 00000000000 00000000
v
»
«
»
*»
»
»«
**
»
*
#
z »
OC
*a »t- »Ul "ac »
»
I v
<org
O
O
-ILU>LU
ZZ z z Z Z Z Z ZZ z zy.^ uo lo uolo i/i 1/1 v. ifl i/i
z z z zzzzzzzzIS) IS) IS) IS)IS)IS)IS)IS)IS)IS)IS)
zIS)Ml
fife SpaceDivisionNorthAmericanRockwell
ii
120SD 70-576
■^ oo oc" oc
in ujLL" k-
Z II zCC LUa is) vV- UJ OCIU U UJoc — a.a o— x -o *»vr -<v v. "lv o iriQ LLp ac -a lv ii
CD
" Xl-a a vLU Z UJ
" o aco - acac -»■ -" v.
O
z o o* "v
O
—
O
in I—LU L-
r-
L- J. Z
O O
— 3 " UJc 3 a O II v. — z ocQ — 4 IS) UJZ -» — X — ac a
<O—
3 Oi OZ*Org O
J-~aC-
-4 lv —" "—
-»
m ac n.
* Ac " X *c ac » " m — lv in "UZ *— |_) ►- -H I— " M I-I >_v a (J*< m j - n . ll " acq_j " p + " i ii " n a il 4_j o a -a o - x3 — PL-*—* r* OCIIUJ3COX
t-»tz —' oc o x*zacujjjaz cc n ic x ~ i— " 4 — — o *s_ai-o ooBZIuXU —q -h _j o ac 4 o3oXac "a
■
—Ol * COD ZP UJ 3vo 3 3 «op a: a o
z<
o l- a: _c — o_ a >- o z v. o
CIZ
"1034 <>— Ol- O O " O rg— I ioh«jilJIU I X Z - " " OC t-ee -C ii ii v. a: _ o _ an
<n
— —" u_ —>ujz __* ii — -ror zua zx z>— ►— l- *— to ~ t—So
«OO
—OC
POC
I_«<<<<JH<i-Lu — zukqh «-< — aca — acaxxxxa«x2 1_ oOplu zx 3 ►- oi-ococacaczinacaujzllpp»c*3 c3i_.oluu.op a cocou-oz— —
O
a ►- ►- 3 O O — LOOC— LO 0C LL LL li. LL - LL UJ
i-3 -8*r
O
-H rg rg in*h —i —« —. o r*i
O Ort
O ff O ff ffff ff
o-J-II
*-zoLUzp
oII
*"a.az4X
IIUJX<z
*oLUILLLUJ
LLUJocX3Z
'
a
L-
oUJaza.<JX3za<3P_CoLUa3Z
uO
P3ZoaoCOUJ
LUOOC
3LO
*I—oUJLLLLUJ
rgffO-H
II
UJ*g
lO
X4CCo3aca
*m
ilIS)k—zLUXUJ
<LO
LUoacaauO
ap
4ac .LUzujU
uO
O
L-iS)3Zo4a
oz
"Z3>—4P
axaoLL
Mff-Mn^JNOlffdrMfl^^JXffOHNr, .»" ,nrgrgrgrAmi»>i»\r»i(->-orr^j-.1-J--»--»->>'-J,^mmmJ> in m000000003000000000000000 OOooooooooocaooooooooooooo oo
zrzzzzzzzzzzzzzzzzzzzzzz z zis>is: IS)I/>IS))S)IS)IS)IS)IS)ISIIS)IS) IS) IS)
zLOz3t-a.3
"
Z
LOz3
*—a
C
*
LOo)—IS)
)—4H-LO
"
*IS)o
LO
►-4►-
"
O
azLU
»*»
fife Space DivisionNorth American Rockwell
- 121 -SD 70-576
ff-H
"op-
ui»--4
Q
Xz4OC
a:3LL
o-o
>.
LO3
ff"O
L-a.UJIS)
CO
rt
a
t-
oLUaza4
5 * *" » "r. » #2 * «a " «-" * »» *O f- <J * .lv a ac * »o 4 » Ja » « x " Zz z g. o lo " Z"*aa * " #
l- a o " JLOL-»*LU-_ * -— i/ima.aoig * Ippauji--zac-H " X "a o a 4 * « m.z.-ac.r. * ct "
-aCJ'O-za^J
" o "
ozau.^*a>H
» o -~ * a a " * "a o »">-". " — " a. -ouj-ox.looo o " Z-ooa-_m_jiu-<aeuj " _o «.uj ac a o— z o " — I»»»z»jc_ia__c " _o Z_j_org-3c3330 # v Joaca.«-#zu-3i/>LL " _. Zac a aa * a » < J
T0C«aCO»O».
" » Z3_C-h4»luli/1-»_j " < Z
/.Luaor^a—
olu # Zaoc.-Ho » i »o>>>>z-.wae * £ "* vi _n-- jie o * Z
lIOZX^J'>J»rHOU_ » -x "h- — QUIO. »o "ill > " JZOOoCacpacoo " mj «
OX'P4X»44-H
. *LO_J "
IuOOCO*--hl-O>-»-> »XcO
"zk-a3p3 * * < — m »
"-»
» ►» *
O
Z o # ac i- »P -»»»-"» "VJ34 4P " 4 4 *" .nr*jjj # a ao "—<—
OOIOZOIO
— o_o * x ►- «
O
o<lil_CJJ_ " >tk_ * » 33 lo *ii zpax=coci-»~» c »oc ou x "I— 4CO~k-4l— I— Or*-— »v.
#Z
4 *all— O rH w — —
*03r-
a "3 .._l^_-H.»==_
< »" »*rH<XLOw—
»*LOa
a «Z *Ol*4X_JO » Z HOC I- »— *s.
O4LOOOC44r*-»*
— M LULU* *4<sio l-jccu^klwohh " a o oc *__ a cc o ac;xaa43a-H»ii uju_44Z »looi z — ►-*- — a. ►- — — * o xzujuiaa »xi — »» — a x * z >-.xxa »
ll Zl < «* :a ju * Jj i/. JJ>uj3X v *dq n *
i/iaf
* » »X JJ
OLO4»OLUZ--
IN . "4Z"v~- Z
IXl*o*olooo.
»Z "— OC LU_)X»ff4affffO ♦I- LU p 4 « --IX L 0 ff "lazzo 4o»»mn v-z»03 a lv » — »»— * a — c z >- tx »
LOaCX X r- — 'J p " O o 4 4 — Z 3 a. *ZfflX X Z a 34»4LULULLPCt-«>-3*333 3 — Q) ji4i_3ac_.Hi.i_L "— i/> o o lv ac » ac
■—
aacu.*i- » l- »a u3lJl.l_ll.l_lJ » >< #
O
" LU *" z "ac " m
rt
"lv " «Poo o oa.x3O
rgr*. g. in -O r- ao
ffO
h^ ol
OO
O
O O OO O-h
—th —. —- —« -h
OO
O O O OO OO 000000
OO
O O O
OO OO
000000
***»*«*
L- "o "LU *-> »UJ "ac »
"a "p ». 4 "> »z "— "*ac "O »LL "-" *o »z "X Z *3 ac "►- a #
UJ >~ »OC ' LU "X »"" »— »
" ac . »- Z I- "O " " o "* OC -" -H LU »— 4 -» "" O I ♦ LU *— LU — OC ")- o a iv lo c "o — — o ac ac 0»u_ 3 C_. — 3 4 — »Tiaooc J _!♦-U O 3 I OC 4 »ac lo o lv " _» »
II * »" ii ii o ac "I- ff o "
>UJOLULO
zll»acozoaci-ac "i- —
t— C
Z 3 -h »ZCJ-OOC-L *■jJla.lacociuz»v — OLuaacac»
3 »>- »UJ *ac ""i~> v v
t ff O N I*l
■.
IT—i -4 rg og rg rg rg00000000000000
PUJ>JJp
ZZZZZZZZZ zz z r z zIS) IS) IS) l/l LO l/l l/l IS) IS) IS) IS) ISI 'A IS) IS)
zzzz z z zIS) ISI IS) IS) IS) IS) IS)
fife Space DivisionNorth American Rockwell
122 -SD 70-576
*
*»
»
*»
*— *>- »— — »."za "
_
-t
X
O
»3 "* * >- OC "UJ LU «
_v is) ec t- *
O
oc v. »—3P 4 »
O rg
ec 4 «- P *X
■
ac X iv »o uj ac a »
O
3 " 3 "ii y- n Z isi X *3—
O
*UJ
O
I/* OC > *O OC 4 CO #—3"O *
O
~ QC ►- ►"» »i — oc a m a *O OC Z LU LU LU ♦z ac v o o #—oaoz o ♦
* — cC L- -» 4 — 4 #« 4 LU I— **poocp a a Q »—. UJ -~ LU "- IP — »i-oo a— ao -j v -i * ia. cc — acin cc ac
404*
LUQO 4 43 ><>#
OJL
00 O " OU. Z » i
03-Hiik- ocac — »< llfl »
»
JL - " JJIU »i— x I " z >- m*- L-k-ac*3 . — — 3 — -<03rgu.u.a»zi-acocxaa crz 4 4 a. »
J-tOZZ-OOZ " Z *a. zii— ac
aC>-
ac O — M "i -__OZPP3Z — 3 »Z — —Z— PP>-—
b-ZZZ*
LUu.u.a:4LL44ujo:LLUjacacac*_ — ZX — ooaca —
3C333»_->-►-»
UJ LU LU *ac ac ac "rg «*U i^3 ijU U
*
***»
**»
»
-H ~ "ac »♦ z »
a *o oc oc »OC Z 4 #4ao »v oc — *X4* »zoo »
UJ »ii a »
">" OC II z »oc z cc *t- a a a *ZXXLIUJ 4 4 LU »o o ac "»»
*i o o
*«»
*»
**
*»— "— »
CC
*LU "X *I- »3 ♦— "p #4 "r- "a »i- "— »LO "__
"4 »— "I *rg — »— ""g II l/l »— "II LO X »— I- *a i — »
Ul I- p »ac 4 *— lo >- *~"— a *I- I -. — L- »LU L- Q O + *z ii cc lo a *
»
4 " -> OC »— ac * o I-X4 »ZjJO— Pl-O » I— I4 at " — ♦4 r- -I
O
I IS) "
O
3 a_ _j a ie »3 rH oc ro 4 »
CC O 4 — »UJ II o II *3- (J II II Z ♦
OC UJ LO — oc oc *r- I— LO"-
LJ*-3*
Z Z -> I v. I lv ►- »UJ— Ih-r-
ZJJ*
L-
O
OC *"**ioo o o
aczCDaLO
*oUla»
— p
* 3O
O
vi a.a a*
_i
LOo —lv 4 IIa * ox o
-t
—Zui «-a r* ~— ii aJo e ooc zkiiiii- * azo lolv uj aa o
i
s ff rg ro i/> HDF«"3_ffrHrgrgrgroiO'O'OO^iOfO^000000000000000000000000
NIO ». iT.»■*»■-»"-»"00000000
*0 I"- Iff HfJI.■j- -j" in in inO O 00000
O
O 00000
Ifl O !*■m in in in00000000
zzzzzzzzzzzzisiiS)iS)iS)iAiS)iS)iS)i/)iS)isiiS)zz z zIS) IS) IS) IS)
zzzz z z zIS) IS) IS) IS) IS) IS) IS)
zz z zISI IS) IS) IS)
fife Space DivisionNorthAmerican Rockwell
123 -SD 70-576
«««**
**»**»»*a "— »a »a_ "3 "LO """ »
*
rt
»o »IP II *Q' ** *" O "ac vi "Z
O
»* — *O *-v *a — *— "-i *♦ "— LO ** »
* o *v uj "LU
O
»Q Z »—. — *-oc ♦~z " »— a. a *p a -< vi *3uo * "
t
»
C
*
r<
in " o »a
»
o * m o aao o *
O
LU O » O II ♦ii -" aiu -> ii non ii ii ii "ff — a ii — ♦
__. ii*— — >—
ocii-HiNLOLuaz*
— i- * —Z I X nc o vi ac »— zo —
-raoXL^Loa
—
oa*
* — lv z a. — acacacaac>-»oa:3u.c44acllnclCuj»uj__----oxozk-h-ujou.ac»a
■
»»"J- »
m *o o
»
*♦ z" -r "» 4 OC
"
O
UJ
" ro — L-» — 4« >» Z Ul» <*> — a:* — zoo" - 4 o:* Z X 4 ac» 4 >» i- x a— » X <*)
rg * r- —. i/i lv p-H » LO
O
4
* » lo . lv z a
-t
* LO LO P 4
O
II " LU 4 Z UJ_C ♦ P lo ~«" » LU — - X LO— * LO o oc o -lo » lv ac a aa* » ■V34UIP a
O*
rr»_jPL3LULUuj* — < a o I l.a » > lo vi l. aZ * - 4 O L- Ul» » ac»-
"OZZo
—* » LUO IU * 4 4 -h oll* rO al XI IS) Z 4
">* X-> UJ >" — *-4 »» 3uji-oaxu.>
* — » ZQCLOZ43—QO
O*
34P0X4Ul n » I- " X L- 0C I l/l Ul
O
» » ouj
O.PZ
ac— -<»
LUOLULU434P
ll* -> Z U U Z 4
" — * 3- 4 Z O i/» Zlo - * 3r <<Quj- in* — » Loai-ax ao * * _uat-4L-i-acUJ * *
OUJZOP4
O "J*
OOLUU.3O
Z
»
* 4 O 3 li-
O
lo«-»
-V4CJOUJU.
" — *
k-muiaac^a
Z P* l_ wLI/11. CC I*l
* 3* aC-o_aP4p — aca 3 * — -" 1-3 JUJ Ul—J L- _J a *
__
iLflt -»
Jl I —.
_■
coLL LOP 3 0 » 4 JJ U. 3 4 _* l/> XLL POO LO » o J LL 13 Z 433 3a — * ~-h Z li. CX Z L-L- zX -a _i— » -. a. . ~3lu — via — I
LO g_ ". 3 __. * in 4 -H
,^c
I— e_ X Ul OUIO
»
»
C
* » —
O
- -*UJJU._IhUIL3Iffi^oaff »co—* — -a — 40 o — x i—
-h
Off
O »— — i— — — l— LU — U. 4
>llOff OffZ*L-l— 41-LOZDCLO—
IS) a.Ac * 4 L-—
OC*44X4Z3LUZ
4 —l-Oi*jOZC33»XXoCXa 13zm _ < - _D4i— „ a ac QiCu l-o
>■
luo iLi_3_j'jjii3.. _.Oo o - »a_— _caLoaca:*u.u. u."»r*-s.»*v "* -^a * cacara.'aci.racarv's4 » rr,LU » O rH rg
O
oc * o o o ff
* ff ff ffo o
a_a*O-_ro*j'in O N X ff
O
-him
0000000000000000000000000000
rO
>$"
Jl .0 r~ 00 ff O H rg Xlr*r*h-h-p- r-p»
coco^co
00000 oo 000000000
OO
0000zzzzzzzzzzzzzzIS)iS)*S)iSIiS)tS)IStiSIISIiSIIS)IS)IS)IS)
ZZZZZZZ z z z zIS) I/) IS) IS) IS) IS) IS) l/l l/l l/l l/l
fife Space DivisionNorthAmerican Rockwell
rt
- 12U -SD 70-576
i__.
o -_) LOa <o —X "
OC
o »-vi a —X Xi- a— LO LUUJ 4 h-
»
0. L-«n LL
lO
Ul —■m a -« v— o aac uj 4 uj— a a i-LO P l/l
O
4*4 UJ
OC
> "" 1x a uj "».O -I LO "
OC
* —
"so*-
» — *>X U "- * </> « ""— noLO r~ U. - "♦ —< z — a — gc "4 UJ - — Z *lv z » >- » m *
O
a_ -" r- o
»>■
—
uO
x z uj — a lv l-►- o4 -» " -h a I
i/ioziu>
" " a_a — ac a — a » axlux lv — " x i/>
X
O
-J H- *» — "1z *- a uj a. —"v ""— a vi x " *
4U.L/100
—
i/i
—00 — xo " * — -«.4—mo — n —I_Q_ " 1-UJK
O
:« vi »_» — — o — oc —►- 4 X in uj *o
O
» Z
Z — t-0»»P — 3"UJ4LU- x in x 4 — lo1
ZOO
— 00 — o —L- L- " "
rg
— " — -O.U.— — — —
"»
— <-
UixKl-L-h-k-^l-SC
O 44444440O XXXXXXXUJ4 acacccacocecacoo
" o » ooooaoooz"* - SU.ILU.LLU.ILU.> LU
c« ra c« -j
#
00 oo in g> r- o ff -<ff o oo ff o ff
ff ff ff ff
O
—0 co co oo oo 0 ff ff0000000 o0000000
O
Z Z Z ZZ ZZ zi/i i/i i/i i/i m it i/i is)
o■»II
zvLUzp
*-HoNI-aoz4X
IIUJX4z
oUlLLILUJ
zLOzoL-o.o"
LLlvocXoz»a»
3UJoz
«>
CL
4zQza<o_J
Pi__oUl
O
□zLO
-1azv_M
ao39UJ
UJoacol/l
*I-oUJILLLUJ
zI/)zoL-a.o"
CM
IIUJr>g
IS)
X4ocoaaca»
o
Oi
NtowzUJZUJL--4I—IS)
IPvecoLO
*v.vi/l
H--4t-l/l
*
oUJt—4acLUzUJliIS)
o►-1/1ozo4
o
oz
v¥-IS)
r--4►-IS)
*
*
IL--4p
azov
*ozUJ
**
fife Space DivisionNorthAmerican Rockwell
EXAMPLE SIMULATIONS
SD 70-576
Space DivisionNorth American Rockwell
—ISJ .-4
I
00 -*
-* o
I
M OI
■c
rv -» Of\lIH
m O f^
- 126 -SD 70-576
o
<zv©x
oczvUJO
i-4-4
H
QC
zCD
«r o.
ooIo cc
UJ Z1
O */»
CO
I
fl___■ao
Cm
UJecva
o
l/tocaat3tLU
U.aacLUtoXZ
NIrtUJvoIvv.oacUJCDX
___>
z
HOUJ
acav.atUJCOXZ
ooo—nvUlaca.oaV-ZUJOatUJo.
oo
MatvatatUJ►-XUJvatUJo.
r»jm
itaZoa
iA<03
<X
t-
in
iA iAUJ <-J
UJ<S\ oLU CC© <-J -J
<.
> l*<
U UJoi a.UI
t-
ac=1x
UJO uj
z v< z»- <
Q.
*-uj Cllj uj
< U4
on en "—at i<n— a-u. Ac —LU LLc u-z v. at-UiJa
►-
OJ__
lLO — <
fc-
*- liJi/i z at_r __; ivo Iv
►-
Z -— oc< UJ© fc-— <— UJZ ac— L 5z o
< ac ofx < v
t-
X.J
L/. UJ *tuj Z O-J < UJ
ZWl — l/l— at x —O Oo -j o auj O uj
*" U UJ H» Oa. la.LU
t-
P- UJv z v«""U< Z U <
>a
< Z — <UJ - Xo > I ■- > <JZ O □ U.
Q
3«J < O — < XhIU I lva at Z Ln ac
***>
ijj -J O -J <u<z < <v UJ< t/. UJ Z L/l (X3oI — o
h- 3t
t-
at e--7 4
►-
< «nlv O U. -J O -33a 3 xO v. l/l U.uj O O uj C Zvn a a —co ix < ac <___! lv O -J UJ OiA to —I tO
X < — X UJO3* 3 XZ Zc- Z -— a. h-a uj uj — uj z-v x v x uj
o .-" v v, »- x— < — *"i/i v. v.Z — —Ov
QC
<0O-JO t/>0 <ac xUJ o1 3*- XUJ iA
<iL.O "*acO vi
* -JQC <O >-Ja i/»v
t-
z v.< —zX
t-
o zO<z
LU —X X*- o
Ov.vLn
oa ac4LUv x« —UJ»- ZaUJu_v—v"«
m
ItaczUJo
HatzCO
iA
IN. O
CO O
O —I
* oI
"O —I
OJ —I
«ro
oj—!«- o
I
ao
* IL_) CCUJ
z>
o »/»
ovi
UJLUatiA
f-
OUJa.UJv<
UJaX
o—i"oi
fsj
t
rg
I
II
OO
mI
tlUJacOv
"H
ataatacUJv.oacUJtoXZ
(0
RiAUJ
OXvv.oaUJCDXz
o
oUJvacav.atUJCOXZ
IA
CO
ai—
UJecaoo
zUJoatUJa
rgp^
Mataatatat
zLUoatUJa.
m
Hatz__:aUJo
fife SpaceDivisionNorthAmerican Rockwell
k-
J
- 127 -SD 70-576
w o—UJ
O.OJvv
"* OI/.
o> — rgO —
c* —IOUJ
►-
Ul-IUJec
—I0* —— O
_jUJoax
OJ oI
_A O "r oi
tr o-4
m
acZ*vUJo
—HatzCO3iA
o —■C —I
rg O1
* oI
ttoo
tt IO COUJ 3O i"
erI
rg
o
<oI
II
aot-
rsj
nUJacov
ni/iatoatacUJ
v.aatUJCOX3z
HiAUJ
axvv.oatUJcoXZ
o
IIoUJoatav.aUJCDX3Z
mCDHt-UUJacacOv*-ZUJoatUJ
"
acvacacUJi-zUJvatUJ«.
en
Naztto
UJo
o
Haczttv
UJv
Hatzmi/l
o
<*
i
rg o
r>
—<c oi
ttoo
tt tv av>uj 3o */i
I
0»
oI
r-
iittca
"org
tlataovI/)
o
Hl/latOacacLU
LLoacUJtox3Z
IIiAUJOOXvLLaatox3Z
O
H□UJwacov.a
COX3z
oooIIh-uatataij
t-zvaUJa.
oo
Nataacocuj
t—
zLUvaUJa.
o**>R
aczttv
UJa
»-
*- ot
oUJ►-a.UJ(J
■# —(/»
o ——I "—I
I
oUJ
oUJ
UJat
o oI
rg O
"0 — _jUJOOX
rg OI
IAfl
N
atz
__t
*->UJa
iatzCO3
"o —I— oI
* o
m oI
t* ot
ttoo
t-
* IU CDUJ 3O LA
oI
g>i
I
Mtt0O
mt
ttU-JaoviA
—Hi/latoecocUJ
v,Oa.UJCOX3X
0>
ttl/>UJvraxwv.vatviCOX3z
o
IIoUJ
aoIXaUJODX3Z
<r
"COVHI-UUJececovt-zUJoatUJa.
HatoatacUJ
zUi(JatUJa.
if*en
NatZttv
UJo
"atzttvUJUl
Hacz_c3l/l
lf\ OI
*»
—rg O
» —
tt
ao
* IU '0uj 3O Lrt
CO
f-
-0»
H
ttvol-
cr
MUJatt_)ol/»
o
HinatoatatLU
v.aaUJenX3z
co
Hl/lLUvUX
<_>
ILCJaLUCDX3Z
rg
HvUJoatov.OLUJCDX3Z
O
"ooIII-vLUa_acOv
zUJvocUJ
O,
O
"O
HatoatacUJ
zLU
acUJ
0*i*i
itatzttoUlO
4|m Space DivisionNorthAmerican Rockwell
rst -*" i
<M O
r-a
t
rg O -*o
rg ot
rsi -"
f*. oI *^ O
I
-o OI I
loi
as imoOD
N N I.O -i O* "o o »
128 -SD 70-576
zttvUJa
—HOfZCD34/1
h- —rg ——
ttOOI-
* IO
cQ
UJ 3a**
*0
r»
rg
H
OO
iUJacOviA
fliACCaacacUJLLOatUJcoX3Z
IIiAUJUaxILaatLUtoX3Z
HOUJvacOv.atUJCOX3Z
oo—Ah«OUJotocOoI-zUJ
atUJ0>
Oo
IIataatatvii-zUJvatUJa
aczttvUJO
zttvUJo
Natzcc3t/l
(*. o
CD —I
ttooI-
* Io COUJ 3o */_
rg
I
I
COI
IIttOo1-
00
I
HUJaovl/»
HiAacaacac
a.oaUJCOX3Z
It</ludooX*_>
v.aaLUCOX3z
f\J
IIOUJvocov.atUJCOX3z
"Oo—R»->oUJntatoot-zUJvatUJa.
oo
Hacoececvit-zUJ
atUJa.
CO
"**eczntvUJo
X ►"
ir. —Ovii—CLUJ!__>a< 00 —
aUJ
t—
OlUJUU<i/>
iI
I*l — rg
rg o—I
OUJI-vUJUJat
—oI
_ooI
3UJI-oUJ-»UJec
m o .jUJooX
rg — -jUJOox
o_ o rg —irI
<0
RacZttvUJo
——ItatXCD3
* o
rg O
O —I
f*» —I
ttooI-
Iv COUJ 3O iA
i
oi
i»*t
H
oa
friI
UJatoviA
*
1l/lSoatatUJ
OatUJ■OX3Z
CO
Hi/lUJvoxvv.aatUJCOX3Z
rg
ROUJUaoLL
aUJmx3Z
ITr»COti
►-
vUJatatovI*zUJvacUJa
unrg
RacoecetUJ.-ZUJU___UJa.
«*igr
RatzttvUJo
Raczttv
UJo
——flatzCO3i/l
m oI
-4 —-4— o
<r
o
ttoa
* I
L,
J COUJ 3O i/>
m
rg
rg
HttOa
Pa,
"Om
RUJavvl/l
RWt0LoacatUJ
ILoetUJCDX3Z
CO
IIiAUJvoX
IXaatUJCDX3z
rg
ROUJ
atov.aUJCDX3Z
"ccitI-vUJatacov
z111vacUJ
*—■acaatacUJi-zUJvacUJa.
*r
Hat
oUJo
fife Space DivisionNorth American Rockwell
aUJ
\
129 -SD 70-576
rg O
UJaa<iA
rg ——I v*> — f*.I— —
—t
I
IN OI
O O—rg O(4
OUJoUJ
UJctt
CO — a o UJaax
cr —l
o* — I"oin
"r o
CO —I
H
atziCoUJa
—NOtZaa3iA
I— oI
m o
'
oott Io COUJ 3O V>
—ICO
erl
c*
ttttooI-
rqmI
ttUJaoi/i
o
iii/iacOeta.
u<
v.OecLUCCX3Z
0"
Ml/lonxvLLOecUPCCX3Z
o
HCJUJacOLL
aUJ03X3Z
oooHI-vUJacecoL->H-Z
UJvaUla
oo
flocvccacUJt"*zUJvatUJ&
rg
NatziCi_>UJO
RaczttvUJa
Meczo3L/l
(*l —I— OI
I
ttaatt iO CDuj 3a i/i
CDI
flI
H
Cofr-
rgCSI
IIUJac
Q
Ul/l
t
—H
iAOLacca.
ILUatUJCOX3Z
*CO
Hi/lUJvaxvLLOaUJccX3Z
rg
NO
atoLL
aLUCOX3z
"r-coH
OLUataa
i-zUJUatUJa.
inrg—MatOacecLU
H-ZUJvacUJa.
<oIT
Haczi£UUJo
L- i-
0* oI
oUJi—exUJvLj
oUla.ujvv< o —t
l/l iA
o —— "* **.rg —I
I*l —o o—I
oUJt-vUJUJac
oUJt-vUJ-1UJec
IP "*I
rg O—iI
* — _jUJoo
-jUJaax
rg —— O* O
eri
if. — O■n
IT —oIMI
HOCzttvUJa
HeczfO3i/l
m —« o
ifi oI
ttooI-
.c Iv xUJ 3O fc"
"r
rg
ir
nttooI-
f«
UJavvI/1
rg
Ni/latOatacUJ
LLCaLUtoa3z
c
NiAUJUOIvv.vaUJa_i
3z
O
MOUJavv.XM.1-.X3Z
CDl«-P*
H
OUJIt.atov
zUJvatUJa.
rg
"rgrg
ROCoatatUJ
zUJLJ«UJ&
amiti
HiXzTataUJ
IIatZtt
" vujo
ReczCD3i/l
— oI
<r
o
r- —ttoo
* IC. cPuJ 3O iA
ITI—
r-
ttL.a
cc
itUJavoi/i
O
Rl/»aoecOlUJv.oa.UJer»3Z
f-
Ri/iUJvoXvv.oOLUJccX3z
c.
ROUJvacov.atUJtox3Z
o"ooR
t-
oUJatacovt-zLUoacUJa.
oo
ROCoatatUJi-z
LUvaUJa.
if*
Hatz
UJa
fife Space DivisionNorthAmerican Rockwell
»»
130 -SD 70-576
!
CO —I
UJ
a.UJvol/l
rgo
I-toI
h-oi
aUJvUJnUJ
OC
in o _JUJaax
-4 —— ODI
oo
ReczttvUJa
roczOD3
— —I
**lo
o —a* —
oant Il_) _oUJ 3
Q
i/i
am
amamI
O—C*
MntOa
om
RUJacOvl/l
"HWlaoocatUJ
v.OatUJoax3Z
er
Mi/lUJooXvILoatUJCDX3Z
oUJvaoILatUJCOX3Z
CTCOCO
I—vUJacacovI-zUJvacUJOl
■4—HOCo
OCatUJ
zUJvacUJ01.
Vac<XX3_/l
o10
ftazttvUJO
o
HutaoacacUJiLOacUJ03X3Z
gfm
liiAUJvoXwILoacUJCDX3Z
n
«aviUacov.AcUJCOX3z
mrgoRI-UUJacaco
zUJvacUJa
in
NacoatatUJ
zUJvaUJa.
N»
I*l o
oUJI-___.UJvviA
o — *—oI
fM O
oUJvUJUJac
rg — -jUJooX
♦— c*
0» rgoI rg
RofZttvUJo
—H
atz_D3l/l
rg OI
«___
amam
CO —a<oo
tt IU COUJ 3
O
wt
CD
Bntcja
gr
RUJac(jvVI
HwtatoececUJiLDacUJcc___3Z
er
Ri/»UJv
Q
Xvv.vatUJCDX3Z
O
oUJva.oLLaUJCDX3Z
er"CO
COflt»vUJacocowI*zUJvatUJOL
■acoacatUJt~zUlvaUJ&
crtn
iiazttvUJo
fife Space DivisionNorthAmerican Rockwell
REFERENCES
t1131 -
SD 70-576
fife Space DivisionNorth American Rockwell
REFERENCES
Balzer, R. M. A mathematical model for performing a complextask in a card game. Behavioral Science, 1966, 11,219-226.
Berlekamp, E. R. Program for double-dummy bridge problems—A new strategy for mechanical game learning. Jour-nal of the Association for Computing Machinery,1963, 10., 357-364.
Clarkson, G. P. E. A model of the trust investment process.In E. A. Feigenbaum and J. Feldman (Eds.), Computers
and Thought. New York: McGraw-Hill, 1963. Pp.329-346.
De Groot, A. D. Thought and Choice in Chess . New York:Basic Books, 1965.
Ernst, G. W. and Newell, A. Some issues of representationin a general problem solver. Proceedings of theSpring Joint Computer Conference, 1967, 583-600.
Feigenbaum, E. A. The simulation of verbal learning behav-ior. In E. A. Feigenbaum and J. Feldman (Eds.),Computers and Thought. New York: McGraw-Hill, 1963Pp. 297-309.
Feigenbaum, E. A. and Feldman, J. Simulation of cognitiveprocesses. In E. A. Feigenbaum and J. Feldman,Computers and Thought. New York: McGraw-Hill,1963. Pp. 269-276.
Feldman, J. Simulation of behavior in the binary choiceexperiment. In E. A. Feigenbaum and J. Feldman(Eds.), Computers and Thought. New York: McGraw-Hill, 1963. Pp. 329-346.
132 -SD 70-576
- 133 -1SD 70-576
A
fife Space DivisionNorth American Rockwell
Findler, N. V. Human decision making under uncertainty andrisk. Kybernetik. 1966, 3., 82-93.
Frijda, N. H. Problems of computer simulation. Behavioral
Greenblatt, R. D. The Greenblatt chess program. Proceed-ings of the Fall Joint Computer Conference, 1967,801-810.
Gregg, L. W. and Simon, H. A. Process models and stochastictheories of simple concept formation. Journal ofMathematical Psychology. 1967, 4, 246-276.
Hunt, E. B. Computer simulation: Artificial intelligencestudies and their relevance to psychology. In P. R.Farnsworth (Ed.), Annual Review of Psychology. PaloAlto, Calif.: Annual Reviews, 1968. Pp. 135-168.
Johnson, E. S. An information processing model of one kindof problem solving. Psychological Monographs, 1964,78 (4, Whole No. 581) .
Laughery, K. R. and Gregg, L. W. Simulation of humanproblem-solving behavior. Psychometrika , 1962, 27 ,265-282.
Newell, A. On the analysis of human problem solving proto-cols. Presented at International Symposium onMathematical and Computational Methods in the SocialSciences, 1966. AD658863.
Newell, A., Shaw, J. c, and Simon, H. A. Chess-playingprograms and the problem of complexity. IBM Journalof Research and Development, 1958, _2, 329-335.
Newell, A., and Simon, H. A. GPS, a program that simulateshuman thought. In E. A. Feigenbaum and J. Feldman(Eds . ) , Computers and Thought. New York : McGraw-Hill, 1963a. Pp. 279-293.
Newell, A. and Simon, H. A. Computers in psychology. InR. D. Luce, R. R. Bush, and E. Galanter (Eds.).,Handbook of Mathematical Psychology. New York :Wiley, 1963b. Pp. 361-428.
Science, 1967, 12_, 59-67.
fife Space DivisionNorthAmerican Rockwell
Samuel, A. L. Some studies in machine learning using thegame of checkers . IBM Journal of Research and De-velopment, 1959, 3, 210-229.
Simon, H. A. and Kotovsky, K. Human acquisition of con-cepts for sequential patterns. Psychological Re-view, 196 3, 70_, 5 34-546.
Simon, H. A. and Newell, A. Information processing computerand man. In W. R. Brode (Ed.), Science in Progress .New Haven: Yale University Press, 1966. Pp. 333--362.
Slagle, J. Unpublished lecture notes for UCLA Extensioncourse, The heuristic programming approach to arti-ficial intelligence, 1967.
Solomonoff, R. J. Some recent work in artificial intelli-gence. Proceedings of the lEEE, 1966, 54_, 1687--1697.
Thiele, T. N., Lemke, R. R., and Fu, K. S. A digital com-puter card-playing program. Behavioral Science,1963, 8, 362-368.
Turing, A. M. Computing machinery and intelligence. InE. A. Feigenbaum and J. Feldman (Eds.), Computersand Thought. New York: McGraw-Hill, 1963. Pp. 11--35.
Weizenbaum, J. Eliza, a computer program for the study ofnatural language communication between man andmachine. Communications of the Association forComputing Machinery, 1966, 9_, 36-45.
Williams, T. G. Some studies in game playing with a digitalcomputer. Unpublished doctoral dissertation,AD634 821, Carnegie Institute of Technology, 1965.
13U -SD 70-576