-
Introduction
On October 5, 1960, the American BallisticMissile Early-Warning
System station at Thule,Greenland, indicated a large contingent of
Sovietmissiles headed towards the United States .l For-tunately,
common sense prevailed at the informalthreat-assessment conference
that was immediatelyconvened : international tensions weren't
particu-larly high at the time . The system had only recent-ly been
installed . Kruschev was in New York, and allin ;all a massive
Soviet attack seemed very unlikely .As a result no devastating
counter-attack waslaunched . What was the problem? The moon
hadrisen, and was reflecting radar signals back toearth . Needless
to say, this lunar reflectionhadn't been predicted by the system's
designers .
Over the last ten years, the Defense Depart-ment has spent many
millions of dollars on a newcomputer technology called "program
verification" -a branch of computer science whose business, in
itsown terms, is to "prove programs correct" . Pro-gram
verification has been studied in theoreticalcomputer science
departments since a few seminalpapers in the 1960s , but it has
only recentlystarted to gain in public visibility, and to beapplied
to real world problems . General Electric,to consider just one
example, has initiatedverification projects in their own
laboratories :they would like to prove that the programs used
intheir latest computer-controlled washing machineswon't have any
"bugs" (even one serious one candestroy their profit margin) . 3
Although it usedto be that only the simplest programs could
be"proven correct" - programs to put simple listsinto order, to
compute simple arithmetic functions -slow but steady progress has
been made in extendingthe range of verification techniques . Recent
papershave reported correctness proofs for somewhat morecomplex
programs, including small operatingsystems, compilers, and other
material of modernsystemsystems, compilers,
aPrepared for the Symposium on UnintentionalNuclear War, Fifth
Congress of the InternationalPhysicians for the Prevention of
Nuclear War,Budapest, Hungary, June 28 - July 1, 1985 .
Copyright (c) 1985 Brian Cantwell Smith
THE LIMITS OF CORRECTNESS'
Brian Cantwell Smith*
18
What, we do well to ask, does this newtechnology mean? How good
are we at it? Forexample, if the 1960 warning system had beenproven
correct (which it was not), could we haveavoided the problem with
the moon? If it werepossible to prove that the programs being
writtento control automatic launch-on-warning systemswere correct,
would that mean there could not bea catastrophic accident? In
systems now beingproposed computers will make launching decisionsin
a matter of seconds, with no time for any humanintervention (let
alone for musings aboutKruschev's being in New York) . Do the
techniquesof program verification hold enough promise sothat, if
these new systems could all be provencorrect, we could all sleep
more easily at night?These are the questions I want to look at
today .Arid my answer, to give away the punch-line , is no .For
fundamental reasons - reasons that anyone canunderstand - there are
inherent limitations towhat can be proven about computers and
computerprograms . Although program verification is animportant new
technology, useful, like so manyother things, in its particular
time and place,it should definitely not be called verification.Just
because a program is "proven correct", inother words, you cannot be
sure that it will dowhat you intend .
First some background .
2 . General Issues in Program Verification
Computation is by now the most importantenabling technology of
nuclear weapons systems :it underlies virtually every aspect of the
defensesystem, from the early warning systems, battlemanagement and
simulation systems, and systemsfor communication and control, to
the intricateguidance systems that direct the missiles to
theirtargets . It is difficult, in assessing the chancesof an
accidental nuclear war, to imagine a moreimportant question to ask
than whether these per-
*Xerox Corporation : Intelligent Systems Laboratory,Palo Alto
Research Center, 3333 Coyote Hill Road,Palo Alto, California 94304
USA ; Stanford Univer-sity : Department of Philosophy, and Center
forthe Study of Language and Information, VenturaHall, Stanford,
California 94305 USA ; and Compu-ter Professionals for Social
Responsibility :P .O . Box 717, Palo Alto, California 94301 USA
.
-
vasive computer systems will or do work correctly .
Because the subject is so large, however, Iwant to focus on just
one aspect of computersrelevant to their correctness : the use of
modelsin the construction, use, and analysis of computersystems . I
have chosen to look at modellingbecause I think it exerts the most
profound and,in the end, most important influence on the systemswe
build . But it is only one of an enormous numberof important
questions . First, therefore,- in orderto unsettle you a little -
let me just hint at someof the equally important issues I will not
address :
1 . Complexity: At the current state of the art,only very simple
programs can be proven correct .Although it is terribly misleading
to assumethat either the complexity or power of a com-puter program
is a linear function of length,some rough numbers are illustrative
. Thesimplest possible arithmetic programs are mea-sured in tens of
lines ; the current state ofthe verification art extends only to
programsof up to several hundred . It is estimated thatthe systems
proposed in the Strategic DefenseInitiative (Star Wars), in
contrast, will re-quire at least 10,000,000 lines of code .5
Byanalogy, compare the difference between re-solving a two-person
dispute and settling thepolitical problems of the Middle East .
There'sno a priori reason to believe that strategiessuccessful at
one level will scale to the other .
2 . Harman interaction : Not much can be "proven",let alone
specified formally, about actualhuman behaviour . The sorts of
programs thathave so far been proven correct, therefore, donot
include much substantial human interaction .On the other hand, as
the moon-rise exampleindicates, it is often crucial to allow
enoughhuman intervention to enable people to over-ride system
mistakes . System designers,therefore, are faced with a very real
dilemma :should they rule out substantive human inter-vention, in
order to develop more confidencein how their systems will perform,
or shouldthey include it, so that costly errors can beavoided or at
least repaired? The Three-Mile Island incident is a trenchant
exampleof just how serious this trade-off can get :the system
desiqn provided for considerablehuman intervention, but then the
operatorsfailed to act "appropriately" . Which strategyleads to the
more important kind of correct-ness?
A standard way out of this dilemma is tospecify the behaviour of
the system relativeto the actions of its operators .
But this, aswe will see below, pressures the designers tospecify
the system totally in terms of inter-nal actions, not external
effects . So you endup proving only that the system will behave
inthe way that it will behave (i .e ., it will
raise this line level 3 volts), not do whatyou want it to do (i
.e ., launch a missile onlyif the attack is real) . Unfortunately,
thelatter is clearly what is important . Systemscomprising
computers and people must functionproperly as integrated systems ;
nothing isgained by showing that one cog in a misshapenwheel is a
very nice cog indeed .
Furthermore, large computer systems aredynamic, constantly
changing, embedded in com-plex social settings . Another famous
"mis-take" in the American defense system happenedwhen a human
operator mistakenly mounted atraining tape, containing a simulation
of afull-scale Soviet attack onto a computer that,just by chance,
was automatically pulled intoservice when the primary machine ran
into a pro-blem . For some tense moments the simulationdata were
taken to be the real thing .6 Whatdoes it mean to install a
"correct" module in-to a complex social flux?
3 .
Levels of ballure :
Complex computer systemsmust work at many different levels . It
fol-lows that they can fail at many differentlevels too . By
analogy, consider the manydifferent ways a hospital could fail .
First,the beams used to frame it might collapse . Orthey might
perform flawlessly, but the operatingroom door might be too small
to let in ahospital bed (in which case you would blame
thearchitects, not the lumber or steel company.)Or the operating
room might be fine, but thehospital might be located in the middle
of thewoods, where no one could get to it (in whichcase you would
blame the planners) . Or, to takea different example, consider how
a lettercould fail . It might be so torn or soiledthat it could not
be read . Or it might lookbeautiful, but be full of spelling
mistakes .Or it might have perfect grammar, but disastrouscontents
.
Computer systems are the same : they canbe "correct" at one
level - say, in terms ofhardware - but fail at another (i .e ., the
systemsbuilt on top of the hardware can do the wrongthing even if
the chips are fine) . Sometimes,when people talk about computers
failing, theyseem to think only the hardware needs to work .And
hardware does from time to time fail,causing the machine to come to
a halt, oryielding errant behaviour (as for example whena faulty
chip in another American early warningsystem sputtered random
digits' into a signal ofhow many Soviet missiles had been
sighted,again' causing a false alert7 ) .
And the connec-tions between the computers and the world
canbreak :' when the moon-rise problem was firstrecognized, an
attempt to override it failedbecause an iceberg had acgidentally
cut anundersea telephone cable .
But the more impor-tant point is that, in order to be reliable,
asystem has' to be correct at every relevant level :the hardware is
just the starting place (andby far the easiest, at that) .
Unfortunately,however, we don't even know what all the rele-vant
levels are . So-called "fault-tolerant"computers, for example, are
particularly goodat coping with hardware failures, but the
soft-
-
ware that runs on them is not thereby iproved .9
4 . Correctness and Intention : What does correctmean, anyway?
Suppose the people want peace,and the President thinks that means
havinga strong defense, and the Defense departmentthinks that means
having nuclear weaponssystems, and the weapons designers
requestcontrol systems to monitor radar signals, andthe computer
companies are asked to respondto six particular kinds of radar
pattern, andthe engineers are told to build signalamplifiers with
certain circuit character-istics, and the technician is told to
writea program to respond to the difference betweena two-volt and a
four-volt signal on a parti-cular incoming wire . If being correct
meansdoing what was intended, whose intent matters?The
technician's? Or what, with twenty yearsof historical detachment,
we would say shouldhave been intended?
With a little thought any of you could extend thislist yourself
. And none of these issues eventouch on the intricate technical
problems thatarise in actually building the mathematical modelsof
software and systems used in the so-called"correctness" proofs .
But, as I said, I want tofocus on what I take to be the most
importantissue underlying all of these concerns : thepervasive use
of models . Models are ubiquitousnot only in computer science but
also in humanthinking and language ; their very familiaritymakes
them hard to appreciate . So we'll startsimply, looking at
modelling on its own, and comeback to correctness in a moment .
3 .
The Permeating Use of Models
When you design and build a computer system,you first formulate
a model of the problem youwant it to solve, and then construct the
computerprogram in its terms . For example, if you wereto design a
medical system to administer drugtherapy, you would need to model a
variety ofthings : the patient, the drug, the absorptionrate, the
desired balance between therapy andtoxicity, and so on and so forth
. The absorp-tion rate might be modelled as a number propor-tional
to the patient's weight, or proportionalto body surface area, or as
some more complexfunction of weight, age, and sex .
Similarly, computers that control trafficlights are based on
some model of traffic -of how long it takes to drive across the
inter-section, of how much metal cars contain (thesignal change
mechanisms are triggered by metal-detectors buried under each
street) . Bicyclists,as it happens, often have problems with
automatictraffic lights, because bicycles don't exactlyfit the
model : they don't contain enough iron totrigger the
metal-detectors . I also once saw atractor get into trouble because
it couldn't moveas fast as the system "thought" it would :
thecross-light went green when the tractor was onlyhalf-way
through the intersection .
To build a model is to conceive of the worldin a certain
delimited way. To some extent you
must build models before building any artifact atall, including
televisions and toasters, but com-puters have a special dependence
on these models :you write an explicit description of the modeldown
inside the computer, in the form of a setof rules or what are
called representations -essentially linguistic formulae encoding,
inthe terms of the model, the facts and datathought to be relevant
to the system's behaviour .It is with respect to these
representations thatcomputer systems work .
In fact that's really whatcomputers are (and how they differ
from othermachines) : they run by manipulating representa-tions,
and representations are always formulatedin terms of models . This
can all be summarizedin a slogan : no computation without
represen-tation .
The models, on which the representations arebased, come in all
shapes and sizes . Balsa modelsof cars and airplanes, for example,
are used tostudy air friction and lift . Blueprints can beviewed as
models of buildings : musical scores asmodels of a symphony . But
models can also beabstract . Mathematical models, in particular,
areso widely used that it is hard to think of any-thing that they
haven't been used for : from wholesocial and economic systems, to
personality traitsin teen-agers, to genetic structures, to the
massand charge of sub-atomic particles . These models,furthermore,
permeate all discussion and communi-cation . Every expression of
language can beviewed as resting implicitly on some model of
theworld .
What is important, for our purposes, is thatevery model deals
with its subject matter at someparticular level of abstraction,
paying attentionto certain details, throwing away others,
groupingtogether similar aspects into common categories,and so
forth . So the drug model mentioned abovewould probably pay
attention to the patients'weight, but ignore their tastes in music
. Mathe-matical models of traffic typically ignore thetemperments
of taxi drivers . Sometimes what isignored is at too "low" a level
; sometimes too"high" : it depends on the purposes for which
themodel is being used . So a hospital blueprint: wouldpay
attention to the structure and connection ofits beams, but not to
the arrangements.of proteinsin the wood the beams are made of, nor
to theefficacy of the resulting operating room .
Models have to ignore things exactly becausethey view the world
at a level of abstraction('abstraction' is from the Latin
abstrahere, 'topull or draw away') . And it is good that they do
:otherwise they would drown in the infinite rich-ness of the
embedding world . Though this isn'tthe place for metaphysics, it
would not be toomuch to say that every act of
conceptualization,analysis, categorization, does a certain amountof
violence to its subject matter, in order toget at the underlying
regularities that groupthings together . If you don't commit that
act ofviolence - don't ignore some of what's going on -you would
become so hypersensitive and so over-come with complexity that you
would be unable toact.
To capture all this in a word, we will say
-
that models are inherently partial .
All thinking,and all computation, are similarly partial .
Fur-thermore - and this is the important point -thinking and
computation have to be partial : that'show they are able to work
.
4 . Pu.ZZ-blooded Action
Something that is not partial, however, isaction . When you
reach out your hand and grasp aplow, it is the real field you are
digging up, notyour model of it . Models, in other words, maybe
abstract, and thinking may be abstract, andsome aspects of
computation may be abstract, butaction is not . To actually build a
hospital, toclench the steering wheel and drive through
theintersection, or to inject a drug into a person'sbody, is to act
in the full-blooded world, not ina partial or distilled model of it
.
This difference between action and modellingis extraordinarily
important . Even if your everythought is formulated in the terms OT
some model,to act is to take leave of the model and partici-pate in
the whole, rich, infinitely variegatedworld . For this reason,
among others, action playsa crucial role, especially in the human
case, ingrounding the more abstract processes of modellingor
conceptualization . One form that grounding cantake, which computer
systems can already takeadvantage of, is to provide feedback on how
wellthe modelling is going . For example, if an indus-trial robot
develops an internal three-dimensionalrepresentation of a wheel
assembly passing by ona conveyor belt, and then guides its arm
towardsthat object and tries to pick it up, it can usevideo systems
or force sensors to see how well themodel corresponded to what was
actually the case .The world doesn't care about the model : the
clawswill settle on the wheel just in case the actuali-ties mesh
.
Feedback is a special case of a very generalphenomenon : you
often learn, when you do act, justhow good or bad your conceptual
model was . Youlearn, that is, if you have adequate sensory
ap-paratus, the capacity to assess the sensed exper-ience, the
inner resources to revise and recon-ceptualize, and the luxury of
recovering fromminor mistakes and failures .
5 .
Computers and Models
What does all this have to do with computersand with
correctness? The point is that computers,like us, participate in
the real world :
they takereal actions . One of the most important factsabout
computers, to put this another way, is thatwe plug them in . They
are not, as some theoreti-cians seem to suppose, pure mathematical
ab-stractions, living in a pure detached heaven . Theyland real
planes at real airports ; administer realdrugs ; and - as those of
you here today know alltoo well - control real radars, missiles and
com-mand systems . Like us, in other words, althoughthey base their
actions on models, they have con-sequence in a world that
inevitably transcends thepartiality of those enabling models . Like
us, inother words, and unlike the objects of mathematics,they are
challenged by the inexorable conflict bet-ween the partial but
tractable model, and the
actual but infinite world .
And, to make the only too obvious point : wein general have no
guarantee that the models areright - indeed we have no guarantee
about much ofanything about the relationship between model andworld
. As we will see, current notions of"correctness" don't even
address this fundamentalquestion .
In philosophy and logic, as it happens, thereis a very precise
mathematical theory called"model theory" . You might think that it
would bea theory about what models are, what they are goodfor, how
they correspond to the worlds they aremodels of, and so forth . You
might even hope thiswas true, for the following reason : a great
dealof theoretical computer science, and all of thework in program
verification and correctness,historically derives from this
model-theoretictradition, and depends on its techniques .
Unfor-tunately, however, model theory doesn't addressthe
model-world relationship at all . Rather,what model theory does is
to tell you how yourdescriptions, representations, and
programscorrespond to your model .
The situation, in other words, is roughly asdepicted in Figure
1, below . You are to imaginea description, program, computer
system (or evena thought - they are all similar in this regard)in
the left hand box, and the very real world inthe right . Mediating
between the two is the in-evitable model, serving as an idealized
or pre-conceptualized simulacrum of the world, in termsof which the
description or program or whatevercan be understood . One way to
understand themodel is as the glasses through which` the programor
computer looks at the world :
it is` the world,that is, as the system sees it (though not,
ofcourse, as it necessarily is) .
The technical subject of "model theory", asI have already said,
is a study of the relation-ship on the left . What about the
relationship onthe right? The answer, and one of the main pointsI
hope you will take away from this discussion', isthat, at this
point in intellectual history, we'have no theory of this right-hand
side relation-ship .
COMPUTER
M0DE
Figure 1 : Computers, Models, and the EmbeddingWorld
-
There are lots of reasons for this, some verycomplex . For one
thing, most of our currentlyaccepted formal techniques were
developed, duringthe first half of this century, to deal with
mathe-matics and physics . Mathematics is unique, withrespect to
models, because (at least to a firstlevel of approximation) its
subject matter is theworld of models and abstract structures, and
there-fore the model-world relationship is relativelyunproblematic
. The situation in physics is morecomplex, of course, as is the
relationshipbetween mathematics and physics . How apparentlypure
mathematical structures could be used tomodel the material
substrate of the universe is aquestion that has exercised physical
scientistsfor centuries . But the point is that, .whether ornot one
believes that the best physical models domore justice and therefore
less violence to theworld than do models in so-called
"higher-level"disciplines like sociology or economics,
formaltechniques don't themselves address the questionof adequacy
.
Another reason we don't have a theory of theright-hand side is
that there is very little agree-ment on what such a theory would
look like . Infact all kinds of questions arise, when one
studiesthe model-world relationship explicitly, aboutwhether it can
be treated formally at all, aboutwhether it can be treated
rigorously, even if notformally (and what the relationship is
betweenthose two), about whether any theory will be morethan
usually infected with prejudices and precon-deptions of the
theorist, and so forth . The inves-tigation quickly leads to
foundational questionsin mathematics, philosophy, and language, as
wellas computer science . But none of what one learnsin any way
lessens its ultimate importance .
In theend, any adequate theory of action, and, conse-quently,
any adequate theory of correctness,will have to take the
model-world relationshipinto account .
6 .
Correctness and Relative Consistency
Let's get back, then, to computers, and tocorrectness . As I
mentioned earlier, the word'correct' is already problematic,
especially asit relates to underlying intention . Is a
programcorrect when it does what we have instructed itto do? or
what we wanted it to do? or what historywould dispassionately say
it should have done?Analysing what correctness should mean is
toocomplex a topic to take up directly . What I wantto do, in the
time remaining, is to describe whatsorts of correctness we are
presently capable ofanalysing .
In order to understand this, we need tounderstand one more thing
about building computersystems . I have already said, when you
design acomputer system, that you first develop a model ofthe
world, as indicated in the diagram . But youdon't, in general, ever
get to hold the model inyour hand : computer systems, in general,
are basedon models that are purely abstract . Rather, if youare
interested in proving your program "correct",you develop two
concrete things, structured in termsof the abstract underlying
model (although these arelisted here in logical order, the program
is veryoften written first) :
1 . A specification : a formal description in somestandard
formal language, specified in termsof the model, in which the
desired behaviour isdescribed ; and
2 . The program : a set of instructions and repre-sentations,
also formulated in the terms of themodel, which the computer uses
as the basis forits actions .
How do these two differ? In various ways, ofwhich one is
particularly important . The programhas to say how the behaviour is
to be achieved,typically in a step by step fashion (and often
inexcruciating detail) . The specification, however,is less
constrained : all it has to do is tospecify what proper behaviour
would be, independentof how it is accomplished . For example, a
specifi-cation for a milk-delivery system might simply be :"Make
one milk delivery at each store, driving theshortest possible
distance in total ." That's justa description of what has to happen
. The program,on the other hand, would have the much more
diffi-cult job of saying how this was to be accomplished .It might
be phrased as follows : "drive four blocksnorth, turn right, stop
at Gregory's Grocery Storeon the corner, drop off the milk, then
drive 17blocks north-east, . . ." . Specifications, to useuse some
of the jargon of the field, are essentiallydeclarative ; they are
like indicative sentencesor claims . Programs, on the other hand,
areprocedural : they must contain instructions thatlead to a
determinate sequence of actions .
What, then, is a proof of correctness? Itis a proof that any
system that obeys the programwill satisfy the specification .
There are, as is probably quite evident, twokinds of problems
here . The first, often acknow-ledged, is that the correctness
proof is in realityonly a proof that two characterizations of
some-thing are compatible . When the two differ - i .e .,when you
try to prove correctness and fail - thereis no more reason to
believe that the first (thespecification) is any more correct than
the second(the program) . As a matter of technical
practice,specifications tend to be extraordinarily complexformal
descriptions, just as subject to bugs anddesign errors and so forth
as programs . In factthey are very much like programs, as this
intro-duction should suggest . So what almost alwayshappens, when
you write a specification and a pro-gram, and try to show that they
are compatible, isthat you have to adjust both of them in order
toget them to converge .
For example, suppose you write a program tofactor a number C,
producing two answers A andB . Your specification might be :
Given a number C, produce numbers A andB such that A x B = C
.
This is a specification, not a program, because itdoesn't tell
you how to come up with A and B . Allit tells you is what
properties A and B should have .In particular, suppose I say : ok,
C is 5,332,114 ;what are A and B? Staring at the specificationjust
given won't help you to come up with theanswer . Suppose, on the
other hand, given this
-
specification, that you then write a program - say,by
successively trying pairs of numbers until youfind two that work .
Suppose further that you thenset out to prove that your program
meets your spe-cifications . And, finally, suppose that thisproof
can be constructed (I won't go into detailshere ; I hope you can
imagine that such a proofcould be constructed) . With all three
things inhand - program, specification, and proof - youmight think
you were done .
In fact, however, things are rarely that sim-ple, as even this
simple example can show . In par-ticular, suppose, after doing all
this work, thatyou try your program out, confident that it mustwork
because you have a proof of its correctness .You randomly give it
14 as an input, expecting 2and 7 . But in fact it gives you back
the answersA = 1 and B = 14 . In fact, you realize upon fur-ther
examination, it will always give back A = land B = C . It does
this, even though you have aproof of its being correct, because you
didn't makeyour specification meet your intentions . You wantedboth
A and B to be different from C (and also differ-ent from 1), but
you forgot to say that . In thiscase you have to modify both the
program and thespecification . A plausible new version of thelatter
would be :
Given a number C, produce A and B suchthat A J 1 and B J 1 and A
x B = C .
And so one and so forth : the point, I take it, isobvious . If
the next version of the programgiven 14, produces A = -1 and B -
-14, you wouldsimilarly have met your new specification, butstill
failed to meet your intention . Writing "good"specifications -
which is to say, writing specifi-cations that capture your
intention - is hard .
It should be apparent, nonetheless, thatdeveloping even
straightforward proofs of "correct-ness" is nonetheless very useful
. It typically forcesyou to delineate, very explicitly and
completely, themodel on which both program and specification
arebased . A great many of the simple bugs that occurin programs,
of which the problem of producing 1and 14 was an example, arise
from sloppiness andunclarity about the model . Such bugs are
notidentified by the proof, but they are often un-earthed in the
attempt to prove . And of course thereis nothing wrong with this
practice ; anything thathelps to erradicate errors and increase
confidenceis to be applauded . The point, rather, is to showexactly
what these proofs consist in .
In particular, as the discussion has shown,when you show that a
program meets its specifica-tions, all you have done is to show
that two for-mal descriptions, slightly different in character,are
compatible . This is why I think it is some-where between
misleading and immoral for computerscientists to call this
"correctness" . What iscalled a proof of correctness is really a
proof ofthe compatibility or consistency between two formalobjects
of an extremely similar sort : program andspecification .
As a community, we computer scien-tists should call this
relative consistency, anddrop the word 'correctness' completely
.
What proofs of relative consistency ignore is
the second problem intimidated earlier . Nothingin the so-called
program verification process perse deals with the right-hand side
relationship :the relationship between the model and the world
.But, as is clear, it is over inadequacies on theright hand side -
inadequacies, that is, in themodels in terms of which the programs
and speci-fications are written - that systems so commonlyfail
.
The problem with the moon-rise, for example,was a problem of
this second sort . The difficultywas not that the program failed,
in terms of themodel . The problem rather, was that the modelwas
overly simplistic ; it didn't correspond towhat was the case in
-the world . Or, to put itmore carefully, since all models fail to
corres-pond to the world in indefinitely many ways, aswe have
already said, it didn't correspond towhat was the case in a crucial
and relevant way .In other words, to answer one of our
originalquestions, even if a formal specification hadbeen written
for the 1960 warning system, and aproof of correctness generated,
there is no reasonto believe that potential difficulties with
themoon would have emerged .
You might think that the designers weresloppy ; that they would
have thought of the moonif they had been more careful . But it
turns outto be extremely difficult to develop realisticmodels of
any but the most artificial situations,and to assess how adequate
these models are . Tosee just how hard it can be, think back on
thecase of General Electric, and imagine writingappliance
specifications, this time for a refri-gerator . To give the example
some force, imaginethat you are contracting the refrigerator out
tobe built by an independent supplier, and thatyou want to put a
specification into the contractthat is sufficently precise to
guarantee that youwill be happy with anything that the supplier
de-livers that meets the contract .
Your first version might be quite simple -say, that it should
maintain an internal tempera-ture of between 3 and 6 degrees
Centigrade ; notuse more than 200 Watts of electricity ; cost
lessthat $100 to manufacture ; have an internal volumeof half a
cubic meter ; and so on and so forth .But of course there are
hundreds of other pro-perties that you implicitly rely`'on :
it should,presumably, be structurally sound :
you wouldn'tbe happy with a deliciously cool plastic bag .It
shouldn't weigh more than a ton, or emit loudnoises . And it
shouldn't fling projectiles outat high speed when the`door' is'
opened .
In general,it is impossible, when writing specifications,
toinclude everything that you want :
legal contracts,and other humanly' interpretable
specifications,are always stated within a background ofcommonsense,
to cover the myriad unstated and unstatableassumptions assumed -
to' hold in force .
(Currentcomputer programs, alas, have no common sense, asthe
cartoonists know so well .)
So it is hard to make sure that everythingthat meets your
specification will'really be arefrigerator ; it is also hard to
make sure thatyour requirements don't rule out perfectly
goodrefrigerators . Suppose for example a customer
-
plugs a toaster in, puts it inside the refrigeratorand complains
that the object he received doesn'tmeet the temperature
specification and must there-fore not be a refrigerator . Or
suppose he triesto run it upside down . Or complains that itdoesn't
work in outer space, even though you didn'texplicitly specify that
it would only work withinthe earth's atmosphere . Or spins it a
10,000 rpm .Or even just unplugs it . In each case you wouldsay
that the problem lies not with the refrigeratorbut with the use .
But how is use to be specified?The point is that, as well as
modelling theartifact itself, you have to model the relevantpart of
the world in which it will be embedded .It follows that the model
of a refrigerator as adevice that always maintains an internal
temperatureof between 3 and 6 degrees is too"strict to coverall
possible situations . One could try to modelwhat appropriate use
would be, though specificationsdon't ordinarily, even try to
identify all the rele-vant circumstantial factors . As well as
therebeing a background set of constraints with respectto which a
model is formulated, in other words, thereis also a background set
of assumptions on which aspecification is allowed at any point to
rely .
? . The Limits of Correctness
It's time to summarize what we've said so far .The first
challenge to developing a perfectly"correct" computer system stems
from the sheercomplexity of real-world tasks . We mentioned at
theoutset various factors that contribute to thiscomplexity : human
interaction, unpredictable fac-tors of setting, hardware problems,
difficultiesin identifying salient levels of abstraction, etc .Nor
is this complexity of only theoretical concern .A December 1984
report of the American DefenseScience Board Task Force on "Military
Applicationsof New-Generation Computing Technologies" identifiesthe
following gap between current laboratory demon-strations and what
will be required for successfulmilitary applications - applications
they call"Real World ; Life or Death" . In their estimationthe
military now needs (and, so far as one cantell, expects to produce)
an increase in the powerof computer systems of nine orders of
magnitude,accounting for both speed and amount of informationto: be
processed . That is a 1,000,000 .000-foldincrease over current
research systems, equivalentto the difference between a full
century of theentire New York metropolitan area, compared to oneday
in the life of a hamlet of one hundred people .And remember that
even current systems arealready several orders of magnitude more
complexthat those for which we can currently developproofs of
relative consistency .
But sheer complexity has not been our pri-mary subject matter .
The second challenge tocomputational correctness, more serious,
comes fromthe problem of formulating or specifying an appro-priate
model . Except in the most highly artificialor constrained domains,
modelling the embeddingsituation is an approximate, not a complete,
endea-vour . It has the best hopes of even partial suc-cess in what
Winograd has called "systematicdomains" : areas where the relevant
stock of objects,properties, and relationships are most clearly
andregularly pre-defined . Thus bacteremia, or ware-house
inventories, or even flight paths of air-
planes coming into airports, are relativelysystematic domains,
at least compared to conflictnegotiations, any situations involving
intentionalhuman agency, learning and instruction, and soforth .
The systems that land airplanes are hybrids- combinations of
computers and people - exactlybecause the unforeseeable happens,
and because whathappens is in part the result of human
action,requiring human interpretation . Although it isimpressive
how well the phone companies canmodel telephone connections, lines,
and evendevelop statistical models of telephone use, at acertain
level of abstraction, it would neverthe-less be impossible to model
the content of thetelephone conversations themselves .
Third, and finally, is the question of whatone does about these
first two facts . It isbecause of the answer to this last question
thatI have talked, so far, somewhat interchangeablyabout people and
computers . With respect to theultimate limits of models and
conceptualization,both people and computers are restrained by
thesame truths . If the world is infinitely rich andvariegated, no
prior conceptualization of it, norany abstraction, will ever do it
full justice .That's ok - or at least we might as well say thatit's
ok, since that's the world we've got . Whatmatters is that we not
forget about that richness- that we not think, with misplaced
optimism, thatmachines might magically have access to a kind
of"correctness" to which people cannot even aspire .
It is time, to put this another way, thatwe change the
traditional terms of the debate .The question is not whether
machines can do things,as if, in the background, lies the implicit
assump-tion that the object of comparison is people . Plansto build
automated systems capable of making a"decision", in a matter of
seconds, to annihilateEurope, say, should make you uneasy ;
requiring aperson to make the same decision in a matter ofthe same
few seconds should make you uneasy too,and for very similar reasons
. The problem isthat there is simply no way that reasoning of
anysort can do justice to the inevitable complexityof the
situation, because of what reasoning is .Reasoning is based on
partial models . Which meansit cannot be guaranteed to be correct .
Which means,to suggest just one possible strategy for action,that
we might try, in our treaty negotiations,to find mechanisms to slow
our weapons systemsdown .
It is striking to realise, once the comparisonbetween machines
and people is raised explicitly,that we don't typically expect
"correctness" forpeople in anything like the form that we pre-sume
it for computers . In fact quite the opposite,and in a revealing
way . Imagine, in some by-goneera, sending a soldier off to war,
and giving him(it would surely have been a "him") final
instruc-tions .
"Obey your commander, help your fellow-soldier", you might say,
"and above all do yourcountry honour" . What is striking about this
isthat it is considered not just a weakness, but apunishable
weakness - a breach of morality - toobey instructions blindly (in
fact, and for rele-vant reasons, you generally can't follow
instruc-tions blindly ; they have to be interpreted to thesituation
at hand) . You are subject to court-
-
martial, for example, if you violate fundamentalmoral
principles, such as murdering women andchildren, even if following
strict orders .
In the human case, in other words, our socialand moral systems
seem to have built in an accep-tance of the uncertainties and
limitations inherentin the model-world relationship . We know that
theassumptions and preconceptions built into instruc-tions will
sometimes fail, and we know that instruc-tions are always
incomplete ; we exactly rely onjudgment, responsibility,
consciousness, and soforth, to carry someone through those
situations -all situations, in fact - where model and world
partcompany . In fact we never talk about people, interms of their
overall personality, being correct ;we talk about people being
reliable, a much moresubstantive term . It is individual actions,
fullysituated in a particular setting, that are corrector
incorrect, not people in general, or systems .What leads to the
highest number of correct humanactions is a person's being
reliable, experienced,capable of good judgment, etc .
There are two possible morals here, forcomputers . The first has
to do with the notionof experience . In point of fact, program
verification is not the only, or even the most common,method of
obtaining assurance that a computersystem will do the right thing .
Programs areusually judged acceptable, and are typically accep-ted
into use, not because we prove them "correct",but because they have
shown themselves relativelyreliable in their destined situations
for somesubstantial period of time . And, as part of
thisexperience, we expect them to fail : there alwayshas to be room
for failure . Certainly no onewould ever accept a program without
this in situtesting : a proof of correctness is at best
addedinsurance, not a replacement, for real life expe-rience .
Unfortunately, for the ten million linesof code that is supposed to
control and coordinatethe Star Wars Defense System, there will
never,God willing, be an in situ test .
One answer, of course, if genuine testing isimpossible, is to
run a simulation of the realsituation . But simulation, as our
diagram shouldmake clear, -tests onZy the Zeft-hand side
reZation-ship .
Simulations are defined in terms of models ;they don't test the
relationship between the modeland world . That is exactly why
simulations andtests can never replace embedding a program inthe
real world . All the war-games we hear aboutand hypothetical
military scenarios, and electronicbattlefield simulators, and so
forth, are allbased on exactly the kinds of models we have
beentalking about all along . In fact the subject ofsimulation,
worthy of a whole analysis on its own,is really just our whole
subject welling up allover again.
I said earlier that there were two morals tobe drawn, for the
computer, from the fact that weask people to be reliable, not
correct . The secondmoral-is for those who, when confronted with
thefact that genuine or adequate experience cannot behad, would say
"oh, well, let's build responsibilityand morality into the
computers - if people canhave it, there's no reason why machines
can't haveit too ." Now I will not argue that this is inhe-
rently impossible, in a metaphysical or ultimatephilosophical
sense, but a few short comments arein order . First, from the fact
that humans some-times are responsible, it does not follow that
weknow what responsibility is : from tacit skills noexplicit model
is necessarily forthcoming . We sim-ply do not know what aspects of
the human conditionunderlie the modest levels of responsibility
towhich we sometimes rise . And second, with respectto the goal of
building computers with even humanlevels of full reliability and
responsibility, Ican state with surety that the present state
ofartificial intelligence is about as far from thisas mosquitos are
from flying to the moon .
But there are deeper morals even than these .The point is that
even if we could make computersreliable, they still wouldn't
necessarily alwaysdo the correct thing . People aren't provably
"cor-rect", either : that's why we hope they are respon-sible, and
it is surely one of the major ethicalfacts is that correctness and
responsibility don'tcoincide . Even if, in another 1,000 years,
someonewere to devise a genuinely responsible computersystem, there
is no reason to suppose that it wouldachieve "perfect correctness"
either, in the senseof never doing anything wrong . This isn't a
failurein the sense of a performance limitation ; it stemsfrom the
deeper fact that models must abstract, inorder to be useful . The
lesson to be learned fromthe violence inherent in the model-world
relation-ship, in other words, is that there is an inherentconflict
between the power of analysis and concep-tualization, on the one
hand, and sensitivity tothe infinite richness, on the other .
But perhaps this is an overly abstract way toput it . Perhaps,
instead, we should just rememberthat there will always be another
moon-rise .
Notes
1 . Edmund Berkeley, The Computer Revolution,Doubleday, 1972, pp
. 175-177, citing news-paper stories in the Manchester
GuardianWeekly of Dec . 1, 1960, a UPI dispatch pub-lished in the
Boston Traveller of Dec . 13,1960, and an AP dispatch published in
theNew York Times on Dec . 23, 1960 .
2 . McCarthy, John, "A Basis for a MathematicalTheory of
Computation", 1963, in P .Braffortand D . Hirschbert, eds .,
Computer Programmingand Formal Systems, Amsterdam :
North-Holland,1967, pp . 33-70 . Floyd, Robert, "AssigningMeaning
to Programs", Proceedings of Symposiain Applied Mathematics 19,
1967 (also in F .T .Schwartz, ed, Mathematical Aspects ofComputer
Science, Providence : American Math-ematical Society, 1967) . Naur,
P ., "Proof ofAlgor.ithms by General Snapshots", BIT Vol . 6No . 4,
pp . 310-316, 1966 .
3 . Al Stevens, BBN Inc ., personal communication .
4 . See for example R .S . Boyer, and Moore, J .S .,eds ., The
Correctness Problem in ComputerScience, London : Academic Press,
1981 .
5 . Fletcher, James, study chairman, and McMillan,Brockway,
panel chairman, Report of the Study
-
on Eliminating the Threat Posed by NuclearBallistic Missiles
(U), Vol . 5, BattleManagement, Communications, and Data
Pro-cessing (U), U .S . Department of Defense,February 1984 .
6 . See, for example, the Hart-Goldwater reportto the Committee
on Armed Services of theU .S . Senate : "Recent False Alerts from
theNation's Missile Attack Warning System"(Washington, D .C . : U
.S . Government PrintingOffice, Oct . 9, 1980) ; Physicians
forSocial Responsibility, Newsletter,"Accidental Nuclear War",
(Winter 1982),P . 1 .
7 . Ibid .
8 . Berkeley, op . cit . See also Daniel Ford'stwo part article
"The Button", New Yorker,April 1, 1985, p . 43, and April 8, 1985,p
. 49, excerpted from Ford, Daniel, TheButton, New York : Simon and
Schuster,1985 .
9 . In point of fact, developing software forfault-tolerant
systems is an extremely trickybusiness .