CIT. - dtic.mil · SUBJECT TERMS (Continue on reveuse if necessary and identify by block nuamber) FIELD GROUP $UB-GROUP dnsign, ergonomics, human …

f5 -al" i

* COGNITIVE

IDI-~U 2.t2 VA6

CIT.

N pr

tiwd I03 -d =A MU

b .dN 22 0

1/

ICOMPUTATION VIA

DIRECT MANIPULATION

Final Report

Contract N00014-85-C.0133

December 1, 1984 to February 29, 1988

Office of Naval Research

ECTE a,

This research was supported by Contract N00014-85-C-0133, NR 667-541 from the Personnel and Training Re-

search Programs (now the Cognitive Science Programs), Psychological Sciences Division, Office of Naval Re-search. Research was also supported by a grant from the System Development Foundation, by thp Navy Person-nel Research and Development Center, and by a fellowship from the John D. and Catherine T. MacArthurFoundation. The views and conclusions contained in this document are those of the authors and should not be in-terpreted as necessarily representing the official policies, either express or implied, of the sponsoring agencies.Approved for public release; distribution unlimited. Reproduction in whole or in part is permitted for any purposeof the United States Government.Copyright 01988 by the Institute for Cognitive Science.

Unclassif~iedSlEcURITY CLASSIFICA'ICN OF THIS PAWE

REPORT DOCUMENTATION PAGEIa. REPORT SECURITY CLASSIFICATION 11b. RESTRICTIVE MARKINGSUnclassified

12s. SIECURITY CLASSIFICATION AUTHORITY 3. DISTRIBUTION / AVAILABILITY OF REPORT

2b. DECLASSIFICATION / DOWNGRADING SCHEDULE

0. PERFORMING ORGANIZATION REPORT NUMBER(S) S. MOWiTORING ORGANIZATION REPORT NUMBER(S

6a. NAME OF PERFORMING ORGANIZATION 6b. OFFICE SYMBOL 7a. NAME OF MONITORING ORGANIZATIONInstitute f or Cognitive Scec N appicale - Cognitive Science ProgramUniversity of California, San Di to Office of Naval Renveaith-6c. ADDRESS (City. State, aNW i Cadc) 7b. ADDRESS (City, State. and ZIP Code)C-015 800 North Quincy StreetLa Jolla, CA 92093-0115 Arlington, VA 22217-5000

$a. NAME OF FUNDING /SPONSORING Ob. OFFICE SYMBOL 9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER

O RG A IZA IO NOf s ~ k a le) N 0 0 0 1 4 -8 5 -C -0 13 3Sc. ADDRESS (City, State, and ZIP Code) 10. SOURCE OF FUNDING NUMBERS

PROGRAM IPROJECT ITASK IWORK UNITELEMENT NO. INO. LNO. I ACCESSION NO.

_________________________61153N RR04206 0U4206-OA IR 667-54111. TITLE (Include Security CGaWfication)

Computation via l~irect Manipulation

12. PERSONAL AUThOR(S)Donald A. Norman and Edwin L. Hutchins Jr.

13s. TYPE OF REPORT 13b. TIME COVERED 814. DATE OF REPORT (YearMonth,Oa) 5 PAGE COUNTVinal IFROMl12101/84 To02/29/8 1988 August 1 ý;7 20

16. SUPPLEMENTARY NOTATION

17. COSATI CODES 1S. SUBJECT TERMS (Continue on reveuse if necessary and identify by block nuamber)FIELD GROUP $UB-GROUP dnsign, ergonomics, human factors, direct manipulation,

171 IA human-computer interaction, representation of knowledge,

(.19. ABSTRACT (Continue on reverse if necestavy and identify by block number)

Interfaces to complex equipment can often impose severe difficulties for the user. In part, these difficultiesare caused by the abstract nature of the interaction. that many modern interfaces present to the operator. Anew class of interfaces, theý`direct manipulation interface, -p-pears to offer improvemnents in ease of useand understandability because. the abstraction of the normal interface is replaced with what might be calledthe'"model world metaphor,'f~where- the user can feelas -if th.5, operations are done- directly ýupon, theexternal environment. Research uinder this contract examined in detail the nature of directness in the useof computer interfaces. The research demonstrates that the concept of-l'directneSS"ý is a complex one,involving at least four different aspects of the interface, including two gulfs, one for execution and one forevaluation, and two different kin-ds of mappings: semantic mappings and referential distance. The ex-perimental and theoretical work reported under this contract examines the complexities of the

-20. DISTRIBUTION /AVAILABILITY OF ABSTRACT 21. ABSTRACT SECURITY CLASSIFICATIONO UNCLASSIFIEOIUNLIMITED E3 SAME AS RPT. 0 DTIC QSERS Unclassified

22a. NAME OF RESPONSIBL' iNDIVIDUAL 22b. TELEPHONE (nclude Atea Cd) 22c. OFFICE SYMBOLIUr. Susan Chipm.. (202) 69-38OR14C

DD FORMA 1473, 84MAR 83 APR edition may be used until txhaivt)ed. SECURITY CLASSIFICATION OF THIS PAGE

All other editions are obsolete. Unclassified

19. ABSTRACT (continued)

differences among interface styles, demonstrates the importance of visibility and sound inthe performance of tasks, and presents a new, detailed analysis of the general attributes ofcognitive artifacts, including an important new theoretical construct. the object-symbol.,\These analyses allow for a deeper understanding of the differences among existing human-smach;ne interfaces and provide the background for the development of a new class ofinterfaces that promise superior performance.

9I CC ~

Contents

Di• d z ah ah• a ...... ... .. .......... .................. ..... ................ ........ ........ ............. ......... ........ 2The Gulfs of Execution and Evaluation ................................................................................................ 3An Empicai Exploration of ................................... .... 5Ev yday st ing and Auditory Icons .............................. .................................................................. 7

.,. ................................... ... ........................................... 7~ 7

Audtq cons Bsed on Everyday Listening ................................................................................... 8S~ ~ 8

Computational Power in Artifacts ....... 9Ch.ack ist as Cognitive Atifact ... ......... 9Problem Solving as Re-Representation ............................................................... .... 10

Five Ways to Do Distance/Rate/Time Problems .............................................................................. 10Artifcts - Amplifiers or Transformers of Cognition .................................................................. 147 rC geO eS :oy .................................................................................................................................. 15

The Design of Appropriately Constrained Artifacts ................................................................................ 167h o( ý ............................................................................................................................................... 17

Reinterpreting the Attraction of Direct Manipulation Interfaces ................................................. 18

20NTIS GRA&Z

DTIC TABUnannzounced4 3

By,Distribution/Availability Codes

IAail and/or

Dist Special

Computation via Direct Manipulation

DONALD A. NORMAN and EDWIN L. HUTCHINS

INTRODUCrION the contract period consisted of the follow-ing individuals:

This is the final progress report for contractN00014-85-C-0133, NR 667-541, Computa- Larry Bierma (NPRDC)tion via Direct Manipulation: Cognitive Bill Gaver (UCSD and Apple Computer)Science Construction Methods for Compu- Michael Goeller (UCSD)tational Systems, Expert Assistants, and Jim Hollan (NPRDC: resigned in JanuaryGraphical Direction of Complex Systems, 1987 to beroome technical director of thewith the Personnel and Training Research Human Interface Program at the Micro-Programs of the Office of Naval Research electronics and Computer Technology(also known, informally, as "the Bridges Corporation [MCC; Austin, Texas])Contract"). This contract was performed Ed Hutchins (NPR)C, theru UCSD)jointly by members of the Institute for Cog- Masumi Ishikawa (UCSD, visiting fromnltiVL Science at the University of Zalifo'- MITI Electrotechnical Laboratories, Ja-nia, San Diego (UCSD) and members of the pan).Intelligent Systems Group of the Navy Per- Tim McCandless (NPRDC: resigned in Jan-sonnel Research and Development Center uary 1987 to join the Human Interface(NPRDC). Key members of the research Program at MCC)team were Donald Norman and David Barbara Morris (UCSD)Owen from UCSD, Jim Hollan, Ed Hutch- Donald Norman (UCSD)ins, and Miriam Schustack from NPRDC, Mark Rosenstein (NPRDC: resigned in Jan-and Colleen Seifert, an Office of Naval uary 1987 to join the Human InterfaceTechnology and American Society for Engi- Program at MCC)neering Education Postdoctoral Fellow. David Owen (UCSD)The total research group over the course of Miriam Schustack (NPRDC)

S"H ia.,L. tfl "L la•nt tf l •-lr t u IJfa n-2$/faw m Jl' r lV, r,•

2 mNAL REPoRT oNR N0014-lS.C.

Colleen Seifert (Office of Naval Technology series of studies aimed at understandingand American Society for Engineering howv computers can augment human cog-Education Postdoctoral Fellow at nitive abilities.NPRDC) One of the most important products of

Ron Stanonik (NPRDC) our research hus been a new understand-Hank Strub (UCSD) ing of the context that surrounds our origi-Mark Wallen (UCSD) nal goals. As our understanding of thisDavid Wargo (UCSD) context grew, topics that had not seemedLouie Weitzman (NPRDC: resigned in January closely related to the issues of DM began to

1987 to join the Human Interface Program at seem very relevant indeed. The theme ex-MCC) pressed in the formal title of the contract

Larry West (UCSD) - direct manipulation - emphasizes theMelissa Monty hittaker (UCSD and Xerox) central focus: the study of tools that mightJiaiie Zhang (UCSD) be easier to understand and easier to use

than ccnventional ones. But as we pur-

DIRECT MANIPULATION sued this analysis, we discovered that to do

INTERFACES the study properly we needed to under-stand the wider range of issues surround-

Computers are clearly the most powerful ing the ways that people perform theirand most flexible information processing tasks. This research project has thereforetools ever. Given the dominant perspec- broadened to consider a number of thetive in cognitive science that human cogni- overall issues relevant to the study of hu-tion is best thought of as a form of infor- man action and the interactions betweenmation processing or computation, it people and designed artifacts. This has ledseems obvious that computers have a to a number of related studies:greater potential than any other technologyto enhance or augment human cognitive * The expeeimental comparison of sys-cmpabilities. But the problem of getting the tems that have different properties. Thishuman information processing system to was an experimental analysis of the dif-work with the machine has not been easily feience in acquisition and performancesolved. Interacting with computers often of these two different styles of interac-seems difficult and frustrating. Traditional- tion with two systems for creating draw-ly, in order to get a computer to do any- ings that were formally equivalent inthing useful, one needed to learn a lan- power (Schastack, in preparation).guage in which the computer could begiven instructions. • The development of a direct manipula-

ln recent years, primarily in the wake of tion system for the analysis of statisticalthe advent of new input/output technolo- data. This programming developmentgies, a vaguely defined set of characteristics and analytical study allowed us to de-parading under the banner of "direct ma- velop the implications of such systemsnipulation" (Shneiderman, 1982) promised on behavior (Owen, 1987, 1988).to make the computational power of com-puters more easily accessible. We set out to * Studies of the role of naturalisticdiscover whether and how this might be sound. Systems that are "direct" pro-true. We started off with a general analysis vide rich and immediate feedbackof the characteristics of direct manipulation about the actions and system state.(DM) interfaces (see Hutchins, Hollan, & Sound can provide information aboutNorman, 1986), and this study led as to a the nonvisible aspects of the interaction

COMPUTATION VIA DIRECT MANIPULATION 3

and of the workings of the system. But kinds of conceptual and mental modelsfor the sound to provide a rich source that they form.of information in a way that is readilyinterpretable, we discovered that it 3. The role of social interaction in the per-must be "naturalistic," related in natu- formance of cooperative work, especial-ral ways to the kind of physical interac- ly how social groups share responsibill-tions users expect from their conceptual ties in the performance of a task,models of the system (Gaver, 1986, including the fact that many of the in-1988). dividuals may have only limited un-

derstanding of the complete task do-The development of design rules. The main and that training and errorvarious studies of this research contract, correction are often integral parts of so-combined with our earlier studies of er- cial task performance, and help dictate arors in human action (work supported number of the ways in which the task isby previous ONR contracts), led to the structured and performed.recent book The Psychology of EverydayThings, which provides a summary of 4L The study of the tools themselves, thosethe issues and suggestions for design artifacts that aid human cognition.(Norman, 1988).

The four topics provide the theme forThe study of socially distributed cogni- the analyses presented in this report. Intion. One new important direction of this report, rather than provide a simpleresearch has been developed through review of the published papers, books, andthe study of naturally situated cognition technical reports that have come from thisand, in particular, the study of the inter- project, we provide a new synthesis. Thisactions and cognitions of the members report, therefore, is more than just a sum-of navigation teams aboard large navy mary: It is a new technical formulation ofships (Hutchins, in press; Seifert & the issues, one that we believe providesHutchins, 1988). The computations un- promise for the understanding of existingdertaken by a navigation team are so intelligent systems, suggestions for the de-completely embedded in the manipula- velopment of new systems, and a frame-tion of computational tools that even work for future research efforts.the participants are often unaware ofthe fact that they are doing computa-tions: The specialized navigational in- The Gulfs of Execution and Evaluationstruments provide a setting in which In doing a task with an arti.fact, whether itall the computation happens outside be computer or some simpler technology,the people who perform it. we can identify several aspects of the per-

formance that affect the way in which theThese studies combine to form a cohe- task gets done. When a person does an ac-

sive attack on the understanding of four es- tion, there are two major aspects - - the ex-sential aspects of performance on tasks: ecution side of the action and the evalua-

tion of the resulting world-state. In our1. Detailed analyses of the tasks. earlier paper (Hutchins, Hollan, & Nor-

man, 1986), we described two gulfs that lie2. Analyses of the individuals, including between the user and the artifact. First con-

their information processing structures, sider the gulf of execution, which lies be-their knowledge structures, and the tween the user's goals and the possible

4 FINAL ROMP OCN N0014-6C.33

actions on the interface. In order to use the face design is the minimization of effort re-device, the user must span this gulf by de- quired to bridge these gulfs. That is, wetermining which actions on the interface would like to make the gulfs narrow. Butwill accomplish the current goals and by what is it that makes them wide?performing those actions. The task of figur- Each gulf has two components of dis-ing out what to do and then doing it takes tance: semantic distance and referential dis-effort. If it takes a lot of effort, we say the tance.,gulf is wide. The other gulf is the gulf of Semantic distance is the relationshipevaluation. It is spanned by perceiving, in- between the user's intentions and theterpreting, and evaluating the state of the meanings of the expressions that are possi-world and comparkig it to the desired or ble in the interface language. It refers toexpected (goal) state (Figure 1). We argued how well the interface language expressesthat one of the most critical issues in inter- the user's intentions. Can the user do what

is intended with simple expressions or arecomplex chains of operations required?

go* High-level progrnmming languages can beWhat we seen as attempts to reduce semantic dis-

want 10 tance. The same argument applies to theevaluation side of action. Here, semanticdistance refers to the amount of differencebetween the interpretation of the informa-tion available from the environment and

Mx M C paX Rwatn the information needed to evaluate the en-What we do Cohparpng what vironment.to the wd happenaed Wtoh Wppe Peferential distance refers to the differ-

wO wanted to happen ence between the user's understanding ofthe meaning of an expression and the

1, f# INS •user's understanding of its form. Basically,Of Of when one decides to do a particular action

sequence (that is, when one forms an in-tention to act), before the action can bedone, its specific manner - its form -must be determined. To specfy an action isto map from its meaning to its form.When the meaning and form are similar,the mapping can be simple (analogical) andthe perceived referential distance is small.

FIGURE 1. The gulfs of execution and molAtion. When the form of an expression is un-Each gulf is unidirectional: The gulf of execution ex- like its meaning, the mapping is complex.tends from user goals to system state; the gulf of eval- Symbolic interfaces, for example, are typi-uation extends from system tate to user goals. Every cally high in referential distance becauseexpression in the interface language has a meaning the relationships between the meaning ofand a form. Semantic distance r"lects the relation-ship between the user's intentions and the meaningsof expressions in the interface language for both in-put and output. Referential distance reflects the rela- t In the earlier work (Hutchins, Hollan. & Norman.tionship between the physical form of the expression 1986), referential distance was called articulatory dis-and its meaning. The easier it is to get from the form tance. The name articulatory is unfortunate, however,of the expression to meaning, the smaller the referen- for reasons that should become clear in the followingtial distance. discussion.

comPUTAToN VIA DuECT MANIPuLAON 5

the expression (a symbolic expression) and - does not inherently possess any mean-

the form by which it must be executed is ar- ing: Our understanding of it is entirelybitnry. The user must do the work to conventional and probably had to betrax.4ate meaning to form. This work is taught upon first encounter. The claim toth, construction and maintenance of struc- meaning similarity is "second order": Theture that spans the referential distance. elements of the contrast set 4, and * have

The same problem applies in reverse a structural relationship to each other thaton the evaluation side of the action. Here. 1 similar to the meaning relationship be-referential distance refers to the difference tween the words left and right. It is the rela-between the forms available in the envi- tionships that are similar, What matters isronraenz and the meanings that are to be not the mappings between individual sym-extracted from them. When form and bols and individual meanings (becausemeaning are similar (such as when an up- that may be arbitrary), but rather the map-ward-moving line is to be interpreted as an pings between sets of symbols and sets ofincrease in the value of the variable of in- meanings.terest), referential distance is small. When We can illustrate the differences be-form and meaning are dissimilar or unre- tween semantic and referential distance bylated (as wt en the change in value must be considering the results of an experimentalcomputed from a series of numbers), refe- comparison of two interfaces that have dif-rential distance is large. ferent interaction styles.

Simple mappings between form andmeaning occur when the form of the ex- An Empirical Exploration of DMpression is similar to the meaning of the Aatexpression or when the structural relations In an empirical study of DM interfacesamong forms are similar to the structural (Schustack, 19xx), users were given a taskrelations among meanings. Consider the to perform using two drawing programsmeanings of the words left and right. Now that were very different in style but withimf.agine two sets of expressions for these almost identical functionality. One of themeanings, first, the words themselves - interfaces had many features that shouldleft and right - and second, some symbols help reduce referential distance: For exam-- 4, and -*. The first set, the words, has ple, it was WYSIWYG (what you see ishigh referential distance because the form what you get), it provided instantaneousof the symbol left has little to do with the feedback on each action as it was per-direction being indicated except by the arbi- formed, and all possible actions were con-trary convention of the translation of let- tinuously visually represented. It wasters to words and words to meanings. We mouse-driven, and it allowed the user tothink the relationship is "naturalr only be- move around the picture space by analo-cause we have Icarned It so well. But note gous movements on the mouse pad. Thewhat happens when we change languages other interface had none of these character-to, say, Spanish or French: the physical istics: Users wrote programming scripts inform of the symbols changes dramatically symbolic language, relying on their memo-(from left to izquierdo or gauch) while the ry or the documentation to generate themeaning remains unchanged. elements and the syntax, and getting feed-

Now consider the pictorial symbols 4. back on the syntactic acceptability of the en-and -*. These symbols have a small refe- tire script (and the resulting picture, if therential distance. Notice that ihis is not be- syntax is all correct) only after movingcause the symbol 4. has a structure similar from script-editing mode to executionto the meaning left. In fact, the structure of mode.

I-

.16 FINAL WXPOIM ONR N="&C4-C013

These interface differences affected the this effect not only reduced the speed bene-nature of the errors that users made, the fit of the visual-style interface for certainrelative difficulty of the various drawings, pictures, but in a few cases reversed it to ahow lonS it took users to reach a criterion speed benefit for the command-style inter-level of performance, and how well they face. One of the consequences of the stylis-exploited the more sophisticated features of tic differences between the interfaces is thatthe programs. there are certain tasks that are quite well

The visually oriented interface allowed suited to the features of one interface andusers to complete the pictures significantly less well suited to the other. For example,faster than the command-language style in- the placement of labels (alphabetic strings)terface. This was a very robust finding that in a picture can be done quite easily In theheld up over the set of pictures as a whole command-style interface but is more cum-(that is, the time to complete the series was bersome in th~e visual-style interface. Forsignificantly shorter for the visual-style in- this task, the command-style interfaceterface), and it also tended to be true of the dearly had less semantic distance than theindividual pictures. This global advantage visual-style interface.in completion time occurred even if the The performances with the two inter-time spent in initial document reading and faces also differed in the quality of the pic-practicing is excluded. Even though the us- tures produced (evaluated by rating eightera of the command-style interface spent independent dimensions uf each copy pro-more time on these preliminary activities duced). By and large, the copies producedbefore they even started copying the first were extremely good reptoductions of thepicture, the copying task itself still took originals in an absolute sense. The picturesthem substantially longer than it took us- got an overall average rating of over 4.5 onera of the visual-style interface, a 5-point scale (where 5 represented perfect

The first, very simple picture (which reproduction). There were two major char-was a circle in the middle of the picture acteristics of the quality of the pictures thatspace) took iners of the command-style in- are notable. First, the users of the com-terface about a luf hour to complete, while mand-style interface were overall signifl-the visual-interface users were finished in cantly more accurate in their pictures, andclose to five minutes. Ju3t getting enough were more accurate on every rated dimen-control over the command-style interface sion on which there were any differences.to make it draw the simplest picture im- The characteristics of the command-styleposes a substantial burden on the new t,-er. interface made some aspects of fine-tuningThere was no such "start-up cost" apparent fairly simple and straightforward (al-for users of the more visual interface-- though not necessarily fast); in the visual-they showed no speed-up from the first to style interface users ran a substantial risk ofthe second picture, while the command- ruining whatever version they already hadstyle interface users brought their times in trying to make small changes. Thus, thedown from the half hour it took for the command-style users were more willing tofirst picture to about 10 minutes for the keep working until they were completelysecond. satisfied, whereas the visual-style interface

But, even limiting the focus to simple users stopped when the picture was "goodquantitative measures, the situation is not enough" (which was objectively quite goodall one-sided. Some of the pictures had in most cases). Second, echoing the findingcharacteristics that made them more quick- for the completion times above, there werely accomplished in one interface than in certain pictures that were more suited tothe other. In terms of time-to-completion, the strengths of each interface. Pictures

COMPUTAMIN VIA om~r3C MANIULATION 7

with features that were differentially eay about events in the world. Yet relativelyIto accomplish in the two interfaces showed little attention has been given to how thisdifferences between the interfaces in their happens. Instead, most studies of sound and

accuracy OIn both directions). hearing have been attempts to understandThese results illustrate the importance either how music is perceived, how the au-I

of realizing that there is more to direct ma- ditory system transduces sound, or how tonipulatlort than short referential distance. measure and reduce environmental noise.On the one hand, the visual-style interface There are, of course, notable exceptions tohad low referential distance, but the prob- this rule (e.g., work on auditory streaming,lems subjects had in cleaning up rough localization, and speech), but on the whole,

drawings and placing ttxt labels indicate it music, transduction, and noise seem to de-had high semantic distance with respect to fine the major interests in audition.

style interface, on the other hand, appeared tempt to define and explore a new way ofto have high referential distance - the understanding how people hear. In this re-meanings of the commands were arbitrari- search, we were concerned with under-Ily related to their form - but low semantic standing everyday listening, the act of gain-distance. With the visual-style interface, it ing information about events in the worldwas easy to specify something to do, but of- by listening to the sounds they make. Heten difficult to specify Just exactly what one takes as a basic hypothesis that people do

wanted. With the command-style interface listen to events, and that they can obtainit was possible to specify exactly what one information about various aspects of theseIwanted, but difficult to generate the expres- events by listening to them. From this be-sion that would do it. ginning, he goes on to address two primary

questions about these abilities: What dowe hear?, and How do we hear it?

Eveydy istnig ndAuditory 1cons2 Work on a basic understanding of eve-Everday istningandryday listening is complementary to the de-

An important method for enhancing di- velopment of auditory icons. On the one

This can be done Visually or auditorily. may be mapped to attributes of events inLittle Is known about the role of sound in the computer are suggested by an under-providing this kind of information, Pait of standing of what we hear; and the ways toIour research efforts have examined the convey these attributes clearly to users drerole of sound. The studies, conducted by precisely the concern of research on howGayer, represent an attempt to understand we hear. On the other hand, work on au-how people listen to the world, and how ditory interfaces is invaluable in testing thesuch an understanding can help in devel- results of basic research in a more naturaloping auditory interfaces for computers. setting, and helps generate intuitions about

everyday listening. Pursuing the basic and

Everyday Listening applied research together has proved to be

There can be little doubt that sounds con-vey a great deal of information to listeners Auditory Icons

2ni section consists of edited excerpts from Wil- Though research on everyday listening isliam W. Gaver's doctoral dissertation, "Everyday Lis clearly only in its very early stages, alreadytening And Auditory Icons." an application of this perspective on

8 FRNAL REPORT: ONR NW145-C-o133

audition has proved to be quite useful. This while other techniques rely on musicalapplication involves using everyday manipulations of sound to convey data.sounds to provide information to comput- This has crucial effects on the kinds ofer users about events in the interface, mappings between sounds and informa-When sounis are used in this way, the re- tion that can be created using these strate-sults are called auditory icons because of gies. Because the attributes of everyday lis-their similarities to visual icons. tening are those of sound-producing

Auditory icons hav ebeen described in events, a nomic or iconic mapping canthree papers that discuss his work on audi- hold between auditory icons and events intory icons. First is the paper that intro- the computer. Systems that rely on musi-duced this idea (Gaver, 1986), and which cal manipulations of sound can only em-lays the framework for further research in ploy metaphorical or symbolic mappings.this area. Second is a technical report writ- Because of this difference, auditory iconsten recently for Apple Computer, Inc. (Gay- are likely to be more intuitively obvious toer, 1988), which is a survey of current tech- users, more easily learned and remem-niques for using sound, and which bered, and more likely to increase users'explores the issues involved in creating au- feelings of working directly in the task do-ditory icons in more detail. Last is a paper main, rather than on a computer.that describes the SonicFinderTM, a proto- It is important to note here that thetype auditory interface developed by Gaver promise of auditory icons is entirely basedat Apple Computer, Inc. (Gaver, in press). on the way of understanding "everyday lis-This interface illustrates the use of auditory tening." That such a new area of researchicons in an actual system. should prove so fruitful is heartening:

These three papers show the develop- The appeal of a-iditory icons must be takenment of auditory icons. In addition, the as support for the notion of everyday lis-Apple technical report provides a broader tening.view of how sounds can be used in inter- In addition, work on everyday listeningfaces, and the paper on the SonicFinder and auditory icons is intimately related.makes the ideas more concrete by provid- Auditory icons depend on an understand-ing specific examples of how auditory icons ing of everyday listening so that computermay be constructed. events can be reliably and usefully mapped

to sound. Conversely, work on auditory ic-yListening and Auditory Icons ons can help provide hypotheses and evi-

dence for research on everyday listening.

A number of people have recently been ex- Finding useful auditory icons often makesploring the possibilities of using nonspeech apparent interesting issues about whataudio in interfaces (e.g., Bly, 1982; Mansur, events people hear, issues that may lead to1984; M•zrich, Frysinger & Slivjanovski, productive empirical examinations. In1984; Morrison & Lunney, 1985; Sumikawa, this way, work on auditory icons supple-Blattner & Greenberg, in press). But our ments studies of physics, protocol studies,strategy behind auditory icons is considera- and more rigorous experimental work inbly different from those used by other re- helping to understand everyday listening.searchers, and has several advantages as away of conveying information from com- COGNITIVE ARTIFACTSputers.

The crux of the differences between au- As we studied direct manipulation systemsditory icons and other methods of using and contrasted them with command-sound is that auditory icons are based on language systems, we became aware thatan understanding of everyday listening, the questions really focused upon a more

L 'Qý Mm _

COMPUTATION VIA DIREcr MANIPULATION 9

general concern: the manner by which peo- by humans, and then view the problem ofple interact ;.Ath the world. This takes o., human computer interaction (HCI) as onespecial conc...rn with computer systems, for of specifying a division of labor betweenwith the computer, there is only a second- human and computer and designing aorder interaction with the world. The com- communication protocol that can be usedputer contains a representation of objects to coordinate the efforts of the two partici-and information in the world and the in- pants. This is, of course, an important pos-teraction is done upon this representation. sibility for HCI, but it is not the only one,The indirectness leads to the gulfs that we and assuming it prematurely may lead oneidentified in our earlier studies. And the to overlook some other important aspectsindirectness leads to both the power and of interactions with arlifacts in general andthe difficulty of the interaction. The major computers in particular.feature of a direct manipulation interface is Our view of cognition assumes neitherits attempt to minimize the indirectness by that cognition happens all "in the head"minimizing the distinction between the nor that useful cognitive artifacts necessar-appearance and requirements of the repre- ily achieve their results by augmenting orsenting world of the computer and the ex- amplifying the cognitive abilities of users.ternal. physical world - the represented Instead, we look for the power of cognitiveworld. Still, direct manipulation interfaces artifacts in the interaction of what is insideare really only one example of a more gen- the person and that which is outside theeral issue: the study of artificial systems in person.general arid, especially, the characteristicsof artifacts that aid b-.:,nan cognition. Checklist as Cognitive Artifact

Most artifaicts aid physical requirementsand enhance physical abilities. Some arti- One simple (noncomputer) artifact that wefacts, however, increase our cognitive abili- have examined in some detail is the sim-ties: We use the term "cognitive artifacts" ple checklist, used to organize tasks when-to refer to these objects. Although cogni- ever it is essential that a list of actions alltive artifacts are less numerous and less be performed, oftentimes in particular or-well studied than physical artifacts, they der (Hutchins, 1986). The checklist organiz-play an increasingly important role in our es the behavior of the task performer in alives. Maps, drawings, blueprints, lists, way that may not be possible for the unaid-tables, calendars, books, slide rules, calcula- ed performer. In order to use a checklist astors: These are all examples of cognitive ar- a guide to action, the task performer musttifacts. Computers are a special class of arti- coordinate with both the checklist and thefact and they deserve special treatment, for environment in which the actions are tothey can do things that are difficult or im- be taken.possible for other artifacts. Still, they are ar- Achieving coordination with thetifacts, and an understanding of the psy- checklist requires the actor to invoke proce-chology of our interaction with things dures for the use of the checklist. These in-must be relevant to our interactions with clude reading skills and a strategy of se- -

computerized things. quential execution that permits the taskperformer to ensure that the steps will be

Computational Power in Artifacts done in the correct o~rder and that each stepwill be done once and only once. The fixed

The computer can function as a powerful linear structure of the checklist permits theand independent information processing user to accomplish this by simply keepingdevice. As a result, we often concentrate track of an index that indicates the first un-upon its power to do tasks previously done executed (or last executed) item.

10 FINAL REPORT: ONR N001485-C-0133

While the task performer is following problem solver and a good artifact will,the checklist, high-level control of task- therefore, use representations that matchrelated behavior is given to the structure of those of their users. In fact, Simon has sug-the artifact. It might seem that the actor al- gested that "solving a problem simp'vternates coordinating with the structure of means representing it so as to make the sc-the checklist and coordinating with the lution transparent" (Simon, 1981, p. 153).struct~are of the task world. However, the Many artifacts can serve as representation-coordination with the two media is in fact al transformers, taking the representationsimultaneous to the extent that under- of the problem from one format into a for-standing a step in the description may de- mat better suited for use by people.2

pend upon understanding the state of theworld in which it is to be carried out. Theexperience of the meanings of the descrip- Five Ways to Do Distance/Ra.e1 Timetions of the steps incorporates experience of Problemsthe task world, and the doing of the actionsin the task world incorporates the experi- An example from Hutchins's (in press)enre of the meaning of the task steps. The study of ship navigation demonstratesimportance of this is that in this mediated how the structure of representational arti-performance, the person provides continu- facts can change the cognitive activities re-ous coordination among several structured quired to do a task. Imagine a navigatormedia, some internal, some external, who has just plotted a position fix and

It is easy to show that a novice task per- needs to compute the ship's speed based onformer has no internal representation of the distance the ship has moved in the in-the entire checklist. The large-scale struc- terval of time that elapsed between the cur-ture of the task performance is in the or- rent fix and the previous one. This is aganization of the checklist, not in the mind standard distance/rate/time problem. Forof the task performer. With practice, of this particular exercise assume that the twocourse, the novice may learn the procedure fix positions are 1,500 yards apart and that 3and the large-scale structure of the task minutes of time have elapsed between themay come to be represented in the mind of fix observations: What is the ship's speedthe performer. As was the case with the (in knots)?typewriter keyboard, internalizing such Consider the cognition required understructure may lead to much more rapid five different conditions:and efficient performance. That, however,is another story. At this point in our argu- Condition 1: The performer has the fol-ment we wish only to illustrate the fact that lowing resources: paper and pencil,people can make use of structure in their knowledge of algebra, knowledge ofenvironment to transform their cognitive arithmetic, lnowledge that there areabilities without having any enduring in- 2,000 yards in a nautical mile and 60ternal representation of that structure. minutes in an hour, and knowledge of

the equation D = RT.

Problem Solving as Re-Representation

Cognitive artifacts, whether computerized 3 Hill (1988) argues that artificial intelligence servesoFimarily as the developer of representational vehi-

or not, are always vehicles for representa- des that permit the human intellect to express itselftions of the problems they are used to in ways more powerful than previous methods; moresolve. The way a problem is represented powerful in the sense that they match better human

can radically change what is required of the capabilities and structures.

COMPUTATION VWA DIECr MANIULATION 11

Condition 2: The task performer has thesame resources as it Condition 1, exceptthat instead of a paper and pencdl, thetask performer has a four-function -* -pocket calculator.

Condition 3: The task performer has athree-scale nomogram (see Figure 2)"Iand knows how to use it.

-2 2

Condition 4: The task performer has anautical sliae-rule (see Figure 3) andknows how to use it.

Condition 5: The task performer has no-material implements at all, but knows --how to use what navigators call the "3

minute" rule. -t-:

In Condition 1 (paper and pencil), the _task performer will first have to use theknowledge of algebra to manipulate the --formula D = RT to the form R = D/T so that - -mrate can be solved for directly from the giv- +en values of D and T. The distance in yards + ,will have to be converted to the equivalent -- 5number of miles (repdring knowledge of ,the number of yards in a mile and the rele- A ",

vant arithmetic). The time in minutes willhave to be converted to the equivalent 2" -. o 0

number of hours (requiring knowledge of :--,the number of minutes in an hour and, 2 " nagain, arithmetic). The distance measuremust be divided by the time measure (re- 30 Z1 30= 30

quiring arl*'metic again) to get the rate. Of - 2s =- =course, these things can be done in a differ- 40 n"

ent order; for example, the division could n W"

come before either of the unit conversions, 50 40- sOor between them, but in any case all these an -=m

things must be done at some point in order Go Go

to solve the problem. V m7oN

D-RT~~~~~~: .. . I -0-l• m a

A - D-T P1500 yards - 1500/2000 - 3/4 miles W: r3 minutes a 3/60 = 1/20 hours L [-R - (3/4) 20 - 15 knots

FIGURE 2. The time/distance/speed nomogram.

12 FTNAL RORT: oNR N0014sC-0133

DISTANCE

S, • |L.ACX-TANDS/

Is 20

F[GTRE 3. Navigation slide rule.

Although the arithmetic and algebraic fail completely for lack of ability to orga-skills required to solve this problem are nize and coordinate the various parts ofreasonably simple, they would tax the abili- the solution with each other.ties of many navigation practitioners in the i: Condition 2 (the calculator), the pro-navy. The problem is not that the a:'ithme- . cedures for doing the arithmetic operationstic is difficult, but rather that it is neessary of division and multiplication are restruc-to figure out what to do and how to fit the tured so that instead of decomposing theintermediate steps together while in the problem t3 a set of operations on paper,real, operational setting of time and social values are keyed into the calculator andpressures and task demands. One may be buttons pushed. Depending upon the orderperfectly capable of doing every one of the in which the .steps are taken, it may be ne-component subtasks In this problem, but cessary to remember a previous result and

COMPUTATION VIA DIRECI MANIPULATION 1?

enter it into a later operation, after other problem, and this tool is designed s'ecifi-operations have intervened. This version cally to make its solution easy.of the task would probably also iax the abil- Condition 4 (the nautical slide rule)itles of many navigation practitioners, be- provides an analysis similar to that of Con-cause the hard part is not doing the arith- dition 3: A slide rule is simply a generail-metic but in deciding how to coordinate ized, mechanical version of the nomo-the arithmetic operations with each other. gram, where the movement of the scalesThe calculator gives no support for that and of the sliding hair line corresponds topart of the task. the setting rf the points and the drawing of

Conditions 1 and 2 (paper and pencil the line on the nomogram. In this case,and calculator) are alike in that they use one aligns the distance index with the de-genieral computational knowledge which sired distance on the distance scale, alignsgives little help in structuring the actions. the elapsed time index with the desiredBecause of this, the computation is corn- time on the time scale, and then the speedplex, and it contains many steps, especially index points to the speed in knots.if we count as a step the writing of each In both the nomogram and the slidesymbol or each key press of the calculator. rule, having the scales in the units nor-

Now consider Condition 3 (the nomo- maily encountered eliminates the need togram). A nomogram is a printed form con- convert one kind of unit into another, buttaining scales, usually linear or logarith- much more important is the fact that thesemic, so arranged so that a straight line two conditions eliminate the need for anydrawn through the known values on the knowledge of algebra. The nomogram andscales intersects the other scales at the solu- slide rule transform the task from one oftion points. In the case of this navigation computational planning - figuring outproblem, there are three logarithmic scales: what to divide by what - to one of simpleone for rate (usually calibrated in knots), manipulation. In the first two conditions,one for distance (usually calibrated in both knowledge of the numbers and the equa-yards and nautical miles), and one for time tions provide little assistance in knowing(usually calibrated in seconds, minutes, exactly what to do. When using the nomo-and hours). A straight line through any gram or the slide rule, the structure of thetwo known points on any two scales inter- artifac-s themselves make it almost impos-sects the third scale at whatever value sible to fail: just enter the knowledgesolves the D = RT equation. Thus, to solve known into the device and it provides thethe particular problem under discussion, value for the missing term. The relations Done simply makes marks at 3 minutes on = RT, R = DIT, and T = DIR are built intothe time scale and at 1500 yards on the dis- the structure of the nomogram and slide-tance scale. Then one draws a line through rule.those two marks with a straightedge: The With the proper artifact, although theanswer of 15 knots can then be read on the task performer still needs to know some-speed scale where the drawn line intersects thing, the knowledge that is required is lessthe scale. This procedure is easy for every- complicated and less general than that re-one, and well within the limits for naviga- quired with just paper and pencil or calcu-tion practitioners in the navy. lator. What needs to be done can be in.-

Of course, Condition 3 has an unfair ad- ferred from the structure of the artifactsvantage over Conditions 1 and 2. The hard themselves. They constrain the task so thatwork was done in making the nomogram many errors that are easy (and frequent)in the first place. But that is part of the with pencil and paper or calculator are notpoint. This is a very frequently occurring even possible with the artifact. Much of the

14 FINAL REPORT: ONR N00148S-C-0133

computation is done by the tool (or by the plification is an artifact of a commonly as-designer of the tool). The person can get by sumed perspective. When we concentratedoing less because the tool does more. on the product of the cognitive work, cul-

But now consider Condition 5, the use tural technologies, from writing and math-of a specialized internal artifact. A general ematics to the kinds of tools we have con-computation rule (the 3-minute rule) is sidered here, appear to amplify thethat the number of hundreds of yards a cognitive powers of their users. Usingship travels in 3 minutes is its speed in these tools, people can certainly do thingsknots. 4 In our problem the ship traveled they could not do without them. When we1,500 yards in 3 minutes, so we instantly shift our focus to the process by which cog-"see" that its speed is 15 knots. The naviga- nitive work is accomplished, however, wetor need only remove the last two digits see something else: Cognitive abilities offrom the distance in miles: Voila! 15. unchanged power are reorganized in inter-

Now the reader may really cry "foul action with the structure of the technology.play." "This i3 a special example with spe- There are two important things to no-cial, made-up numbers that permit this tice about these alternative ways to do antrick to be used," you might say. True, but algebraic task. First, the very existence ofbecause the rule is so -i;ple, navigators try these five different ways is evidence of a lotto determine distances in yards every 3 of effort directed toward avoiding the useminutes. Navigators are capable of measur- of algebraic reasoning, and arithmetic. Sec-ing their distance every 2 minutes or even ond, these tools and techniques permit theevery minute, but three minutes is more task performer to avoid doing algebra, rea-common, not because it meets the needs of soning and arithmetic, replacing these ac-the ship better than the other intervals, but tivities with aligning indices on scales, orbecause It meets them well enough, and it simply deleting zeros from numbers.makes this computation so convenient.5 These tools transform the task by mapping

it into a representational medium fnSwhich the method or answer is apparent.Artifacts -Amplifiers or TransformersNoetathrquedpcdrsmy

ofCgiinNote that the required procedures mayIof Cognition. be trivial, but they are not obvious. One

It has now become commonplace to speak can scarcely imagine a simpler procedureof technology, especially information pro- than the application of the 3-minute rule,cessing technology, as an amplifier of cog- yet many who use it have no idea why itnitive abilities. Cole and Griffin (1980) works and would never have discovered itshow, however, that the appearance of am- on their own. We return to this point later.

Let us now summarize the argumentso far. The power of cognitive artifacts de-

4 One hundred yards is 1/20 of a nautical mile and 3 rives from the fact that they form or trans-minutes is 1/20 of an hour. So the number of hun- form cognitive tasks. An artifact that trans-dred yards traveled in 3 minutes is the number of forms a task can radically change the kindmiles that will be traveled in 1 hour, which is the of mental structure that is required tospeed in knots. bridge the gap between user intentions and

SThe nautical slide rule and nomogram are normal- actions on the device interface (the world).ly only used when the ship is away from land and the They also create entirely new relationshipsfix intervals are much longer than 3 minutes. When to the world. A cognitive artifact that trans-the cycle is performed on the shorter intervals of I or fonms tasks reduces semantic distance by2 minutes, speed is normally computed by conversionto the 3-minute standard. For example, if the ship simplifying what must be done to solve atravels 800 yards in 2 minutes, it would travel 1200 problem. This is very clear in the case ofyards in 3 minutes, so its speed is 12 knots. navigation instruments that replace

~Lft~K

COMPUTATION VIA DIRECT MANIPULA71ON 15

algebraic ard arithmetic operations with To do the task symbolically, one wouldsimple manipulations of external or inter- have to multiply the fractions together.nal structure. Even the simple checklist re- One way to do that would be to notice %hatduces the semantic distance for its user. the 3 in the numerator of 3/4 cancels the 3Lacking the checklist, the novice must dis- in the denominator of 2/3, giving 2/4,cover the steps that need to be done and an which is easily transformed to 1/2. The op-order in which they can be applied. With erations are easy for one experienced withthe checklist, the task is transformed: read- the relevant arithmetic, but if one justing and following instructions take the thinks about the problem it has the poten-place of procedural reasoning. tial to be difficult (taking 3/4 of 2/3 seems

hard, something like, perhaps the taking of5/7 of 8/9). Moreover, the arithmetic pro-ceeds according to syntactic rules that re-

In a study of weight watchers in Southern spond to the form of the expression andCalifornia, de la Rocha reports6 the follow- have nothing whatever to do with cottageing strategy is employed by a weight watch- cheese. Only after the syntactic operationser. The menu for the meal calls for 2/3 of a have been completed, is the new symboliccup of cottage cheese, but the dieter only form, 1/2 cup, interpreted in the world ofwants to make 3/4 of the recipe amount. cottage checse. It is the sensitivity tc formHow can the dieter computer the proper rather than meaning and the suspensionamount of cottage cheese? In the case re- of interpretation during syntactic opera-ported, the person filled a measuring cup tions that makes it a formal system. For-up to the 2/3 mark with cottage cheese, mal systems are thus liberated from thedumped the measured cottage cheese on a constraints of concrete reality. But the gainplate, patted it out into a pancake, drew an in power Is at the expense of clarity.X on the top that divided it into four "pie The theme of liberation from con-sections" of equal size, and then used three straints of concrete reality is prevalent inof the four sections. the philosophical and developmental liter-

The recipe provides an initial set of ature, and the history of Western civiliza-symbolic constraints, but instead of operat- tion is full of demonstrations of the enor-ing on the symbolic representation, the mous power of formal systems. Yet, thisrepresentation was interpreted into the poses a computational burden on people inphysical medium of cottage cheese. This the requirement for syntactic knowledgeprovided a mapping of the symbolic state- and mediation of the formal operations:ment of the problem onto manual action We call this the burden of syntax.in the physical world. In reviewing this As observers and actors we may certain-and related examples of real world arith- ly intend what we do in the world and in-metic reasoning, Resnick (1987) notes: terpret the consequences. But the world i'-

self neither interprets nor intendsThey got reliable arithmetic results by consequences. We take the point of view oftreating the material tl•ey were working an observer, interpreting actions and states:with as part of their calculation process, The world is a system with neither repre-rather than by just operating on sym- sentations nor interpretations - the worldbols. (p. 14) simply responds (this view is presented in

expanded form in Gaver, !988). Thus, al-though our acdons have meaning to usand to other observers, they do not to the

6This account originally appeared in a chapter by world. The notion of "the meaning of anLave, Murtaugh, and de Ia Rocha (1984). expression" is a linguistic notion. It implies

16 FINAL REPORT: ONR N0014-85-C-0133

a reference gap - a relationship between is not possible to try to remove a le thatone thing and some other thing that it does not exist.8

"represents" or "stands for." The referencegap in turn implies an interpreter, an agent The Design of Appropriatelythat can bridge the gap and make a map- Constrained Artifactsping from the expression that does the rep-resenting to the thing that is reprewented. The symbolic world is an artificial world. IfThis reference gap does not exist for events syntax were the only constraint, the protec-in the world. It occurs only when there are tion against the impossible and the non-symbolic relations. This difference of per- sensical is weak because it places a heavyspective provides two apparently incom- burden on the knowledge of the user andpatible realms: a realm of action and objects the correctness of the operations.and a _,alm of symbols. Computers pro- The weight watcher had two means tovide -as a way to put these two together, but solve the problem - by manipulatingit is difficult to describe from the perspec- symbols or by manipulating the cottagetive of either realm. 7 cheese. The culturally valued technique

From the perspective :f the symbolic (taught in schools) is to do symbolic ma-realm, we want the expressions in the in- nipulations on the symbolic representa-terface language to be composed of the ob- tions. The preferred technique of people isjects and actions in the world cf interest, to manipulate the actual objects.From the perspective of the realm of ac- Given this apparent conflict betweentions and objects, we want the objects and cultural and individual preferences, thereactions in the world to be the elements of are at least two roads to take. One is tothe symbolic expressions that refer to them. mechanize the manipulation of symbolicThe key to the sensation of directness is structures. That road leads to the comput-that technology permits the design of an in- er. Another is to stay in the world of man-terface-language/model-world that can ap- ual manipulation. This second road leadsproximate this state of affairs to an extent to a special class of computational artifactsimpossible in any previous medium. in which the syntax of the represented do-

By presenting the world of action as the . tin is built into the physical properties ofinterface language Its•lf, we can collapse the representing object. But how can we getthe symbolic reference gap. And if the mod- syntactic constraints out of physical proper-el world is properly designed, it can be !he ties?kind of artifact we sought. Consider a sys- Locking the door to your car with thetern like the Macintosh where files are re- keys inside is an annoying error. One couldmoved by dragging icons that represent imagine a checklist for leaving the car thltthem to the trash can. From the point of could prevent this sequence error, but thenview of our analysis, what is especially nice the checklist would be a nuisance andabout this interface is that the physical con- would likely often be skipped. There is,straints of the world of file icons make cer-tain classes of syntactic errors impossible: It

8 Unfortunately, the collapse of the reference gap b3never complete. There is still the mouse protocol and

7 Gaver (1988) describes exactly this dichotomy, but the key prerses instead of getting our hands on thein the context of distinguishing between cognitive and objects. Even touch screens leave a gulf. And the ob-ecological (Gibsonlan) views of psychology. Gaver's jects of interest may or may not actually be the objectwork provides a valuable step in the argument we of interest. What the object of interest is may dependpresent here. upon the user's goals at any moment.

-- - - ____ ~~ fl m7SFW.S

COMPUTA110N VIA DREC"T MAN.•utLATncw 17

however, an infallible way to ensure that the trick - symbolic, numeric computa-keys are not locked in the car. That is to tion is required.make the car door so that it can only be If a representing world Ls necessary, thelocked with the key operated from the out- design question for the artifact, then, isside. Norman (1988) called this type of how to achieve the appropriate constraintsscheme a "forrIng function," for it provid- i, the representing world so that all possi-ed w, physical constraint so .tat failure at ble actions model meaningful actions Inone stage prevents the next step from hap- the represented world (and so that non-rmning. meaningful actions are not possible).

Sequential dependencies among actions Some artifacts, such as the nomogram,imply a kind of syntax of action. Doing the exploit the fact that they are physical asaction in one order is a welt-formed action, well as symbolic. The usual analysis ofand violating sequential dependencies pro- symbols ignores the fact that symbols areduces a malformed action. But sequential also physical things: The physical proper-relations are only one kind of relation that ties are thought to be irre-levant to theircan be constrained. T"here are other sorts of symbolic, computational, or representa-srtax of action. Insrting a disk upside tional properties. Certainly, the manipila-down or backwards in the disk drive is a tion of the symbols has nothing to do with"syntax" error that ca& be eliminated by 4.e- the constraints of the domain of inter-est.signing disks and drives with physical con- In fact, for many purposes the symbolsstraints that make only meaningful actions need not be thoaght of as things at all. In-possible. Now let us see how to carry this stead, thkoy are treated as if they exist inmethod to the deign of artifacts so that their own realm, separated from the realmphysical constraints preclude erroneous ac- of "real" things. rEternal representations oftior.s. arithmetic are thought to be helpful only

intelligent artifacts are "representing because they ease memory load, not be-worlds": That is, they forui a representa- cause there is anything in the physicaltion of some aspect of the external, physical structure of their external representationworld and do their ope. atioius upon this that is essential to the computation.internal, representing wo-Id. If th2 artifactis to be readily usaoiu., the constra.n-.ts of therepresenting world must have a meaning-ful relation-hip to the constraints of the We now have on hand the conceptualrepresented world. Furthermore, there pieces required to construct the new kindmust be a reason to do things in the artifact of artifact. First, select the objects that arerather than directly i'n' the world - wvhy to serve as the symbols in the system- Notemanipulate the revrv:sentation if one can that these are bot, objects and symbols -manipulate the real thing? In the case of they arn o-symbols (object-symbols) Thesematl ;matics, the reason is otten that some o-symbols serve as the interface betweenimportant aspeN t of the represented world the world of objects and the world of sym-cannot be direLdy operated upon. In the bols, each being simultaneously in bothcase of the cottage cheese example, the ma- the wor.d of objects and of symbols. Now,nipulations of the real object are messy, al- it, the world of objects. the items can ex-though still do-able: Other kinds of objects ploit the powers of forcing functions, us-are not apt to oe as susceptible to manipula- ing physical constaints in such a way totion. It, the case of computing the speed of force the symbolic interpretations to bethe ship, no physical manipulation will do meaningful.

18 FINL URrPf1 MRK N0044C-850133

The nomogram for cormputing boat speed producing syntactic violations. This, we be-(Figur 2) was constructed in this way. The lieve, is what is at the heart of the attrac..physical constraints for the layout of the tiveness of direct manipulation interfaceslogarithmic scales (whicl -re the symbolic to computers.representatione for speat-, distance, and True, DM interfaces have another attrac-time) make certain classes of syntactic er- tion: the naturalness of the mappings be-rors impossible. tween the represented and representing

worlds. But this very naturalness also pro-

Reinterpreting the Attraction of Dit vides an opportunity to be exploited by

Manipulation Interfaces making the manipulations of the repre-senting world follow the physical con-

We are now able to return to the problem straints of the represented world. This isarea that triggered this investigation: the one reason why the visible file folders met-study of direct manipulation interfaces. aphor for organizing computer files can beConsider what one wants in an artifact. so appealing. But it is not just any repre-One wvnts structures that simplify the task aenting world that works this way: DMfor the person, that permit new computa- works when the physical constraints oftions, or that produce new interpretations, manipulating the representatiorn prevent itall easier, faster, and more accurately or from representing anything nonmeaning-precisely than could be done without the ful or impossible.artifact. And a major enhancement of the We have now considered several modesartifact comes when it is so constructed that of computation that turn out to fill theone can't do wrong, where the structures four spaces of a two-by-two array: One di-by their very construction are incapable of mension represents the kinds of con-

straints - syntactic or physical; the otherI I I dimension represents the kind of actionPAtIIaV Actts done by the artifact - passive or active.

- n The combination gives the four computa-tional modes that are shown in Figure 4.

wor REFERENCESCheese Bly, S. (1982). Sound and Lomputer infor-

COMM"published doctoral dissertation, Law-rence Livermore National Laboratoryand University of California, Davis.

Cole, M., & Griffin, P. (1980). Cultural am-m-ga- __ T'-o' plifiers reconsidered. In D. R. Olson

(Ed.), The social foundations of lan-

guage and thought. New York: Norton..iVII4 Gaver, W. W. (1986). Auditory icons: Using

sound in computer interfaces. Human-) 4 Computer Interaction, 2, 167-177.ti Gaver, W. W. (1988). Everyday listening

-J and auditory icons. Unpublished doc-toral dissertation. University of Califor-

FIGURE 4. Four modes of artifacts. nia, San Diego.

COMPUTAlON VIA DutBcr mANiPuLLATON 19

Gaver, W. W. (1988). Sound in the inter- Norman, D. A. (1986). Cognitive engineer-face. (Technical Report 8891-WG1). Cu- Ing. In D. A. Norman & S. Draperpertino, CA: Apple Computer, Inc. (Eds.), User centered system design:

Gaver, W. W. (in press). The SonicFind- New perspectives in human-computererTM, a prototype interface that uses au- interaction. Hillsdale, NJ: Lawrence Ed-ditory Icon•s Human-Computer Interac- baum Associates.tion. Norman, D. A. (1988). The psychology of

Hill, W. C. (1988.) The mind at A4: Horse- everyday things. New York: Basic Books.less carriage, mouse trap, card trick, Owen, D. (1987). Direct manipulation andclock (Tech. Rep.). Austin TX: Microel- procedure/ reasoning (ICS 8801). La Jol-ectronics and Computer Technology. ia: University of California, San Diego,

Hutchins, E. (1986). Mediation and auto- Institute for Cognitive Science.matizatlon. Quarterly Neusletter of the Owen, D. (1988). Directly manipulatingLaboratory of Comparative Human data: An implementational study of di-Cognition, 8(2), 47-58. red manipulation (ICS 8801). La Jolla:

Hutchins, E. (in press). The technology of University of California, San Diego, In-team navigation. In J. Galegher, R. stitute for Cognrive Science.ICraut, & C. Egido (Eds.) Intellectual Resnick, L, (1987, Decmuber). Learning inteammork: Social and technical bases of school and out (1987 AERA Presidentialcollaborative work. Hillsdale, NJ: Law- Address). Educational Researcher.rence Erlbaum Associates& Schustack, M. (in preparation). How inter-

Hutchins, E., Hollan, J., & Norman, D. A. face style influences user performance(1986). Direct manipulation interfaces. (NPRDC Technical Report). San Diego,In D. A. Norman & S. Draper (Eds.), CA: Navy Personnel Research and De-User centered system design: New per- velopment Center.spectives on human-computer interac- Seifert, C., & Hutchins, E. (1988). Learningtion. Hillsdale, NJ: Lawrence Erlbaum from error. Unpublished manuscript.Associates. Shneiderman (1982). The future of interac-

Lave, J., Murtaugh, M., & de Ia Rocha, 0. tive systems and the emergence of di-(1984). The dialectic of arithmetic in gro- rect manipulation. Behavior and Infor-cery shopping. In B. Rogoff &J. Lave mation Technology, 1, 237-256.(Eds), Everyday cognition: Its develop- Simon, H. A. (1981). The sciences of the ar-meat in social context. Cambridge, MA: tificial (2nd ed.). Cambridge, MA: MITHarvard University Press. Press.

Mansur, D. L (1984). Graphs in sound: A Sumikawa, D. A., Blattner, M. M., &numerical data analysis method for the Greenberg, R. M. (in press). Earcons:blind (UCRL-53548). Unpublished Mas- Structured audio messages.ter's thesis, Lawrence Livermore Na-tional Laboratory and University of Cal-ifornia, Davis.

Mezrlch, J. J., Frysinger, S., & Slivjanovskl,R. (1984). Dynamic representation ofmultivariate time series data. Journal ofthe American Statistical Association. 79,

K ~34-40.Morrison, R., & Lunr.ey, D. (1985). (contri-

bution to panel on communicating withsound]. CHI "85 Proceedings. 118-119.

2D F•IAL RPORT. ON N0014.&C43

Appendbi Draft Checklist for the S. Redo the task and system analysis,Design of an Artifact Are all meaningful operations possible?

Are all nonmeaningful operations impos-It is tempting to use the analyses of this re- sible?port as design rules. This operation is pre- What kinds of social interactions are sup-mature for the full understanding of cogni- po.ted? What kinds are made more dif-tive artifacts and, in particular, the o- ficult?symbol, do not yet exist. Nonetheless, it is Does the user operate directly on the o-possible to provide a draft checklist of de- symbols?sign procedures that follow the principles Are the affordances right-does the first-studied in this research project. The princi- time or casual user need extra guld-pies include those discussed in this report ance?plus the analysis of human action into sev- Is the feedback right? Car the user tellen stages of action (Norman, 1966, 1988) and what has been done?an emphasis on the p'oper role of con- How are errors:straints, affordances, and mappings (Nor- Discovered?man, 1988). This checklist needs elaboration Recovered from?and verification, but it illustrates how the Corrected?work conducted under this research con-tract could be applied to the design setting. 6, Watch typical users in a realistic situs-

tion.L Do a task analysis. Can they use the artifact appropriately?What needs to be done? Do the o-symbols work properly-are theyWhat are the variables to be controlled, to interpreted properly?

be determined? Does the artifact fulfill the design goals?What operations or actions need to be con- Do they wish to use the artifact in ways not

trolled? fully supported, such as:What feedback is needed to the user? For repeated operations, keeping someWhat errors are possible? previous values?

In a different order than contemplated?2. Do a system analysis. For a different purpose?Where does the task fit in the general over- Etc.

all picture? And, if so, does the design support theseHow do the various participants interact needs?

with one another? Do they make errors?What are the hidden goals of the task? Do they discover all their errors?If the task or method of doing the task Can they recover (gracefully) from error?

changes, what side-effects result in the Would they stop their current methodsrest of the organization? and switch to the artifact?

3. Select the o-symbols 7. Repeat design effort from 2, with empha-sis on all "no" answers, to 6.

4. Do the design.Develop the forcing function.Make operations visible.Make results visible.Get the affordances right.Follow the seven stages as design guides.

0 0 n

4A -,n

64 r- &A f

* ~bd ji; as" i~~ l . j

4 4A . P1 -

o~~~ a44.

:: -, .~ .- 'o ,r I'm9 E

'M !a ¾Lm M. a 'A 0 'A

dc a 04 3. In.. a~ 3.4 C

i

42A'00

-Is

C a ',0-.4 .4L

88*-f4 act-

,n

4)Ul

r-0 age w r-0 , c = b. CD 0 0%

0 C w 0 j N c , = dc pi ti ol 0. a 0:3 . $4 ý 0 ý 0 > u

0 0 00 0 to'A u U 0

mý4 c m L) c In u c

4 .4 Ad >. 0 Ch c U)% 1n40 0 to 0 6. 'm 01 .1 11

cc w 0 U IU du14 0 .1 Ic

Im U 2 jd a,0 6,In 4 0 0 0 u -0w 14 C u 14 1. 4; .1 wo C; co

CA 44 j N a (" CK 0 a a0 10 0 C 11 li 1ý 0 C 6 lz 0) - 0. j Qj 0 w V jtft 0 A W u ej a c o- X u N w

z a Ic m -U a 's. 0 AX 0 AC :0 li- c r-.- 0 60

w 0. 0 n a- W =C,3 6. c a v W C: Uto w 04 L4 =a0 z U (n Q Ic -C In in P a p im cn 14 t. 0 U 0 =0 2. aw 5 a tun -a

r

a 0A410

m 0 mm m 0. 0 en 01 w ICc 24 In a cr 0In %4 0 0 v 6 Q N u N 1 m oj 2

- cq U,, z C4 to 0 no0 11 a r 41 c u o r- c w 1 0 0, 1-0 X, >M' 06. 0% >. c 6, 0%

Z, r -2 00 -t; 0In I 1 %4 M 14 1 M H

rz O.C N 0 0 W :1 0 U u . 10 ICw c 10 v x 0 94 0 CW - 41 to a w 0

0 4 c 0 U id ýj cu C 64 b. a, H w060 x t, 'E = = 0 D. c 0 6. 0 ),z x .8 IS.0 4 0 617 .2 a : 0 c 0 0 m C4 a 0 c 0 40 N 0, 10 'rfj Vw ý W " 'a : r lz In a , - = z 114 a Vý a 0 0 W L. 4)0 en 0 li r. 0 m 6. 0 - -7 0 0 wC 0 0' ýO 4 9 0

8 U 0 11 'A C m -1 . 0 U U ý C M- X f. 90 0 c w o 0 c 08 2 1 1 1 om wq 'j W' 11 r 0Ic

01 14 ý 'ji Z 0 4 al

(A M CK ta 01 1. x 0 x v a 09 1. 0 0 a a U c 0 In r- 94 of.

0In0

c0

tp rID

r6. In41 Aj j :3a 0.

cc >4 c02m 10, 40 0 - N -- olC4 > w - M 10 r-ON c 0 -C

0 0 j r0 a r 6m. w licc 0 0 C6 In 10 A. 6. 0 u c o wCD - 6. 41 0 'S '1 41 ý - 100 a 0 - r)c- m 0 0 0 0 C >. C 6

Cý 41 r U 0 0 1-1 ON 10 4 0 - w M- 4, c>. im D. C>0 6 Ic me >7 Ci r- ro- a 11 No MW 0,41 410 W $4 Ol :.. ý ý W Q L. 0 4 '1 0 0- u IN 'a 0 m 0 N2 0 4 - 0 w Qj V 0 c 0 wo 2 j 0 0A c jýo 'o, 4 10 o D 0 '1 0 W t -Q 4

Imi cl 0 W 0) j 0ix 0 0 =0> . . w 0'Q 0 M W W 0 0 0 W 14 'r W Cl M w 0 c 4 w 4) 0

0 U a r. >- U)ti v 0 m 0 Ic m Ln

(4 u N a I.:M 4) 0

u 0 140 M I w 'A 0 > u8 c 0) -C - r 00 0 m 0 w w 0 4).- W Q, . Oý c 10 4; 0 = W -0 t4w r C, 0w - w 14 a, 61 Iwo "o a, 0 wm

0 41 0 0' 0 C 06 x 0 m 0 r 4)C. w c ýD " c 4) -0 - 91 m 0 0 w 0 >, u0 0 w = z 41 u 0 0 14 C. 0 - 10 li0 L. 01 OD

L. 0 r w U. 0 c . - "D 14 Cý - > m U U co . . . - L, . - U .3 do 'Ali ý4 C. ý4 0 ý m 0 ý4 ý c 40. C. u w C 0)4) 11,10. 1 0 0 0 0 0 06Ch V 4; 4 > 01m u m c a 0 >1 Ukw Ul 0 4 c '1 0 4, 7 w Cý 'I 'i c j C j:p W 0 c 0 x 0 w 0 C tP

= E.0 w to 5 W" W 0 uz t, " w m 0, 4 j 2 w w .0 dl W IJ M W96 C u C x I w U w V w a x 41 14 r U ý4c 0 A c 0

CPC4 g v (ý w w C6 '1 06 z C6ý4 0 U*, 0 li w CL 1:0, -W -"w 64. ci a') ý c 4) m w w c Q 0 4)Ic fA (n x M x w U Q -C - -C a m a co 0 , 0 a am a , D in = 0 m a x m m

* U

0 0

Nu -WNC 4 - k.

01 1. 0 0.O 4

4, 44 2 31 W, 0 04 6- 8CI

c -, 4. 04 a A0U1.%14 A- 444 0 C 400

0, LP W4, a 99. 9. a4 -0 I.- ~0 44 .44 .4 6 06CM -0 C3 414, 14 .QN C-

4, . 0 4 0 4J C, O N 0 4 4A,4 , U 04- 4

464,4. 4, 0 U M8 OR a 461 Wl o a I.5 Zd VC 01's I"a WmA. 4- 4- a. A A

04 ~ § ~ m u ~ 4 ~ ~ 4b N 0 3 0 4

444~~~00 A44 ~ ~R42. %4 .14 ,

0 0. a -0 04, 0 14 4 4 0 44- In~ 0 W 0 .o

r-40. 04 Udl UO- 44 441 4Iow a-i MW 444 em4- 2 00 Z

o

m44 0 .3Xc C 64, 0 lo o a R - U a U0 w

Ccz-3 - a @,4,0 na'" 1, 0 t. 9KW o j "

4, Wic 4, c CU c440

Q44 04 'a"0 a4~4 u0 ... q - . .4 ,-..4

C 44OM4, N SO 44-

U a -4 U U v W40a W44 U00 4, (44, $4 a 0 0 aC 00 0" . 44 0. @0 N .I

N4I UG 24,4 a04 :'4 I40 aQ V" 04 6j 440 " % M44ND'44C, FA 0 to A00 406 CU - 0 0~I W(4 4

@1. am44. 0 4D c 0- @U'4 44 4,,.444 4l.4C~4~ .444 44 4 C .. ( .4 U %4

4444 W 0O 0 Z.4 O. c U a~3 0 ~ I c0~ 0 0 00U 0-44

0

o a-5 , I- - 1 0

Q- tC - 614E' ýV4. 00 -4 1.

4, 4. 0 V

u 04

0464 W 113Wa l l~ ~ I04- 0 36, 4C 00 -0404- ,00 0IS N . N- 0 0

In z1 0%.4 - . .* ~ 4. '4,04.4 440 @ ,3.4 -4g.~04 4,04 C,~44C 4,0 a,,4 .CI 4

4,4404 U "0 In-4 ~1 4 ~.0 04 cX 'A @ a c 40- %446 qa C4 4.-0

3 4444 ml 0 0..04 mzm4 44444a 04 '46 144m4.C414 0404 444 a110 444 c 44444) 44.40.

4iC

f, W 0 4 .4000za

4, "4r444, 0N aC*1 44.1 a 104 0 04L -4 4ýc0u4,c 0 04

0 -, '4 aOW 0 14 L.4.

4.0)W U Al . a 0 4, c c in c @0c c 1m 4Wvto c0 W ~ 4 a1 In c 0 L. 14c

>140 At.44mm N 0 -m

CU W4 04( 0.) 4404 0) W,4 c 4, N N O94. 0 z t.444 I-40. -C 1. 0 44 04 4 W N 4 44 a 4 C) 0 U4.4.

,n

00

Al. I .ý ý

ft~ 0. 0 00 946 A4 C4 Wn ' , 0

4. 6 4 *4 0- wf . . f I* 4i A 'A 0 2 04 0'. 43 N1 w4 A w4~ 44

*M 0 N 0 - 0!.4- 00 '

b~t2 1-4 12U -4OI6 12 A 0

jt .2 Ogg ,41,,a

96 a -V r 0.0 aO N (9.4 A a %

00

0 t,

4.1 a4 00 4C 4 0 01 " a* 0j 1

w4 0 0 NoC4 a 4j V.U3 0 a .0 m 4. 0-

CA ea. um aS.n W) a, 04a40 X 4,00,to a- - 0 t40 9

3. 0 .w 04 ,C 0 00W*.C'IxA cz~O - 0 IS W-.' OVUS 4aý. U 0j - r 2

4460. 0 04 ' rA4 w4 1141 c IA 12 4Za64a0 0: we on 2 w'44. 44. w4 .0 w3 04. w.4In

0 t

0

@4 4

cc 0 Z C'2 0 c.*U 'i'4

' In 0 U j i .n 0 an UO41 in40 4

N ~ ~ *44 44 N .14 -4 W: 4 :" e4 * 0N4 " IN 0 . cm1 aC4 a wC

D4N rU '4N Do. IOU 0 *N u444.4iAn ~ IN 'I "'A IA C4 44 4OC N 4InN cc41 UNN 41 id 4'4 IA U . 1 C* 4 444 404 4 4 c 4..0 40 40 ~ 4

UNC- W4 RO 0U 'sn W0 24 04 =M~ 8C ', CO =..-U 044U 4..U AlI S 4 6.40 66%4~.4 4 1 44. '4 4 2 442 b >

.1 -'-4 .4 a4 01 0.0 a .400 99 .4.M 4

%4 4 00. 4 44 0 44 % . 4 .4 4. - .4 40 0 0 40 >1 %404 ~ ~.4 0 w4 0' *4 00 ~ 0 0 . h A 4 4I 4 0..

49

0

41 c

.4 0 iIn 0 C1 0

0 '4 0 399 0 1 4 '4O P 0 44 '4 -

'4 0n '0'4 0 41 0 &. 0 0 60 44 0 w &J vs

r 0 4 n 0 44 41 0 '4.4 a 4.40 a N

-0 - N N a.-4 N N In Q 044 4 U444- q cc.I

(A C4 44 h 0 U.' 0. . 0 r4C 4 4

c u 4w L40 1.4 B.0 0 * -450 go5 04 >o*..r I~e~ a45~ '4ýU r0 144 11,41 '444 0 4 0l 00

V 0C -0 li6uý 0 '44 %. As.4 13 CL; 1 04 '46 au 0 -4 4.4 D . W0a n4'a~ UO. tB 0 U 0.0 4 0 W IC14 &JOU SC U Z'43. 6.4 a4 41 C'4 C '4 & .0U C3 c~ W4.. '4 0.-. 4.0 0, a4 o~

'Am Is .4 M - 4 .4 4 044 m gI '4I. C-41 40 4 4 0'U, c A .4 uIW E-.Z 0 lu 05 0-00 W0. 0 4 0 04

0C

*00

0 0 V"Co r t Cfn C Af a 'a -

&* 4.4 VW 0 4h c.W 0 41A14 14 f

A 4.4 444 - 0900r-

64 $ 4 4 ad 41. 9 H N 04

a~ 0... -.44 o. OwR ~6~S gm14 4 4 0 CIs. 4 4 . 6 U S

414

a Sn a 24

0 64 a4

r~ r WC- .04 A54 44 cC -4

Ad og.. 0- 0 c 09 w 140-," 0 ;1 N N A4 40 r

N U 6 00 9 '0 0 8 we. 4 * 4 .r-~~~~~~* 60 Nt .60. ON0 4 -C4 - . 0 6 ~

03 044 ~ go~ 1.4 044 N 6 0 ~ U a- . .

CA 0ý ; I. ; 4.C.z. . _ aaCI . ;0 4 .44 1 6 6-.. *4? 4 14 6 ", ZO- . ad0 o 4 0 . . 46641044J tP - M1 0 a V .41 440 p1 41 'N -4 '9

t:4 2. 1t. CO 41-4 - r 1,4 1.4 ., OMO

644 1. X 41 940"0 I4148D 6 0l I 0'A C 0)- 84C444 -9 s6 C 0 .. 2id0 *c *64 00 0. '0. U C 0 4c004

So

464

C 60 fn0

A4 . 0. U 144

N46 c 41 -40 %cn LO 41 in %04N 4 .0 a9 0 --1 0. 604 00c"4- 0o 0S N

- a6 04.4.%4 4 640 Ad I.O 00 C1 *N 6 0 N w6 04 44

r44 .. l0 44 0- 0 0 4106 14 W0 A)4 CA O C14.4 k .6.* 6 w~ 0 C64 .8 0 4 6

w 06 a9. .46 w 0 -4 0. r w 00 2" 14 4.6 * *4 4t 4 a -94 0. a u4 W.40 =0 M.066 U64 k414 k W [email protected]~~ 14 t S 6~40) 0 ~ C2 m )0A.

44 C'A s.S 44f 10 04 10 4 U 4 1 . 4 1 1 40 0 a~~ 440 44 O.4 4 0 to -4 0 0 4 1~ w)0 . 02

t: 0ýCON C4 C,-& 0Ql

4. w041 L 0 C4 0*00 - 0 t40t"w0, ,o j 0C9 41 c1. 0 C .>1 64 f - o C

C3 4.4 a0. ..4 -0 4040 t3 w . 0 4116 W"' 41 41 11 6 O6 41

S 41 N 4 U1 044O . -w4 :0 01. 4 0, .

00 U -0 .0 U1 e 60 . 0 a41 Ai 06 10 44 0844444.4~ ~~~~ ~~~ 'A COU 41004 44.4 - 6N 4.4 4

.6 446 -4140 .4..0.-0 1 6 6 0 ,.44 ... 41 *0 0 "4 h-..46 0 c 6-4 40 w' 6 X L; In

t40 .-4 w40 1 00 0 1 U0.~. 0. 4.4.4064 444 30 Q4.4-46 6~.0L w681 -4 444 1U4

rm a CA 4.6.0 '-4 414MO m .46 u 64 400 4 A -4 r a., a-, a4

0 $

00Up

0