Manfred Kochen: In Memory of an Information Scientist ...garfield.library.upenn.edu/essays/v12p166y1989.pdfKochen M, Cohen L & Wrdff Y. Information systems and clinical resesrch by

EUGENE GARFIELDINSTITUTE FOR SCIENTIFIC lNFORMATION@3501 MARKET ST. PHILADELPHIA PA 19104

IManfred Kochen: In Memory of an

Information Scientist Pioneer quaWorld Brain-ist

Number 25 June 19, 1989

For the sake of historical accuracy, it isimportant to point out that even before wepublished our recent essay on citation be-havior, 1 we had planned to reprint the fol-lowing paper by Manfred Kochen. z Then,while preparing an introduction to appearwith Koehen’s paper in Current Contentsm ,

we Ieamed that he had died suddenly in Col-orado. We have been deprived of a dearfriend and of the chance of conveying to thisdedicated scholar this further evidence ofour esteem for his work. Perhaps his fami-ly and friends will derive some small solacefrom this posthumous recognition of his con-tribution to science.

Biographical Data

Fred Kochen was born in Vienna, Austria,on July 4, 1928. After narrowly escapingthe threat of Nazism by boarding one of thevery last boats to leave Lisbon, Portugal,Fred and his family eventually made it toNew York City. In his adolescent years heattended Brooklyn Technical High School,where he was awarded a Mechanical Coursediploma in 1947. He met his future wife,Paula, in New York as well: they weremarried in 1954.

Fred’s education included a BS (1950) inphysics from the Massachusetts Institute ofTechnology, Cambridge, followed by MA(1951) and PhD (1955) degrees in mathe-matics from Columbia University, NewYork. He was an anrdyst with the John vonNeumann computer project at the Institutefor Advanced Study, Princeton, New Jersey(1953- 1955), and continued his postgradu-ate work at Harvard University, Cambridge

(1955- 1956), where he worked on mathe-matical models in the behavioral sciences.From 1956to 1964, he was a researeh math-ematician at the IBM Thomas J. Watson Re-search Laboratory, Yorktown Heights, NewYork.

From 1965 onwards, he was a researchprofessor at the University of Michigan,Am Arbor, and from 1970 to 1989 he wasalso a professor of information science atthe University of Michigan Medical School.While he was a professor at the MedicalSchool, he was (1981-1989) an adjunct pro-fessor of computer and information sys-tems, as well, in the School of BusinessAdministration.

Even before his collaborative work withvon Neumann at the Institute for AdvancedStudy, Fred had a long history as a consul-tant. Between 1948 and 1955, he held part-time consultancy positions in spectroscopy,aeroelasticit y, and photogrammetry. In the1960s he was a consultant to RAND, RCALaboratories, the United Aircraft Corpora-tion, and EURATOM (Ispra, Italy); in the1970s, to the Advanced Research ProjectsAgency, the National Institute of Education,the Science Center of Berlin (Federal Re-public of Germany), and the National Sci-ence Foundation’s (NSF) Division of Sci-ence and Technology. Just last year he wasa consultant to the US Library of Congress,Washington, DC.

Fred Kochen authored and edited over 250publications (including eight books), primar-ily in information science, artificial intelli-gence, and the behavioral sciences. Table 1presents a selected chronological list of hisworks. He was also a member of several

166

professional and honorary societies: theAmerican Association for the Advancementof Science (AAAS), the American Mathe-matical Society, the American Physical So-ciety, the American Society for InformationScience (ASIS), and Sigma Xi.

The direct connection between Fred andISI@began in 1972 when I first came acrosshis paper on a proposed world informa-tion retrieval system~ (which he termed“WISE,” for’ ‘world information synthesisand encyclopedia’ ‘). At that time, I waswriting “The World Brain as seen by an in-formation entrepreneur,’ ‘b for the AAASmeeting on Reorganizing Information Re-sources to Improve Decision-Making, whichwas held in San Francisco in February 1974.H .G. Wells’s concept of the World Brainshad been an inspiration to me early on, andI was delighted it was being recognized assignificant by others. Fred and I corre-sponded from 1973, but our contacts mayhave been even earlier. For example, Fredwas closely associated with Merrill Floodat Columbia. in the early spring of 1954,I had discussed the possibility of taking mydoctorate with Professor Flood, and, a fewyears later, Fred had many contacts with myclose friend the linguist Casimir Borkowski.

Kochen the Man:Some Colleagttes Remember

Whetr Fred Kochen was nominated to bepresident of ASIS back in 1986, he inter-preted his discipline in these words:

I am an informationscientist.I interpretit very broadly. For me, it includesthestudyof howbrainbecomesmind and ofthe evolution of social organs with mind-like properties, such as scientific commu-nities; how to design and use computer in-formation systems in business; and newroles for information professionals as ref-erential consuirants, catalytic brokers, andchief information officers.6

That definition of the information scien-

tist is worthy of continued discussion in whatmay never be a settled debate. But definingoneself as an information scientist only re-veals so much about a person, espzirdly one

m many-sided as Fred. Feeling that my own‘emarks could not adequately describeFred’s varied contributions, I asked several:olleagues to share some of their thoughtsibout him.

Henry Small, director of corporate re--ch, 1S1,recalls his association with Fred,%peci~y his involvement in clustering tech-~iques and science mapping. (In fact, Fredhelped organize an NSF-ISI sponsored con-ference on clustering held at 1S1in Septem-ber 1986.)

Fred was very interested in getting a han-dle on all human knowledge. Some of hiswork, for example, on discovery and cre-ativity, was quite original. From my viewhe was more a theorist rather than an em-piricist. I remember hlm telling me thathe had anticipated co-citation and someother techniques used here at 1S1, On apersonal level, he was quiet, and to manyhe may have apyared introverted. But hewas like Derek [J. de Solla Price] in thathe was inspirational and had the uncom-mon knack of bringing people together fora common goal.7

Robert Lindsay, research scientist, Men-talHealth Research Institute, University ofMichigan, had this to say about Fred andhis connection with the World Brainconcept:

I knew Fred for 30 years, having first methim about 1958. 1 wasn’t intimately in-volved with Fred’s work on the WorIdBrain notion. However, I think that it wasa sort of guiding principle of his, ratherthan a specific project. A variety of thingsthat he worked on were directed towardsachieving information systems that wouldmake information readily available to peo-ple who needed it. He worked on prob-lems of computer-based matcbirsg systemsand information retrieval, and the’ ‘smallworld” problem. He approached theWorld Brain notion indirectly through allhk work. He had an enormous range ofinterests, and breadth of knowledge, andbrought great energy and humanity to hkwork and to his personal relationships. s

As a teacher, Fred was one of the best.According to Wyme Chin, one of hisstudents:

167

Table 1: Selerted fist of wnrks by Manfred Kocherr arranged in chronological order.

Kachen M. Extending the humsn record. 16d/. Arm’. SOC.Infiwrn.Sci. 14(6):25-6, August-September 1988.Srnit P H & Kwhen M. hrfomrstion impediments to innovation of on-line database vendors. h@rrr.

Process, Marrage. 24:229-42, 1988.Koehen M. Advanced infomrstion technology and small manufacturers. Bull. Amer. Sot. Inform. Sci.

14(4):24-7, April 1988,Kachen M & Hastings H M, eds. Advances in cognirive science: steps toward convergence. Boulder, CO:

Westview Press, 1988.283 p.Kachen M. How well do we acknowledge intellectual debts? J. Dar. 43:54-64, 1987.Kochen M, Lee C & Weattand C. The adsptive man-mnchlne non-arithmetic information processing system

revisited, (Williams M E & Hogsn T H, ccfs.) Proceedings of the Natimral Online Meeting, 5-7 May 1987,New York. Medford, NJ: Learned Information, 1987. p. 253-9.

Kochen M. Ethics and information science. J. Amer. Sot. inform. Sci. 38:206-10, 1987.Kochen M, Cohen L & Wrdff Y. Information systems and clinical resesrch by residents in internal

med[cine. Merhadr Inform, Med. 24:85-90, J985.Kocben M. Infomratinn science research. The search for the nature of information. J. Amer. Ser. h@rrr.

Sci. 35:194-9, 1984.Kneben M. Towards a paradigm for information science: the influence of Derek dc Solla Price. J. Amer.

Sot. inform. Sci. 35:147-8, 1984.Kuehen M. Information and society. (Williams M E, ed. ) Annual review of information scierrce and

technology, white Plains, NY: Knowledge Industry, 1983. p. 277-304.Kurhen M, Crictmran R & Bbaivaa A. Distribution of scientific experts as recognizd by peer consensus.

Scientometrics 4:45-56, 1982,Kuchen M & Blaivas A. A medel for the growth of mathematical specialities. Scientometrics 3:265-73,

1981.Knchen M & Derrtach K W. Decenmalizarion: sketches toward a rational theorv Cambrid~e. MA:

Oclgeschlager, 1980.384 p.Kuehen M, Dynamics of scholarly communication (Bcnenfeld A R & Kazlauskas E J, eds. ) Communicating

information: pracetdirrg$ OJ (he 4.3rd ASIS Afmrad .Mee/ing, 5-10 Octoter 1980, Anaheim, CA. whitePlains, NY: Knowledge Industry, 1980. p. 233-5.

Kachen M. Can the behavioral sciences contribute to the foun&tions of infomration science? (Manin S K,ed. ) Irrforrrwion polifics; proceedings of the 39rh ASLS Anrusd Meetin8, 4-9 October 1976, San Frrrrvciaco,CA. Washington, DC: American So-ciety for [formation Science, 1976. p. 81.

Dmrohue J C & Knehen M, eds. hrfiormarion for /he community. Chicago, IL: Ameriran LibraryAssociation, 1976.282 p,

Kncben M. Pn’rrciples of information retrieval. New York: Wiley, i974. 203 p.Kochen M & Tagliacoazo R. Matching authors and readers of scientific papers. hrfomr. Storage J?eoiev.

10:197-210, 1974.Knchen M. lnregrnrive mechanisms in literature grnwth. Westpnrt, CT: Greenwood press, 1974.275 p.Kncben M & Badre A N. Questions and shills of represenrstion in problem solving. Amer. J. Psychol.

87:369-83, 1974.Knrhen M. WISE: a world information synthesis and encyclopedia. J. Dec. 28:322-43, 1972.Knchen M. Directory design for networks of information and referral centers. f.ibr. Quart. 42:59-83, 1972.Kaehen M. Automatic question-answering of English-like questions about simple diagrams, J. Arm. Compur.

Mach. 16:26-48, 1%9.Krxhen M. Stability in the growth of knowlcd8e. Amer. Dec. 20:186-97, 1969.Kochen M & Derrtach K W. Toward a rational theory of decc.ntralization: some implications of a

mathematical approach. Amer. Poli:. Sci. Rev. 63:734-49, 1%9.Kochen M & Tagfiacnazo R. A study of cross referencing. 3. Dec. 24:173-91, 1968.Koehen M. The growrh of knowledge: readings on organuation and rewieval of information. New York:

Wiley, 1967, 394 p.

I have worked with quite a few professors Flood, formerly University of Michiganin my years of graduate studies,.. but Pr-ofessorKochen] was my first true mentor,

professor of mathematical biology and pre-

With extreme patience and clarity, hevious to that professor of industrial engirteer-

wouldaskpenetratingquestionsatroutmy ing at Columbia, fondly remembers Fred

researchthatwouldreflecthis intellectual both in and out of academe.understanding . . . . In many instances, itwould [onfy] be months later that certain For several years, whale Fred was rnarra-questions he continually raised finally ger of a department at the JBM Thomasmade sense to me,... [H]e had developed J. Watson Research Laboratory, I workedan exprtise in guiding a student’s path with him and his colleagues as a consul-towards “intellectual errtightemnent’ ‘9. . . . tant, He generated ideas that kept us all

168

excited and very busy, many of them wellahead of his time as we krokback on themnow. One among these projects that Iworked closely on with Fred, and others,was the pioneering investigation of auto-mationpossibfhies for the Library of Con-gress. Fred continued an active and pro-ductive research and teaching career in li-brary automation and library science un-til his death-a notable tendencyof Fred’sto stay with an impxtant basic problem....When Fred decided to return to academiclife, it was my great good fortune to bringhlm to the Mental Health Research Insti-tute in the Department of Psychiatry at theUniversity of Michigan Medical Schoolasa colleaguein the Systems ScienceGroup.In addition to his own wide-ranging re-search and teaching efforts in severalschools of the university at Arm Arbor,and his many external consultingand pro-fessional societyactivities, Fred partneredjoint research effortswith Anatol Rapoportand Karl Deutsch and me, giving gener-ously of his time and ideas on everyoccasion. 10

Belver Griffith, College of InformationStudies, Drexel University, Philadelphia, re-calls Fred’s command of information:

My main memory of Fred Kochen was ofhis optimism, openness, and far-rangingcuriosity .. . . Fred [was] always like a ladwho knows that tomorrow is his Bar Mitx-vabandthat itwilfke a great one.... Therewas, however, very little that Fred did notunderstandnod ahnost nothingthat he was

1.

2,3.4.

5.6.7.8.9.

10,

willing to dismissout of hand. A rare com-bination to find in anyone-and one ofsuch great service to his friends, col-leagues, and professions.. . . [We] aregrateful for.. his enduring vision and forthe memory of a mind both deft andgraceful.9

I don’t think I can add much to these suc-cinct and moving statements. I can say thatI will not only miss him, but that his pio-neering work will be reflected in theseessays regularly.

As for Fred Kochen’s paper, on reflec-tion and in the aftermath of the InternationalCongress on Peer Review and Ethical Be-havior in Science held in May in Chicago,I am persuaded that the issue of citation be-havior is a fundamental one. Fred Kochenrealized that electronic access in publishingcan be useful as well as a curse. Every oneof us, veterans and students alike, needs tobe reminded how important it is to reflecton the question of how wefl we acknowledgeour intellectual debts.

*****

My thanks to C.J. Fiscus and PeterPesavento for their help in the preparationof this essay.

e!9%91s1

REFERENCES

Garfield E. Citation behavior-an aid or a hindrance to information retrieval? Current Cornerm(18):3-8, 1 May 1989.

Kochen M. How weil do we acknowkdge intellectual debts? J. Ooc. 43:54-64, 1987.—-. WISE: a world information synthesis and encyclopedia. J. Dec. 28:32243, 1972.Garfteld E. The World Brain as seen by an information entrepreneur. Presented at the American

Association for the Advancement of Scknce Symposium on ‘‘Reorgardzhrg trrfonnstion Resources toImprove Deckion-Msking,” February 1974. San Francisco, CA. (Reprinted in: Kochen M, ed.lnfmmarion for oerion. New York: Academic Press, 1975. p. 155-6Q sod Essays of an b@rrrotirmsciennsr. Philadelphia: 1S1Press, 1977. Vol. 2. p, 638-45.)

Wetts X3G. Worfd Bruin. Garden City, NY: Doubleday, 1938. 130p.Kochen M. Candidate’s mission statement for ASIS president. 1986. p. 3; 8. (BatIot.)Small H. Personst communication. 3 May 1989.Lindsay R. Personal communication, 8 May 1989.Manfred Kochen 1928-1989, Remembrances of a scholar and a gentle man. J. Amer. Sot, lnforrn. Sci.

4tk223-5, hdy 1989.Ftond M. Personal communication. 12 May 1989.

169

Repnmed with pernmssmn from J. Dec. 43 54-64, 1987. Copyright 1987 Ashb. The As$c.amen for hdonnation Managemem.

HOW WELL DO WE ACKNOWLEDGEINTELLECTUAL DEBTS?

MANFRED KOCHEN

Mental Heafih Research hrsfirare, Universi~ of MichiganAnn Arbor, Michigan, 48109, USA

Authors of scientific articIes often resd a paper that fails to cite their prior work when they feel it should have.A survey of university faculty shows the extent to which such opinions abound. If justified,they reflectnon-useof bibliographic search methods, their inadequacy or non-s&holarly use of the result. Principles for the design ofa new kind of automsted or semi-automated document retrievsl system are fonmdatcd. They are ansfyaed snd sbnwnlikely to improve the scholarly qusfhy of scientific work as represented by the blbtiographies in rnanuseripts re-porting that wnrk.

INTRODUCTION

WHEN THE EDITOR OF A SCHOLARLYJOURNAL mxives a snmmscnpt,he selectsqual-ified referees. He aaks for their expert opinionsabut ifasuitability for publication. Sometinresheselects the refereea from the 8et of authors citedby the author in the rnarruacnpt.‘fltereferees evaf-uate the paper. They judge the quafify of tbe listof references at the end of the manuscript alongwith its other aspects. We will in this paper alsocall this list of references the bibliography. Dur-ing the short time a referee spends in scanningthat fist, he may think of major omissions. He mayquestion the inclusion of some of the references.But he rarely offers a detailed critique of thisbibliography.

A paper that conformsto the norms of scholarlyperfection would explicitly cite every past publi-cation to which if owes an intellectual debt. Notknowing what he should acknowledge his intel-Iecmafdebt to is no excuse for omission, any morethan ignorance of the law can excuse its viola-tion. Acknowledgementof intellectual debt is notthe ottiy function of the paper’s bibliography.1[tirtdieatesthe author’sactuafsourceof ideaa, whichmay not be the tme origin of the idea. [t directsthe reader to fusther information. It meets others’expectations about the content of a scholarlypaper. There are many other reaaom for citing.

In what follows, we ask aeversl questionsaboutthese bibliographies.

1. How close or far from such ideal bibliogra-phies are the ones published in journal articlestoday? We ask that only for one of the functionsof a bibliography, however.

2. How important is this aapeetof seholarabip,and to whom?

3. How should we distribute among authors,referees, editors, readers and sponsors resprsi-bilities for producing good blbliograpbies or forimproving them?

4. Could an automatic or semi-automated ref-erence retrieval system he expeeted to improvesignificantly the scholarly quality of bibliogra-phies, by building on what the author providedwith his manuscript or perhaps by performing ade rtovo search, given the rnarrtrscript?If SO,how?A major improvement in one of the fimetions ofbibliography could reduce the effectiveness ofanother. For example, the atrdtormay recommenda work that he has not read; that improves thevalue of the bibliography in direeting the readerto sources, but not its bonesty in reporting thesources actuafly used by the author.

5. If the answer to 4 is Yes, how much effortis it worth expendingon it, and how should thesecosts bc distributed?

The primary contribution of this paper lies inthe formulation of new pnncipleg underlying theatmfysisto question 4, particularly in the recom-mendation to develop expert systems. The issueis seen not onfy as improving the bibliographiesat the end of manuscripts submitted to a scientif-ic journal pcr se, but in improving the scholar-ship of the entire scholariy research proeeas, inwhich publication is a final stage. Tbtrs, it raisesdeeper questions about the role of document re-trievaJ in scholarship and the research process,fmdthe two-wayinteractionbetweenthe pmceseaof literature searching, screening,comprehension,

170

evaluation, organisation and utilisationon the onehand,z and of adding to knowledge on the other.

Theac issues are &o significant for researchand practice in information retrievsd(IR). One ofthe basic problems in IR research is how to mea-sure ‘recall’or hit rate. This is defined as the frac-tion of atl relevant documents that are retrieved.Estimating the denominator of this fraction, i.e.the number of relevantdocuments, has challengedinformationscientistsfor severaldecades, becauseof the difficulty of defining ‘relevance’, of per-forming controlled experiments in large collec-tions and of trarrsfemingconclusions to red situ-ations. By asking authors tojudge errors of osnis-sion in bibliographies, we can roughly estimatethe number of relevant papers.

These issues are afaoof practical import for IRfor what they imply about citation indexing andrelated searches. Citation-basedretrieval dependscritically on proper acknowledgementof intellec-tual debts.

ON THE QUALITY OF REFERENCES INPUBLISHED ARTICLES

If the primary criterion of quality is the extent towhich all prior publications to which a given ar-ticle owes some intellectual debt are acknowl-edged, then quality is probably quite low. It hasbeen estimated that at most 10 per cent of whatis published is a genuine contribution to knowl-Cdge.s Quke possibly, the quafity of bibliogra-phies is similarly low. Qurdity there is also diffi-cult to measure. To estimate it roughly, we con-ducted a mail survey to determine from a sampleof faculty members in a major research trrriversi-ty how frequently they encounter published arti-cles in their specialities that they feel neglect toacknowledge intellectual debts.

The first conceptusddifficultyarises in detirdng‘intellectual debt’. If an author uses a hhhertolittle-known concept, method, or result, withoutwhich he couldnot substantiatethe claim advancxxlin his article, he owes a substantial inteklualdebt to the author of that concept, method, resultor issue. Those debts are frequently acknowl-edged. Thus, authors who use the original con-cept of a fuzzy set have cited Zadeh’s 1965 arti-cle that introduced it; chemists using Lowry’smethod for protein analysis have gerreraflycitedhis seminal paper reporting it; studies based onthe result that people can enccde into short-termmemory only about seven plus or minus twochunks, such as digita in a telephone number tobe remembered long ermughto dial it, usually ac-

knowledgethe scmimdpaper by G. A, Mdler thatpresented it. And few who contribute to the issueof the ‘Tragedy of the commons’ fail to cite G.Hardin’s pioneering work.

Priority of a discovery is often difficult to es-tablish. Thus, a court settlement awarded the pat-ent priority for the electronic computer toAtanasoff rather than to Eckert and Mauchty.Does that mean that an author of an article or pat-ent that builds on these early designs should citeAtarsasoffrather than Eckert and Mauchly? Sup-pose he independently discovers the same designthey used, without having seen their patents,models or papers. Does he owe either, neither ortroth some intellectual debt? If he has acknowl-edged his intellectualdebt to Eckert and Mauchly,then, Iogically, he owes an intellectual debt toAtarrasoff because Eckert and Mauchfy, by thecourt decision, owe him that debt, and ‘—owesan intellectual debt to—’ (abbreviated by dJ canbe viewed as being a transitive relation (a hered-iW proP@Y:fF (x$9 AND W), THEN(d)).But painstaking historical scholarship can otlenuncover obscure antecedents of widely ackriowl-cdgcd seminal reports of diacovenes, and thkwouldimply that reference lists ceaseto be perfectwhen such priorities are established because theyfail to acknowledge debt to the originator.

The most workable definition is probably onethat refers to ‘reasonableand proper effort’ in thedeterminationof priority. ha meaning dependsonthe consistent interpretation of precedents as wellas on advances in retrievat technology. Modernonline bibliographicsearchingmakes it reasonableto expect a far more thorough search and identi-fication of prior literature than was possible with-out it.

Of course, it is widely understood that classicssuch as the original ‘publication’ of the Pythago-rean Theorem or widely known concepts, suchas the definition of a number, generrdly do notrequire citation.

A second conceptual difficulty arises in defin-ing the magnitude of the intelleaual debt. Howbig a debt must be incurred to warrant citation?The author of concepts or reds that he reportedin an older publication may estimate, on encoun-tering the same items in a recent publication, themagnitude of the debt much more highfy than theauthor of the recent article.

What is most inimical to the spirit of scholar-ship is the deliberate omission of acfrnowkxlge-rncnt of art intellectual debt so that the author canfraudulently advertise his claim to priority of dis-covery. It is close to plagiarism. Such omissionsshould be weighted heavily in assessing quality.

171

But can it be ascertained that an omission is de-liberate and that the intent of the author is dk.-honest?

Failure to cite an article that should have beencited can be attributed to any one of the followingfour failures:

1. no attempt was made to search the literature,2. the literature was searched, but not well

enough; the document retrieval system usedwas not gocd enough;the queriesped werenot good enough; or not enough effort wasexpended,

3. relevant documents were available but notread or not used,

4. the item that should have been cited wasretrieved and at least lookedat but not cited(a) because of an attention lapse orcarelessness; (b) deliberately, because theauthor did not deem it worthy of citing; (c)because the author did not understand it orits relevance; (d) for the less honorablereasons afhsded to above.

The questionnaire used in the survey assumesthat respondents are experts in their fields; havepublished in those fields; keep up with currentliterature; examine the bibliographies of the arti-cles they ti, recogniseart idea, result or methodthey encounter in their readings as similar to onesthey encountered previously or that they them-selves originated; appreciate distinctions betweenthe form, content, clarity and expression of theseideas as encountered and the corresponding at-tribution to similar ideas. We also assumed thatwe could recogniseand compensatefor or againstrespondents’ biases in seeking recognition due tothem. The questionnairewas sent to twenty namesselected randomly from listings of faculty mem-bers in each of three departments: mathematics,history, psychology. Of the sixty questionnairessent out, twenty-one usable responses were re-turned. Of these, only one had been in the fieldof specialisation less than ten years, and seven-teen more than fifteen years. Fifteen of them hadauthored or co-authored more than twenty publi-cations in that field. To keep current with spe-ciality literature, seventeen relied on personaljournal subscriptions (not exclusively); fifteen onrepnnts/preprints; seventeen on libraries; five onordine searching; eight on bookstores; sixteen oncmtferences.Everyonescansthe bibliographicref-erences sOor most of the time in the articles theyread.

Ten of the twenty-one said that in the most re-cent article they read, the author failed to cite rel-

evant prior work, (seven said no, and four didn’tkrtow). Of these ten, two felt that onfy five sucherrors of omissionswere made, whifefasr thoughtfive or more references were omitted that shouldnot have been. The remaining four didn’t know.Six of the ten felt that these omissions were toauthors who are widely rexognised in the field.On the average, 30 per cent of the articles theyread omitted references to prior work that shoufdhave been cited, but the variance is very high.Two of the twenty-one felt that 75-ltXt per centof the articles in their fields left out works thatshould have been cited, and four of the twertty-one felt that less than 10 per cent of the articleswere flawed in this way,

On the whole, one respondent said that the lit-erature does not adequately acknowledge intel-lectual debts; eleven said ‘somewhatadequately’,eight said ‘adequately’, one didn’t know, Five ofthe twenty-one rarely encountered works thatshould have cited their own pubIishedworks; fivesaid this occurred often, and eleven said ‘occa-sionally’. Some of their comments were: ‘in myfield (adolescentpsychology), only a handfulciteadequately. I take it to be a part of life. I don’tcite adequately, by the way.’ ‘I have recentiy be-come interested in philosophy and have writtenin that area, I have been shocked at the extent towhich intellectual debts are not acknowledged.’‘Mathematics is fairly careful about this. Surveyarticles are often cited in lieu of direct (ongiml)sources, Most omissions of this type are made byyoung people not yet in control of the literature,or old people at odds with one another.’ ‘Thereis a good deal of superficial, “protective” cita-tion, in which a work is cited (e.g. “see also”)without actually taking accounts of its argumentor conclusions.‘ ‘Some researchers unwittinglyrept earlier works-things done some fifty yearsago. 1ssgeneral, the three or four such cases haveobtainedthese early (uncited)resufts by more ele-gant modem methcds (mathematics).’‘I think thatthe problem has decreased with increased sp-edand scope of information dissemination and re-trieval. The older literature was much worse thannrdem literature in my perception.In part, peopleare ignorant. They are harried into publicationwith little time taken for scholarship. It happensto me. But also people are unbelievably peevishabout citing competition. Slg, established inves-tigatorspretend that the others (and their studentsandtheir students’ students) do not exist. But it’siust another fact of life. The situation is hopdessbut not serious.’

Suppose that about 30 per cent of the articlespublishedfailed to acknowledgeintellectualdebts.

172

Supposethat on the average, five prior works thatshould have been cited were omitted in tfds 30per cent of the articles. Suppose tirrtfrer that theaverage article has ten relevant prior articles in70 per cent of the citing articles. The other 30per cent should have cited fifteen instead of onlyten relevant prior works. Then, about 10per centof the past literature should have been cited butwasn’t. That might & about the magnitudeof fail-ure to acknowledge intellectual debts. In otherwords, most (70 per cent) of the quafity of bibli-ographies is qnite good in that they cite most ofwhat they should cite. But a significant minorityof publications is quite poor, citing only perhapstwo-thirds of what it should.

Given the fornd&ble difficulties in judging thequality of a bibliography, it is naturrd to ask howimportant the task is. The considerable effort de-voted to document retrieval could be viewed asculminatingprimarily in improved reference lists.But that is only the most visible end-product ofthe entire scientificresearch process. Presumably,a more thorough and successful search for rele-vant, important and valid concepts, metbcds, re-sults, and issues that appeared in prior publica-tions results in improved contributions to knowl-edge. Such searches are deeply integrated into allphases of the research process, and scientific in-quiry has been suggested as a model for ordinesearching.4 Having contributed significantly byusing prior works that were selected early in theresearch process, how important is it to checkwhether the resulting contribution has alreadybeen produced by someone else?

IMPORTANCE OF HIGH-QUALITYREFERENCE LISTS

To the sponsor of research, unplanned duplica-tion of contribution seems like an inefficient al-location of his scarce resources. To those con-cerned with the utilisation of human resources,e.g. employers, it seems as if opportunities forbetter use of scarce, hlgfdy trained talents, weremissed. E&tots of journafs in which there is highdemandfor tbe allocationof scarcepagesto marnr-sc@.$ @lI.wto publish repxts of previouslypub-lished contributions. It wordd therefore be pru-dent to search the literaturerepeatedlyat all stagesof the research process that leads to a publication.

The list of referencw in the manuscript that issubmitted to a journal is a composite of atl thesesearch results. If it omits stating key intellecturddebts, it casts justified doubt on the quality of theDrocessad its Droducts.To ti sure, carelessness

resutting in a low-qualitybibliographymay do in-justice to a good process and product, and a par-ticularly well-prepared bibliographymay deliber-ately mask and conceala poor processor product,but we expect these to be exceptions rather thana rule. This is a hypothesis that should be exper-imentally tested. The price of such carelessnessshouldbe rejectionby a journal, and good refereeswill rarely be deceived by a gd bibliographyinto accepting a poor manuscript.

Another reason to attach importance to a bigb-quality reference list in a publication is that it be-comes part of the archival record, not to mentionits role in citation indexes.5 As such, it is ac-cepted as an authoritative source for further bib-liographic work. Poor bibliographies contributeto the propagation of errors and these are veryhard to detect and correct. (There was a good ar-ticle in the early days of the information retrievaldiscipline-perhaps in the early IXiOs-that dem-onstrated how an error in a citation was propa-gated by uncritical, unchecked copying from onebibliography to another. My inability to recall orretrieve the citation to this article is an exampleof retrieval failure, even given strong clues.) Theintegrity and qrrafity of the cumulative archivalrecord depends on the quality of bibliographiesadded to it.

Recognition of priority is a powerful incentiveto researcherswho engagein the arduousand frus-trating, often tbantdess, effort to add to knowl-edge. Making light of errors of omission is there-fore a disservice to the motivatingforces in schol-arship. Put positively, ensuring that contributionswill be appropriatelyrecognisedat least by publiccitations of intellectual debts can increase the in-centives that attract appropriate people to Iivcs asscholars and keep them productively (and happi-ly) engaged in such pursuits. With the trends to-wards use of lifetime citation countsb-10 inawards of tenure, awards of promotion, salary in-creases, etc., these incentives have a materiatcomponent.

Given that quality of bibliographies is impor-tant, how are and should responsibility and au-thority for such quality control be dktributed?Now, authorsbear the primay responsibtity. Edi-tors rely mainly on referees who are experts ina speciality, and they are not generally aware ofany special responsibility for the qurdity of thereference lists in manuscripts they review. Theyjudge the manuscript in its entirety. Editors havethe authority, including that of rejecting a mamr-script because of a poor bibliography. Readershave neither responsibility nor authority direct-ly, but they can refuse to read or subscribeto jour-

173

mds with papers that they judge to have poorblbtiograph]es and, as authors themselves, theycan refuse to cite such papers, write letters to theeditor, and decide not to submit their manuscriptsto suchjoumrds. Sponsors can refise to sponsorwork by authors theyjudge to publish papers withlow-quality bibliographies or urge and help theirgrantees to improve in tils regard.

The present distributionof quafitycontrol is notcommensuratewith the importanceof quality bib-liogmpfties. An improved distribution would givethe referees and edhors greater responsibility, Inthe next section we present some new ideas forhelping them carry it out.

PROPOSED REFERENCE SEARCHMETHODS TO IMPROVE SCHOLARSHIP

The intent is to help an editor to check the qtadi-ty of the list of references used in a submittcxlmanuscript in the sense discussed so far, Inputconsists of all or part of the manuscript. It is as-sumed that these inputs are availablein computer-readable form, such as a diskette.

The simplest procedure is to enter the list ofreferences provided by the author and search on-line citation indexes for recent articles that citethem singly, in pairs, in triples, etc. This is a ver-sion of co-citation analysisl[ used at search time,The editor then asks the referees to judge whetherthe retrieved referencesthat are not in the author’sbibliography are serious errors of omission. Pub-lishers are not likely to cover the costs of suchsearches unless forced to by competitivepressuresor by standards set and enforced by professionalsocieties.

The atroveprocedure is biased toward recencyand quite costly as well. A more traditional sub-ject search would remedy thk. For journals suchas JACM, which require tie author to submit key-words with his manuscript, any of the search sys-tems based on Boolean combimtions of the key-words can be used. Again, referees are providedwith the resulting bibliographies, in which thoseitems used by the authors are deleted or marked,and they are asked to judge if the remainder con-tains major errors of omission. Such keyword-driven generation of references may improve theextent to whichothers’expectationsabout the con-tent of a paper are met but at the expense of indi-cating bonestty which sources the author actuallyused. Both of these as~ts should, of course, betaken into account. Again the cost of this mightbe home by the publishers if that were requiredby market forces.

Havinga completemanuscriptin machireread-able form provides opportunities, however, formore sophisticatedrefererw searchingthan is fros-sible either by references or keywords with cita-tion indexes or Boolean searching, respectively.To be sure, lexical and logicsd content anafysisof clear text is still not well developed and costfy.But a variety of simple statistical and lexical/log-ical methods of the kind proposed in the1950s]2-14can be applied. Most of the methcdsthat have been discussed in the literature in rela-tion to indexing~5can be applied, with mo&fi-cation, to formulate search queries. They are stilllimited by the way the bulk of literature to besearched is made available for searching: by in-dex terms, till text of titfea and occasionally ab-stractsand references.Thus, the most we can hopeto obtain from the analysis of a manuscript is thekey concept, methods, findings or issues that itsauthor claims to add to knowledge or that theauthor used, for which he owes their authors in-tellectual debts; as so approximation, such con-cepts, methods, futdbrgs and issues must be ex-pressed in the language suitable for searching anonfine database.

One idea is to look up each text word-stem ina dictiorrmyor alphabetisedauthoritylist of searchterms, counting the number of times each entryin the list is consulted. Four micro-thesauri ofterms, used to write about concepts, methods,tindings and issues respectively, are consultednext. A findings thesaums, for example, containsverbs such as ‘tind/formd’, ‘show’, special ad-jectives and nouns not in the search vocabularies.The output of each thesaurus is a grouping ofterms that can be combkxl with one or moresearchterms to expressa concept, method, tindirrgor issue, and a Boolean query is composed ofthose searchterms and usedto searchvariousdata-bases. Automatic methods for query formulationhave been inves.tigatedlGand are relevant here.

Another idea is to divide the number of timesa term in the authority list is consuftedby the totalnumberof worda in the text and compare that fre-quency with a stored frequency with which thatterm occurs in general use aa weflas in documentsthat have beerrjudged relevant to that term wherethat is available. If the frequency of occurrencein the document is much greater than the storedfrequency-or close to that of a relevant docu-ment—then it is used for a search.

A more timely and amb~tiousundertakingis thedevelopment and use of expert systems. That isthe main propusalput forth here. An expert systemis needed for each speciality. For example, tobuild an expert system for ‘fuzzy set theory and

174

information retrievaf’, 17a knowledge engineerwouId capture an expert’s (say Abraham Book-stein’s) expert knowledge of the literature in thattopic. This could be in the form of productionrules, such as ‘If finding (or concept, or method,or issue) p is salient, then consult referencesa,b,c,...’ Here p can be viewed as a proposition.To illustrate with simple propositions in a well-known domain of discourse, arithmetic and num-ber theory, consider propositions about concepts,findings, methods, and issues respectively:

1. ‘A number is prime if and onfy if its onlydivisors are 1 and itself.’ (The primary conceptis italicised.)

2. ‘Every number can be expressed as a prod-uct of powers of primes, in a unique way,’ knownas the jimdarnental theorem of arithmetic or asthe prime factorisation theorem. (The names ofthe finding are italicised: a keyword might bepn”me factorisation.)

3. ‘Whether a given rrurnber is prime canalways be found by the method of the sieve ofEratosthenes.’

4. Gokibach’s conjecrrnw ‘Every even numberis the sum of two primes.’

Just kcause a concept is a needed prerequisitedoes not mean that the first publication to intro-duce the concept needs to be cited. The conceptsof number or prime, for example, are known tojust about anyone, and though some people havecontributed to their profound explication (e.g.Pcano), no individualis regardedas the discovererof these concepts. It could, indeed, be argued thatevery new concept is a sociaf rather than an indi-vidual product, and no person has a right to claima discovery as solely his own. Even though dis-wveries are often made independesstfyby severalpersons within a few years, when the logic andmaturity of the specialityhas sufficientlyripened,the practice of crediting an individuaf-even ca-priciously as in a lottery-serves as a pwerfulincentive to dkcoverers. 1s

These propositions use ‘functions’, interpretedhere in the sense of a programminglanguageusedin artificial intelligence such as Interlisp, whichare applied to domains and renrm unique values:e.g. ‘Sum or appiiea to pairs of numbers as inthe list (PLUS 23) and, when executed, returnsa number, 5, in this case. Predicates are specialkinds of function that return only vafues T (true)or NIL (usually false); ‘is prime’ applies to inte-gers, as in (PRIME 97), which returns T in thiscase.

We have implied that wncepts, findings, meth-ods and issues are the kinds of entities that wm-prise knowledge and are added to it, and are used

in the production of knowledge. TM list is notintendedto be wmplete. Explanations,discourses,critiques, and histories are a few of the otherelements found in scholarly publications. ‘Con-cepts’ are intended to include ideas (e.g. the idea

of an imaginarynumber), principlesand laws (e.g.the principle of duality, the law of effect), defi-nitions, and other seminal mentafconstructs (e.g.atoms, quarks, libido). By ‘tindings’ we meantheoremstogetherwith their proofs, tested hypoth-eses together with the evidence and statistical in-ferences for acceptingor rejectingthem, reasonedinclusions, justified recommendations, prin-cipledpolicies, facts and their sources, trenda, arxigenerally justified, tme beliefs about the worldor ourselves, ‘Methods’ generally have names,often associated with a discoverer, as do findingsoccasionally .s There are mathematical, experi-mental, observational and many other kinds ofmethods. Often a new mathematical method ispresented as a theorem. The term ‘issues’ is usedhere to include open questions and conjectures,controversies, choice points, foci of debates.Generally, a publication contributes not one buttwo or more of tkcse four kinds of knowledge.11generally uses all four, Citations acknowledgedebts to these, but they serve other fimctions aswell.1

Ordy the first half of any production mle in theproposed expert system has been illustrated, andin a domain in which expertise on referenceswould be too elementary to be useful and hencefar from the research frontier. Thus, no authoris expectedto acknowledgehis intelkctrad indebt-edness to Goldbach, Eratosthenes or the authorof the paper reporting the first proof of the fim-dametmd theorem of arithmetic. Nor would anexpert on number theory be expected to knowthese references. However, one can quickfymoveto the leading edge of number theory by askingabout the frequency of primes between 1 and anynumber n, the famousprime number theorem. Anexpert would know that Hadatnard and de laVallee Poussin first proved in 1896that the ratioof that frequency to n/log n gets closer and closerto 1 as n gets larger and larger, and that morerewntly, P. Erdos and A. SeIbergfoundmore ele-mentary proofs for this frnding.19This reference,incidently, acknowledges debt to a source of in-formation for both the facts and the referenceswithin my assertion. Ordine bibliographic searchsystems could provide the expert with (or checkhk memory of) the most recent exact referencesand thus build the expert system.

The expertise to be captured pertains primar-ily to knowledge of the literature. If that is not

175

. ..

to be superficial, it also requires some expertisein the subject matter. Quite often, the most cre-ative contributors to a speciality do not have ex-pert knowledgeof the Literature.(Theyoften knowonly as little of the work of others as they needor in fact used to make their own contributions;a Nobel laureate once said, ‘If I need a good bookin my field, I write one’.) But literature experts(quite frequently Ph.D. students) have quite a bitof the propsitionrd expertise illustrated above.The building of an expert system should capturethe expertise of several such Ph.D. students. ht-decd, the system of comprehensive Ph.D. exam-inations could add to as well as draw upon sucha corpus of production rules and the network ofassociated propesit ions.

How would such expert systems be used? Thefirst step is to determine for each manuscript theconcepts, fiochngs,methods or issues it claims toadd or that it uses in the production of its contri-bution in a reamer requiring citation. The authorcould be required to identify those passages in hispaper, if any, that are transformable into the ap-propriate propositions. The referees could beasked to formulatesuch propositions. A computerprogram could search the text for the names ofmethods, findings, issues, concepts or phrases inan authority list likely to identifi them, and forfunctions and predicates likely to comprise thesought-forpropositions,and output the wantedps.Any of these three methods should generate thepropositions of ps needed to enter the productionrules of the expert system, which then returns anexpert-quality list of references.

What are the principles on which to ground thedesign of such an expert system, that woufdmakeit feasible? The first is that production rules cap-ture the knowledge of experts (e.g. in diagnosisand treatment of bacterial infcctions20)and facil-itate automatic inference-making.?] The combi-nation of such production rules with the knowt-edge of human experts and the output of comput-erised literature searches is new, as far as I amaware.

A second principle is that concepts, findings,methods and issues can be represented as propo-sitions using functions and predicates as under-stood in the languages and systems22used in 4ar-tificial intelligence’ research (e.g. LISP, PRO-LOG) and that such propositional expressionsare useful for formulating queries for onfineSearch]ng.

A third principle is the existence of program-ming languages, such as KEE,23 for the devel-opmentof expert systemsthat facilitatethe writingof systems of production rules, with consultation,

inferencesand questionansweringavailableto theuser.

These make the constructionof such expert sys-tems well within the state of the art. Their devel-opmentcan be expectedto improveuponthe qual-ity of bibliographies in manuscripts, because itbrings to bear the combinedpower of the indexedliter’ahsrethrough computerisedsearches with thatof human experts on the literature in each spe-ciality, as well as inferences from these data andrefinements and improvements resrdting fromtheir use. If authors don’t use them, editors canevaluate the bibliographies in their manuscriptsin comparison with bibliographies generated inthis way.

CONCLUSION: fNFORMATfON RETRIEVALAND SCHOLARSHIP

The development, maintenance and usc of aidsto improved scholarship, such as proposed in theprevious section, is costly and risky. The percep-tion of benefits is subtle. Some practically-mindedleaders, includingkey decisionmakers in the ptsb-Iishing industry, often relegate scholarship intolow-priority categories with unessentialluxuries,the pastimes of esoteric academics, the compul-sions of perfectionists. Indeed, the magazine ar-ticles and reports that busy executives pay mostattention to seldom have bibliographies or listsof references. Pedantic scholars are often cmi-caturcd as the antithesis of hard-headed, driving,decisive real-world managers.

Yet, scholarship, like the arts, is a worthy endin itself. It epitomises a humanistic tradition, thepreservation of which gives more meaning to thepursuit and attainment of practical endeavors.This must be kept in perspective as a high prior-ity, particularly as needs for survival, security,bdongingness and esteem~A are increasinglybeing met, permitting societies to attend to theneeds for self-actualisation and collective well-being. Scholarship is one of the higher forms ofse]f-expression as well as a manifestation of‘group mind’, of co-operative advancement ofknowledge that enriches civilisation.

Because subtlety, sophkication and high cul-turedness is needed to appreciate the importanceof scholarship as an end in its own right, thatmessage must also be supported in a way thatreachesUsemore practicallyminded scientificandthe associated technological advances on whichdepxtd our competitivepositionin worfd markets,in geopolitical arenas and military fhcatres, ourliving standarda,etc., rely heavilyon a highqual-

176

ity record and system of communication. As@rtted out above, the quafity and integrity of therecord-and therefore its use for effective com-munication and making good use of the work ofothers-requires good scfkolarshipin the sensedis-cussed. The practical importance of this isenormous.

As knowledgecontinuesto doubleevery decadeor two, to become more specialised, and to re-quire more studyto attain its leadingedges, schol-arship becomes correspondingly more dlfticult.The very technologiesthat are helping to advanceknowledge can, however, also be used to man-age it by arnpli~lng the productivity of scholars.The ideas proposed here contribute to our knowl-

edge ahout how to do this. Enterprisingpublishersor leaders in other information industries shouldcqnsider seriously the recommendation to launchthe steps recommended here toward expert sys-tems that support the evaluationand improvementof bibliographies in the manuscripts submitted tohigh-quality scholarly journals.

ACKNOWLEDGEMENTS

Thanks are due to D. West for discussing someof the ideas in this paper, and to the NatiorudSci-ence Foundation for partial support under grantIST-8301505.

REFERENCES

1. Moravcsfk M J &MurugesanP. sameresultsonthe fmction and quality of citations. Sot. Stud. Sci. 5:85-92, 1975.2. Wstker D E, The organization and use of infonnawm: contributions of information science, computational linguistics and

aruficisl intelligcnm. J. Amer. tic. /nJionn.SCI.32:347-63, 19S1.3, Coping wifh the biomedical literature expfosion-a qualitative apprcuch, 22-23 May 197S, Pocantico Hills, NY,

New York: Rockefeller Foundation, 197S.

4. Harrer S P. Sckntific inquiry: a mcdel for online searching. J. Amer. So& Infom. .$ci. 35:110-7. 19S4

5, Garfield E, cd, Science Cimion Imie-x; social .Srience Cimtion Index; ArTs & Jfwmmiries CikXion Index

Philadelphia Institute for Scientific Information.

6. ---—. Is cirstion analysis a legitimate evatualion tool? Sciemomemics 1:359-75, 1979,

1, -—-—. What’s in a nsme? Tlc eponymic route to immortahty. .%mys of an infornuuion scientist. Phitadelphiw ISI

k%.SS, [954. Vol. 6. p, 354-95.8. -------- How to use citation anatysis for faculry evaluations, and when is it relevant? Part 1. J6id. p. 354-62.

9. GeIler N L, De Csnf J S & Davies R E. Lifetime-citation rates to compare scientists’ work. SoC. Sci. Res.

7:345-65, 197s.10. -----—. Lifetime-citation rates: a mathematical model to compare scientists’ work. J. ,4mer. Sot. Inform. Sci.

32:3-15, 1981.II, St’nsll H & Grimth B C. ‘t_twstructure of scientific literatures. Sci. SIud. 4:17-40, 1974.

12, Luhn H P. A statistical approach to mechanical ending and searchins of literary information, IBM J. Res. Devefop.

4:3G9-17, 1957,

13. Rogers P J & Tardtnoto T T. A computer fwogram for classifying plants. Science 132:11 15-S, 1960.

14. Kkng C W. Table fook-up procedures in data processing. Yorktown HeiShIs, NY: IBM Corporation, 1%2.15. Cteveland D B, Clevetsnd A D & Wk.? O B. less than fidl-texi indexing, using a non-ltoolean searching model.

J. Amer. Sot. Inform. Sci. 35:19-28, 19S4.16. Salton G, Buckky C & Fox E A. Automatic query formulations in information retrieval. J. Amer. SW Inform. Sri.

34:262-sO, l!Jg3.17. Bocd@efn A. Fuzsy requests an approach to weighted Boolean searches. J. Amer. SW. lnfmm. Sci. 3 I :240-7, 19S0.

18. Merron R. S,nglerons and muttiples in scientific dkwery. Proc Amer. Phil. Sac. 105:470-86, 1%1.

19. Ksc M & Uhun S M. Marhaadcs md fogic. New York Praeger, I%S. p. 4.

20. Shorkliffe E H, Bucktamn B G & FeJgenhmnn E A. Knowledge engineering for medical decision-making a res’iew of

computer-based clinicsf decision aids. Pmt. IEEE 671207.24, 1979.

21, Kactten M. Adaptive mechanisms in digitsl concept processing. IEEE Trans. htd. AppL S3:305- 14, 1964.

22, Quaile M A. The fkend[y dandelion primer, Pittsburgtt, PA: University of Pittsburgh learning Research and

Development Center, 19S4.

23. KEE mining mimd. Menlo Psrk, CA: Imellicorp, 19S5.

24. Maslow AH. Toward a psychology of being. Princeton, NJ: Van Nosrmnd, 1%S.

177