ED 253 126 AUTHOR TITLE PUB DATE NOTE PUB mg EDRS PRICE DESCRIPTORS DOCUMENT RESUME HE 017 927 Kroc, Richard J. Comparing Citation Rates with Other MeasureS of Scholarly Productivity. Oct 84 ' .28p.; Pgper presented at the Joint Meeting of the American Educational. Research Association Division J and the Associationipor the Study of Higher Education (San Francisco, CA, October*28-30, 1984). Reports Research/Technical (143) Speeches/Conference Papers (150) MF01/PCO2 Plu0Posta0. *College FaJUlty; Conferences; *Evaluation Criteria; Financial Support; -Higher Education; Institutional CharacteristicS; *Productivity; Reputation; *Scholarship; *Schools of Education *Wrip iting for Publication IDENTIFIERS *Faculty Publishing 'ABSTRACT The construct validity of scholarly productivity was investigated, with attention to definifions and measurement approaches. A nonrepresentative sample of 51.schools of education was selected to include very produgtive schools. Measures were made of the following variables: citations, rankings froth previous studies, publications, conference participation, funding, and general institutional characteristics. Publication counts were estimatedlrom data in the Educational Resources I4lformation Center (ERIC) system. Each school of educption's participation in the American Educational Research Association conference. was assessed for 1981 and 1982. Information on grants awarded to schools of education during 1978-Y982 was obtained from the Smiths9nian Science Information Exchange. Finally, the Higher Education General Information Survey provided data on _five variables: percentage of aoctoraes granted, total research expenditures, total overall expenditures, government grants and contracts per full-time equivaldncy, and, average salary. Conclusions include the following: program size did no significantly determine prestige; surveys on reputation did not appear to be reliable; and. citation counts seemed to b the best measure of scholarly work. (SW) ., 1 ***************************************************1**********Ao***** Reproductions Supplied by EDRS are the best that can be made from ihe original document. ******************.************,*****t***********************v********** 0.1
26
Embed
DOCUMENT RESUME ED 253 126 HE 017 927ED 253 126 AUTHOR TITLE PUB DATE NOTE PUB mg EDRS PRICE DESCRIPTORS DOCUMENT RESUME HE 017 927 Kroc, Richard J. Comparing Citation Rates with Other
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
ED 253 126
AUTHORTITLE
PUB DATENOTE
PUB mg
EDRS PRICEDESCRIPTORS
DOCUMENT RESUME
HE 017 927
Kroc, Richard J.Comparing Citation Rates with Other MeasureS ofScholarly Productivity.Oct 84
' .28p.; Pgper presented at the Joint Meeting of theAmerican Educational. Research Association Division Jand the Associationipor the Study of Higher Education(San Francisco, CA, October*28-30, 1984).Reports Research/Technical (143)Speeches/Conference Papers (150)
'ABSTRACTThe construct validity of scholarly productivity was
investigated, with attention to definifions and measurementapproaches. A nonrepresentative sample of 51.schools of education wasselected to include very produgtive schools. Measures were made ofthe following variables: citations, rankings froth previous studies,publications, conference participation, funding, and generalinstitutional characteristics. Publication counts were estimatedlromdata in the Educational Resources I4lformation Center (ERIC) system.Each school of educption's participation in the American EducationalResearch Association conference. was assessed for 1981 and 1982.Information on grants awarded to schools of education during1978-Y982 was obtained from the Smiths9nian Science InformationExchange. Finally, the Higher Education General Information Surveyprovided data on _five variables: percentage of aoctoraes granted,total research expenditures, total overall expenditures, governmentgrants and contracts per full-time equivaldncy, and, averagesalary. Conclusions include the following: program size did nosignificantlydetermine prestige; surveys on reputation did notappear to be reliable; and. citation counts seemed to b the bestmeasure of scholarly work. (SW)
.,
1
***************************************************1**********Ao*****Reproductions Supplied by EDRS are the best that can be made
from ihe original document.******************.************,*****t***********************v**********
0.1
to-
-
Comparing Citation Rates with.Other Measures
"PERMISSION TO REPRODUCE TINS
MATERIAL.HAS BEEN GRANTED. BY
TO THE EDUCATIONAL RESOURCES
.INFORMATION CENTER (ERIC)"
of Scholarly Productivity *
. U.S. DEPARTMENT OF EDUCATIONNATIONAL INSI IIUT( ,I.DOCAT ION
t OUCAT IONAL Itl SOIATC.IS 4NrpliNIA I IONCLNITIt (Intel .
(..411,441.'etonlimt hat, tUtUo toptOthtflUti aA
Richard J. K r o c mcmvtul hum the, tmus011 et oftiointe*on
ofiti,nallmodMIt(11 Imbues kive boon made to 11114110VO
telpot 1)4:1) itmlity.
P us .11 owmoos *Anted it 11:*do(*.mom do not ectrwtin, roptoent offIc141NIE
UniversAy of Colorado
pn511n/n of putt V.
In.assessments of excellence in higher education, uestions
of scholarly productivity are often pivotal.. Research and
publication are the driving forces in the modern university,
evaluators of university programs are attuned ;() this fact.
From i6divAual.promotion and tenure decisions to overall...
institution-al prestige, scholarly research is a fUndament.I1141 ,
issue.
A variety' of procedures hive been used 'to measure
productivity. Some researchers have surveyed faculty or
and
\
administrators to obtain ranjdngs of universities and
departments. ,Others have-used the, quantity or quality of
publications to rate programs, sometimes including presenttAans
of papers at conferences. Another measure is the type and .
amount of funding obtained by a ,department, and, at -thy
university level, characteristics such as faculty salaries and
research expenditures have been ] -inked to faculty productivity.-.
Fln14h aelly, citation rates, a measure of peer recognition, have
been used to- assess scholarly accomplishment. In some way,
of these measures has' been associated.wi.th productt sty.
.This study ..addressed tke construct validitx oft cholarly
productivity: Mow this concept has :been defined, and how
.:S
Paper p sented at the Joint Meeting of the American Educational ResearchAssoc.]. ion DivIston J and the Association for the Study of) Higher Education,San Fr n4pcd, C.41ifornia, October-2830, 1984.
Sr
-.)111yr
At*
ti
*2
various ways of measuring it compared. In particular, data
pertaining to schools of ed.ucation were examined. The
. perspective was that of an administrator or evaluator who must
,collect and assess information regarding productivity,
discovering in the 'process what data are available for making
.valid, useful comparisons among 'schools of educatico, whatti
problems occur in gathering this data, what relationships 'exist
amoog measures, and what interpretations can be made. These
issues are of considerable practical significance, reflecting
processes that often have a direct impact 611 the future of a
program, a school or a career'.
Interrelationships among variousfmeasures of productivity
have been explored indifferent ways by several researchers.
Hagstrom (1971) determined that a single, unrotave'd factor was
sufficient to account for most of the variance among 188
university.science and mathematics departMents on ten variables
related to institutional quality. Astin and Solmon discovered
two factors,'"scholarly excellence of faculty" and. "commitment
to teaching, ". in their factor analysis of a questionnaire on
justification for. thinking of schonanly productd vity vi,n schools
of education as a''unitary concept.
Finally, citation analysis, a reliable process with a1
compelling and Logical basis, produced data whilich .showed a
stronger re1ationship than did any other measures with funding
4.
4.
15
and' institutional variables. The evidence in this. study implied
that *citation counts are most central to the concept of.
fprodu.aivity: Although all measures are somewhat flawed,
particularly when used to evaluate indiNidua) careers, the
citation rate may* be the best single measure 'of scholarly work.
e
I
rr
d.
ti
14.
:
v
REFERENCES .
. fa
Anderson, R. C. Narin, & McAllister, P.7- Publitation,ratings at Journal of.t.he American Societz.for,WorMatio Science, 1178, M-1)-10:101.
Astin A. W , & Solmon, L. C. Are reputational ratings neededto measure quality. Change, 1981, 13, 14-19.
Blau; P. M.J, & Margulies, R. 7. The reputations of America.pprofessional schools. Change, 1975, 6(10),'42 -47.
.The Cartter Report on the leading schools of education, raw, and.
business. Change, 1`577, 9(2,), 44-48..
Cochran, W. G. 'Samplingrtechniques. New York: John Wiley and' Sons, 1953.. .
Faculty size in institutional proa'uctivity. Schoolof Edueition,.bn-FieT'ilIy ofPennsylvaTra,1§81.
Eash, M. .Educational research productivity of institutions ofhigher education. American Educational Pesearch Journal,1983, 2Q(1),
Endler, N. S., Rushton, J. P., & Roediger, H. L. Productivity. and scholarly. impact (citations) of Briitish, Canadiah, andU.S. departments of'-psychology (1975). American
tsno9.91q, 197a,.33, '1,064 - 1,082,
Hagstrom, W, 0. Inputs; outputs, and the prestige of univer-sity science departments. Sociology of Education, 1971,,44, 375-397.
Kroc, R. J. Using citation analysts to assess scholarlyproductivity. Educational Researcher, 1984, 13(6),17-22.
Ladd, E. C. & Upset, S. M. The 1977 survey of the Americanprofessoriate. Chronicle oftilhfs Education, 17(18),7-8.
93Schubert, W.'H. Contributions to AERA annual pro.9rams as an
(
indicator of i'ns itutjonal productivity. EAucationalResearcher, 1979 8(7), 13-17.
,Sieber,, S. D. The organization of educational researcW. NewYork: Columbia 'University, BureaTWTApplied SocialResearch, 1966. .
am.
Solmon, L. C., & Astin, A. W. Departments %Jithout distin-guished graduate programs. Chan.91, 1981, 23-28.
..
4
NR
r
.Walberg, H. J. University distinction in educational research:An exploratory turvey. educational Researcher, 1972,1(1), 15-16.
West, C. K.. Productivity ratings of instituti,QA based onpublication in the journals of the American EducationResearch. Association: 1970-1976. Educational Researcher1978, 7(2), 13-14.
ti
'4."?sil.
If
4
4
w.
.
4
..1
Ar
Table
List .of Universities
University of ArizonaArizona State bniveFsityBoston UniversityUniversity of California at BerkeleyUniversity of talifornia at Los AngelesUniversity of California at Santa BarbaraCat lic)Universityn varsity of Chicago
City University of NewyorkUniversity of ColoradoColumbia UniversityUniversity of ConnecticutCornep. UniversityUniversity of FloridaGeorgia State UniversityHarvard University
,,
University of HoustonUniversity of Illinois: Champaign ,UrbanaUniversity of Illinois: Chicago-CircleIndiana University,University of IowaUniversity of KansasUniversity of MichiganMichigan State UniversityUniversity,of MinnesotaUniversity of MissouriUniversity of Montana
. University of NebraskaNew York UniversityUniversity of North Carolina: chip*i HillNorthwestern Un,,ivetsityOhio' State University:University of Oregontniversity of PennsylvaniaPennsylvania State UniversityUniversity of PittsburghPurdue University..UEINWEsity of RochesterRutgers University
. University of Southern CaliforniaStanford UniveriitySyracuse UniveriityTemple University(University of TexasPeabody-Vanderbilt' University 4University of Virginia.Virginia Polytechnic InstituteWashington University
; Unfvertity. of WashingtonUniversity of Wisconsin
Table 2
Correlations Among Other Studies Producng Rankings of Education Program* and Citation Path
a
4. 'V iiII Ili t
^ ...I4 :s. .-.S.
*V
'0 ....l 0
w 43 ..C a .1.. o--
t- a .04-1 .. 44 .4 a '.0. .. - .a.a 0 ..I
41 - i ' i P:4
O.0
0,.1% i i , ..1. I a
Mean Citation*Mean Citation RankTotal Citation Plank
ercent'o1 raculty 141tillo CitationCarttar 119771Slam and Margulies 119751Ladd and Lipsett (19791:Siabor-11966)Skilful'', 119721
Schubert 119791pole 119011
M*Al 11979)t ask (190)1 AEA Totalg ash 119m1) ADA Per r?EEash419011 Total Solicit,*Kash 419011 Article* Per rTE