Pringle Research Eval

Thomson Reuters and Research Evaluation in the Humanities

James Pringle, VP, Development

Academic & Government, Thomson Reuters Scientific

[email protected]

September 19, 2008

Citations in the 21st Century

• The “User-Generated Content” of formal scholarly communications

• Fundamental to effective evaluation & management

• Frequently mis-used and misunderstood

• May have a place in humanities research evaluation

3

Changing Scholarly Communications Value Chain

A Contentious Issue

“…impact factors and citations are dominating minds, distorting behaviour and determining careers….

…The main villains are fashion, the management cult, and the politics of our time, all of which favour numerical evaluation of ‘performance’ and reward compliance.”

--Lawrence, P. A. (2007).

The Management Challenge

• “Metrics democratization” leads to confusion

• Reasonable efforts by university administrators to derive objective measures for promotion and tenure decisions free of bias.

• Pressures on these administrators to upgrade the reputation and quality of their programs to compete in global academic markets.

• Global competition in scholarship, leading to national policies designed to increase global competitiveness and demonstrate national achievement.

Some Rules for Good Use of Citation Data

• Consider whether available data can address question

• Compare like with like

• Use relative measures, not just absolute counts

• Recognize skewed nature of citation data

• Confirm data collected are relevant to question

• Ask whether the results are reasonable

---Using Bibliometrics in Evaluating Research

http://scientific.thomsonreuters.com/bibliometrics2/

7

Institutional Research Evaluation Workflow

Supporting evaluation workflow:

Tools

Content

Workflow Services

8

Government agencies/funding organizations

IndividualsFaculty, staff, students

University DepartmentsInstitutional research, academic

affairs, tech transfer, etc.

We want to know how the funds that we have granted have been effective in promoting research.

External Entities

University ManagementManagement, including committees,

provost, vice provosts

I’m responsible for seeing how we’re doing overall in performing in terms of bringing in research dollars in overall publications.”

For those aspirational peers...I want to go in and evaluate the faculty and those departments to give me a direct comparison, then I get every single faculty member and run them through the citation index”

“We have built a research information system which is fully integrated with all of the systems—library, research office, grants, and the university -- it’s CV driven, and so every researcher in the institution can access a web based, online view and over time create a research profile for themselves containing up-to-date publications”

“Every author should have a unique identifier in order to retrieve unequivocally his/her articles.

Citation Metrics in Research Evaluation

9

US, NSF: biennial Science & Engineering Indicators report (1974 - )

European Union: EC’s DG XII (Research Directorate)

UK: Office of Science & Technology; Higher Education Funding Council

Canada: NSERC, FRSQ (Quebec), Alberta Research Council

France: Min. de la Recherche, OST - Paris, CNRS

Germany: Max Planck Society, several gov’t labs, DKFZ, MDC

Italy: CRUI (University Rectors) MURST (Ministry of Research, CNR

Spain: CSIC (Spanish Science Agency), CIRIT (Catalonia)

Japan: National Institute of Informatics, Ministry of Education, Ministry of Economy, Trade & Industry

People’s Republic of China: Chinese Academy of Science

Korea: Korea Research Foundation, Korea Advanced Inst. Of S&T

Australia: Australian Academy of Science, gov’t lab CSIRO

Turkey: ULAKBIM

Others include: New Zealand, S. Africa, Portugal, Ireland, Switzerland, Austria, Poland, Czech Republic, Singapore, Malaysia, Thailand, Sweden, Norway, Denmark, Finland, Mexico, Brazil, Chile, Argentina, Uruguay, Russia… and more!

Nations and Agencies using TS Evaluative Reports

Citations and Evaluation in the Humanities• Medium of Scholarly Communication

– Importance of monograph (e.g. Cronin & La Barre, 2004)

• Disciplinary Role of Citations– Intellectual debt or literary artifact?

• Usefulness of journal citations in humanities evaluation– High level of citations to non-journal literature (e.g. Larivere, 2006)

– Useful in aggregate for specific disciplines? (e.g. Oppenheim, 2008)

– Useful as indicators of non-journal literature? (e.g. Butler, 2006)

• Characteristics of the ISI Citation Databases– Regional coverage issues (Archimbault, 2006; Nederhof, 2006)

– Increasing Coverage in ISI Citation Databases

10

Growth of Journal Coverage in WoS: 2002-2008

Web of Science

2002 2008 Growth Variance

Total Journal Coverage

8,607 10,185 1,578 +18%

11

Growth of Journal Coverage in WoS Focus: Regions

Web of Science: Regions

2002 2008 Growth Variance

Asia Pacific 497 767 270 +54%

European Union

1684 2155 471 +28%

Middle East/Africa

54 109 55 +102%

Latin America

63 160 97 +154%

12

Growth of Journals in Arts & Humanities 2002-2008

2002 2008 Growth Variance

A&H Coverage 1122 1238 116 10%

Overall Growth:

Regional Growth:Region 2002 2008 Growth Variance

Asia Pacific 27 40 13 48%

European Union

532 626 94 18%

Latin America 7 16 9 129%

Middle East/Africa

8 15 7 88%

North America 548 541 -7 -1%

Growth of Papers in Humanities Fields1981-2007

15

ERIH Category # ERIH Titles (Total)

# Shared  w/ WoS

Percent in WoS

Anthropology, Evolutionary 36 27 75%Anthropology, Social 195 54 28%Archaeology 390 50 13%Art, Architectural & Design history 418 76 18%Classical Studies 208 41 20%Gender Studies 108 42 39%History 757 192 25%History & Philosophy of Science 146 55 38%Linguistics 542 149 27%Literature 728 215 30%Musicology 160 58 36%Pedagogical & Educational Research

434 151 35%

Philosophy 246 89 36%Psychology 532 503 95%Religious Studies_Theology 305 58 19%Total 5205 1760 34%

Comparison with European Reference Index for the Humanities (ERIH) Journal List

European Reference Index for the Humanities (ERIH), http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/erih-initial-lists.html

16

ERIH Category ERIH "A List" Titles

Included in WoS

Percent of "A List" in WoS

Anthropology, Evolutionary 8 8 100%Anthropology, Social 29 19 66%Archaeology 69 31 45%Art, Architectural & Design history

101 47 47%

Classical Studies 47 16 34%Gender Studies 28 21 75%History 98 72 73%History & Philosophy of Science

25 20 80%

Linguistics 83 55 66%Literature 170 93 55%Musicology 20 16 80%Pedagogical & Educational Research

73 50 68%

Philosophy 23 19 83%Psychology 184 184 100%Religious Studies_Theology 53 20 38%Total 1011 671 66%

Comparison with ERIH “A” List

17

ERIH Category ERIH "B List"

"B List" in WoS

Percent of "B List" in WoS

Anthropology, Evolutionary 20 17 85%Anthropology, Social 87 32 37%Archaeology 159 14 9%Art, Architectural & Design history

192 24 13%

Classical Studies 86 20 23%Gender Studies 54 19 35%History 303 98 32%History & Philosophy of Science

80 31 39%

Linguistics 195 69 35%Literature 343 104 30%Musicology 83 27 33%Pedagogical & Educational Research

274 97 35%

Philosophy 104 42 40%Psychology 264 260 98%Religious Studies_Theology 168 35 21%

Total 2412 889 37%

Comparison with ERIH “B” List

References• Archambault, E., Vignola-Gagne, E., Côté, G., Larivière, Gingras, Y., “Benchmarking Scientific Output in

the Social Sciences and Humanities: The Limits of Existing Databases,” Scientometrics, 68(3) 329-342, 2006.

• Butler, L., Visser, M., “Extending Citation Analysis to Non-Source Items,” Scientometrics, 66(2): 327-343, 2006.

• Cronin, B., and La Barre, K., “Mickey Mouse and Milton: Book Publishing in the Humanities”, Learned Publishing, 17(2): 85-98, 2004.

• European Reference Index for the Humanities (ERIH), http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/erih-initial-lists.html

• Larivière, V., Archambault, E., Gingras, Y, Vignola-Gagné, “The Place of Serials in Referencing Practices: Comparing Natural Sciences and Engineering with Social Sciences and Humanities,” Journal of the American Society for Information Science and Technology, 57(8): 997-1004, 2006.

• Lawrence, P. A. (2007). "The Mismeasurement of Science." Current Biology 17(15): R583-585.

• Nederhof, A., “Bibliometric Monitoring of Research Performance in the Social Science and Humanities: A Review,” Scientometrics, 66(1) 81-100, 2006.

• Oppenheim, C., Summers, M., “Citation Counts and the Research Assessment Exercise, Part VI: Unit of Assessment 67 (Music),” Information Research—An International Electronic Journal, 13(2): 342, June 2008.

• Using Bibliometrics in Evaluating Research, http://scientific.thomsonreuters.com/bibliometrics2/

18

19

Thank You!