Estimating the Quality of Business and Management Journals from the UK Research Assessment Exercise John Mingers, Paola Scaparra, Kate Watson Kent Business.

Estimating the Quality of Business and Management Journals from the UK Research Assessment ExerciseJohn Mingers, Paola Scaparra, Kate Watson

Kent Business SchoolCentre for the Evaluation of Research Performance

IMEC 2011, Hong Kong

John Mingers

• Many different journal rankings each with its own biases and prejudices

• They are based on often arbitrary criteria. They can be by peer review or behavioural (e.g, impact factors)

• The original Kent ranking was simply a statistical combination of other rankings

• “Objectivity results from a combination of subjectivities” (Ackoff)

Journal quality

Journal rankings

1. Journal rankings

Why are they so contentious?

John Mingers

Journal quality

Journal rankings

Paper quality

Researcher quality

• Paper quality is unknown unless we peer review it – hence the RAE; so is researcher quality – no little Lion markSo we impute them from the journal ranking

• THEORY 1: The quality of a journal purely reflects the quality of its papers (Editors/publishers/common sense)

• THEORY 2: Low quality papers may be published in high quality journals and vice versa (RAE)

• It matters in terms of publication strategy and decision-making Is journal a good proxy for quality?

John Mingers

2. Reconstructing the 2008 RAE Grades

Submission statistics for the last three RAEsAdapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a)a Totals differ slightly between different sources. Figures for 2008 are after data cleaning as described later

1996 2001 2008

No of submissions 100 97 90

No. of staff submitted 2300+ 3000+ 3300

Total no. of outputs 8000+ 9942 12575

No. of journal papers (% of total)a

5494 (69%) 7973 (80%) 11625 (92%)

No. of journal titles 1275 1582 1639

Mean outputs/journal 4.3 5.0 7.1

Mean outputs/institution 80.0 102.5 139.7

John Mingers

Output Type

Description 1996 2001 2008

A Authored book 431 285B Edited book 77 60C Chapter in book 863 332D Journal article 7973 11374E Conference contribution 295 85G Software 3 1H Internet publication 24 318N Research report for

external body80 98

T Other form of assessable output

184 22

Total 9942 12575

Number of publications by output typeAdapted from Geary et al (2004), Bence and Oppenheim (2004), RAE (2009a). Categories with zero entries have been suppressed

John Mingers

Pareto curve for the number of entries per journal in the 2008 RAE

John Mingers

Numbers of journals in the RAE and the ABS list

John Mingers

2008 Rank

2001 Rank

Journal Name 2008 Total 2001 Total

ABS Grade

ABS subject area

1 2 Journal of Management Studies 219 116 4 GEN MAN

2 8 Human Relations 171 78 4 ORG STUD

3 3 Journal of the Operational Research Society 153 113 3 OR&MANSCI

4 5 European Journal of Marketing 146 90 3 MKT

5 10 Organization Studies 144 75 4 ORG STUD

6 14 European Journal of Operational Research 137 61 3 OR&MANSCI

7 6 International Journal of Operations and Production Management

134 85 3 OPS&TECH

8 9 International Journal of Human Resource Management

133 76 3 HRM&EMP

9 1 Journal of Marketing Management 125 127 3 FINANCE

9 11 Journal of Business Finance and Accounting 125 65 3 MKT

11 4 British Journal of Management 108 107 4 GEN MAN

12 13 Work, Employment and Society 103 64 4 HRM&EMP

13 7 British Journal of Industrial Relations 99 84 4 HRM&EMP

14 Regional Studies 97 4 SOC SCI

15 Research Policy 95 4 SOC SCI

16 16 Service Industries Journal 92 59 2 MKT

17 Critical Perspectives on Accounting 89 3 ACCOUNT

18 20 Organization 83 55 3 ORG STUD

19 Accounting, Organisations and Society 82 4 ACCOUNT

20 Journal of Business Ethics 81 3 ETH-GOV

20 Journal of Business Research 81 3 GEN MAN

Journals most frequently submitted to RAE

John Mingers

Initial model (QP1)Let: j index the journals (j = 1 .. no. of journals)g index the grades 0* - 4* (g = 0 .. 4) i index the universities (i = 1 .. no. of institutions)eig be the estimated proportion of research at grade g for university ipjg be the estimated proportion of the outputs of journal j graded at grade guig be the actual proportion of research at grade g for university inij be the number of entries of journal j submitted by university i   

s.t.

for each institution (i) and grade (g)

for each journal (j)

2.1 The LP Model

John Mingers

2.2 Validity of the Results ABS 2009 Kent 2007 Geary

medianABS 1* or 4*

RAE grade 0.42 (574) 0.37 (575) 0.42 (416)

ABS 2009 .69 (574) 0.48 (394)Kent 2007 0.49 (394)Geary medianRAE grade 1* or 4* .61 (183)

Type Modal grade Mean grade No of outputs No. of institutions

Authored book 2 2.37 285 68

Book chapter 2 2.24 333 65

Edited book 2 1.87 61 28

External report 1 1.35 102 35

Other 1 1.31 169 47

John Mingers

Journal Title % 4.0 % 3.0 RAE Entries ABS 2009 1* to 4*

FT Top 40 y/n?

Management International Review 100.0 0.0 27 3 yJournal of the Academy of Marketing Science

100.0 0.0 18 4

Accounting, Organisations and Society 100.0 0.0 82 4 yHarvard Business Review 100.0 0.0 19 3 yJournal of Applied Psychology 100.0 0.0 33 4 yAbacus 100.0 0.0 25 2Journal of Marketing Research 100.0 0.0 17 4 yJournal of Financial Economics 100.0 0.0 39 4 yAcademy of Management Review 100.0 0.0 30 4 yFiscal Studies 100.0 0.0 13 2Review of Financial Studies 100.0 0.0 27 4 yInternational Journal of Industrial Organization

100.0 0.0 14 4

American Economic Review 100.0 0.0 17 4 yInformation Systems Journal 100.0 0.0 18 3Organization Science 100.0 0.0 28 4 yMIT Sloan Management Review 100.0 0.0 23 3 yMarketing Science 98.2 0.0 12 4 yCommunications of the ACM 91.8 5.2 13 3Financial Accountability and Management 87.5 4.3 30 3European Journal of Information Systems 85.7 13.6 49 3

TOP-20 JOURNALS BASED ON % 4* RECONSTRUCTED FROM RAE OUTPUTS

John Mingers

ABS Grades RAE Estimated Grades

All journals

Journals not in RAE

Journals in RAE and our list

All our list

Journals not in ABS

Journals in ABS and our list

4* 10% 4% 15% 17% 13% 18%

3* 24% 12% 31% 29% 28% 31%

2* 37% 39% 37% 28% 26% 28%

1* 27% 45% 17% 23% 19% 22%

0* 3% 13% 2%GPA 2.17 1.74 2.43 2.34 2.09 2.41

PROPORTIONS OF JOURNALS IN PARTICULAR RANKS COMPARING ABS WITH RAE

John Mingers

Conclusions from Table 9

• Overall RAE grades were higher than overall ABS grades (cols 1, 4) but this was because of selectivity of submissions

• This can be seen by comparing the ABS submitted with the ABS not submitted (cols 2, 3)

• Comparing those journals that are in common the level of grading is very similar (cols 3,6)

• In the RAE , ABS journals were graded more highly than non-ABS journals (cols 5,6)

John Mingers

Scattergram showing association between GPA and proportion of an institution’s submitted journals that are in ABS

John Mingers

There are at least 3 possible explanations of this:

Higher % ABS journals

Better RAE grades

Higher quality of department

Higher % ABS journals

Higher % ABS journals

Better RAE grades

“RAE Bias”

“Better depts. more mainstream”

“Greater selectivity”

Higher quality of department

John Mingers

Journal Title RAE Grade

Entries % 4.0 % 3.0 % 2.0

Management Science 4 29 69.9 29.0 1.1Group Decision and Negotiation 4 4 100 0 0Operations Research Letters 4 4 100 0 0Computational Optimization & Applications 4 3 100 0 0

SIAM Journal on Optimization 4 3 100 0 0Journal of Heuristics 4 3 100 0 0Decision Sciences 4 3 100 0 0European Journal of Operational Research 3 137 0.0 98.2 1.8

Computers and Operations Research 3 20 0.0 67.9 32.1Operations Research 3 20 0.0 98.9 1.1Journal Royal Statistical Society A 3 9 26.2 44.8 28.8Theory and Decision 3 9 0 80.0 19.9IEEE Trans. on Systems, Man, & Cybernetics 3 7 0 84.4 15.5

Mathematical Programming 3 7 0 98.3 1.6Advances in Applied Probability 3 5 0 81.0 18.9Annals of Operations Research 3 5 0 96.2 3.7Journal of Combinatorial Optimization 3 4 0 96.5 3.4Computational Statistics & Data Analysis 3 3 0 82.0 17.9

TOP JOURNALS IN OPERATIONAL RESEARCH RANKED BY IMPUTED RAE GRADE

John Mingers

3. Technical conclusions

• Rankings are just a heuristic device and should not be taken as synonymous with quality

• We can use the RAE data to reconstruct the judgements they made

John Mingers

4. Strategic questions

• Current measurement regimes are hugely distorting to research:• Narrow focus on types of outputs – ie “4*”

English language journal articles• Narrow focus on types of measurements• Narrow focus on types of impact

• Should we stop now and develop a system that aims to evaluate quality in a variety of forms, a variety of media, through a variety of measures with the ultimate goal of answering significant questions?Adler, N. and Harzing, A-W. (2009) “When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings”, Academy of Management Learning and Education 8, 1, 72-95