Peer reviewer training part I: What do we know about peer review? Dr Trish Groves Deputy editor, BMJ.

Peer reviewer trainingpart I:

What do we know about peer review?

Dr Trish GrovesDeputy editor, BMJ

What do editors want from papers?

• Importance• Originality• Relevance to readers• Usefulness to readers and, ultimately, to

patients• Truth• Excitement/ “wow” factor• Clear and engaging writing

Peer review

• As many processes as journals or grant giving bodies

• No operational definition--usually implies “external review”

• Largely unstudied till 1990s

• Benefits through improving what’s published rather than sorting wheat from chaff

What is peer review?

• Review by peers

• Includes: internal review (by editorial staff) external review (by experts in the field)

BMJ papers

• All manuscripts handled by our online editorial office at http://submit.bmj.com

• The website uses a system called Benchpress

• Reviewers recruited by invitation, through volunteering, and by authors’ suggestions

• Database also includes all authors

• We monitor reviewers’ workload for BMJ

• We rate reviewers’ reports using a 3 point scale

BMJ peer review process I

• 7000 research papers, 7% accepted

• approximate numbers at each stage:– 1000 rejected by one editor within 48 hours

– further 3000 rejected with second editor

– within one week of submission 3000 read by senior editor; further 1500 rejected

– 1500 sent to two reviewers; then 500 more rejected

– approx 1000 screened by clinical epidemiology editor and more rejected

BMJ peer review process II

• 400-500 to weekly manuscript meeting attended by the Editor, an external editorial adviser (a specialist or primary care doctor) and a statistician..

• …and the full team of BMJ research editors, plus the BMJ clinical epidemiology editor

• 350 research articles accepted, usually after revision

• value added by commissioned editorials and commentaries

BMJ peer review process III

• always willing to consider first appeals--but must revise the paper, respond to criticisms, not just say subject’s important

• perhaps 20% accepted on appeal• no second appeals; always ends in tears;

plenty of other journals

What we know aboutpeer review

Research evidence

Peer review processes

• “Stand at the top of the stairs with a pile of papers and throw them down the stairs. Those that reach the bottom are published.”

• “Sort the papers into two piles: those to be published and those to be rejected. Then swap them over.”

Some problems

• Means different things at different journals • Slow• Expensive• Subjective• Biased • Open to abuse • Poor at detecting errors• Almost useless at detecting fraud

Is peer review reliable?(How often do two reviewers agree?)

NEJM (Ingelfinger F 1974)• Rates of agreement only “moderately better than chance”

(Kappa = 0.26)• Agreement greater for rejection than acceptance

Grant review • Cole et al, 1981 – real vs sham panel, agreed on 75% of

decisions• Hodgson C, 1997 – two real panels reviewing the same

grants, 73% agreement

Are two reviewers enough?• Fletcher and Fletcher 1999 - need at least six reviewers,

all favouring rejection or acceptance, to yield a stats significant conclusion (p<0.05)

Should we mind if reviewers don’t agree?

• Very high reliability might mean that all reviewers think the same

• Reviewers may be chosen for differing positions or areas of expertise

• Peer review decisions are like diagnostic tests: false positives and false negatives are inevitable (Kassirer and Campion, 1994)

• Larger journals ask reviewers to advise on publication, not to decide

Bias

Author-related• Prestige (author/institution)• Gender• Where they live and work

Paper-related• Positive results• English language

Prestigious institution bias

Peters and Ceci, 1982

Resubmitted 12 altered articles to psychology journals that had already published them

Changed:• title/abstract/introduction - only slightly• authors’ names• name of institution, from prestigious to unknown

fictitious name (eg. “Tri-Valley Center for Human Potential”)

Peters and Ceci - results

• Three articles recognised as resubmissions

• One accepted

• Eight rejected (all because of poor study design, inadequate statistical analysis, or poor quality: none on grounds of lack of originality)

How easy is it to hide authors’ identity?

• Not easy

• In RCTs of blinded peer review, reviewers correctly identified author or institution in 24-50% of cases

Reviewers identified (open review) – results of RCTs

Asking reviewers to sign their reportsin RCTs made no difference to the qualityof reviews or recommendations made

• Godlee et al, 1998• van Rooyen et al, 1998• van Rooyen et al ,1999

Open review on the web

Various experiments and evaluations are underway…

What makes a good reviewer? – results of RCTs

• Aged under 40

• Good institution

• Methodological training (statistics & epidemiology)

What might improve the quality of reviews?

• Reward/credit/acknowledgement?

• Careful selection?

• Training?

• Greater accountability (open review on web)?

• Interaction between author and reviewer (real time open review)?