peer review final 10-27-10.ppt · PDF filePEER REVIEW Bob Garry ... Hendler, J./ Reinventing Academic Publishing-Part 1; ... • Spend enough time writing and discussing each application

9/19/2011

1

PEER REVIEW

Bob GarryIllustration:James Wang Paradigm Magazine

9/19/2011

2

license agreement: cartoonstock

9/19/2011

3

9/19/2011

4

NIH Study Section Service•Member of AIDS Study Section (SS) 1988-90•Member of AIDS Mol. Biol. SS (1990-96)•Chair, VATID Biodefense SS (2002)•Chair, SBIR-STTR Biodefense SS (2002-06)•Co-Chair Regional Centers of Excellence for Biodefense SSs (2008)•Chair: New Approaches to HIV Vaccines SS (2009, 2010)•Member “Bugs and Drugs” SS (2007 – date)

9/19/2011

5

Definitions• Peer

[1] Someone “of equal standing with another … especially belonging to the same societal group…or [having the same] status” [2] “A noble with a hereditary title, i.e., a peerage, and in times past, with certain rights and privileges not enjoyed by commoners.”

• Review[1] A critical inspection or examination[2] A second or repeated viewing of past events, circumstances or facts.

Wicktionary

9/19/2011

6

Peer Review is performed by:• Publishers: Manuscripts (publish or

perish)• Agencies or foundations: Grants (follow

the money)• The Feds (FDA, CDC, USDA, EPA,

patent office, etc.)• Institutional committees (IRBs, IACUC,

IBC, promotions and tenure, misconduct, etc.)

9/19/2011

7

EDTIOR

REVIEWERS

INPUT

PEER REVIEW

PEERREVIEW

OUTPUT

Submission of manuscripts, grant applications, etc.

Publications, funded grants, etc. OR heartache and rejection

9/19/2011

8

Traditional publishing houses

Elsevier Wiley Springer

40% of all journals are published by the “big three.”

Nature

9/19/2011

9

“Society” journals

9/19/2011

10

Federal Research Public Access Act

Each agency of >$100 million must:• Require researchers to submit an

electronic copy of published papers.• Ensure that the manuscript is

preserved in a stable digital repository.• Require that free, online access for

manuscript occurs within than six months.

9/19/2011

11

The Federal Research Public Access Act does not apply to:

• laboratory notes, preliminary data analyses, author notes, phone logs, or other information used to produce the final manuscript.

• classified research or research that results in works that generate revenue or royalties for the author (such as books). Patentable discoveries are exempt only to the extent necessary to protect copyright or a patent.

9/19/2011

12

Open Access Publishers

9/19/2011

13

Open access to articles: 2009Björk B-C, Welling P, Laakso M, Majlender P, Hedlund T, et al. (2010) Open Access to the Scientific Journal Literature: Situation 2009. PLoS ONE 5(6): e11273. doi:10.1371/journal.pone.0011273

9/19/2011

14

Peer Review: limitations/problems

• slow• expensive• time consuming • highly subjective• prone to bias• easily abused• poor at detecting gross defects• almost useless for detecting fraud.

9/19/2011

15

Nobel winning science:Problems or outright rejection

Rejecting and resisting Nobel class discoveries:accounts by Nobel LaureatesJUAN MIGUEL CAMPANARIOScientometrics, Vol. 81, No. 2 (2009) 549–565

9/19/2011

16

Nature rejection of Kreb’s cycle manuscript

Hans A. Krebs, PhD(1900-1981)Nobel Prize, 1953

Rejecting and resisting Nobel class discoveries:accounts by Nobel LaureatesJUAN MIGUEL CAMPANARIOScientometrics, Vol. 81, No. 2 (2009) 549–565

9/19/2011

17

Sir Winston ChurchillBritish politician (1874 - 1965)

“It has been said that democracy is the worst form of government except all the others that have been tried.”

“It has been said that peer review is the worst system of scientific accountability except all the others that have been tried.”

9/19/2011

18

9/19/2011

19

Open review(“Where everybody knows your name”)

• Reviewers sign their reviews.

• Has also been used to describe a process whereby community scientists can post (signed or unsigned) reviews –better name: Public review.

9/19/2011

20

Open reviewing: Pro• Maintains balance ( judgemental role vs.

helping authors• Adds credibility• Reviewers become more accountable• Eliminate abuses of the system (maybe)• Little justification for secrecy• Less disagreeable/more polite (maybe)

9/19/2011

21

Open reviewing: Cons• Reprisal, especially by big “Kahunas”• Creates an “old boy” network • Resentment and animosity• Higher acceptance rate?• Not broke - don’t fix it

9/19/2011

22

The Internet permits more interactive science publishing

• Peer review need no longer be a one-step process

• Online comment systems allow analysis after an article is published.

• Some journals use a two-stage system formally (for ex. Physics).

9/19/2011

23

Nature's public review trial –aka open comment (2006)

• 1,369 papers were received for review during the trial period.

• 71 authors (or 5%) agreed to their papers being displayed for open comment.

• 33 received no comments; 38 (54%) received 92 technical comments (more than half [49] were to 8 papers).

9/19/2011

24

The future of scientific publishing• Several models are being investigated:

eg. open publication and archiving.• Many journals are adding community-

oriented features. • Online scientific interaction outside

journals (Science Blogs) is becoming increasingly important

Hendler, J./ Reinventing Academic Publishing-Part 1;IEEE Intelligent Systems 22, 2-3, 2007.

9/19/2011

25

Should I review the paper?

Am I qualified?

Do I have a Conflict of Interest?

Do I have the Time?

9/19/2011

26

Peer Review of Manuscripts Do’s

• Support work of high quality• Set the standards of the journal and of the

field • Consider the appropriateness of the paper

for the journal • Appropriately challenge flawed papers (but

be collegial). • Provide your scientific expertise (editorial

assistance – rewriting paper not needed)

9/19/2011

27

Peer Review of Manuscripts Don’ts

• Don’t give your scientific stamp of approval to a flawed paper.

• Don’t demean the authors.• Don’t agree to review unless you have the time

(and don’t delay papers on purpose or not) • Don’t pass the paper on to someone else to

review? • Don’t contact the author about the work or the

paper? • Don’t undermine the confidentiality of the review

process.

9/19/2011

28

Peer review at NIH: Process Overview

• The goal: evaluate applications in a way that is fair, equitable, timely, and free of bias.

• Review is a TWO-step process:1.Scientific Review Group (SRG). 2.Institute and Center (IC) National Advisory

Councils or Boards. • Only applications favorably recommended

by both the SRG and the Advisory Council may be recommended for funding.

9/19/2011

29

NIH CONFLICT OF INTEREST RULES:

• Employment• Financial Benefit • Personal Relationships (Relatives)• Professional Associates • Standing Review Group Membership • Longstanding Disagreements • Multi-Site Or Multi-Component Project • Request For Applications (RFA) Or Request

For Proposals (RFP)

9/19/2011

30

NIH CONFIDENTIALITY AND NON DISCLOSURE

RULES: • You must sign a CONFIDENTIALITY

AND NON-DISCLOSURE OF MATERIALS AND PROCEEDINGS agreement to participate in an NIH review.

9/19/2011

31

9/19/2011

32

Study Section: Seating chartProgram Staff

Scientific Review Officer (SRO)

Chair

Administrative AssistantReviewers

Telephone reviewers

http://www.youtube.com/watch?v=HMO3HoLJuJY

9/19/2011

33

(Fr. trier, separate, sort, sift or select) all applications are reviewed, but generally only a fraction (40-60%) are discussed

Triage

9/19/2011

34

Scored Review Criteria Significance

• Is an important problem or a critical barrier addressed?

• Will knowledge, technical capability, and/or clinical practice be improved?

• Will the proposed research change concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

9/19/2011

35

Scored Review Criteria Investigator(s)

• Are researchers well suited to the project? • Do Early Stage Investigators or New

Investigators have appropriate experience and training? Have established investigators demonstrated an ongoing record of accomplishments?

• If collaborative or multi-PD/PI, do the investigators have complementary/integrated expertise; is leadership approach appropriate?

9/19/2011

36

9/19/2011

37

Scored Review Criteria Innovation

• Will current research or clinical practice paradigms be shifted?

• Are the concepts, approaches or methodologies, instrumentation, or interventions novel?

• Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

9/19/2011

38

Schachman H K J. Biol. Chem. 2006;281:6889-6903

9/19/2011

39

Scored Review Criteria - Approach

• Will approaches accomplish the specific aims of the project?

• Are problems, alternative strategies, and benchmarks for success presented?

• Will the strategy establish feasibility and will particularly risky aspects be managed?

• Are human subjects minorities children, etc. justified

9/19/2011

40

Scored Review Criteria Environment

• Does the scientific environment contribute to the probability of success?

• Is institutional support, equipment and other physical resources adequate?

• Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?

9/19/2011

41

9/19/2011

42

Additional Review Criteria(Scored)

• Protections for Human Subjects• Inclusion of Women, Minorities, and

Children• Vertebrate Animals• Biohazards• Resubmission• Renewal• Revision

9/19/2011

43

Additional Review Considerations (unscored)

• Budget• Applications from Foreign Organizations• Select Agents• Resource Sharing Plans• Period of Support

9/19/2011

44

Overall Impact• “Reviewers will provide an overall

impact/priority score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the scored review criteria, and additional review criteria (as applicable for the project proposed).”

9/19/2011

45

New scoring (10-90)

9/19/2011

46

Second Level of Review • NIH program staff members examine

overall impact/priority scores, percentile rankings and summary statements and consider IC's needs.

• Program staff provide a grant-funding plan to Council.

• Council considers IC’s goals and needs and advises the IC director.

• IC director makes final funding decisions.

9/19/2011

47

9/19/2011

48

9/19/2011

49

Peer Review of Grants Do’s

• Support work of high quality• Appropriately challenge flawed

applications• Be collegial to the applicants and

the other members of the study section and program officers.

• Provide your scientific expertise.• Spend enough time writing and

discussing each application.

9/19/2011

50

Peer Review of Grants Don’ts

• Don’t give your scientific stamp of approval to a flawed application.

• Don’t demean the applicants, the other members of the SS or program officers.

• Don’t let your students write the review? • Don’t contact the applicants about the

work or the application? • Don’t undermine the confidentiality of the

review process.

9/19/2011

51

Sir Winston ChurchillBritish politician (1874 - 1965)

“It has been said that peer review is the worst system of scientific accountability except all the others that have been tried.”

W.C. FieldsAmerican comedian(1880 - 1946)

““If at first you don't succeed, try, try again. Then quit. There's no point in being a damn fool about it.”

9/19/2011

52

Ethical concerns may arise during peer review

9/19/2011

53

9/19/2011

54

http://ori.hhs.gov/education/products/niu_authorship/mistakes/index.htm#

9/19/2011

55

“On one hand, reviewers and editors must take all appropriate steps to preclude publication of duplicate, plagiarized or fraudulent papers. On the other hand, the suspicion of scientific misconduct can have a devastating impact on a scientific career, even if deliberate malevolence is eventually disproved. Because of this, the reviewer should carefully review the facts underlying his/her concerns.”

Sara Rockwell, Ph.D. Yale University School of Medicine

9/19/2011

56

– “Carefully reviewed studies sometimes turn out to be wrong because later attempts at repetition fail. But peer review requires authors to provide more data and more confirming material, making it likelier that careful efforts at confirmation will follow. Fraud is something quite different, and very hard to detect… The reporting of scientific results is based on trust. It's better to trust our colleagues, despite the fact that on rare occasions one of them might disappoint other scientists and those hoping for cures.”

Donald Kennedy , Editor-in-Chief of Science. Science 2006: Vol. 311; no. 5758, p. 145 -on the occasion of the Woo Suk Hwang scandal

9/19/2011

57

Research Misconduct NIH reviewers

“Research misconduct is defined as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results, but not honest error or differences of opinion. It is vital that you do not make allegations of potential misconduct in the critique. Instead, such concerns must be brought to the attention of the SRO in a confidential manner, preferably before the study section meets.”

9/19/2011

58

9/19/2011

59

• Dr. XXX has entered into a Voluntary Exclusion Agreement in which she has voluntarily agreed, for a period of three (3) years, beginning on July 22, 2010:

1. To exclude herself from any contracting or subcontracting with any agency of the U.S. Government and from eligibility for, or involvement in, nonprocurement programs of the U.S. Government referred to as ``covered transactions'' pursuant to the HHS Implementation of OMB Guidelines to Agencies on Governmentwide Debarment and Suspension at 2 CFR 376, et seq.; and

2. To exclude herself from serving in any advisory capacity to PHS, including but not limited to service on any PHS advisory committee, board, and/or peer review committee, or as a consultant.

Posted on http://ori.hhs.gov/ and sent to federal grant recipients via email

9/19/2011

60

.

Schachman H K J. Biol. Chem. 2006;281:6889-6903©2006 by American Society for Biochemistry and Molecular Biology

PLEASE – DO NOT BE THIS GUY!!!