Improving Peer Review CSR Initiatives toni scarpa [email protected] 301-435-1109 PRAC Nov. 3, 2008 National Institutes of Health U.S. Department of Health and Human Services
May 24, 2015
Improving Peer Review
CSR Initiatives
toni scarpa [email protected]
301-435-1109
PRAC
Nov. 3, 2008
National Institutes of Health
U.S. Department of Health and Human Services
New Data and the Drivers for Change
CSR’s Efforts to Enhance Peer Review
The NIH Director’s Peer Review Initiatives
Improving Peer Review
New Data and the Drivers for Change
CSR Peer Review: 2008
• 77,000 applications received
• 56,000 applications reviewed
• 16,000 reviewers
• 240 Scientific Review Officers
• 1,800 review meetings
$13.
7
$15.
6
$17.
8 $20.
5 $23.
3 $27.
1
$28.
0
$28.
6
$28.
6
$29.
1
$29.
5
$29.
5
$0
$5
$10
$15
$20
$25
$30
1998
1999
2000
2001
2002
2003
2004
200520
0620
0720
0820
09
1st Driver: The NIH Budget
Doubling
0
20000
40000
60000
80000
1998 2000 2002 2004 2006 2008
2nd Driver: Number of Applications Submitted
Historical Growth
SBIR Applications
7000
6000
5000
4000
3000
2000
1000
01999 2000 2001 2002 2003 2004 2005 2006 2007
INST
CSR
R01 Applications Reviewed at CSR vs. Other ICs
0
5000
10000
15000
20000
25000
30000
35000
2001 2002 2003 2004 2005 2006 2007 2008
IC
CSR
R21 Applications Reviewed at CSR vs. Other ICs
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
2001 2002 2003 2004 2005 2006 2007 2008
IC
CSR
0
2
4
6
8
10
12
3rd Driver: Reviewer’s Load
Applications Per Reviewer
October Council Rounds
4th Driver: CSR Budget
0
10
20
30
40
50
60
70
2004 2005 2006 2007 2008 2009
CSR Reviewer Cost Constant $
$ M
illion
s
4th Driver: CSR Budget
0
10
20
30
40
50
60
70
2004 2005 2006 2007 2008 2009
CSR Reviewer Cost Constant $
$ M
illion
s
Annual Savings in Reviewers’ Expenses Budget
• Non-refundable tickets with one possible change
$15 million• 3,000 fewer reviewers
$3 million• 15% reviews using electronic platforms
$5 million • One meeting a year on the West Coast
$1.8 million
Next Savings?
• Replace shipping of CDs with online downloading
Cost of 18,000 CDs $750,000
Cost of shipping $200,000
Handling $50,000
• 5% more reviews using electronic platform
SREA budget $1,500,000
CSR budget -$100,000
• More meetings in Baltimore
Savings/100 meetings $500,000
• Review of fellowship applications
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
2. Recruiting CSR Staff
3. Shortening the Review Cycle
4. Holding Out-of-Town Meetings
5. Recruiting the Best Reviewers
6. Performing a Second Cycle of IRG Reviews
7. Revising Study Section Guidelines
8. Training Chairs
9. Managing Conflicts of Interest
10. Advancing Additional Review Platforms and Processes
CSR’s Efforts to Enhance Peer Review
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
1. CSR Organization of Peer Review
Divisions Scientific Review OfficersIntegrated Review Groups
1. CSR Reorganization
Rene
Etcheberrigaray
Translational and Clinical Sciences
Cardiovascular and Respiratory Sciences
Surgical Sciences, Biomedical
Imaging and Bioengineering
Musculoskeletal, Oral And Skin Sciences
Oncology 2 – Translational Clinical
Vascular and Hematology
Physiological and Pathological Sciences
Endocrinology, Metabolism, Nutrition & Reproductive Sciences
Immunology
Infectious Diseases& Microbiology
Digestive, Kidney &Urological Systems
Neuroscience, Development and Aging
Brain Disorders &Clinical Neuroscience
Molecular, Cellular &Developmental Neuroscience
Integrative, Functional & Cognitive Neuroscience
Emerging Technologies &Training in Neuroscience
Biology of Development &
Biobehavioral & Behavioral Processes
Risk, Prevention& Health Behaviors
Epidemiology & Population Sciences
Healthcare Delivery & Methodologies
AIDS &Related Research
AIDS, Behavioral and Population Sciences
Basic and Integrative Biological Sciences
Biological Chemistry & Macromolecular
Biophysics
Bioengineering Sciences& Technologies
Genes, Genomes &Genetics
Oncology 1 – Basic Translational
Cell Biology
Interdisciplinary Molecular & Training
Don
Schneider
Update on CSR Realignments
Don Schneider
Update on CSR Realignments
Don Schneider
1. CSR Organization of Peer Review
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
2. Recruiting CSR Staff
2. Recruiting CSR Staff
3 Division Directors
6 Integrated Review Chiefs
20 Scientific Review Officers
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
2. Recruiting CSR Staff
3. Shortening the Review Cycle
3. Shortening the Review Cycle
The Goal
• To provide applicants a review and score within 3 months of application submission. This will permit resubmission of applications (when doable and desirable) 4 months earlier than in the past.
RO1 A1 Resubmission Within 4 Months of Original Application
0
100
200
300
400
500
600
2006 2007 2008
New PI
Estab.PI
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
2. Recruiting CSR Staff
3. Shortening the Review Cycle
4. Holding Out-of-Town-Meetings
4. Holding Out-of-Town Meetings
• To facilitate participation from colleagues on the West Coast, one meeting out of three each year will be held in the metro areas of:o Los Angeleso San Franciscoo Seattleo Chicago
Benefits• Great satisfaction• Significant savings• Ranking of hotels
5. Near-Term Solutions for Recruiting and Retaining the Best Reviewers
Move a meeting a year to the West Coast
Develop a national registry of volunteer reviewers Searchable database with 3,500 reviewers
Provide tangible rewards for reviewers No submission deadlines for chartered members of study sections (effective February 2008). 1,574 chartered members used flexible deadlines during the last 6 months
Provide flexible time for reviewers Choice of 3 times/year for 4 years or2 times/year for 6 years
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
2. Recruiting CSR Staff
3. Shortening the Review Cycle
4. Out-of-Town Meetings
5. Recruiting the Best Reviewers
6. Performing a Second Cycle of IRG Reviews
6. Second Cycle of IRG ReviewsScheduled 2008-5 IRGs Scheduled 2009-12 IRGs (Continued)
Brain Disorders and Clinical Neuroscience (BDCN) and Molecular, Cellular and Developmental Neuroscience (MDCN)
Bioengineering Sciences and Technologies (BST) andSurgical Sciences, Biomedical Imaging and Bioengineering (SBIB)
Biology of Development and Aging (BDA), Emerging Technologies and Training in Neuroscience (ETTN) andIntegrative, Functional and Cognitive Neuroscience (IFCN)
AIDS and Related Research (AARR) andEndocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR)
Scheduled 2009-12 IRGs Scheduled 2010-8 IRGs
Biobehavioral and Behavioral Processes (BBB) andRisk, Prevention and Health Behavior (RPHB)
Vascular and Hematology (VH) andCardiovascular and Respiratory Sciences (CVR)
Healthcare Delivery and Methodologies (HDM) andEpidemiology and Population Sciences (EPS)
Musculoskeletal, Oral and Skin Sciences (MOSS) and Digestive, Kidney and Urological Systems (DKUS)
Immunology (IMM) and Infectious Diseases and Microbiology (IDM)
Genes, Genomes and Genetics (GGG) andBiological, Chemical and Macromolecular Biophysics (BCMB)
Oncology 1—Basic and Translational (OBT) andOncology 2—Translational Clinical (OTC)
Interdisciplinary Molecular Sciences and Training (IMST) and Cell Biology (CB)
6. Performing a Second Cycle of IRG Reviews
• All pertinent study section chairs were invited• Advice on study section and IRG• Revision of study sections description/outline• Training of the chair
6. Performing a Second Cycle of IRG Reviews
Second Cycle of IRG Reviews
Don Schneider
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
2. Recruiting CSR Staff
3. Shortening the Review Cycle
4. Holding Out-of-Town Meetings
5. Recruiting the Best Reviewers
6. Performing a Second Cycle of IRG Reviews
7. Revising Study Section Guidelines
7. Revising Study Section Outlines
• Cellular Signaling and Regulatory Systems
• [Roster]
• The Cellular Signaling and Regulatory Systems (CSRS) study section reviews applications that focus on the initiation and execution of programs that control cellular homeostasis and physiology. A distinguishing characteristic of these applications is an emphasis on signaling networks and the coordination of processes related to cell proliferation, survival, and growth.
• Cell cycle regulation, mitosis, meiosis, checkpoint controls and regulation by ubiquitination• Proteolytic mechanisms associated with cell cycle, senescence and death• Programmed cell death and apoptosis, particularly their regulation in the context of stress,
growth, and transformation. • Proliferation and growth control by the nucleus; signaling pathways regulating transcription• Integrative cell physiology, e.g., stress, clocks, cellular modeling; cell differentiation and
transformation• Basic studies of cytokine signaling• Application of state-of-the-art technologies such as imaging and computational modeling of
cellular signaling networks
• Study sections with most closely related areas of similar science listed in rank order are:
• Molecular and Integrative Signal Transduction• Intercellular Interactions• Membrane Biology and Protein Processing• Molecular Genetics A• Molecular Genetics B
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
2. Recruiting CSR Staff
3. Shortening the Review Cycle
4. Holding Out-of-Town Meetings
5. Recruiting the Best Reviewers
6. Performing a Second Cycle of IRG Reviews
7. Revising Study Section Guidelines
8. Training Chairs
8. Training Study Section Chairs
Scope and Schedule • 120 new chairs every year, more if “permanent” SEPs
chairs are included• 5 meetings in January 2009 for chairs recently appointed• 5 meetings in July 2009 for chairs appointed next June
Training Program• Share data • Explain the new changes and the significance• Share the best practice (ours and theirs)• Answer questions and address concerns• Make Chairs more effective stakeholders
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
2. Recruiting CSR Staff
3. Shortening the Review Cycle
4. Holding Out-of-Town Meetings
5. Recruiting the Best Reviewers
6. Performing a Second Cycle of IRG Reviews
7. Revised Study Section Guidelines
8. Training of Chairs
9. Managing Conflicts of Interest
9. Managing Conflicts of Interest
Report of Conflict of Interest Committee
Noni Byrnes
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR’s Review Groups
2. Recruiting CSR Staff
3. Shortening the Review Cycle
4. Holding Out-of-Town Meetings
5. Recruiting Reviewers
6. Performing a Second Cycle of IRG Review
7. Revising Study Section Guidelines
8. Training of Chairs
9. Managing Conflicts of interest
10. Advancing Additional Review Platforms and Processes
10. Advancing Additional Review Platforms and Process
• Video (VED)
• Internet (AED)
• Editorial Board
• Order of Discussion
10. Advancing Additional Review Platforms and Processes
Editorial Board Reviewo Heavy triage based on innovation and potential science
transformation by a small study section of distinguished, broad-science reviewers (the editors)
o Specific science reviewed by appropriate reviewers (the editorial board)
o Final ranking by the editors
10. Advancing Additional Review Platforms and Process
New Transformative R01
Betsy Wilder
10. Advancing Additional Review Platforms and Process
Electronic Discussion Survey
Andrea Kopstein
NIH Enhancement of Peer Review
• The Charge from Dr. Zerhouni:
““Fund the best science, by the best scientists, Fund the best science, by the best scientists,
with the least administrative burden…”with the least administrative burden…”
http://enhancing-peer-review.nih.gov
Two advisory committees to the NIH Director
The NIH Director’s Peer Review Initiatives
The NIH Director’s Peer Review Initiatives
Corporate NIH Peer Review Enhancement: Implementation
Raynard Kington
This is CSR
September 2008
Recognizing Outstanding Reviewers
• Going above and beyond the call of duty
• Having a significant impact on the quality of peer review
• Educating and inspiring other reviewers
• Setting the highest standard of peer review excellence
The First Marcy Speer Award, 2007
Marcy Speer
2008 Marcy Speer Outstanding CSR Reviewer Award
Dr. David J. Sahn, M.D. Oregon Health & Science University
The 2nd Marcy Speer Award, 2008