Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular.

Center for Scientific ReviewNational Institutes of HealthDepartment of Health and Human Services

Toni Scarpa

NIH Peer Review: Continuity and Change

APS/Cardiovascular Section

Salt Francisco, April 2006

Peer Review: An N.I.H. “Conception”

• Is the heart and soul of NIH• Has produced an effective partnership between the

federal government and research institutions• Has created the best academic medical centers, the best

biomedical/behavioral research and biotechnology• Has made possible the best cures and the best prevention• Has been admired and imitated here and abroad• Has protected NIH against outside influence

Center for Scientific Review

Applications received for all of NIH and applications referred for CSR review, FY 1998-2004

0

20,000

40,000

60,000

80,000

1998 2000 2002 2004Fiscal year

Nu

mb

er o

f ap

pli

cati

on

s

Applications received for all of NIH

Applications assigned for review by CSR

CSR 4 Review Divisions & 23 IRGs

Division of Biologic Basis of Disease

Elliot Postow, Ph.D.

AIDS and RelatedResearch IRG (AARR)

Ranga V. Srinivas, Ph.D.

Endocrinology, Metabolism, Nutrition, and Reproductive

Sciences (EMNR)Sooja Kim, Ph.D.

Immunological SciencesIRG (IMM)

Calbert Laing, Ph.D.

Oncological SciencesIRG (ONC)

Syed Quadri, Ph.D.

Infectious Diseases andMicrobiology IRG (IDM)

Alex Politis, Ph.D.

Division of Physiology and Pathology

Michael Martin, Ph.D.

Cardiovascular SciencesIRG (CVS)

Joyce Gibson, D.Sc.

Integrative, Functionaland Cognitive Neuroscience

IRG (IFCN)Christine Melchior, Ph.D.

Renal and Urological Sciences IRG (RUS)

Daniel McDonald, Ph.D.

HematologyIRG (HEME)

Joyce Gibson, D.Sc.

Digestive Sciences IRG (DIG)

Mushtaq Khan, Ph.D., DVM

Division of Clinical andPopulation-Based Studies

Anita Miller Sostek, Ph.D

Surgery, Radiology, andBioengineering IRG (SRB)

Eileen Bradley, D.Sc.

Health of the Population IRG (HOP)

Robert Weller, Ph.D.

Risk, Prevention, and HealthBehavior IRG (RPHB)Michael Micklin, Ph.D.

Brain Disorders and ClinicalNeuroscience IRG (BDCN)

Dana Plude, Ph.D.

Behavioral & BiobehavioralProcesses IRG (BBBP)Karen Sirocco, Ph.D.

Division of Molecular and Cellular Mechanisms

Donald Schneider, Ph.D.

Bioengineering Sciences and Technologies IRG (BST)

Sally Amero, Ph.D.

Biology of Development and and Aging (BDA)

Sherry Dupere, Ph.D.

Biological Chemistry and Macromolecular Biophysics

IRG (BCMB)John Bowers, Ph.D.

Cell Biology IRG (CB)Marcia Steinberg, Ph.D.

Genes, Genomes, and Genetics IRG (GGG)

Richard Panniers, Ph.D.

Molecular, Cellular and Developmental

Neuroscience IRG (MDCN)Carole Jelsema, Ph.D.

Respiratory Sciences IRG (RES)

Mushtaq Khan, Ph.D., DVM

Musculoskeletal, Oral, and Skin Sciences IRG (MOSS)

Daniel McDonald, Ph.D.

CSR Mission Statement

To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.

Time

C

om

ple

xity

an

d I

mp

act

CSR Operations

C

urrent S

ystems

New Systems?

Necessary Changes in CSR Peer Review Operations

• Increase communications between CSR, the ICs, our reviewers and applicants

• Increase uniformity

• Increase efficiency

• Facilitate work of IC program staff

Changes in CSR Operations

Potential of Knowledge Management Tools for Peer Review

Collexis Software or Others

• Knowledge management solutions• Fingerprinting and text retrieving• Disease coding

Benefits for Peer Review

• Assigning applications to Integrated Review Groups or Study Sections

• Selecting reviewers (one application, multiple applications)

Nine pilots are underway to begin to assess these benefits

Study Section Realignment

• Review of one IRG every month• Total review every 2 years

Required Changes in Current Systems

• Shorten the review cycle

This is Not an Ford Assembly Line

Receipt ReferEvaluate Scientific

Merit of Applications

EnterpriseArchitecture@mail.nih.gov

Shortening the NIH Review Cycle, Initial Steps

For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05)

We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)

Desirable Changes in CSR Review

• Shorten the review cycle

• Address concern that clinical research is not properly evaluated

• Improve the assessment of innovative, high- risk/high-reward research

“The judging of grants has become a charade.”

The American Society for Cell Biology

“The judging of grants has become a charade. To be funded, the experimental plan has become a litany of experiments already accomplished so that everything is feasible. When grants come back with unfundable scores, new investigators may not have sufficient resources to do the experiments that “show feasibility.”

Zena WerbPresident, ASCB

Newsletter August 2005

Possible Changes in Current Systems

• Shorten the review cycle

• Address concern that clinical research is not properly evaluated

• Improve the assessment of innovative, high- risk/high-reward research

• Do more to recruit and retain more high-quality reviewers

Expanding Peer Review’s Platforms

Electronic Reviews• Telephone Enhanced Discussions• Video Enhanced Discussions• Asynchronous Electronic Discussions

Study Sections

Necessity ● Clinical reviewers Preference ● Physicists, computational biologists

New Opportunities ● Fogarty, International Reviewers

The First NIH Study Section

1945

The Last NIH Study Section

2005

If we didn’t have any peer-review system and we had to design one from scratch, what would it look like?

Possible New Systems

Questions—Applications

• Should we have 3 or 365 deadlines for most applications?

• Should applications (Rs) be shortened? Should appendices be eliminated or reduced in size?

• Is there more value in having 2-3 reviewers reading 25-page applications or 10-15 reviewers reading 5-page applications?

Questions—Study Section Meetings

• What is the ideal number of members to have serve on a study section?

• Is one study section with 50-70 reviewers efficient?

• What is the intellectual contribution of individual reviewers in large study sections?

Questions—Scoring

• Is it proper or desirable to have 50-70 reviewers voting on priority scores for each application referred to their study section?

• Is consensus always good? Or should we focus on score variance?

Applications Received for All of NIH FY 1998-2004

0

20,000

40,000

60,000

80,000

1998 2000 2002 2004Fiscal year

Nu

mb

er o

f ap

pli

cati

on

s

Number of Research Grant Applications/Applicant

1.05

1.1

1.15

1.2

1.25

1.3

1.35

1.4

1.45

1980

1982

1984

1986

1988

1990

1992

1994

1996

1998

2000

2002

2004

CSR Applications Reviewed, Regular and SEP May Council Only

0

2000

4000

6000

8000

10000

12000

14000

16000

1999 2000 2001 2002 2003 2004 2005

Applications Regular

Applications SEP

Study Section Application/Reviewer Ratio October Council Only

0

0.5

1

1.5

2

2.5

3

3.5

4

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Regular

SEP

Two groups of challenges/opportunities

• Reviewers– Decrease the number of reviewers and increase the

quality– Increase the number of applications reviewed

without extra workload– Recruit and retain the best reviewers

• SRA– Increase efficiency– Recruit and train

Possible Short Term Approaches for Increasing Efficiency for Reviewers and CSR

• Replace Many SEPs with Smaller Parallel Study Sections

• Enlarge Study Section Membership and Decrease Frequency of Participation

• Pre Meeting Streamlining• Various Review Platforms• Hybrid Review Platforms• Staggering Application Deadlines• 2 instead of 3 reviews• Shorten Applications• More Structured Applications and Reviews

Size of Grant Applications

• RO1 • Will increase number of applications reviewed by

reviewers• Will decrease the number of reviewers in a study section• May be combined with a change in format of the

application, more consonant to review criteria• May be combined with scoring individual criteria (i.e.

relevance, innovation, etc• Strong support by councils and scientific leadership• May result in more innovation

This is CSR

Coronary Heart DiseaseAge-Adjusted Death Rates in U.S.: Actual (blue) vs Expected (yellow)

500

400

300

200

100

01950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000

Dea

ths

per

100

,000

Year

~ 514,000 Actual Deathsin 2000

~ 1,329,000 Projected Deaths in 2000

815,000 Deaths Prevented in 2000

The True Value of Peer Review