preliminary evaluation of the program

INTERIM REPORTINTERIM REPORTEVALUATIONEVALUATION

OF PILOT TO SHORTEN THE OF PILOT TO SHORTEN THE REVIEW CYCLEREVIEW CYCLE

Presentation to Peer Review Advisory Committee22 May 2006

Bettie J. Graham, Ph.D.for the Committee

THE LONG-TERM GOAL THE LONG-TERM GOAL

TO ALLOW AMENDED TO ALLOW AMENDED APPLICATIONS TO BE APPLICATIONS TO BE SUBMITTED FOR THE VERY SUBMITTED FOR THE VERY NEXT REVIEW CYCLENEXT REVIEW CYCLE

RECOMMENDATIONSRECOMMENDATIONS(Amended Applications)(Amended Applications)

PILOT R01* APPLICATIONS IN A FEW CSR STUDY PILOT R01* APPLICATIONS IN A FEW CSR STUDY SECTIONS (3 cycles)SECTIONS (3 cycles)

EVALUATEEVALUATE MAKE DECISION TO/WHEN TO MOVE TO NEXT PHASEMAKE DECISION TO/WHEN TO MOVE TO NEXT PHASE

EXPAND TO ALL R01* APPLICATIONS IN ALL CSR EXPAND TO ALL R01* APPLICATIONS IN ALL CSR STUDY SECTIONS STUDY SECTIONS

EVALUATE EVALUATE MAKE DECISION TO/WHEN TO MOVE TO NEXT PHASEMAKE DECISION TO/WHEN TO MOVE TO NEXT PHASE

EXPAND TO ALL APPLICATIONS IN ALLEXPAND TO ALL APPLICATIONS IN ALLCSR STUDY SECTIONS CSR STUDY SECTIONS

EVALUATE EVALUATE

OVERARCHING PRINCIPLEOVERARCHING PRINCIPLE

MAINTAIN CORE VALUES OF NIH MAINTAIN CORE VALUES OF NIH PEER REVIEWPEER REVIEW

Expertise appropriate to review Expertise appropriate to review applicationsapplications

Fairness/Objectivity in reviewing Fairness/Objectivity in reviewing applicationsapplications

Timeline for Shortening the Review CycleTimeline for Shortening the Review Cycle

End of Survey Collection Data End of Survey Collection Data

11/20 R01* Amended

7/20 R01* Amended

Feb Mar Apr May Jun Jul Aug Sep Oct

3/1Type 2Receipt

and amendeds

4/15 - 5/15Mailouts

Sep - OctCouncils

2/1Type 1Receipt

3/15 End of Appls

to IRG

6/20 - 6/25 SS for RO1 * Applications

Releasedand in Commons

5/15 - 6/15Meetings

Cycle 1 (Feb – Oct)

Jun Jul Aug Sep Oct Nov Dec Jan Feb

6/1Type 1Receipt

Jan - FebCouncils

8/15 - 9/15Mailouts

10/20 - 10/25SS for RO1* Applications

Releasedand in Commons

9/15-10/15Meetings

7/1Type 2Receiptminus

New PIAmended

7/15Deadline for Type 1and Type 2 in IRG

7/25 Last appl toIRG/end

of appls to IRG

Cycle 2 (Jun – Feb)

Cycle 3 (Oct – Jun)

Oct Nov Dec Jan Feb Mar Apr May Jun

10/1Type 1Receipt

1/15 - 2/15Meetings

May - JunCouncils

12/15 - 1/15Mailouts

2/20 - 2/25SS for R01* Applications

Releasedand in Commons

11/25 - Last appl to IRG/end of appls to IRG

11/1Type 2

Receipt/allamendeds

R01*

11/15Deadline for Type 1and Type 2 in IRG

End of Survey Collection Data End of Survey Collection Data

10,025 Total R01 applications received

18,958 Total applications received

2,466 New investigator R01s

received

Oct 2006 ApplicationsOct 2006 Applications

PURPOSE OF PURPOSE OF EVALUATION EVALUATION

TO PROVIDE INFORMATION TO TO PROVIDE INFORMATION TO NIH DECISION MAKERS WHO NIH DECISION MAKERS WHO WILL DETERMINE IF/WHEN TO WILL DETERMINE IF/WHEN TO PROCEED TO THE NEXT PROCEED TO THE NEXT PHASE. PHASE.

DESCRIPTION OF DESCRIPTION OF PILOT EVALUATION PROCESSPILOT EVALUATION PROCESS

40 STUDY SECTIONS FROM 10 IRGS WERE SELECTED40 STUDY SECTIONS FROM 10 IRGS WERE SELECTED

STAKEHOLDERS INCLUDE:STAKEHOLDERS INCLUDE:

REFERRAL OFFICERSREFERRAL OFFICERSSCIENTIFIC REVIEW ADMINISTRATORSSCIENTIFIC REVIEW ADMINISTRATORSREVIEWERSREVIEWERSPROGRAM ADMINISTRATORSPROGRAM ADMINISTRATORSAPPLICANTSAPPLICANTS

QUALITATIVE AND QUANTITATIVE INFORMATION WILL QUALITATIVE AND QUANTITATIVE INFORMATION WILL BE COLLECTEDBE COLLECTED

EVALUATION OF PILOTEVALUATION OF PILOT

QUALITATIVE---SURVEYSQUALITATIVE---SURVEYS

QUANTITATIVE---IMPAC II QUANTITATIVE---IMPAC II DATADATA

PARTICIPATING IRGSPARTICIPATING IRGS[NIH(25 ICs); FDA (1); CDC (4)] [NIH(25 ICs); FDA (1); CDC (4)]

Bioengineering Sciences and Technologies (2)Bioengineering Sciences and Technologies (2)

Biological Chemistry and Macromolecular Biophysics (6)Biological Chemistry and Macromolecular Biophysics (6)

Genes, Genomes and Genetics (6)Genes, Genomes and Genetics (6)

Endocrinology, Metabolism, Nutrition and Reproductive Endocrinology, Metabolism, Nutrition and Reproductive Sciences (3)Sciences (3)

Infectious Diseases and Microbiology (3)Infectious Diseases and Microbiology (3)

Oncological Sciences (2)Oncological Sciences (2)

Digestive Sciences (5)Digestive Sciences (5)

Respiratory Sciences (3)Respiratory Sciences (3)

Health of the Population (6)Health of the Population (6)

Surgical Sciences, Biomedical Imaging, and Bioengineering Surgical Sciences, Biomedical Imaging, and Bioengineering (4)(4)

IMPLEMENTATION OF PILOTIMPLEMENTATION OF PILOT

APPLICATIONS REFERRED TO STUDY SECTIONS EARLIER.APPLICATIONS REFERRED TO STUDY SECTIONS EARLIER.

REVIEWERS HAVE APPROXIMATELY 4 WEEKS TO EVALUATE REVIEWERS HAVE APPROXIMATELY 4 WEEKS TO EVALUATE APPLICATIONS.APPLICATIONS.

STUDY SECTION MEETINGS HELD EARLIER.STUDY SECTION MEETINGS HELD EARLIER.

SUMMARY STATEMENTS WRITTEN WITHIN ONE WEEK FOR R01* SUMMARY STATEMENTS WRITTEN WITHIN ONE WEEK FOR R01* AND 30 DAYS FOR ALL OTHER APPLICATIONS.AND 30 DAYS FOR ALL OTHER APPLICATIONS.

ADMINISTRATIVE NOTE ON R01* SUMMARY STATEMENTS ABOUT ADMINISTRATIVE NOTE ON R01* SUMMARY STATEMENTS ABOUT PILOTPILOT

APPLICANTS DISCUSS FEASIBILITY OF EARLY RESUBMISSION WITH APPLICANTS DISCUSS FEASIBILITY OF EARLY RESUBMISSION WITH PROGRAM DIRECTORS, IF APPLICABLEPROGRAM DIRECTORS, IF APPLICABLE

AMENDED APPLICATIONS MAY BE SUBMITTED 20 DAYS AFTER AMENDED APPLICATIONS MAY BE SUBMITTED 20 DAYS AFTER RECEIPT DATE FOR TYPE 2/AMENDED APPLICATIONS.RECEIPT DATE FOR TYPE 2/AMENDED APPLICATIONS.

COMMUNICATIONS COMMUNICATIONS TRIANGLETRIANGLE

ApplicantApplicant

Program ReviewProgram Review

COMMUNICATIONSCOMMUNICATIONSTRIANGLE (cont’d)TRIANGLE (cont’d)

SCIENTIFIC REVIEW ADMINISTRATORSSCIENTIFIC REVIEW ADMINISTRATORSEileen Bradley is meeting with CSR SRAs.Eileen Bradley is meeting with CSR SRAs.

PROGRAM STAFFPROGRAM STAFF Philip Smith (DK) is meeting with IC Program Philip Smith (DK) is meeting with IC Program

Staff and is providing feedback to CSR Staff.Staff and is providing feedback to CSR Staff.

APPLICANTSAPPLICANTSNotice has been placed in the NIH Guide to Grants Notice has been placed in the NIH Guide to Grants

and Contractsand ContractsAddendum will be added to R01* Summary Addendum will be added to R01* Summary

StatementsStatements

EVALUATION OF REFERRAL PROCESS EVALUATION OF REFERRAL PROCESS

WHAT % OF APPLICATIONS WERE WHAT % OF APPLICATIONS WERE REFERRED 6 WEEKS AFTER THE TYPE REFERRED 6 WEEKS AFTER THE TYPE 1 RECEIPT DATE?1 RECEIPT DATE?

WHAT % OF APPLICATIONS WERE WHAT % OF APPLICATIONS WERE SCANNED AND IN IAR 6 WEEKS AFTER SCANNED AND IN IAR 6 WEEKS AFTER THE TYPE 1 RECEIPT DATE?THE TYPE 1 RECEIPT DATE?

EVALUATION OF REFERRAL PROCESS EVALUATION OF REFERRAL PROCESS

WHAT WORKED?WHAT WORKED?

WHAT DID NOT WORK?WHAT DID NOT WORK?

HOW DID THIS PILOT AFFECT YOU HOW DID THIS PILOT AFFECT YOU PROFESSIONALLY AND PERSONALLY?PROFESSIONALLY AND PERSONALLY?

WHAT IMPROVEMENTS WOULD YOU WHAT IMPROVEMENTS WOULD YOU RECOMMEND?RECOMMEND?

EVALUATION OF REFERRAL PROCESS EVALUATION OF REFERRAL PROCESS

REVIEWER SURVEYREVIEWER SURVEY

DO YOU KNOW WHAT THE GOALS OF THE PILOT ARE?DO YOU KNOW WHAT THE GOALS OF THE PILOT ARE?

DID YOU RECEIVE YOUR APPLICATIONS AT LEAST 4 DID YOU RECEIVE YOUR APPLICATIONS AT LEAST 4 WEEKS PRIOR TO THE MEETING?WEEKS PRIOR TO THE MEETING?

DID YOU HAVE SUFFICIENT TIME TO REVIEW YOUR DID YOU HAVE SUFFICIENT TIME TO REVIEW YOUR APPLICATIONS?APPLICATIONS?

DID YOU CHANGE ANY OF YOUR USUAL PRACTICES DID YOU CHANGE ANY OF YOUR USUAL PRACTICES TO ACCOMMODATE THIS PILOT?TO ACCOMMODATE THIS PILOT?

WHAT IMPROVEMENTS WOULD YOU RECOMMEND?WHAT IMPROVEMENTS WOULD YOU RECOMMEND?

FINAL REVIEWER SURVEYFINAL REVIEWER SURVEY

FINAL REVIEWER SURVEYFINAL REVIEWER SURVEY

DATA COLLECTION ON APPLICANTSDATA COLLECTION ON APPLICANTS

Data will be collected from all Data will be collected from all chartered study sections in the pilot chartered study sections in the pilot that review R01 grant applications.that review R01 grant applications.

Comparison groups will be:Comparison groups will be: Pilot new PIs who submit earlyPilot new PIs who submit early Pilot new PIs who do not submit earlyPilot new PIs who do not submit early Non-pilot new PIs.Non-pilot new PIs.

DATA COLLECTION ON APPLICANTSDATA COLLECTION ON APPLICANTS

RESUBMISSIONSRESUBMISSIONS

When did applicants have access to When did applicants have access to their summary statements?their summary statements?

How many resubmitted early?How many resubmitted early? How many resubmitted later and How many resubmitted later and

when?when? How many did not resubmit ?How many did not resubmit ?

DATA COLLECTION ON APPLICANTSDATA COLLECTION ON APPLICANTS

OUTCOMESOUTCOMES

Differences in the priority scoresDifferences in the priority scores Differences in the funding ratesDifferences in the funding rates Differences in the time between Differences in the time between

first review and fundingfirst review and funding

WHAT NEXT?WHAT NEXT?

ANALYZE DATA AS THEY BECOME ANALYZE DATA AS THEY BECOME AVAILABLEAVAILABLE

DEVELOP INSTRUMENTS FOR OTHER DEVELOP INSTRUMENTS FOR OTHER STAKEHOLDERS (Scientific Review STAKEHOLDERS (Scientific Review Administrator survey next)Administrator survey next)

PROVIDE FEEDBACK TO NIH PROVIDE FEEDBACK TO NIH LEADERSHIPLEADERSHIP

KEEP NIH INFORMEDKEEP NIH INFORMED

THE COMMITTEETHE COMMITTEE

Bettie J. Graham, Ph.D. (Co-Chair)National Human Genome Research Institute

Robert Hammond, Ph.D. Consultant

Teresa Levitin, Ph.D.National Institute on Drug Abuse

Jane Steinberg, Ph.D.National Institute of Mental Health

Susan Streufert, Ph.D.National Institute of Child Health andHuman Development

ConsultantGeorgine Pion, Ph.D.Vanderbilt University

Center for Scientific Review Eileen Bradley, D.Sc. (Co-Chair)Chuck Dumais, M.S.Teresa Lindquist , M.S.Elliot Postow, Ph.D.Terra L. Vinson, M.P.A.