Disclosure I do not have any relationship(s) with commercial interests.

04/19/2023 LSUHSC-NO ORS Training Series 1

LSU School of Medicine-New Orleans (LSUSOM-NO) is the provider of Continuing Medical Education for this activity. The planning and presentation of all LSUSOM-NO activities ensure balance, independence, objectivity and scientific rigor.

The LSU School of Medicine-New Orleans designates this educational activity for a maximum of 1.5 AMA PRA Category 1 Credit(s) ™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.


Disclosure

I do not have any relationship(s) with commercial interests.

A commercial interest is any entity producing, marketing, re-selling, or distributing health care goods or services consumed by, or used on, patients.


The NIH Peer Review Process


Welcome

– Presenter:• Nicole G. Hammill, MBA• Coordinator of Grants and Development• Office of Research Services

– 433 Bolivar Street, Room 206E, New Orleans, LA 70112– [email protected]– (504) 568-4970 tel– (504) 568-8808 fax

– http: http://www.lsuhsc.edu/administration/academic/ors/grants_contracts_processing.aspx

http://www.lsuhsc.edu/administration/academic/ors/grants_contracts_processing.aspx




Office of Research ServicesDirector:

Dr. Kenneth E. Kratz

Staff:

Nicole G. Hammill – Pre-award (Grants and Contracts)Rose Castay – IACUC and IBCDyan Melson – IRBLynn Arnold – IRBMisty White – IRB Anissa McDougle – Conflicts of Interest

Responsibilities:

• Pre-award, sponsored project activity; this includes evaluation and routing for signatures all grant applications, research agreements, and clinical trial agreements.

• Conflict of Interest Program based upon Chancellor’s Memorandum #35 “Individual and Institutional COI in Sponsored Projects”.

• The AAHRPP “Fully Accredited” Human Research Protection Program and Institutional Review Board (IRB) which provides oversight for the protection of human subjects participating in research.

• The Institutional Animal Care and Use Committee (IACUC) which provides oversight for the welfare of animals used in research.

• The Institutional Bio-safety Committee (IBC) which in collaboration with the Office of Environmental Health and Safety provides oversight of bio-safety issues and recombinant DNA research.


Helpful Administrative Information

• Most of the numbers, dates, names, and titles commonly needed for the completion of grant applications can be found here:

http://www.lsuhsc.edu/administration/academic/ors/docs/Helpful_Administrative_Information.pdf

Updated frequently!






• Mandated by law – PHS Health Act • Defined in federal regulation – 42 CFR

52h• Further defined in NIH policy • Per year:

– Nearly 80,000 applications– Over 18,000 reviewers

Two-Tiered Process


• First Tier: Initial peer review

– Recommendations on scientific and technical merit – Scientific Review Groups (“SRGs” or “Study Sections”)

• Second Tier: Advisory Council or Board

– Recommendations to the Institutes/Centers (“ICs”) on funding, appeals, program priorities

– “Council”

• Final funding decisions – IC Director

Two-Tiered Process


• Application is received at the NIH Center for Scientific Review (“CSR”).

• First Tier assignments include:– Scientific Review Group (SRG)– Study Section– Institute/Center (IC)– Scientific Review Officer

• Second Tier assignments include:– Advisory Council or Board

Referral of Application


• Funding considerations include:– Specific ICs (each IC has its own budget,

priorities, and paylines)– Dual assignment (to more than one IC) is

possible– Input from Program Officer

• Funding decisions– Ultimately made by the director of the IC

Referral of Application, con’t.


CSR Review

• Most R01s, fellowships, and small business applications

• Some Program Announcements (PAs, PARs)

• Some Requests for Applications (RFAs)

IC Review

• IC-specific features

• Program projects

• Training grants

• Career development awards

• RFAs

Referral to a Scientific Review Group

The review locus is stated in the Funding Opportunity Announcement (“FOA”)


Sample Review LociU54 - IC REVIEWhttp://grants1.nih.gov/grants/guide/rfa-files/RFA-CA-11-003.html

R01 - CSR REVIEWhttp://grants.nih.gov/grants/guide/pa-files/PA-10-067.html

Applications that are complete and responsive to the FOA will be evaluated for scientific and technical merit by an appropriate peer review group convened by the National Cancer Institute and in accordance with NIH peer review procedures (http://grants1.nih.gov/grants/peer/), using the review criteria stated below.

As part of the scientific peer review, all applications will:

Undergo a selection process in which only those applications deemed to have the highest scientific and technical merit, generally the top half of applications under review, will be discussed and assigned an impact/priority score;

Receive a written critique; and

Receive a second level of review by National Cancer Advisory Board.

Applications submitted for this funding opportunity will be assigned on the basis of established PHS referral guidelines to the ICs for funding consideration.

Applications that are complete will be evaluated for scientific and technical merit by (an) appropriate scientific review group(s) in accordance with NIH peer review procedures (http://grants.nih.gov/grants/peer/) using the review criteria stated below.

As part of the scientific peer review, all applications will:

Undergo a selection process in which only those applications deemed to have the highest scientific and technical merit, generally the top half of applications under review, will be discussed and assigned an impact/priority score;

Receive a written critique; and Receive a second level of review by appropriate national advisory council or Board.

http://grants1.nih.gov/grants/guide/rfa-files/RFA-CA-11-003.html

http://grants1.nih.gov/grants/guide/rfa-files/RFA-CA-11-003.html

http://grants.nih.gov/grants/guide/pa-files/PA-10-067.html

http://grants.nih.gov/grants/guide/pa-files/PA-10-067.html

http://grants1.nih.gov/grants/peer/

http://grants.nih.gov/grants/peer/


• Include in the application’s Cover letter: – Application title– FOA # and title– Request:

• Assignment to particular SRG or study section– SRG rosters are posted 30 days before the SRG meeting:

» http://era.nih.gov/roster/index.cfm » http://www.csr.nih.gov/committees/rosterindex.asp

• Assignment to particular IC for funding consideration

– Provide:• Disciplines involved, if multidisciplinary• Explanation for late application, if necessary

Not all requests can be honored

Requesting Referral to a Specific SRG

http://era.nih.gov/roster/index.cfm

http://www.csr.nih.gov/committees/rosterindex.asp


• First level of peer review– Designated Federal Official– Extramural scientist administrator– Identifies and recruits reviewers– Manages conflicts of interest– Oversees arrangements for review meetings– Presides at review committee meetings– Prepares and releases summary statements

The Scientific Review Officer (SRO)


• Recruiting Criteria include:– Expertise– Stature in field– Mature judgment– Impartiality– Ability to work well in a group– Managed conflicts of interest

• NIH attempts to ensure:– Balanced representation

• Gender• Geography• Diversity• Seniority

– Availability

Peer Reviewers


• “Chartered” SRGs– Multi-year terms– Formal appointment process– May include temporary members for special

expertise

• Special Emphasis Panels (SEPs)– Ad hoc membership– Often meet only once

Types of Scientific Review Groups (SRGs)


• Regular reviewers – permanent and temporary– Preliminary impact/priority scores, criterion scores,

written critiques– Final impact/priority scores

• Other Contributing Reviewers (“mail” reviewers)– Written critiques, criterion scores, preliminary

impact/priority scores– Cannot submit final impact/priority scores

Types of Reviewers


• For each application: – ≥ Three qualified reviewers are assigned (“2 + 1”)– Assignments are made by the SRO

• Based on the scientific content of application• Expertise of the reviewer• Suggestions from the PI on types of expertise –

– not names!• Suggestions from Program staff• Suggestions from SRG members• Managing conflicts of interest• Balancing workload• Assignments are confidential

Reviewer Assignments


• Potential COIs between a reviewer and an application:– Financial – Employment– Personal – Professional– SRG membership – Other interestsTwo COI vouchers are submitted by each SRG

member.

Reviewer Conflicts of Interest (COI)


• Make recommendations on merit - not funding!– Scientific and technical merit – Budget and project duration – Protection of human subjects, inclusion plans, vertebrate

animals, biohazards– Resource Sharing Plans– Other administrative factors

• Provide:– Impact/priority scores– Criterion scores– Written critiques

Scientific Review Groups (SRGs)


• All confidential materials, discussions, documents are deleted, retrieved, or destroyed

• Reviewers sent guidance with applications• Application information provided on secure

websites or protected portable devices• All questions must be referred to SRO • SRG meetings are closed to the public• Program staff may observe SRG meeting

Do not contact reviewers directly!

Confidentiality


• Overall Impact:– Likelihood for the project to exert a sustained,

powerful influence on the research field(s) involved • In consideration of:

– At least five scored criteria• Receive individual, numerical scores• Additional criteria in certain announcements

– Additional review criteria• As applicable for the project proposed• Do not receive individual, numerical scores• Additional criteria in certain announcements

Application Scoring


• Applications for:– Research Grants– Cooperative Agreements– Significance– Investigator(s)– Innovation– Approach– Environment– (FOA-specific criteria)

• Other review criteria apply to other mechanisms• See “Review Criteria at a Glance” (

http://grants.nih.gov/grants/peer/reviewer_guidelines.htm)

Scored Review Criteria

http://grants.nih.gov/grants/peer/reviewer_guidelines.htm


• Does the project address an important problem or a critical barrier to progress in the field?

• If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved?

• How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

Review Criterion: Significance


• Are the PD/PIs, collaborators, and other researchers well suited to the project?

• If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training?

• If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)?

• If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?

Review Criterion: Investigator(s)


• Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions?

• Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense?

• Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

Review Criterion: Innovation


• Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project?

• Are potential problems, alternative strategies, and benchmarks for success presented?

• If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?

Review Criterion: Approach


• If the project involves clinical research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed?

Review Criterion: Approach (con’t.)


• Will the scientific environment in which the work will be done contribute to the probability of success?

• Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed?

• Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?

Review Criterion: Environment


• As applicable for the project proposed, reviewers:– Consider in determining scientific and technical merit– Do not give separate scores for these items– FOA-specific criteria – Protections for Human Subjects– Inclusion of Women, Minorities, and Children – Vertebrate Animals – Resubmission Applications– Renewal Applications– Revision Applications– Biohazards

Additional Review Criteria


• As applicable for the project proposed, reviewers:– Address each item– Do not give scores for these items– Should not consider them in providing an overall

impact/priority score– FOA-specific considerations– Select Agent Research– Applications from Foreign Organizations– Resource Sharing Plans– Budget and Period Support

Additional Review Considerations


• Numerical scores– 1.0 (exceptional) to 9.0 (poor)– Final impact/priority score - average of individual

scores x 10– Individual criterion scores– Ranked by percentile for certain mechanisms– Not Discussed (ND) – streamlining– Other designations (DF = Deferred, AB = Abstention,

CF = Conflict, NP = Not Present)– Final impact/priority scores range from 10 through 90.

NIH Scoring System


• Preliminary scores (before the SRG meeting)– Entered by assigned reviewers and discussants in

secure website– Made available to other SRG members

• Final overall impact/priority scores (at the SRG meeting)– Voted by private ballot– All eligible SRG members vote

• Reviewers are instructed to revise their criterion scores after the meeting.

NIH Scoring System (con’t)


Impact Score Descriptor

High Impact1 Exceptional2 Outstanding3 Excellent

Moderate Impact4 Very Good5 Good6 Satisfactory

Low Impact7 Fair8 Marginal9 Poor

Score Descriptors


• Allows discussion of more meritorious applications– Less meritorious applications tabled at the

SRG meeting, designated Not Discussed (ND)

– Requires full concurrence of the entire SRG– Summary statement:

• Reviewer critiques• Individual criterion scores• No numerical, overall impact/priority score

Streamlining


• Score order of review– SRG discusses most meritorious applications first– Entire SRG decides when to stop, which

applications will not be discussed in panel

• Other order of review (e.g., IC assignment, mechanism)– SRO prepares a list of average preliminary scores– Distributes to SRG– Entire SRG decides which applications to discuss

Streamlining (con’t)


• SRO– Performs administrative review of applications– Recruits reviewers, arranges for meeting date and site– Assigns 3 SRG members to each application– Makes applications available to reviewers

• Internet Assisted Review (IAR) site or on CDs• Usually about six weeks before the SRG meeting

– Instructs reviewers in review procedures– Monitors posting of initial scores and critiques in IAR

• Documents for Reviewers are available at: http://grants.nih.gov/grants/peer/reviewer_guidelines.htm#general_guidelines

Pre-Meeting SRG Procedures




• New summary statement format– Bulleted comments from reviewers, less text– Criterion scores from assigned reviewers– Decreases variability– Increases quality of information in critiques– More succinct, better organized– Encourages evaluative statements– Ensures that reviewers address all review criteria and considerations– Reviewers also write a paragraph summarizing the factors that informed

their overall impact score to supplement the bulleted critiques.

• Critique templates are available at: http://grants.nih.gov/grants/peer/reviewer_guidelines.htm#general_guidelines

Structured Critiques




Templates for Reviewers


Templates for Reviewers (con’t)


































• Reviewers– Examine assignments– Submit Conflict of Interest and Confidentiality voucher– Read applications, prepare written critiques in templates– Enter preliminary scores into IAR– Read and consider other critiques and preliminary

scores– Make travel and hotel arrangements– Preliminary scores and critiques may be due

several days or a week in advance!

Pre-Meeting SRG Procedures


• The NIH will only accept administrative materials resulting from unanticipated events, such as:

• Revised budget page(s) (e.g., change in budget request due to new funding or institutional acquisition of equipment);

• Biographical sketches (e.g., change in senior/key personnel due to the hiring, replacement, or loss of an investigator);

• Letters of support or collaboration resulting from a change in senior/key personnel due to the hiring, replacement, or loss of an investigator;

• Adjustments resulting from natural disasters (e.g., loss of an animal colony);

• Adjustments resulting from change of institution (e.g., PD/PI moves to another university);

• News of an article accepted for publication (a copy of the article should not be sent); and

• News of a professional promotion or positive tenure decision for any Program Directors/Principal Investigators and Senior/Key Personnel.

Post-Submission Materials


• Agenda– Call to Order – Chairperson– Policy and instructions – SRO– Discuss applications one at a time– Where feasible:

• In score order• Cluster New Investigator (NI)/Early Stage Investigator (ESI)

applications• Cluster clinical applications

– Score each application by private ballot after its discussion– Discuss other considerations

• Budget • Resource Sharing Plans

SRG Meetings


Clustering NI/ESI Applications• Applies to R01 applications only• New Investigator (NI)

– PD/PI who has not yet competed successfully for a substantial NIH

research grant– If Multiple PD/PIs, all PD/PIs must meet requirements for NI status.

• Early Stage Investigator (ESI)– PD/PI who qualifies as a New Investigator AND is within 10 years of

completing the terminal research degree or is within 10 years of

completing medical residency (or equivalent)• NI and ESI R01 applications are clustered together in review.

– ESI applications are not separately clustered within the NI/ESI group.– NI/ESI applications are identified for reviewers so there can be

appropriate review in context of career stage.– Expectations of preliminary data and publication track record less than

for established investigators.


Clustering Clinical Applications

• The goal of clustering clinical applications in review is to increase fairness in the process of reviewing clinical applications.

• A clinical application is defined as human subjects research minus Exemption 4.

• The clustering of NI/ESI applications takes precedence over the clustering of clinical applications.


• Discussion format– Members with conflicts excused– Initial levels of enthusiasm stated(assigned reviewers

and discussants)– Primary reviewer - explains project, strengths,

weaknesses– Other assigned reviewers and discussants follow– Open discussion (full panel)– Levels of enthusiasm (assigned reviewers) re-stated– Individual SRG members vote– Other review considerations discussed (budget)

SRG Meeting Procedures


• If 60 applications/SRG meeting– ~ 50% streamlined, 30 applications to discuss and

score

• If 9 hour SRG meeting– ~ ½ hour introduction, streamlining– ~ 1 hour lunch, 2 x 15 minute breaks

• Which Leaves– ~ 14 minutes on average/application– ~ 3 - 4 minutes/reviewer

• Thus, clarity and brevity are essential!

SRG Meeting Procedures (con’t)


• eRA Commons (http://era.nih.gov/commons/index.cfm)– Final Impact/Priority Score available three days after the

SRG meeting– Summary statement available 4 – 8 weeks after meeting– Available also to Program Officers at that time– Confidential document– Available to:

• PD/PIs• NIH officials• Advisory Council members• No one else at LSUHSC-NO will see scores

After the Review

http://era.nih.gov/commons/index.cfm


• First page– NIH Program Officer (upper left corner)

• Name• Contact information

– Final Impact/Priority Score or other designation– Percentile (if applicable)– Codes

• Human subjects• Vertebrate animals• Inclusion plans

– Budget request

• A favorable score does not guarantee funding!

Summary Statement


• Subsequent Pages– Description (provided by applicant)– Résumé and Summary of Discussion (if discussed)– Reviewer critiques – essentially unedited– Follow review criteria for mechanism– Protections for Human Subjects– Inclusion Plans– Vertebrate Animals– Biohazards– Budget– Administrative Notes– Meeting roster

Summary Statement (con’t)


Summary Statement (con’t)

• Sample summary statements are available here: http://www.niaid.nih.gov/researchfunding/grant/pages/appsamples.aspx#rpindex.

http://www.niaid.nih.gov/researchfunding/grant/pages/appsamples.aspx

http://www.niaid.nih.gov/researchfunding/grant/pages/appsamples.aspx


Appeals Process

• First, the investigator must contact the NIH program officer (PO) to discuss the concerns and outcomes.

• Second, if the investigator would still like to proceed with an appeal, he or she should contact The Office of Research Services, since our authorized organization representative (AOR), must concur with the appeal in order for it to be accepted by NIH.


Appeals Process (con’t)• The appeal letter must meet these three criteria:

• Describe the flaws in the review process;

• Explain the reasons for the appeal; and

• Be based on one or more of the following issues: • Evidence of bias on the part of one or more peer reviewers;• Conflict of interest, as specified in regulation at 42 CFR 52h.5.

“Scientific Peer Review of Research Grant Applications and Research and Development Contract Projects”, on the part of one or more peer reviewers;

• Lack of appropriate expertise within the Scientific Review Group (SRG); or

• Factual error(s) made by one or more reviewers that could have altered the outcome of review substantially.


Appeals Process (con’t)

• If the letter does not meet the criteria listed above, then the concern is probably a grievance and not an appeal.

• Also, if the appeal letter is based solely on differences of scientific opinion, it will not be accepted by NIH as an appeal.



• If review staff and program staff support the appeal, the original application will be re-reviewed.

• If the review and program staff do not support the appeal, then the PD/PI can either:• Withdraw the appeal, or;• The appeal letter can be made available

to the IC’s council during the second level of review.



• Finally, if the appeal goes to council, council can either recommend re-review or deny the appeal. The PD/PI will be notified of the decision within 30 days of the council meeting.

• More details are provided in the complete Guide Notice, http://grants.nih.gov/grants/guide/notice-files/NOT-OD-11-064.html.

http://grants.nih.gov/grants/guide/notice-files/NOT-OD-11-064.html

http://grants.nih.gov/grants/guide/notice-files/NOT-OD-11-064.html


Advisory/Council Board

• Second level of review – recommendations on:– Research priority areas– Policy– Appeals– Funding– Quality of SRG review

• Members– Scientists from the extramural research community– Public representatives – Appointed to multi-year terms– Appointed as Special Government Employees


Advisory/Council Board (con’t)

• Balanced representation– Expertise– Stature in field– Mature judgment– Impartiality– Managed conflicts of interest– Balanced representation– Gender, Diversity– Geography, Seniority


Funding Considerations

• Authority of the IC Director– Scientific and technical merit (initial peer

review)– Council recommendations– Relevance to program priorities in IC– Compliance with policies– Number of meritorious applications received– Availability of funds– Advice of IC Program Staff


Timeline

Planning, Writing, and Submitting

Planning: Applicant should start early, collect preliminary data, and determine internal headlines

Writing: Applicant often begins writing application several months prior to application due date.

Submitting: Applicant organization submits most applications to NIH through Grants.gov.


Timeline

Receipt and Referral

Months 1-3Applications compliant with NIH policies are assigned for review by the Division of Receipt and Referral in the Center for Scientific Review (CSR).

CSR assigns application to an NIH Institute/Center (IC) and a Scientific Review Group (SRG).

Scientific Review Officer (SRO) assigns applications to reviewers and readers.


Timeline

Peer Review

Months 4 - 8Initial Level of Peer Review: SRG members review and evaluate applications for scientific merit.

Priority Scores: Available to Principal Investigator on eRA Commons.

Summary Statement: Available to Principal Investigator on eRA Commons.

Second Level of Review: Advisory council/board reviews applications.


Timeline

Award

Months 9-10Pre-Award Process: IC grants management staff conducts final administrative review and negotiates award *

Notification of Award : Institute/Center (IC) issues and sends Notice of Award (NoA) to applicant institution/organization.

Congratulations! Project period officially begins!

* NIH requests additional information needed Just-in-Time (“JIT”) for award.


Video: NIH Peer Review Revealed

• This video provides a front-row seat to a peer review meeting.

• http://www.youtube.com/watch?v=fBDxI6l4dOA

http://www.youtube.com/watch?v=fBDxI6l4dOA

http://www.youtube.com/watch?v=fBDxI6l4dOA


Additional Information and Resources

• Enhancing Peer Review Initiative– http://enhancing-peer-review.nih.gov/

• Office of Extramural Research Peer Review Process– http://grants.nih.gov/grants/peer_review_process.htm

• Peer Review Policies & Practices– http://grants.nih.gov/grants/peer/peer.htm

• Center for Scientific Review– http://cms.csr.nih.gov/AboutCSR/Welcome+to+CSR/

http://enhancing-peer-review.nih.gov/

http://grants.nih.gov/grants/peer_review_process.htm

http://grants.nih.gov/grants/peer/peer.htm

http://cms.csr.nih.gov/AboutCSR/Welcome+to+CSR/


Questions?


Thank You!

Disclosure I do not have any relationship(s) with commercial interests.

Documents

ors training series7mandated

ors training series2disclosure

ors training series8first

research agreements

institutional review

recombinant dna research

nih policy

nih peer review process