VA Research & Development Health Services Research and Development Scientific Merit Review Board CyberSeminar: Reviewer Update and Orientation February 1, 2017 February 2017
VA Research & Development Health Services Research and Development
Scientific Merit Review Board
CyberSeminar:
Reviewer Update and Orientation
February 1, 2017
February 2017
Orientation to the March 2017 Scientific Merit Review Board (SMRB)
• Welcome and Outline of today s CyberSeminar
• Introductions of HSRD SPMs and Staff Transitions
• Broad Overview of HSR&D Priorities
• Poll question
• Peer Review core values / Review Panels
• Reviewer responsibilities
• Preparing Critiques / Using the Rating Scale for Scoring
• Preparing for the Meeting
• Activities during the Meeting, and Follow-up after
• CyberSeminar feedback survey VETERANS HEALTH ADMINISTRATION 1
Available on the HSR&D Website:
http://www.hsrd.research.va.gov/funding/Priorit
yDomains2017.pdf?utm_source=enews&utm_
medium=email&utm_campaign=Jan17
HSR&D Updated Priority Domains
Major Priority Domains
Healthcare Access
Healthcare Equity and Health Disparities
Healthcare Informatics
Aging, Long-Term Care, and Caregiving
Mental and Behavioral Health
Women s Health
Implementation Science
Patient-centered Care, Care Management, and Health Promotion
Health Care Systems
Cross-cutting Methods Domains
1. Non-VA services use data to
better track Veteran care
2. Engagement of Veterans,
providers, and other
stakeholders in research
methods, outcomes, and
implementation
3. Business case and policy
analysis, including systematic
evaluations of programs/policies
pertinent to Veteran and VA care
VETERANS HEALTH ADMINISTRATION 2
Poll
• Have you ever received funding from HSR&D?
- *Yes
- *No
VETERANS HEALTH ADMINISTRATION 3
On which review panel(s) are you serving?
When you finish your participation in this CyberSeminar,
please be sure to indicate your review panel(s) on the
survey. These data, together with the feedback you
provide, will help us to provide the most useful
information for reviewers, this round and in the future.
VETERANS HEALTH ADMINISTRATION 4
HSR&D Review panels; SROs
Panel Panel Description SRO Cathie Plouzek, PhD, PMP HSR1 Health Care and Clinical Management
HSR2 Behavioral, Social, and Cultural Determinants of Health and Care Robert Small
Healthcare Informatics Miho Tanaka, PhD HSR3
HSR4 Mental and Behavioral Health Robert W. O'Brien, PhD
HSR5 Health Care System Organization and Delivery Jeffrey P. Haibach, PhD, MPH
HSR6 Post-acute and Long-term Care Cathie Plouzek, PhD, PMP and
George Fitzelle, PhD
Previously John Holden, PhD
CDA Career Development Award Robert Small
NRI Nursing Research Initiative Robert Small
HS1A Learning Healthcare System Initiatives (LHI) Cathie Plouzek, PhD, PMP
Provider Behavior Miho Tanaka, PhD HS3A Data and Measurement Sciences Jeffrey P. Haibach, PhD, MPH HS5A
HS8A Learning Healthcare System Initiative (LHI) Melissa Braganza, MPH
VETERANS H EA L RaTHndomi ADMIzNed ISTPrRAogrTIOam N Evaluation
5
Application Types
• IIR – Investigator Initiated Research
• PPO – Pilot Project Opportunity
• LHI – Learning Health Research Initiative
– Provider Behavior
– Data and Measurement Sciences
– Randomized Program Evaluation
• CDA – Career Development Award
• NRI – Nursing Research Initiative
VETERANS HEALTH ADMINISTRATION 6
Applications Reviewed and Funded
VETERANS HEALTH ADMINISTRATION
98
108
132 131 137
20, 20.4%
19, 17.6%
17, 12.9%
23, 17.6%
27, 19.7%
50
39 34
46 45
9, 18%
8, 20.5%
6, 17.7%
8, 17.4%
9, 20%
0
20
40
60
80
100
120
140
160
Summer 2014 Winter 2015 Summer 2015 *includes LHI proposals
Winter 2016 *includes LHI proposals
Summer 2016 *includes LHI proposals
Number of IIR Projects Reviewed
Number of IIR Projects Funded
Number of Pilot Projects Reviewed
Number of Pilot Projects Funded
7
Peer Review Core Values
• Fairness - Standard review and scoring criteria for each application.
• Transparency - Only published criteria used for evaluation.
• Expert Assessment - Scientific expertise to evaluate potential impact.
• Impartiality - COI, bias and predisposition must be managed for all participants in process (SRO, reviewers, applicants, observers), to avoid inappropriate influence.
• Highest Ethical Standards - Confidentiality of all discussions, application materials, other aspects. - Potential misconduct (very rare) is assessed by HSR&D Central Office.
VETERANS HEALTH ADMINISTRATION 8
Invitation to eRA Commons Internet Assisted Review (IAR) website
• Indicate concurrence with confidentiality agreement.
• Check all applications for conflict of interest (COI).
• Check your assigned applications for appropriateness of
review assignment.
• If new COI or assignment questions arise, discuss with your
Scientific Review Officer (SRO) as soon as possible.
VETERANS HEALTH ADMINISTRATION 9
Key Criteria Considered in Scoring
• Significance
• Approach
• Impact and Innovation
• Investigator Qualifications, Facilities and Resources
Similar criteria categories are used for Pilots and LHI applications.
VETERANS HEALTH ADMINISTRATION 10
Consider the Significance and Expected Impact
Focus on the significance and overall impact of the specific project (if executed successfully), not that of the field or the condition being investigated.
• Significance refers to scientific importance or value of the project, and its value to Veterans health care and health outcomes. While science is incremental, is the advancement in science large enough to be meaningful and generalizable?
• Impact and innovation refers to the ability of the project to yield results that can be expected to change practice, influence scientific methods, disrupt the previous order of things, or introduce novel ideas or methods.
VETERANS HEALTH ADMINISTRATION
11
Scoring Scale 2017
• 1.0 – 1.5: OUTSTANDING- Exceptionally strong with negligible weaknesses; ready for execution "as is."
• 1.6 – 1.9: EXCELLENT- Strong but with weaknesses that should be addressed prior to execution. Re-review not necessarily required.
• 2.0 – 2.3: VERY GOOD - Strong but with weaknesses that should be addressed in a resubmission.
• 2.4 – 2.8: GOOD - Some strengths, but also key weaknesses that require re-working.
• 2.9 – 3.4: FAIR - Major weakness that requires substantial revision before resubmission.
• 3.5 – 5.0: POOR- Major weaknesses that discourage resubmission.
VETERANS HEALTH ADMINISTRATION 12
Other Criteria and Summaries
• If Multiple Principal Investigators , need MPI Leadership Plan (see MPI eligibility policy in !dditional Guidelines document in Meeting Materials)/
• Response to prior review.
– Note: Each round of review is independent.
• Protection of human research participants.
• Inclusion of women and minorities.
• Overall Impression: Summarize your evaluation in a paragraph that includes the key factors that determine your overall priority score.
• Key Strengths and Weaknesses: List/describe the key strengths and weaknesses of the application, with brief supporting statement for each.
VETERANS HEALTH ADMINISTRATION 13
Support Letters
• HSR&D encourages a broad range of research relevant to VHA.
– Partnered research in collaboration with VHA healthcare system executives, clinicians, and Veterans.
– Highly innovative research that is responsive to VHA scientific priorities, but is not yet considered a high priority by a partner.
• Letters of support may be appropriate when research is connected to a healthcare system partner s goals/
VETERANS HEALTH ADMINISTRATION 14
Unscored Criteria
Criteria that should not be factored into the score of scientific merit.
Budget appropriate to scope of work.
Data Management and Access Plan (DMAP)
• Reviewers comment if Data Sharing Plan is reasonable.
VETERANS HEALTH ADMINISTRATION 15
Preparing Your Written Review
• Updated critique templates for IIRs, Pilots, and LHI.
– Hyperlinks to criteria descriptions are in template appendix.
• One overall score (no individual criterion scores).
• Substantive narratives expected for major review criteria. – As appropriate, identify your comments as Major or Minor.
• Evidence-based: critiques supported by evaluation of the application and understanding of the state of the science.
• Each round of merit review is independent; it is fair to raise new questions about a revised application.
VETERANS HEALTH ADMINISTRATION 16
Critique Comments
• Include full statements/descriptions of Key Strengths, Key Weaknesses in the last sections of your written critique.
• Evaluative (not descriptive) with supporting statements.
• Clear, concise narrative.
• Focus on specific strengths and weaknesses; avoid direction.
• Summarize your evaluation in a paragraph that includes the key factors that determine your overall priority score.
• Use abbreviations and acronyms sparingly
• Express criticism constructively as it will become part of the Summary Statement used by HSR&D and the applicant.
VETERANS HEALTH ADMINISTRATION 17
Preparing for Review Meeting
• Check for conflicts, read assigned application(s), and prepare written critique & prelim. score for each assigned proposal.
• Submit reviews on the IAR website (by February 28, 11:59pm).
• During I!R Read Phase (starting March 1), read the critiques submitted by other reviewers, in order to facilitate discussion.
• Prepare assigned reviewer presentations to be given at meeting.
• Be familiar with unassigned applications. • Participate in review discussions and scoring at meeting;
upload final scores in IAR by last day of meeting. • After meeting, revise written critiques as needed, and upload by end of the I!R Edit Phase (11.59pm ET, March 13).
VETERANS HEALTH ADMINISTRATION 18
Preparation for Review Discussion
• Prepare assigned reviewer presentation, focusing on the factors/criteria driving your score; critique should not be read verbatim.
• Primary reviewer prepares brief description of the proposal (not included in the written critique) to orient the panel.
• Include the major strengths and weaknesses for each of the key criteria
• Additional reviewers (secondary, tertiary, other) add only new comments based on their own expertise, and/or highlight key points of agreement or disagreement.
• Assigned reviewer presentation should reflect any change in evaluation that may have occurred based on review of other critiques.
VETERANS HEALTH ADMINISTRATION 19
Activities During the Meeting
• Introductions of reviewers (including telephone reviewers, if available).
• Opening remarks by SRO and Chair.
• Consider candidate applications for Streamlining. Any non-conflicted panel member may request application be removed from the list and discussed.
• Discussion of applications (~20 minutes per IIR application).
VETERANS HEALTH ADMINISTRATION 20
Streamlining
• Streamlining (not discussing) – Meeting management tool to increase efficiency, focus time on more meritorious proposals.
– Written critiques are sent to all applicant, but streamlined applications lack the Key Points from those discussed.
• Criteria for Streamlining – All initial scores in the range of 3.0 to 5.0 and/or
– Applications that rank in the lower quartile of all the applications on the panel
based on the preliminary scores.
• Panel Process: – Applications that are streamline candidates will be presented one at a time by the
SRO at the beginning of the meeting.
– Any non-conflicted panel member may indicate he/she wants application discussed by simply raising hand (or saying yes if on the phone)/
– Conflicts may be present, so no other comment or discussion is appropriate. Discussion would occur later in the meeting.
VETERANS HEALTH ADMINISTRATION 21
Discussion and Scoring
• Goal of discussion is to identify the overall impact & major strengths/weaknesses of an application, so that each member of the panel can assign an appropriate score to the application.
• Consensus is not the overall goal; discussion can include dissent and a range of opinions.
• Each panel member assigns a final score based on her or his independent understanding of the strengths and weaknesses of the application as expressed in the discussion and the application.
VETERANS HEALTH ADMINISTRATION
Review Order
• Application is introduced; reviewers in conflict leave.
• Assigned reviewers identified, asked to announce their preliminary scores (may differ from those entered in IAR).
• Presentations by the primary, secondary and tertiary reviewers (plus others, if additional assignments).
• All reviewers discuss the application and the critiques.
• The Chair summarizes major strengths, weaknesses.
• Key summary points are reviewed (IIRs only).
VETERANS HEALTH ADMINISTRATION 23
Review Order (Continued)
• After key points are finalized: Primary reviewer may make motion to recommend disapproval (due to research being unethical -- very rare); panel members would then vote.
• The Chair calls for assigned reviewers to provide their final scores, reminding them to use guidelines.
• The Chair asks if anyone is going to score outside the range; if so, panel member must state this, with a brief reason.
• The scores are recorded (on paper and in IAR Final Scoring).
• Budget considerations are not a factor in scientific merit.
VETERANS HEALTH ADMINISTRATION 24
Scoring Out of Range
• Range is set by the final scores of assigned reviewers.
• If unassigned reviewers score outside the range , they must declare their scoring;
• The exact score does not need to be stated, but the direction outside the range and a reason must be indicated.
• SRO will note names of panel members who score outside range.
• Additional review comments can be sent to the SRO, especially if the score is significantly outside the range.
• Difference could be based on:
• a scientific difference of opinion
• different weighting of the review criteria
• a perceived mismatch between the comments/score provided by the assigned reviewers.
• Dissent should be transparent and is NOT discouraged. VETERANS HEALTH ADMINISTRATION
Scoring Factors
• Score the application as submitted, not what it could be.
• Very important to follow HSR&D Scoring Guidelines, found in the Reviewer guidelines in the IAR Meeting Materials folder – Chairs will summarize strengths & weaknesses from discussion.
– Scores should be consistent with review comments (strengths, weaknesses) and with the scoring guidelines.
• Overall scores are generated by the entire panel.
– Reviewers score all proposals for which they are present for the discussion. Ideal is uniform reviewer cohort scoring all proposals.
– Phone reviewers participate in as much of meeting as possible.
VETERANS HEALTH ADMINISTRATION 26
Meeting Follow-up
• Reviewers are required to revise their critiques, as needed, so that they are consistent with their final scores.
• Reviewers are responsible for editing and uploading their final critiques to IAR (by end of Edit phase).
• Reviewers are asked to revise the critiques of not discussed applications. This is especially important if the preliminary evaluation was more enthusiastic than other reviewers and would lead to unclear feedback.
• Program staff in HSR&D will perform additional programmatic and administrative review of applications that are candidates for funding.
VETERANS HEALTH ADMINISTRATION 27
Reviewer Essentials
• Bring your own laptop to the meeting and/or have computer access if teleconferencing.
• Wireless internet access is available in the meeting room.
• Adobeconnect will be used to share information throughout the meeting.
• Download essential items (your written critiques) prior to the meeting just in case there are any issues with the wireless access being interrupted.
• If you are traveling to the meeting, hotel reservations deadline is February 12, 2017.
• Pre-COIs are required for access to eRA Commons (Ethics also required for non-VA).
• eRA Commons links to RFAs work only with VA access because they are attempting to access an intranet source. Use the pdf of RFAs in the meeting materials.
VETERANS HEALTH ADMINISTRATION 28
Important Dates
• February 28, 2017 11:59 pm ET
• March 1, 2017 begins
• March 7, 2017
• March 7-8, 2017
• March 8, 2017
• March 8-9, 2017
• March 9, 2017
• March 10, 2017
• 3 hours after each meeting ends
• March 13, 2017 11:59 pm ET
Preliminary Critiques due in eRA
Read period - ends before meeting
HSR2
HSR1/HS1A, HSR3/HS3A, HSR4/HS4A
HS8A
HSR5/HS5A
HSR6, CDA
NRI
Final Scores due in IAR
Final Edits Critiques due
VETERANS HEALTH ADMINISTRATION 29
Insider s Guide to Peer ReviewAdapted from NIH/CSR
• Get started right away to discover if you have a conflict with an application.
– It is cumbersome to reassign the application late in the game. You want to make sure every application gets due diligence.
• Remember you’re not expected to know everything.
– Peer review is a group process with multiple individuals providing special expertise and different points of view.
• Be clear.
– If you feel there is an immutable and fatal flaw, say so in your review. Otherwise, the investigator may needlessly make changes that do not address the issue.
VETERANS HEALTH ADMINISTRATION 30
Insider s Guide to Peer ReviewAdapted from NIH/CSR
• Recognize that applicants can’t provide all the details/
– 0 in the shorter format / page limits/ �onsider if the investigators have sufficient command of the material to make it likely the project will succeed.
• Don’t try to rewrite an application.
– You are a critic, not a mentor. Your comments should be generous when possible and polite at all times.
• Talk about what matters.
– Review meeting time is valuable, so it is vital that the oral presentations focus on issues that help people arrive at a score. Focus on major strengths and weaknesses. Limit the description of the research- dont try to enumerate every minor flaw/
VETERANS HEALTH ADMINISTRATION 31
Top 10 Things Reviewers Should Not Say in a Review !dapted from NIH
• “This application has 2 great aims and 1 bad one. I would recommend deleting !im 3, and I can give it a good score.”
– The application needs to be evaluated and scored as a whole, as it is submitted.
• “This application is not in my area of expertise . . . “
– If youre assigned an application you feel uncomfortable reviewing, you should tell your Scientific Review Officer (SRO) as soon as possible before the meeting.
• “This application was scored a 245 the last time it was reviewed . . . .”
– Do not mention the previous score an application received, because this could skew the review discussion. Focus on the strengths and weaknesses of the current application, as well as the responses to previous critiques, and score based on the current application and scoring guidelines.
VETERANS HEALTH ADMINISTRATION 32
Top 10 Things Reviewers Should Not Say in a Review !dapted from NIH
• “This I21 application does not have pilot data, which should be provided to ensure the success of the project.”
– I21s are pilot projects; preliminary data are not required.
• “I like this project but I’m giving it a poorer score because the applicant has too much money.”
– You should focus on the application s scientific and technical merit/ (However, you can note an excessive budget request in the budget section, though this should not affect your overall score.)
• “I didn’t read the application, but I scanned it and saw the applicant said XXX. He doesn’t know what he’s doing.”
– It s better for you to ask other reviewers who have read the application carefully what they think about XXX.
VETERANS HEALTH ADMINISTRATION 33
�efore Starting the Q and ! 0
For more information:
• See the guidelines posted under Meeting Materials on the IAR website for your meeting.
• Ask the SRO for your review meeting.
• Watch the video with advice from Dan Berlowitz, the Chair of the overall HSR&D review committee (http://www.hsrd.research.va.gov/for_researchers/merit _review/default.cfm).
• Send general questions to [email protected].
Please complete the CyberSeminar feedback survey that will be presented when you leave the session.
VETERANS HEALTH ADMINISTRATION 34
Thank you! Next on to Questions and Answers
35
Appendix
The following slides are included simply as a reference.
VETERANS HEALTH ADMINISTRATION 36
Pilot projects
• Goals of a pilot (and next steps) should be clear; for example:
– Strengthen the evidence for larger study
– Preliminary data for full-scale proposal (e.g., effect size)
– Establish feasibility of objectives/methods (e.g., recruitment)
– Explore novel research approach through small scale study
• Methods should align with goals, be appropriate for pilot work
– e.g., sample size/population will depend on whether goal is to establish effect size vs. feasibility
• Preliminary data are not required.
• Scale is smaller than an IIR; scientific rigor still important
– Feasible to complete in one year, max budget of $100,000 VETERANS HEALTH ADMINISTRATION 37
RFAs
• IIR – Investigator Initiated Research – Maximum 4 years and $1.1M
• PPO – Pilot Project Opportunity – Maximum 1 year and $100,000
VETERANS HEALTH ADMINISTRATION 38
RFAs
• LHI-Learning Health Research Initiative – Provider Behavior
• Maximum 4 years and $1.1M
– Data and Measurement Sciences • Maximum 4 years and $1.1M
– Randomized Program Evaluation • Maximum 3 years and $750,000 ($250K/year)
VETERANS HEALTH ADMINISTRATION 39
RFAs
• CDA- Career Development Award – Maximum 5 years
– Clinicians: full salary and fringe benefits to support a 6/8ths appointment
– Non-clinicians: minimum 5/8ths appointment up to 8/8ths
– Supplementary project funds first three years of the award capped at $40,000/year for awardees at HSR&D Centers of Innovation (COINs), and $50,000/year for all other CDAs, and are subject to availability.
VETERANS HEALTH ADMINISTRATION 40
RFAs
• NRI – Nursing Research Initiative – NRI IIR
• Maximum 4 years and $1.1M
– NRI Pilots • Maximum 2 years and $100,000
VETERANS HEALTH ADMINISTRATION 41