This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Slide 1
CSR Advisory Council Meeting May 19, 2014 Editorial Board
Review A Few Good Reviewers Don Schneider, Ph.D.
Slide 2
Reviewer pools Past experience Cost considerations Toward
Review by the Best
Format modeled on journal manuscript review First piloted in
2008 with 6 SBIR panels, just in time for TR01s, Challenge grants,
DP1 etc. Two stages First Stage Mail reviewers Second Stage Editors
Past Experience
Slide 5
First Stage/Mail Reviewers Subject matter experts Provide depth
in review Focus on scientific and technical merit 2-3 first stage
mail reviewers per application Submit full critiques Give overall
impact and criterion scores Overall impact scores not factored into
final priority score
Slide 6
Hold face-to-face meeting Recruit broad experts Provide
perspective in review (assign about 15 applications each) Focus on
impact and significance Assign 3 second stage reviewers per
application Consider first stage critiques in review Write overall
impact paragraph Give overall impact score Final priority score
based on second stage only Second Stage/Editors
Slide 7
Provides both depth and breadth in review Optimizes use of the
best reviewers Scales well for large numbers of applications
(second stage discusses a fraction of the applications)
Rationale
Slide 8
Perceived Advantages Involves no travel/teleconference for
first stage reviewers Allows small, interactive face-to-face
meetings Promotes better scoring and assessment of impact Lessens
travel and lodging expenses and inconveniences
Recruitment of large numbers of reviewers Timeline Tight, two
sequential reviews (in the 17 week cycle) More staff time required
(SROs) Some sense of isolation by first stage reviewers
Challenges
Slide 11
Each application examined by at least 5 reviewers Interactive,
thoughtful discussions Overall scoring by second stage members
Reviewers and staff like final review products Review Outcomes
Slide 12
Survey conducted by A Kopstein of reviewers participating in
SBIR pilots 2008 Outcomes were generally positive Majority willing
to participate in either review stage in future Editorial Board
Review: Increases expert review 3/4 ths of respondents Preferred
for their own applications 2/3 rds of respondents Survey