Methods and Uses of Peer Review - Challenges and Lessons Learned From a Canadian Perspective Jennifer Birta, National Research Council Canada New Frontiers in (R&D) Evaluation Conference Vienna, Austria 25 April 2006
Jan 01, 2016
Methods and Uses of Peer Review -Challenges and Lessons Learned From a
Canadian Perspective
Jennifer Birta, National Research Council CanadaNew Frontiers in (R&D) Evaluation Conference
Vienna, Austria25 April 2006
2
Presentation Overview
• Overview of the National Research Council
of Canada (NRC)
• Terminology
• Case Study– Overview of initiative– Peer reviews conducted– Challenges and opportunities
• Conclusions
3
National Research Council Canada (NRC)
2004-2005
# of papers published: 1,286
# of patents issued: 86
# of licenses issued: 105
• Agency of the Canadian federal government
• Principal scientific research body conducting intramural research
• Range of research from fundamental to applied in areas such as:– Molecular sciences– Astronomy– Biotechnology– Environmental technologies
• Provision of innovation support– Industrial Research Assistance Program– Technology cluster strategy
4
National Research Council Canada (NRC)
• 20 institutes and 10 other research centres
• 4,000+ employees
• 1,300+ students, visiting/guest workers
• Laboratories and facilities across the country
• Total expenditures 2004-05: $712M
• Income 2004-05: $103.7M
5
Terminology
PURPOSES
Journal manuscript review
Proposal review
Evaluation review
Review of research results for competence, significance and originality prior to publication
Assessment of proposals for future funding decisions
Performance assessment of level of excellence and impact of past work
METHODS
Paper-based
Panel
Standing Committee
Provide written opinion only, no interaction
Come together to conduct a single review
Conduct many reviews over a term
6
Case Study: Overview of NRC’s Genomics and
Health Initiative
• Horizontal initiative across 10 NRC Institutes• Research focus fundamental and applied (e.g., diagnosis of
disease, crop enhancement)• Annual expenditures ~ $25M (CDN)• Started in 1999, divided into phases of 3 year lengths, internal
competitive process used to select programs• Responsibility of the VP Life Sciences, managed by a coordination
office and a standing committee
7
Case Study: Peer Reviews of NRC-GHI
• Research program selection every 3 years– Competitive approach for funding– Scientific and market-driven assessment criteria– Two-tiered approach used
• Periodic retrospective evaluation– Assessment of relevance, success, alternatives and
effectiveness of design/delivery– Seven methodologies used
• Peer review achievement of objectives and progress made
– Two-tiered approach used
8
Case Study: Challenges
Subjective Bias
• Reliance on human judgement
• Trust scientists’ desire not to diminish their reputations
Confidentiality
• Possibility of inappropriate use of knowledge gained during review
• Precautionary measure to sign a confidentiality agreement
• Best defence not to disclose sensitive or proprietary information
9
Case Study: Challenges
Conflict of Interest• Ability of reviewers to participate without conflict with their own
interests• Difficult to find qualified individuals with no prior knowledge of
program under review• Sources of potential reviewers• Disclosure of conflict of interest
Timing• Short timeframe between two peer reviews• Most qualified individuals had already participated• Result low return rate
10
Case Study: Challenges
Reviewer Fatigue
• More S&T organizations relying on peer review
• Difficulty finding specialists, willing and able to participate
Level of Commitment
• Paper-based reviews lack sense of community that exists with panels/committees
• Dependent on profile/reputation of initiative under review
• Might be increased with provision of honoraria
11
Case Study: Challenges
Supporting Materials
• Existing documentation has limited applicability
• Customized materials required
Costs
• Travel and hospitality expenses, honoraria (if applicable)
• Loss of productivity for reviewers and those being reviewed
12
Case Study: Opportunities
Standing Committees
• Benefit from familiarity and sense of obligation
Integration
• Combine retrospective and prospective portions of reviews
Honoraria
• Revisiting NRC’s policy
13
Conclusions
Way forward…
• Has the support of most researchers/scientists
• Alternatives will do little to reduce pressures on the system
• Remains the most appropriate means to assess scientific meritof past, present and future work
14