1 Utilization-focused Evaluation of a Portfolio of Research, Development, & Demonstration Programs Helen Kim, Larry Pakenas - NYSERDA Rick Ridge – Heschong Mahone Group, Inc. Scott Albert – GDS Associates, Inc. Gretchen Jordan – Sandia National Laboratory American Evaluation Association Conference Portland, Oregon November 2, 2006
24
Embed
1 Utilization-focused Evaluation of a Portfolio of Research, Development, & Demonstration Programs Helen Kim, Larry Pakenas - NYSERDA Rick Ridge – Heschong.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Utilization-focused Evaluation of a Portfolio of Research, Development, & Demonstration Programs
Helen Kim, Larry Pakenas - NYSERDA Rick Ridge – Heschong Mahone Group, Inc.
Scott Albert – GDS Associates, Inc. Gretchen Jordan – Sandia National Laboratory
American Evaluation Association ConferencePortland, Oregon
November 2, 2006
2
New York State Energy Research and Development Authority
(NYSERDA)• Mission
Use innovation and technology to solve New York’s energy challenges in ways that benefit the State’s economy and environment
• Vision
Serve as a catalyst for change – enabling New Yorkers to realize affordable energy, a growing and vibrant economy, greater energy independence, and a cleaner environment
3
NYSERDA and RD&D Funding
• Statutory Funding for RD&D– Levy on interstate sales of gas and electricity– Annual funding ~$13 million for RD&D
• Public Benefits Funds– Annual budget
~175 million total~$45 million for RD&D
• NYSERDA Staff– Total 220– RD&D 40 (29 before Public Benefits Funding)
4
Public Benefits Funded RD&D Program Portfolio
Product Development
21%
Proof of Concept20%
Develop/Improve New Product
64%
Pre-deployment21%
Research for Policy18%
Demonstration45%
Test/Improve Product16%
6
Project Selection
Competitive Solicitations
Proposals reviewed by a Technical Evaluation Panel
• Mix of external and internal reviewers• Score and rank projects• Criteria for selection
– Technical merit (Does it make sense?)– Resources (Does the team have the resources to
succeed?)– Benefits (Will the project result in energy and economic
benefits for New York State?)
8
NYSERDA Select & Manage R&D Projects to:-drive portfolio changes over time to respond to current needs, and-Provide public benefits
Develop new or improved product
Study, Prove Concepts
Demonstrate products, inform markets
Study to inform policy & R&D community
Test & improve products
Dissemination builds common knowledge base-Lab prototypes-Future R&D & product options
-Investment/interest growing-Commercial scale product developed -Potential demonstrated
Policy and Product development and pre-deployment process (5-10 years)
Knowledge for future R&D and productsFirms have credibility & market infrastructure is supportive
Products manufactured as replacement, stand alone, or part of system and purchased by early adopters
External Influences:Cost, Performance of existing technologies; Industry willingness to take risks; Uncertainty of R&D; Energy prices; Government policies
Research for Policy DemonstrationProduct Development Pre-deployment
– Goal: Increase use of compressed air efficiency projects among manufacturers
– Activity Type: Demonstration
14
Assessment Criteria in Year One• Knowledge Creation
– Quantity – Significance
• Knowledge Dissemination– Availability of Knowledge Products– Impact on target audience
• Commercialization Progress– Capital Attraction– Technical Achievement – Market Advancement
• Realized and Potential Energy Benefits• Realized and Potential Economic Benefits• Realized and Potential Environmental and Health Benefits• Value versus Costs
15
Rater Scores by Criteria and Project
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
TruckstopElectrification
Compressed AirProgram
21st Century HVAC Particulate Monitoring Aggregating DG
• Likelihood of Realizing Significant Energy Benefits• Likelihood of Realizing Significant Economic Benefits• Likelihood of Realizing Significant Environmental and Health Benefits• Value versus Costs
21
Year-Two Results
Rater Scores for CHP and Env. Monitoring Programs
00.5
11.5
22.5
33.5
44.5
Knowledg
e Cre
ation
Knowledg
e Diss
eminati
on
Commerci
aliza
tion
Progr
ess
Energ
y Ben
efits
Econom
ic Bene
fits
Enviro
nmen
tal B
enef
its
Value
vers
us C
ost
CHP
Env. MonitoringNot Rated
24
Reviewer Comments
Knowledge (Quantity)• The program has produced a large number of published papers in
quality journals. • Staff have gone the extra mile to summarize these for those interested
in an overview of findings.• Improvements could be made to expand the amount of knowledge (as
opposed to the amount of data) created. .
25
Knowledge (Quality)• The information on sulfur, nitrogen, and mercury deposition and its
effects on surface waters and fauna as well as studies of the responses of surface waters to changes in emissions are of particular value and probably the most important contribution of this program.
Availability of Knowledge Product• The program has done an exemplary job of making results from their
projects available to multiple audiences ranging from the scientific community to policy-makers.
• To the extent possible, links to the actual papers would be very helpful as well.
• EMEP should consider more detailed evaluation of the internet contact information. That information would improve the program’s knowledge of which documents are found to be useful by web users.
Reviewer Comments
26
Reviewer CommentsTarget Audience• Given the mission of NYSERDA over all, and the focus of the program on
impacts of power industry operations, there should be greater efforts to engage the industry.
• EMEP might consider some sort of survey of the relevant target audiences to estimate the extent to which the appropriate target markets are reached.
Environmental• Determining whether the likelihood of increase is “significant” or if those
increases are likely to be “substantial” is extremely difficult to gauge. • A large number of new fish consumption advisories were found to be needed
as a consequence of the monitoring. Some health benefits are likely, but their significance is probably difficult to quantify.
Overall Value versus Cost• Estimating the value of research on a dollar basis is next to impossible,
particularly given that much of the benefit from program-supported research is not likely to be measurable for some time after the research is completed.
• I was amazed that the oversight of the program is conducted with only 2.5 full-time-equivalent employees.
27
Year-Two Results
Feedback from Peer Reviewers
00.5
11.5
22.5
33.5
4
Adequecy ofInformation
Packet
Clarity ofInstructions
Clarity ofcriteria
OverallRelevance of
Criteria
CHP
Env. Monitoring
28
Year-Two Results
Feedback on Relevance of Criteria
0
0.5
1
1.5
2
2.5
3
3.5
4
Knowled
ge Q
ty.
Knowled
ge S
ig.
Knowled
ge A
vail.
Targe
t Aud
ience
Capita
l Attr
actio
n
Tech.
Ach
ievem
ents
Mar
ket A
dvnm
nt
Impr
oved
Out
look
CHP
Env. Monitoring
29
Conclusions
• Logic model was useful in identifying project types and assessment criteria
• Peer reviewers were able to provide useful feedback• The assessment criteria appear to be clear and
relevant• Assessment activity validated the project types• Framework can be applied to a group of related
projects• Although too early to tell, R&D programs appear to
be on track to achieving long-term energy, economic, and environmental benefits
30
Next Steps
• Develop methodologies for measuring economic impacts – Economic case studies
– Macroeconomic impact analysis based on new investments and sales of new products
• Develop ratios to forecast future benefits of product development activities