Using Logic Models in Managing Performance of Research and Technology Programs: An example for a Federal energy efficiency and renewable energy program IAMOT 13 th International Conference on Management of Technology April 4, 2004 Gretchen Jordan, Sandia National Labs John Mortensen, Independent Consultant John Reed, Innovologie George Teather, Independent Consultant Work presented here was completed for the U.S. DOE Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy ( DOE) under Contract DE-AC04-94AL8500. Opinions expressed are solely those of the authors.
21
Embed
Using Logic Models in Managing Performance of Research and Technology Programs: An example for a Federal energy efficiency and renewable energy program.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Using Logic Models in Managing Performance of Research and
Technology Programs:An example for a Federal energy efficiency and
renewable energy program
IAMOT 13th International Conference on Management of Technology
April 4, 2004
Gretchen Jordan, Sandia National LabsJohn Mortensen, Independent Consultant
John Reed, InnovologieGeorge Teather, Independent Consultant
Work presented here was completed for the U.S. DOE Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy ( DOE) under Contract DE-AC04-94AL8500. Opinions
expressed are solely those of the authors.
April 4, 2004 2
Presentation Overview
• Managers are under increasing pressure to demonstrate the value of their programs
• Logic models can help programs identify and explain their value
• Logic models can include strategies that represent research, development, demonstration and deployment (RDD&D) stages
• Logic model example for federal energy program• Logic models help programs identify:
National R&D capabilities, including options if circumstances change
April 4, 2004 11
Logic models help programs identify:
• Measurement areas for which indicators (metrics) may be developed
• Performance targets for each indicator• Evaluation questions
April 4, 2004 12
Each box in the logic model is a potential measurement area
Inputs
Activities
Outputs
Outcomes
Program planning &
assessment
Conduct research
Developtechnology
Demonstrate technology
Deploy technology
Federal funding (millions of nominal $) Federal personnel (FTEs)Private funding (millions of nominal $) # of RD&D facilities
Develop & maintain program
infrastructure
Developing government &
market infrastructure
% programs w/benefit estimates
% program w/program
plans
# of journal articles
# of presentations
# prototypes-initial-intermediate-commercial
Prototype cost & performance
# and % of commercial-
scale technologies
validated
# of tech’s purchased by
gov’t,# of materials disseminated,# of website
hits
# of partnerships
# codes and standards,# personnel
trained,# audits,# state
programs
# journal article citations
# of innovative or advanced commercial
products with improved cost &
performance
# of recommenda-tions for using
advanced commercial
products and practices
# and % of advanced
commercial products by
adoption stage
For
Energy saved (quad. Btu), GW of add’l RE capacity,Expenditure savings ($)
Carbon saved (mmtce)
Programs,CFO, OMB,Congress
Programs,partners
R&Dcommunity
R&D Community, Industry
Relevantindustries
Relevantmarkets
Potentialpurchasers
# of potentially commercializ-
able technologies
Cost and performance of
competing technologies
(varies by technology)
Oil prices ($/barrel)
NG prices ($/tcf)
Electricity prices (c/kWh)
Coal prices ($/ton)
GDP (billion 1996 $)
RE production tax credit (c/kWh)
EE/RE tax credits ($)
CAFÉ standards
(mpg)
Vehicle & power plant
emission standards(varies by pollutant)
ExternalInfluences
# of technology spinoffs
Feedback Loops
April 4, 2004 13
Inputs
Activities
Outputs
Outcomes
Program planning &
assessment
Conduct research
Developtechnology
Demonstrate technology
Deploy technology
Federal funding (millions of nominal $) Federal personnel (FTEs)Private funding (millions of nominal $) # of RD&D facilities
Develop & maintain program
infrastructure
Developing government &
market infrastructure
% programs w/benefit estimates
% program w/program
plans
# of journal articles
# of presentations
# prototypes-initial-intermediate-commercial
Prototype cost & performance
# and % of commercial-
scale technologies
validated
# of tech’s purchased by
gov’t,# of materials disseminated,# of website
hits
# of partnerships
# codes and standards,# personnel
trained,# audits,# state
programs
# journal article citations
# of innovative or advanced commercial
products with improved cost &
performance
# of recommenda-tions for using
advanced commercial
products and practices
# and % of advanced
commercial products by
adoption stage
For
Energy saved (quad. Btu), GW of add’l RE capacity,Expenditure savings ($)
Carbon saved (mmtce)
Programs,CFO, OMB,Congress
Programs,partners
R&Dcommunity
R&D Community, Industry
Relevantindustries
Relevantmarkets
Potentialpurchasers
# of potentially commercializ-
able technologies
Cost and performance of
competing technologies
(varies by technology)
Oil prices ($/barrel)
NG prices ($/tcf)
Electricity prices (c/kWh)
Coal prices ($/ton)
GDP (billion 1996 $)
RE production tax credit (c/kWh)
EE/RE tax credits ($)
CAFÉ standards
(mpg)
Vehicle & power plant
emission standards(varies by pollutant)
ExternalInfluences
# of technology spinoffs
Feedback Loops
Performance targets may also be developed for each box in the logic model
Cost of Hydrogen
($/gallon gasoline equivalent)
2003200
62010
Non-renewables
5.00 3.00 1.50
Renewables 6.20 5.30 3.90
April 4, 2004 14
Arrows between the boxes help identify evaluation questions
Activities
Outputs
Outcomes
For
Demonstrate technology
Test, improve, & validate commercial-
scale technology,Give industry hands-on
experience
Investment by industry in innovative or
advanced commercial products
Relevantindustries
Evaluation Questions:
• Has industry experience lead to investment in these innovative or advanced commercial energy products?
• Have relevant industries gained hands-on experience with the technologies?
• Have EERE’s efforts to test and demonstrate energy technologies led to validated commercial scale technologies? (Activities to outcomes)
• Has EERE engaged the right partners in tests and done it efficiently?
April 4, 2004 15
Potential management / evaluation questions for EERE ask if strategies are working
1. Have program planning and assessment activities increased potential program benefits per federal dollar spent?
2. Has developing and maintaining EERE program infrastructure provided the scientific and technical expertise and facilities required to carryout program activities?
3. Has research conducted by EERE yielded energy-related concepts and designs with possible commercial applications?
April 4, 2004 16
Management / evaluation questions (cont.)
4. Has EERE technology development yielded potentially commercializable energy technologies?
5. Have EERE efforts to test and demonstrate energy technologies led to investment by industry in these innovative or advanced commercial energy products?
6. Have EERE efforts to develop government and business infrastructures led to favorable policies and capable delivery channels for EERE products and practices?
April 4, 2004 17
Management / evaluation questions (cont.)
7. Have EERE deployment activities with end users increased the awareness, appreciation, and adoption of EERE products and practices?
And, altogether has the EERE portfolio of RDD& D programs led to adoption of EERE products and practices and yielded economic, security, and environmental benefits that would not have occurred otherwise?
April 4, 2004 18
Inputs
Activities
Outputs
Outcomes
Program planning &
assessment
Conduct research
Developtechnology
Demonstrate technology
Deploy technology
Federal funding (millions of nominal $) Federal personnel (FTEs)Private funding (millions of nominal $) # of RD&D facilities
Develop & maintain program
infrastructure
Developing government &
market infrastructure
% programs w/benefit estimates
% program w/program
plans
# of journal articles
# of presentations
# prototypes-initial-intermediate-commercial
Prototype cost & performance
# and % of commercial-
scale technologies
validated
# of tech’s purchased by
gov’t,# of materials disseminated,# of website
hits
# of partnerships
# codes and standards,# personnel
trained,# audits,# state
programs
# journal article citations
# of innovative or advanced commercial
products with improved cost &
performance
# of recommenda-tions for using
advanced commercial
products and practices
# and % of advanced
commercial products by
adoption stage
For
Energy saved (quad. Btu), GW of add’l RE capacity,Expenditure savings ($)
Carbon saved (mmtce)
Programs,CFO, OMB,Congress
Programs,partners
R&Dcommunity
R&D Community, Industry
Relevantindustries
Relevantmarkets
Potentialpurchasers
# of potentially commercializ-
able technologies
Cost and performance of
competing technologies
(varies by technology)
Oil prices ($/barrel)
NG prices ($/tcf)
Electricity prices (c/kWh)
Coal prices ($/ton)
GDP (billion 1996 $)
RE production tax credit (c/kWh)
EE/RE tax credits ($)
CAFÉ standards
(mpg)
Vehicle & power plant
emission standards(varies by pollutant)
ExternalInfluences
# of technology spinoffs
Feedback Loops
Evaluations can explain why some goals were met and others were not
(1) Funding was about what was expected
(2) R&D yielded expected results
(3) But, competing technologies improved more than expected
(4) Energy prices were lower than expected
(5) Thus, market penetration was slower than expected
(6) And benefits were lower than expected
April 4, 2004 19
In summary, logic models for RDD&D programs help identify…
Inputs
Activities
Outputs
Outcomes
Program planning &
assessment
Conduct research
Developtechnology
Demonstrate technology
Deploy technology
Federal, state & local government fundingPrivate funding, Personnel, Facilities, Past R&D results