Measuring the Economic Impacts of Technology Transfer from Federal Labs Strategy and Development Branch Frederick (Rick) Kijek June 12, 2007.

Measuring the Economic Impacts of Technology Transfer from Federal Labs

Strategy and Development Branch

Frederick (Rick) Kijek

June 12, 2007

Introduction

Presently undertaking a study to measure economic impacts of NRC

Demonstrate the return on investment from NRC’s R&D programs/activities

Retrospective - 10 years Develop baseline to measure impact of Renewal Strategy Benchmark against other R&D performers Align with S&T Strategies

Technology Transfer is one of the main channels included in the analysis

Several proposed approaches

Presently organizing consensus on methodology No answers or conclusions yet

“Greater Sophistication in Measuring the Impacts of Our Science and Technology Investments” Mobilizing Science & Technology to Canada’s Advantage, Page 88

Basic Design Tenets

Emphasis on quantitative impacts Capture impacts in a consistent, quantitative and credible manner Identify and conceptualize qualitative impacts Contextualize with emphasis on relative (vs absolute) impacts

Objectivity Adopt known and accepted analytical methods, measures and metrics Adopt existing guidelines (TBS) for impact analysis Involve advisory panel of international experts, including representatives from

central agencies (Treasury Board, Finance and Industry Canada) in the review of the methodology

develop “multiple lines of evidence”

Transparency & Repeatability Understandable to NRC decision makers and central agencies Ability to trace and repeat results Methods and assumptions reviewed within NRC

“Canada’s Federal Government will increase its accountability to Canadians by improving its ability to measure and report on the impact of S&T expenditures.”Mobilizing Science & Technology to Canada’s Advantage, Page 88

Proposed Analytical Methods

Adopt a number of analytical methods Benefit Cost Analysis Risk (Data-Variability) Analysis Input-Output Analysis Econometric Analysis Comparative Analysis

Use of multiple sources and techniques Multiple lines of evidence - validation of findings Interweaving of impact story - results of one feed into the next Spreading risk of data availability problems - depending on data

captured

“The government will improve its understanding of Canadian S&T developments and the impact of federally performed S&T....” Mobilizing Science & Technology to Canada’s Advantage, Page 88

Methodological Challenges

Accessing quality and reliable data Establishing an appropriate analytic framework acceptable to

stakeholders at Finance, TBS, Industry Canada Determining the incrementality and attribution of impacts Assigning values to difficult-to-quantify benefits (i.e. social,

environmental, health benefits) and indirect (spill-over) effects of R&D activities

Limiting burden on NRC staff and stakeholders

Lack of international performance benchmarking information

“… and will work with the OECD and other countries to develop metrics that will enable comparisons against international benchmarks of success.” Mobilizing Science & Technology to Canada’s Advantage, Page 88

Measuring Impacts from Technology Transfer

There are few quantitative studies on government technology transfer in Canada

Lack of reliable quantitative measures of utilization and impacts of technology transfer

Most of the studies in this area are based on qualitative measures

There are a few quantitative studies in the United States.

To measure Technology Transfer Impacts

1) Estimate how successful government labs are in terms of transferring the technology they produce

2) Estimate the impact on the technology recipients – firms, spin-off, other labs

3) Measure their collective economic impacts – multiple methods

Previous studies

Two qualitative and focus group studies for Canada: Warda (1999) and Zieminski and Warda (1999)

Quantitative study for Canada: CANMET Study (Partial Cost-Benefit Analysis): Belinko (1994)

Some of the qualitative studies for the United States: O'Brien and Franks (1981), Winebrake (1992), Bozeman (1992), Rogers and Bozeman (1997), Sally (2000), Link and Scott (2005)

Some of the quantitative studies in the United States: Bozeman and Coker (1991), Bozeman (1994), Bozeman (2000), Bozeman

and Pandey (1994), and Adams, Chiang, and Jensen’s (2003)

Approach/Conclusions from Other Studies

Focus Group Approach The Conference Board of Canada (Warda, 1999): private sector points out six

advantages for government laboratories versus universities.

Econometric Approach Measuring both the success of the transfer “out-the-door”& impact on technology

recipient “market impact” Bozeman and Coker (1991): Industry is increasingly viewing the federal

laboratories as a potential technology development partner.

Bozeman (2000): Main comparative advantage of federal laboratories is interdisciplinary team research - difficult at universities. Expensive/unique scientific equipment/facilities of laboratories

Benefit-Cost Approach CANMET Study (1994): Positive B/C Ratio of 8/1

Approach/Conclusions from Other Studies (Contn’d)

Partial Cost-Benefit Analysis The CANMET Impact Study (1994):

methodology was reviewed with officials from Department of Finance and Treasury Board

The lab selected a sample of 121 research projects over a 5-year period  All projects involved industrial partners and resulted in successful technology

transfer with economic return The lab invested $100 million in the R&D for these projects, while industry

invested $264 million – a leverage of 2.6 Industry generated $1.4 billion in actual economic wealth with additional

anticipated return of $1.8 billion for a total of $3.2 billion Based on a total R&D investment for the lab of $364 million over the 5 year

period (total budget), this represented an 8 fold return on investment The average attribution was estimated to be 56%

Measuring Successful Transfer

Econometric Approach

"Out-the-Door" model Once the technical good/process has been adopted by another

organization, then an instance of successful technology transfer has been accomplished

A measure of technology transfer will be regressed on variables such as lab budget, the number of researcher, administrative overhead, diversity of mission

Limitation: the model not concerned if the transferred technology will have any benefit – commercial or non-commercial – for its recipient or the economy at large

Data to be collected mainly from lab directors & program administrators

Measuring Impact on Recipient

Econometric Approach

"Market Impact" model Identifies technology transfer success in terms of commercial

contributions to the technology recipient firm and its level of activity.

Model can be based on a number of metrics for commercial impact, including # of licenses issued by the gov. lab. , recipient sales, value-added, labour productivity

A measure commercial impact will be regressed on variables such as lab budget, the number of researcher, administrative overhead, diversity of mission

Data to be collected from technology recipients

Measuring Impact on Canadian Economy

Benefit Cost Analysis Decide on Partial vs Full BCA Benefits include, license fees, sales and cost savings to technology recipeients Risk (data variability) analysis on all data

Input-Output (Impact) Analysis Use combination of data collected from technology recipients and Statistics

Canada Input-Output Tables Impact on Jobs, GDP, Value-Added

Econometric Analysis Impact on Productivity, Value-added

Comparative Analysis Benchmarking

ANALYSIS METHODS

Extra Slides

Benefit-Cost Analysis

To ultimately measure if benefits of program outweigh cost to taxpayers and by how much

Present value of benefit and cost streams Methodology based on TB guidelines Benefits to include revenues generated, cost savings, value of

services– Attributed increased commercial revenues– Cost savings/productivity improvements

Costs will be broken down by grants, services, overhead To address issues of wealth creation

Econometric Analysis

To assess relationships between the data Estimate elasticities

– a change in the explanatory variable (i.e. increased R&D in that sector) will cause what change in the dependent variable (i.e. increased economic wealth or productivity growth)

– Estimate complimentarity and incrementality Address issues of

– relevance – program satisfies economic needs

» social return of R&D greater than private return

» stimulating increased private sector R&D

– wealth creation using productivity growth as a proxy

Comparative Analysis

To compare relative impacts of non-NRC affected firms or similar programs (nationally or internationally)

Benchmarking NRC Clients, spin-offs & technology transfers Compare to impacts from other R&D performers, Universities, other

countries

Statistics Canada Surveys (Community Innovation Survey, Manufacturers Survey, LEAP, GIFI Data)

Identify established parameters on attribution, incrementality Deal with issues of program reach, improved technical R&D

capability, improved mngmt, mrktg, finance capabilities, supporting of new knowledge creation, enhancing of commercialization, client growth

Input-Output Analysis

To measure national & regional impacts from expenditures Based on national account multipliers Estimate impacts on

Job Creation (direct and indirect), GDP, Imports, Exports

Compare estimates with data collected from survey of companies Provide regional/provincial breakdown Address issue of wealth creation

Risk Analysis

To address inherent uncertainty/variability in data Primarily to be used in Benefit Cost Analysis Establishes ranges of data and probability distribution Uses Monte-Carlo Simulation to identify to what extent data

variability impact range and distribution of benefit & cost streams Provides impact numbers as a distributions around a mean as

opposed to a point estimate