S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 1 Rationales and evolution of public 'knowledge policies' in the context of their evaluation Seminário Internacional CGEE “AVALIAÇÃO DE POLÍTICAS DE CIÊNCIA, TECNOLOGIA E INOVAÇÃO” - Diálogo entre Experiências Internacionais e Brasileiras Rio de Janeiro, 3-5 December 2007 Professor Stefan Kuhlmann University of Twente, The Netherlands
45
Embed
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007page 1 Rationales and evolution of public 'knowledge policies' in the context of their evaluation.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 1
Rationales and evolution of public 'knowledge policies'
in the context of their evaluation
Seminário Internacional CGEE“AVALIAÇÃO DE POLÍTICAS DE CIÊNCIA, TECNOLOGIA E INOVAÇÃO”
- Diálogo entre Experiências Internacionais e BrasileirasRio de Janeiro, 3-5 December 2007
Professor Stefan KuhlmannUniversity of Twente, The Netherlands
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 2
‘Knowledge policy’ issues Growing dependence of socio-economic development and welfare on sustainable
‘knowledge base’, in particular science, technology, innovation – and education
Internationalisation of industrial research, not caring about national borders
International search for creative, highly skilled people; mobility of top researchers
Knowledge production: (re-) discovery of a “mode 1 – mode 2 transition" (M. Gibbons et al. 1994); advanced technology and innovation: "fusion” of heterogeneous trajectories (Kodama 1995)
Public research and innovation policy: push for efficiency, evidence and evaluation; ‘bureaucratised’ semi-industrial R&D (Ziman, 2001, 82)
Search for ‘intelligent’ policy designs: ‘Systemic’ policy instruments; concern about ‘policy mix’ and ‘governance’
Multi-level arenas and governance; [EU: large number of policy actors on national, regional, and transnational levels; re-shuffling of institutional research landscape]
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 3
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 6
Innovation system (IS) - a heuristic
IS = analytical heuristic (NOT normative perspective)!
IS = “biotopes” of such institutions engaged in scientific research, the accumulation and diffusion of knowledge, which educate and train the working population, develop technology, produce innovative products and processes, and regulate and distribute them.
IS extend over schools, universities, research institutions (education and science system), industrial enterprises (economic system), the politico-administrative and intermediary authorities (political system) as well as the formal and informal networks of the actors of these institutions.
One can conceptualise national, regional, sectoral, and technological IS.
Each IS is different. Sustainable innovation systems develop their special profiles and strengths only slowly, in the course of decades or centuries. They are based on evolving exchange relationships among the institutions of science and technology, industry and the political system (= co-evolution).
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 7
Researchandinnovationpolicyinstruments
Instruments in a narrow sense Instruments in a broader sense
1. Institutional funding National Research Centres Research Councils ( in Germany DFG;
Max Planck Society) Applied Research and Techn. Deve-
lopment Organisations (in Germanye.g. Fraunhofer Society)
Universities and other Higher Educa-tion Institutions
Co-operative policy rationale, e.g. "Specific Programs" under the EU Framework Programs, or the "Verbundforschungsförderung" (co-operative R&D between public sector institutes, universities, and industry) in Germany
“System failures” rationale, need for structural change in the innovation system, e.g. government initiatives aiming at overcoming sclerotic institutions and procedures e.g. in the academic research system
Counter rationale ”government failures”, e.g. institutional inertia, lack of reliable information (on efficiency and impacts of policies etc.), lack of continuity and long-term perspective, red-tape procedures, rivalry of bureaucracies
Note: Actual policymaking only seldom follows such rationales !
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 9
Knowledge creation Funding of basic and applied research; (higher) education and training
Knowledge diffusion through networks
Support for R&D and innovation networks (industry, academia, et al.) and clusters; multi-actor programmes; support for knowledge infrastructures (e.g. patent data bases)
Guidance of the search Science and technology foresight exercises; communication platforms/fora for industry, academia, societal organisations and public policy
Market formation Regulatory frameworks for technical standards and norms; ethical regulation; Intellectual property rights (IPR); et al.
Resources mobilization Thematic or sectoral profiling of public investment in science, R&D, and education
Creation of legitimacy/counteract resistance to change
S/T foresight exercises; communication platforms/fora; maintaining policy networks (e.g. multilevel cooperation across regions, nations and trans national levels); fostering institutional adaptation and change
Source: Hekkert et al. 2006
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 10
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 11
Detour (1): ‘One size does not fit all’The PRIME ERA Dynamics concept (work in progress)
NSI too static (building on specialisation of ‘national’ or ‘sectoral’ systems) – difficulty to cope with complex dynamics in knowledge prduction and global socio-economic change
Core hypothesis: different ‘search regimes’ in knowledge production correspond to different institutional settings and policies = evolving ‘configurations’.
Implications for policy design: different knowledge dynamics appearing in different ‘configurations’ will evolve with different policy mixes.
Aim of project: Identification of limited set of ‘ideal-type’ knowledge configurations and characteristics of related institutions and policy-mixes – helping to design policy concepts in a prospective manner (foresight)
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 12
Knowledge configurations correspond to
Specific characteristics of different knowledge dynamics
Public policy initiatives and traditions Historical path dependency
[Degree of ‘Europeanisation’]
The involved actors, their ambition, strategy and power.
Detour (2): Knowledge dynamics and institutional “configurations”
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 13
Knowledge dynamics have three main ‘aspects’*:
Growth = capacity to survive and/or prosper within the same institutional and organisational setting. Indicators: publications, patents, exports
Convergence = modalities of knowledge flows, and in particular opposing ‘individual’ vs. ‘distributed knowledge’ and the collaboration patterns
Complementarities = Technical complementarities = role of large shared infrastructures or equipment (critical
infrastructures)
Cognitive complementarities = collaboration patterns (bilateral vs. multilateral e.g. networks and clusters); critical mass, competences to be assembled to develop a relevant ‘research production unit’
Institutional complementarities = heterogeneous collaboration for efficient productive settings (e.g. strong relationship between clinicians and biologists in biotechnology); frequency of industry-university collaborations
(* Building on Bonaccorsi 2005ff; see also the rich body of literature in innovation economics, sociology of science and in science and technology studies (STS).)
Detour (3): Knowledge dynamics and institutional “configurations”
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 15
Detour (4): ‘Anecdotal’ examples of knowledge dynamics
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 16
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 17
Weaknesses of research and innovation policy(in EU and else) Lack of Governance mechanisms allowing for more ‘systemic’ orchestration
between diverse knowledge and innovation related policy domains (see LEG 2007)
High degree of departmentalisation, sectoralisation of the political administration, and low inter-departmental exchange and co-operation
Heterogeneous, un-linked arenas: often ‘corporatist negotiation deadlocks’
Failing attempts at restructuring responsibilities in government because of institutional inertia
Dominance of ‘linear model’ of innovation in policy approaches
‘Innovation policy’ run in a very specific, narrow field focusing on introduction of new technologies in SMEs, IPR or VC issues etc.
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 18
Need for ‘systemic’ policy instruments
Re-shaping of innovation and research systems
facilitating the construction (Neue Kombinationen) and deconstruction of subsystems, preventing of lock-in
supporting prime movers
ensuring that all relevant actors are involved
Stimulating demand articulation, strategy and vision development
Building cross-linking platforms and "new spaces" for learning and experimenting
Providing and exploiting an infrastructure for distributed "strategic intelligence" (building on technology assessment, foresight, evaluation, benchmarking etc.)
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 19
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 23
Policy Learning
Concept of bounded rationality in policy learning processes
Providing insights in the limited but relevant impact on the quality of decision-making for “rationalisation”-based policies (see Braun/Benninghoff referring to Heclo, Hall, Olsen/Peters)
Concept of single and double loop learning (Argyris/Schön 1978)
first-order (single loop) learning helps “to keep organizational performance within the range set by organizational norms. The norms themselves […] remain unchanged”;
Second-order (double loop) learning concerns “incompatible organizational norms by setting new priorities and weightings of norms, or by restructuring the norms themselves together with associated strategies and assumptions” .
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 24
Strategic intelligence and policy learning
Strategic intelligence is ...
... a set of sources of information - often distributed and heterogeneous
explorative/empirical as well as analytical (theoretical, heuristic, methodological) tools
well known strategic intelligence tools are evaluation studies, performance measurement, benchmarking initiatives, foresight exercises, or technology assessment (TA)
employed to produce “multi-perspective” insight in the actual or potential costs and effects of public or private policy and management, to be 'injected' and 'digested' in political arenas
facilitating policy learning
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 25
Analysing the dynamics of research and innovation systems
Analyses & methods:Evaluation (ex post, monitoring, ex ante)Delphi, scenarios, TAPolicy-analysisInstitutional analysisStatist.-econometrical analysesNetwork analysis
Indicators:corporate dataSectorial techno-eco- -nomic performanceBibliometricsRegulatory data (e.g. norms, standards)
Sectors,technologies:
retrospectively, prospectively
Innovationprocesses:
micro, meso,macro
Actors:
Companies,Science,
Policymakers
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 26
‘Forum’ and ‘Strategic Intelligence’
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 27
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 28
R&D policy ‘evaluation’ may be defined as ...
“... methodology-based analysis and assessment of the appropriateness of S/T policy assumptions and targets, of the related measures and their impacts, and of the goal attainment.”
seeking to assist policy makers, programme managers and programme participants
throughout the whole life cycle of funding programmes
to make all actors involved learn and (re-)adjust
and thus contribute to the overall success (and/or improvement and/or termination) of programmes and funded structures and to learn for future approaches.
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 34
1. ‘Innovation system’ concept; evolution of policy instruments and rationales
2. Detour: ‘One size does not fit all’
3. Weaknesses of present research and innovation policy
See also "Evaluation Toolbox": http://epub.jrc.es/docs/EUR-20382-EN.pdf
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 39
Evaluation methods, quantitative and qualitative Quantitative: Statistical data analysis
Innovation Surveys: basic data describe the innovation process, using descriptive statistics
Benchmarking: comparisons based on a relevant set of indicators across entities
Quantitative: Modelling methodologies Macroeconomic modelling and simulation: broader socioeconomic impact of policy
interventions
Microeconometric modelling: effects of policy intervention at the level of individuals or firms
Productivity analysis: impact of R&D on productivity growth at different levels data aggregation
Comparison group approach: effect on participants using statistical sophisticated techniques
Qualitative and semi-quantitative methodologies Interviews and case studies: direct observation of naturally occurring events to
investigate behaviours in their indigenous social setting
Cost-benefit analysis: economic efficiency by appraising economic and social effects
Expert panels/peer review: scientific output relying on the perception of peer scientists
Network analysis: structure of cooperation relationships and consequences for individuals and their social connections into networks
Foresight/ technology assessment: identification of potential mismatches in the strategic efficiency of projects and programmes
Source: Fahrenkrog et al., RTD Evaluation Toolbox, http://epub.jrc.es/evaluationtoolbox/start.swf
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 40
A professional evaluator should:
Focus on decision context
Understand the stakes
Use relevant methods (and use them well)
Understand the nature of the object (here: R&D and its institutions)
Understand the evolving context (here: knowledge, research & innovation system)
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 41
Specifics of R&D evaluation (1)
R&D is open-ended (success can be re-defined after the fact)
Connection between R&D and effects is non-linear and indirect
Even in application-oriented research it may take 10 years or more for impact to be realized (and attribution then becomes tenuous)
But evaluation is required earlier …
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 42
Specifics of R&D evaluation (2)
R&D functions in a larger whole, thus its productivity and effects depend on what happens there
no simple RoI (return on investment) approaches
Knowledge, Research and Innovation System (KRIS)
links with internationally defined scientific fields and domains of application
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 43
Specifics of R&D evaluation (3)
Content & domain are important
distantiated measures (e.g. number of publications, patents) do not capture enough
judgment of domain experts is necessary (but has limitations as well, cf. peer review)
balance the two – responsibility of the evaluator!
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 44
There is not one evaluative ‘truth’:
Actual R&D policies relate to variety (and competition) of targets, envisaged effects and underlying rationales and assumptions variety of promotion instruments, overlapping, competing ...
Ever more R&D policy interventions aim at multiple purposes and heterogeneous actors (e.g. the set of “socio-economic” targets and clients) increased complexity and interbreeding of input-output-outcome relationships
Specifics of R&D evaluation (4)
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 45
Specifics of R&D evaluation (5)
Standard (quantitative) analyses often not applicable
limited number of cases (sometimes N=1, and evaluator has to mobilize experience/insight in similar cases)
skewed distributions (of productivity, of impact, of uptake in innovation) – so one cannot use sampling
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 46
U r s a c h e u n d W i r k u n g. - Vor der Wirkung glaubt man an andere Ursachen als nach der Wirkung.Friedrich Nietzsche, Die fröhliche Wissenschaft, Aphorismus 217, 1882
Cause and impact. - Before any impact you believe in other causes than after the impact.
S. Kuhlmann, CGEE International Seminar, Rio de Janeiro 2007 page 47
Prof. Dr. Stefan KuhlmannUniversity of Twente Chair Foundations of Science, Technology and Society School of Management and Governance Institute of Governance Studies (IGS)Enschede, The Netherlandse-mail: [email protected]
Thanks for your attention !!
Further info and contactSmits, R. / Kuhlmann, S. (2004): The rise of systemic instruments in innovation policy. In: Int. J. Foresight and Innovation Policy (IJFIP), Vol. 1, Nos. 1/2, 2004, 4-32
Shapira, Ph., Kuhlmann, S. (eds.) (2003): Learning from Science and Technology Policy Evaluation: Experiences from the United States and Europe, Cheltenham (E. Elgar)