Www.uis.unesco.org Innovation data collection: Methodological procedures & basic forms Regional Workshop on Science, Technology and Innovation (STI) Indicators.

www.uis.unesco.org

Innovation data collection:Methodological procedures

& basic forms

Regional Workshop on Science, Technology and Innovation (STI) Indicators for Gulf countriesDoha, Qatar

15 to 17 October 2012

www.uis.unesco.org

Ch. 8 OM - Survey procedures

Guidelines - collection and analysis of innovation data;

Comparable results over time and across countries;

Particular circumstances may require other methodology comparability.

www.uis.unesco.org

The survey approach

The “subject” approach:

• Innovative behaviour and activities of the firm as a whole;

The “object” approach:

• Specific innovations (“significant innovation” of some kind, firm’s main innovation).

www.uis.unesco.org

Populations (1)

The target population:The target population:

• Innovation activities in the business enterprise sector (goods-producing and services industries);

• Minimum: all statistical units with at least ten employees:

• Small: 10-49;

• Medium: 50-249;

• Large: 250 and above.

www.uis.unesco.org

Populations (2)

The frame population:The frame population:

• Units from which a survey sample or census is drawn;

• Based on the last year of the observation period for surveys;

• Ideal frame = up-to-date official business register NSOs;

• If the register forms the basis for several surveys (innovation, R&D, general business), the information can be restricted to innovation.

www.uis.unesco.org

Survey methods (1)

Mandatory Mandatory surveys increase response rates;

Census or sample surveys?Census or sample surveys?

• Sample surveys - representative of target population (industry, size, region) stratified sample;

• Census - costly but unavoidable in some cases.

www.uis.unesco.org

Survey methods (2)

Domains (sub-populations):Domains (sub-populations):

• Subsets of the sampling strata;

• Potential sub-populations: industry groupings, size classes, regions, units that engage in R&D and innovation-active;

• Guidelines:» Same statistical units and classifications;

» Consistence of the methods for results calculation;

» Documentation of deviations in data treatment or differences in the quality of the results from the domains.

www.uis.unesco.org

Survey methods (3)

Sampling techniques:Sampling techniques:

• Stratified sample surveys: size and principal activity;

• Sampling fractions should not be the same for all strata;

Cross-sections: Cross-sections: standard approach - new random sample for each innovation survey;

Panel data:Panel data: alternative/supplementary approach.

www.uis.unesco.org

Survey methods (4)

Suitable respondents:Suitable respondents:

• Methods: e.g., postal surveys, web-based questionnaires, personal interviews;

• Unit’s most suitable respondent - very specialised questions that can be answered by only a few people;

• Try to identify respondents by name before data Try to identify respondents by name before data collection starts.collection starts.

www.uis.unesco.org

Survey methods (5)

The questionnaire:The questionnaire:• Pre-test;

• Simple and short;

• Order of the questions;

• Questions on qualitative indicators - binary or ordinal scale;

• International innovation surveys - attention to translation and design;

• Short-form questionnaires - units with little/no innovation activity previously reported.

www.uis.unesco.org

Survey methods (6)

Combination of Innovation and R&D surveys:Combination of Innovation and R&D surveys: Reduction in the overall response burden;

Scope for analysing the relations between R&D and innovation activities;

Increase in the frequency of innovation surveys;

Country experiences - it is possible to obtain reliable results for R&D expenditures;

Longer questionnaire;

Units not familiar with the concepts of R&D and innovation may confuse them;

Different frames for the two surveys.

www.uis.unesco.org

Survey methods (7)

Guidelines for conducting combined surveys:Guidelines for conducting combined surveys:

• Questionnaire: two distinct sections;

• Smaller individual sections;

• Comparison of results from combined and stand-alone surveys should be done with care - surveying methods should be reported;

• Samples extraction from a common business register.

www.uis.unesco.org

Estimation of results (1)

Weighting methods:Weighting methods:

• Weighting by the inverse of the sampling fractions of the sampling units, corrected by the unit non-response;

• If a stratified sampling technique with different sampling fractions is used, weights should be calculated individually for each;

• Based on the number of enterprises in a stratum;

• International and other comparisons: same weighting method.

www.uis.unesco.org

Estimation of results (2)

Non-response:Non-response:

• Unit non-response: reporting unit does not reply at all;

• Item non-response: response rate to a specific question - % of blank or missing answers;

» Disregarding missing values + applying simple weighting procedures based on the responses received assumes that respondents and non-respondents are distributed in the same way biased resultsbiased results;

» Possibility: imputation methodsimputation methods.

www.uis.unesco.org

Presentation of results

Descriptive analysis: no generalisation of results;

Inferential analysis: conclusions about target population;

• Variance for the results: (average) values for innovation indicators and their coefficients of variation and/or confidence intervals;

Results presentation: metadata (data collection procedure, sampling methods, procedures for dealing with non-response, quality indicators).

www.uis.unesco.org

Frequency of data collection

Every 2 years;

If not economically feasible frequency of 3 or 4 years;

Specify an observation period;

• The length of the observation period for innovation surveys should not exceed 3 years nor be less than 1 year.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts (1)

Information system specificities:

• Relative weakness of statistical systems:

» Absence of linkages between surveys and data sets;

» Lack of official business registers;

• Involvement of NSOs;

• When lacking, basic variables about firms’ performance can be included in the innovation survey.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts (2)

General methodological considerations:

• Survey application:» In-person;

» Trained personnel;

• Questionnaire design: » Sections can be separated to allow different persons in the

firm to reply them;

» Guidance/definitions;

» Language and translation of technical terms.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts (3)

General methodological considerations:

• Frequency:» Every 3 to 4 years (e.g., timed to CIS rounds);

» Update a minimum set of variables every year;

• Purpose of surveys;

• Clear questions;

• Adequate legislative base;

• The results should be published and distributed The results should be published and distributed widely.widely.

www.uis.unesco.org

Example - product innovation/degree of novelty

(Product innovations) Yes No

New or significantly improved goods? ( ) ( )

New or significantly improved services? ( ) ( )

(Degree of novelty) Yes No

New to your market? ( ) ( )

Only new to your firm? ( ) ( )

During the three years 200X to 200Y, did yourenterprise introduce:

Were any of your product innovations during the threeyears 200X to 200Y:

www.uis.unesco.org

Example - innovation activities and expenditures for product and process innovations

(Innovation activities and expenditures forprocess and product innovations)

Yes NoAmount of

expenditureIn-house R&D ( ) ( ) $

External R&D ( ) ( ) $

Acquisition of machinery, equipment and software ( ) ( ) $

Acquisition of external knowledge ( ) ( ) $

Training for innovative activities ( ) ( ) $

Market introduction of innovations ( ) ( ) $

Other (preparations) ( ) ( ) $

During the three years 200X to 200Y, did your enterprise engage in thefollowing innovation activities:

www.uis.unesco.org

Example - co-operation

Within country

Abroad

A. Other enterprises within your enterprise group ( ) ( )

B. Suppliers of equipment, materials, components, or software ( ) ( )

C. Clients or customers ( ) ( )

D. Competitors or other enterprises in your sector ( ) ( )

E. Consultants, commercial labs, or private R&D institutes ( ) ( )

F. Universities or other higher education institutions ( ) ( )

G. Government or public research institutes ( ) ( )

LocationType of innovation co-operation partner

During the three years 200X to 200Y, did your enterprise co-operate on anyof your innovation activities with other enterprises or institutions?( ) Yes ( ) No

www.uis.unesco.org

Example - hampering factors

High Medium LowNot

relevantExcessive perceived economic risks ( ) ( ) ( ) ( )

Innovation costs too high ( ) ( ) ( ) ( )

Lack of appropriate sources of finance ( ) ( ) ( ) ( )

Organisational rigidities within the enterprise ( ) ( ) ( ) ( )

Lack of qualified personnel ( ) ( ) ( ) ( )

Lack of information on technology ( ) ( ) ( ) ( )

Lack of information on markets ( ) ( ) ( ) ( )

Insufficient flexibility of regulations or standards ( ) ( ) ( ) ( )

Lack of customer responsiveness to new goods or services ( ) ( ) ( ) ( )

Economic factors

Internal factors

Other factors

Degree of importanceHampering Factors

If your enterprise experienced any hampering factors during the period 200X-200Y, please gradethe importance of the relevant factors.

www.uis.unesco.org

Example - organisational innovation

(Organisational innovations) Yes NoNew business practices for organisingprocedures

( ) ( )

New methods of organising workresponsibilities and decision making

( ) ( )

New methods of organising external relationswith other firms or public institutions

( ) ( )

During the three years 200X to 200Y, did your enterpriseintroduce:

www.uis.unesco.org

Basic forms - CIS

The Community Innovation Survey (2010) (CIS 2010):The Community Innovation Survey (2010) (CIS 2010):

1. General information about the enterprise

2. Product innovation

3. Process innovation

4. Ongoing or abandoned innovation activities for pdt/pcs innovation

5. Innovation activities and expenditures for pdt/pcs innovation

6. Sources of information and co-operation for pdt/pcs innovation

7. Objectives for pdt/pcs innovation

8. Factors hampering pdt/pcs innovation activities

9. Organisational innovation

10. Marketing innovation

11. Creativity and skills

12. Basic economic information on your enterprise

www.uis.unesco.org

Basic forms - AU/NEPAD

AU/NEPAD Standard Innovation Questionnaire:AU/NEPAD Standard Innovation Questionnaire:

1. General information about the enterprise

2. Product innovation

3. Process innovation

4. Ongoing or abandoned innovation activities for pdt/pcs innovation

5. Innovation activities and expenditures for pdt/pcs innovation

6. Sources of information and co-operation for pdt/pcs innovation

7. Effects/Objectives of pdt/pcs innovation

8. Factors hampering (pdt/pcs) innovation activities

9. IPRs

10. Organisational and marketing innovation

11. Specific innovations by your enterprise

www.uis.unesco.org

Thank you!Thank you!

http://www.uis.unesco.org

[email protected]