Innovation data collection: methodological procedures

www.uis.unesco.org

Innovation data collection:methodological procedures

ECO - UIS Regional Workshop onScience, Technology and Innovation (STI) Indicators

Tehran, Iran8-10 December 2013

Luciana Marins, UIS

www.uis.unesco.org

Ch. 8 OM - Survey procedures

Guidelines - collection and analysis of innovation data;

Comparable results over time and across countries;

Particular circumstances may require other methodology comparability.

www.uis.unesco.org

The survey approach

The “subject” approach: • Innovative behaviour and activities of the firm as a

whole;

The “object” approach:• Specific innovations (“significant innovation” of

some kind, firm’s main innovation).

www.uis.unesco.org

Populations (1)

The target population:

• Innovation activities in the business enterprise sector (goods-producing and services industries);

• Minimum: all statistical units with at least ten employees;

• Classification by size:

• Small: 10-49;

• Medium: 50-249;

• Large: 250 and above.

www.uis.unesco.org

Populations (2)

Statistical unit:

1 country (ETH)

17 countries (AZE, BLR, CHN, COL, CRI, CUB,

DOM, ECU, LSO, PAN, PRY, PER, SRB, UGA, UKR, URY,

ZMB)

4 countries (IDN, MYS, PSE, PHL)

2 countries (HKG, TUN)

Enterprise group

Enterprise

Establishment

Kind of activity unit

Size cut-off point:

12 countries (BLZ, HKG, COL, CRI,

DOM, ECU, IDN, LSO, PRY,

PHL, SRB, TUN)

2 countries (PAN, PER)

6 countries (AZE, ETH, MYS, UGA, URY, ZMB)

1 country (CUB)

3 countries (CHN, PSE,

UKR)

Number of employees

Turnover

Number of employees /Turnover

Number of employees /Other

Number of employees /Turnover / Other

Source: 2012 UIS Innovation Metadata Collection

1

5

1

3

1

0

1

2

3

4

5

6

0 employee 1 employee 5 employees 10 employees 16 employees

www.uis.unesco.org

Populations (3)

The target population (cont.):

• Classification by main economic activity:

• (National industrial classification system);

• ISIC;

• NACE.

ISIC Rev. 4 INDUSTRY

01-03 Agriculture forestry and fishing

05-09 Mining and quarrying

10-33 Manufacturing

35-39Electricity, gas, steam, air conditioning and water supply; sewerage, waste management and remediation activities

41-43 Construction

45-82 Services of the business economy

84, 85Public administration and defence; compulsory social security and education

86-88 Human health and social work activities

90-93 Arts, entertainment and recreation

94-99Other service activities; activities of households as employers and of extraterritorial organisations and bodies

www.uis.unesco.org

Populations (4)

The frame population:• Units from which a survey sample or census is drawn;

• Based on the last year of the observation period for surveys;

• Ideal frame = up-to-date official business register NSOs;

• If the register forms the basis for several surveys (innovation, R&D, general business), the information can be restricted to innovation.

www.uis.unesco.org

Survey methods (1)

Mandatory surveys increase response rates;

Census or sample surveys?• Sample surveys - representative of target population

(industry, size, region) stratified sample;

• Census - costly but unavoidable in some cases.

www.uis.unesco.org

Survey methods (2)

Completion: Survey type:


7 countries (CRI, DOM IDN, LSO, MYS, PSE, PRY)

16 countries (AZE, BLR, CHN, HKG, COL, CUB, ETH, PAN, PER, PHL, SRB, TUN, UGA, UKR, URY,

ZMB)

Voluntary

Compulsory, notspecified

Compulsory,enforceable penalties

Compulsory, notenforceable

14 countries (HKG, CRI, CUB, DOM, ECU, ETH, IDN, MYS, PSE, PAN, PRY, PER, PHL, UGA)

4 countries (AZE, BLR, COL, LSO)

6 countries (CHN, SRB, TUN, UKR, URY, ZMB)

Sample

Census

Combined

www.uis.unesco.org

Survey methods (3)

Domains (sub-populations):• Subsets of the sampling strata;

• Potential sub-populations: industry groupings, size classes, regions, units that engage in R&D and innovation-active;

• Guidelines:» Same statistical units and classifications;» Consistence of the methods for results calculation;» Documentation of deviations in data treatment or differences in

the quality of the results from the domains.

www.uis.unesco.org

Survey methods (4)

Sampling techniques:• Stratified sample surveys: size and principal activity;

• Sampling fractions should not be the same for all strata;

Cross-sections: standard approach - new random sample for each innovation survey;

Panel data: alternative/supplementary approach.

www.uis.unesco.org

Survey methods (5)

Suitable respondents:• Methods: e.g., postal surveys, web-based

questionnaires, personal interviews;

• Unit’s most suitable respondent - very specialised questions that can be answered by only a few people;

• Try to identify respondents by name before data collection starts.

www.uis.unesco.org

Survey methods (6)

Data collection method:


8

2 2 2

1 1

2

1 1 1

2

0

3

6

9

In-person Mail Web In-person/phone

In-person/email

In-person/mail

In-person/web

Web/email/mail

Email/mail/

in-person/phone

Web/email/mail/

in-person/other

Other

www.uis.unesco.org

Survey methods (7)

The questionnaire:• Pre-test;• Simple and short;• Order of the questions;• Questions on qualitative indicators - binary or ordinal

scale;• International innovation surveys - attention to

translation and design;• Short-form questionnaires - units with little/no

innovation activity previously reported.

www.uis.unesco.org

Survey methods (8)

Combination of Innovation and R&D surveys: Reduction in the overall response burden;

Scope for analysing the relations between R&D and innovation activities;

Increase in the frequency of innovation surveys;

Country experiences - it is possible to obtain reliable results for R&D expenditures;

Longer questionnaire;

Units not familiar with the concepts of R&D and innovation may confuse them;

Different frames for the two surveys.

www.uis.unesco.org

Survey methods (9)

Survey combination:


6 countries (HKG, DOM, ETH, LSO, PAN, TUN)

1 country (CRI)

2 countries (PSE, ZMB)

15 countries (AZE, BLR, CHN, COL,

CUB, ECU, IDN, MYS, PRY, PER, PHL, SRB, UGA, UKR, URY)

Yes, R&D survey

Yes, R&D and ICT surveys

Yes, R&D and BusinesssurveysNo, stand-alone

www.uis.unesco.org

Survey methods (10)

Guidelines for conducting combined surveys:• Questionnaire: two distinct sections;

• Smaller individual sections;

• Comparison of results from combined and stand-alone surveys should be done with care - surveying methods should be reported;

• Samples extraction from a common business register.

www.uis.unesco.org

Estimation of results (1)

Weighting methods:• Weighting by the inverse of the sampling fractions

of the sampling units, corrected by the unit non-response;

• If a stratified sampling technique with different sampling fractions is used, weights should be calculated individually for each;

• Based on the number of enterprises in a stratum;

• International and other comparisons: same weighting method.

www.uis.unesco.org


Non-response:• Unit non-response: reporting unit does not reply at all;

• Item non-response: response rate to a specific question - % of blank or missing answers;

» Disregarding missing values + applying simple weighting procedures based on the responses received assumes that respondents and non-respondents are distributed in the same way biased results;

» Possibility: imputation methods.

www.uis.unesco.org


Unit non-response: Item non-response:


3 countries (CUB, ETH, SRB)

16 countries (AZE, BLR, CHN, COL, CRI, DOM, IDN, LSO, MYS, PAN, PRY, PER, UGA, UKR, URY, ZMB)


1 country (PHL)1 country (PSE)

Non-response survey

Re-contacting the firms

Re-contacting the firms andimputation

Re-contacting the firms andother

None

15 countries (AZE, BLR, CHN, COL, CRI, DOM, IDN, LSO, MYS, PAN, PRY, PER, UKR,

URY, ZMB)


1 country (UGA)

2 countries (CUB, SRB)

3 countries (ETH, PSE, PHL)

Non-response survey

Re-contacting the firms

Re-contacting the firms andimputation

Imputation

None

www.uis.unesco.org

Presentation of results

Descriptive analysis: no generalisation of results;

Inferential analysis: conclusions about target population;• Variance for the results: (average) values for innovation

indicators and their coefficients of variation and/or confidence intervals;

Results presentation: metadata (data collection procedure, sampling methods, procedures for dealing with non-response, quality indicators).

www.uis.unesco.org

Frequency of data collection

Every 2 years;

If not economically feasible frequency of 3 or 4 years;

Specify an observation period;

• The length of the observation period for innovation surveys should not exceed 3 years nor be less than 1 year.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts (1)

Information system specificities:

• Relative weakness of statistical systems:» Absence of linkages between surveys and data sets;» Lack of official business registers;

• Involvement of NSOs;

• When lacking, basic variables about firms’ performance can be included in the innovation survey.

www.uis.unesco.org


General methodological considerations:

• Survey application:» In-person;» Trained personnel;

• Questionnaire design: » Sections can be separated to allow different persons in the

firm to reply them;» Guidance/definitions;» Language and translation of technical terms.

www.uis.unesco.org


General methodological considerations:

• Frequency:» Every 3 to 4 years (e.g., timed to CIS rounds);» Update a minimum set of variables every year;

• Purpose of surveys;

• Clear questions;

• Adequate legislative base;

• The results should be published and distributed widely.

www.uis.unesco.org

Basic innovation indicators:examples

www.uis.unesco.org

How do we measure innovation? (1)

Indicators - definition:

• Statistics and data, often gathered through specialised surveys, are the building blocks from which indicators are constructed;

• An indicator can be defined as something that helps us understand where we are, where we are going and how far we are from a specific goal. Therefore it can be a sign, a number, a graphic;

• An indicator quantifies and simplifies phenomena and helps us understand complex realities.

Source: International Institute for Sustainable Development / Adapted from Blakley, W. (2012). Providing and calculating innovation

indicators. Cape Town, South Africa. ASTII/HSRC/UIS Workshop.(PowerPoint Presentation)

www.uis.unesco.org

How do we measure innovation? (2)

Indicators - definition:

• Basic indicators: based on “one question”;

• Composite indicators: combine answers to several questions in order to examine a number of policy-relevant factors and better capture the diversity of innovative firms.

www.uis.unesco.org

Innovation indicators - examples (1)

Product or process innovation:

1. % of firms that implemented product innovation

2. % of firms that implemented process innovation

3. % of firms that implemented product or process innovation (innovative firms)

4. % of firms that developed in-house product or process innovation

5. % of firms that implemented new-to-market product innovation

www.uis.unesco.org



1. % of firms that implemented product innovation

(N) =Number of firms that implemented product innovation (in each economic activity)

*100

(D) = Total number of firms (in each economic activity)

(N) =Number of Manufacturing firms that implemented product innovation

*100

(D) = Total number of Manufacturing firms

www.uis.unesco.org



15

30

45

60

75

Brazil China Colombia Egypt Israel Malaysia Philippines RussianFederation

SouthAfrica

Uruguay

Product innovation Process innovation

Source: 2011 UIS Pilot Data Collection of Innovation Statistics

www.uis.unesco.org


Marketing or organisational innovation:

1. % of firms that implemented marketing innovation

2. % of firms that implemented organisational innovation

3. % of firms that implemented marketing or organisational innovation

www.uis.unesco.org


Marketing or organisational innovation:

3. % of firms that implemented marketing or organisational innovation

(N) =Number of firms that implemented marketing or organisational innovation (in each economic activity)

*100

(D) = Total number of firms (in each economic activity)

(N) =Number of firms in the Electrical machinery industry that implemented marketing or organisational innovation

*100

(D) = Total number of firms in the Electrical machinery industry

www.uis.unesco.org


Inputs:

1. Total expenditures on innovation (as a % of total turnover)

2. Expenditure on innovation by type of expenditure (as a % of total expenditure on innovation)

3. % of firms that performed R&D

4. % of firms that performed R&D on a continuous basis

www.uis.unesco.org


Inputs:

3. % of firms that performed R&D

(N) =Number of product or process innovation-active firms that performed R&D (in each economic activity)

*100

(D) =Total number of product or process innovation-active firms (in each economic activity)

(N) =Number of Services product or process innovation-active firms that performed R&D

*100

(D) =Total number of Services product or process innovation-active firms

www.uis.unesco.org


Key policy-relevant characteristics:

1. % of firms that were active on international markets

2. % of firms that co-operated with foreign partners on innovations

3. % of firms that co-operated with universities or other higher education institutions

4. % of firms that received public financial support for innovation

5. % of firms that applied for one or more patents

6. % of R&D-performing firms that co-operated with other institutions

www.uis.unesco.org



3. % of firms that co-operated with universities or other higher education institutions

(N) =Number of product or process innovation-active firms that co-operated with a specific partner

*100

(D) = Total number of product or process innovation-active firms

(N) =Number of product or process innovation-active firms that co-operated with universities or other higher education institutions

*100

(D) = Total number of product or process innovation-active firms

www.uis.unesco.org


Key policy-relevant characteristics: co-operation

Any type ofco-operation

partner

Other enterprises within your

enterprise group

Suppliers of equipment, materials,

components, or software

Clients or customers

Competitors or other

enterprises in your sector

Consultants, commercial

labs, or private R&D institutes

Universities or other higher education institutions

Government or public research

institutes

Brazil 9.7 1.1 5.0 3.5 1.0 1.9 1.9 n.a.

China n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a.

Colombia 47.8 18.3 31.8 24.9 5.8 20.7 14.9 n.a.

Egypt 7.5 n.a. n.a. n.a. n.a. n.a. n.a. n.a.

Ghana n.a. 28.1 21.1 31.6 17.5 22.8 12.3 8.8

Indonesia n.a. 37.8 66.3 n.a. 18.4 24.5 19.4 11.2

Israel 33.4 8.3 19.6 21.3 14.4 17.3 12.6 8.2

Malaysia n.a. 65.5 55.1 56.1 30.0 84.0 45.0 37.0

Philippines n.a. 91.2 92.6 94.1 67.6 64.7 47.1 50.0

Russian Federation 37.3 12.6 16.9 10.9 3.9 5.1 9.1 15.6

South Africa 33.0 14.2 30.3 31.7 18.6 21.1 16.2 16.2

Uruguay n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a.

EU-27 n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a.

Eurostat min 12.9 2.4 7.1 4.2 2.7 4.4 4.3 1.1

Eurostat max 56.2 23.0 41.5 36.0 30.8 33.9 30.8 26.3

Co-operation partner

Source: 2011 UIS Pilot Data Collection of Innovation Statistics

www.uis.unesco.org



3. % of R&D-performing firms that co-operated with other institutions

(N) =Number of R&D-performing firms that co-operated with other institutions

*100

(D) = Total number of R&D-performing firms

(N) =Number of R&D-performing firms that co-operated with other institutions

*100

(D) = Total number of R&D-performing firms

www.uis.unesco.org

Final remarks

Data collected with innovation surveys are a important component of comparative studies about countries’ competitive performance;

Strategically important for policy-makers;

Data confidentiality;

Data reliability.

www.uis.unesco.org

Thank you!

http://www.uis.unesco.org

[email protected]

Innovation data collection: methodological procedures

Documents

surveys innovation

innovation activities

innovation rd surveyspro

firms main innovation

innovationactive guidelines

business surveys

survey sample

stratified sample surveys