Cognitive testing CIS 2013 Belgian findings

COGNITIVE TESTING CIS 2013BELGIAN FINDINGS

Anneleen Bruylant, Petra Andries and Machteld HoskensECOOM

Presented at het Eurostat CIS Task Force meeting, 3-4 May 2012

THE ORIGINAL QUESTION

• The question aims at getting an estimate for the expenses firms had in year X to perform their innovation activities with regard to the four given areas. Firms often gave their total expenses though, rather than their expenses for innovation.

• The question was rephrased, and Belgium decided to test the longer version.

ISSUE TO RESOLVE

In 2012, how much did your company spend on the following activities or acquisitions? Then please estimate the share of each activity or acquisition that was meant for product or process innovation, as defined earlier.

Activity Expenditures 2012 Estimated share of activityfor product and/or process innovation

• Internal R&D _____________€ ____%(including investments in property and equipment,personnel and related costs specifically meant for R&D)• Purchase of external R&D _____________€ ____%• Purchase of property, buildings, _____________€ ____% machinery and equipment (excluding purchases made for R&D)• Purchase of software _____________€ ____%• Purchase of external knowledge _____________€ ____%(including licenses)• Training paid by the employer _____________€ ____%• Advertisement, market campaigns, _____________€ ____% market research• Design activities _____________€ ____%(in-house and external contracts)

REVISED QUESTION: VERSION 1

• Version 1 was tested on nine subjects, six French-speaking, three Dutch-speaking.

• Main issues:I. Some subjects read the question vertically, instead of horizontally. In the first column they filled in the expenditures specific for innovation, rather than general expenditures. This way, they sought to make the sum of the percentages add up to 100%. So ironically, subjects answered the new question as if it were phrased in the old way, not seeming to notice what the revised question was asking. This problem occurred in 4 out of 9 cases.

RESULTS OF VERSION 1

II. “Design” is too vague a term, as people didn’t know what is meant by it (four out of nine subjects). Different industries had a different take on the term, varying from understanding “design” as referring to everything from user interface to fliers. The vagueness of the terms used (“innovation”, “R&D”) was mentioned several times.III. One of the subjects (an R&D manager) thought the question was difficult to answer, as he didn’t know the total expenditure (only the innovation expenses), and would thus have to gather the information. Also, calculating the percentages was seen as something very difficult to do.

In 2012, how much did your company spend on the following general activities or acquisitions? Then please estimate the share of each activity or acquisition that was specifically meant for product or process innovation, as defined earlier.

Activity Expenditures 2012 • Purchase of property, buildings, machinery and equipment: _____________€, of which ____% for product or process innovation(excluding purchases made for R&D)• Purchase of software: _____________€, of which ____% for product or process innovation• Purchase of external knowledge (including licenses): _____________€, of which ____% for product or process innovation• Training paid by the employer: _____________€, of which ____% for product or process innovation• Advertisement, market campaigns, market research: _____________€, of which ____% for product or process innovation• Design activities: _____________€, of which ____% for product or process innovation(in-house and external contracts)• Internal R&D: _____________€, of which ____% for product or process innovation(including investments in property and equipment,personnel and related costs specifically meant for R&D)• Purchase of external R&D: _____________€, of which ____% for product or process

innovation

VERSION 2

• We moved R&D to the end of the question, hoping this would make the context of the question less focused on R&D and innovation, so subjects wouldn’t answer the question the way it used to be phrased.

• We highlighted the words “general” and “specifically” in the question, hoping this would emphasize the intent of the question.

• We tested this version on only one subject.

RESULTS OF VERSION 2

• There was still confusion about the percentages: the subject read the question vertically, not horizontally. The subject admitted it was actually very clear once it was pointed out to her there was a second column to the question.

• There was also confusion about what to include in the innovation expenses, and what not to include. This was the case for “purchase of buildings, machinery and equipment”, “publicity”, and “design”.

• In 2012, how much did your company spend on the following general activities or acquisitions?

• Then please estimate the share of each activity or acquisition that was specifically meant for product or process innovation, as defined earlier.

VERSION 3

• We split the question in two, thinking that this would make the purpose of the question clearer.

• The question remained the same as in version 2, only the lay-out of the question was changed.

• This version was tested on eight subjects.

RESULTS OF VERSION 3

• Three subjects remarked that the way the question was phrased made it necessary to involve several people to answer the question, instead of just one person.

• One person still read the question vertically, the other seven read it correctly.

• The exact meanings of the following terms were unclear: “external R&D”, “design”, “external knowledge”, “innovation”, “new product”, and “training”.

• “Innovation” and “R&D” were seen as interchangeable terms, which they are not. This was exemplified by the remark one subject made, namely, that the clause “exclude purchases for internal R&D” was confusing, as we later asked for the percentage dedicated to innovation (and “innovation” was seen as synonymous with “R&D”).

• Some subjects were not entirely clear about the distinction between “design” and “internal R&D”, or between “software” and “IP”. Also, the distinction between “external R&D” and “external knowledge” was not clear to several subjects.

• Two subjects pointed out that firms may use a narrower definition of “R&D”, thus under-reporting the actual R&D expenditures. This is because the person who fills in the questionnaire is most likely the same person who applies for R&D subsidies, which are granted based on a narrower definition of “R&D” than the one used in our questionnaire.

• Although the need for rephrasing the original question was abundantly clear, our experience showed that subjects – when faced with the revised question – either answered as if they were answering the original question, or – when noticing the difference – remarked that the new version was more complicated, as they would now have to involve several people to answer the question.

CONCLUSIONS

• Clearer definitions of the terms used are necessary, and they could be repeated for clarity on all relevant pages.

• We propose rephrasing the original question again, as the revised question seemed to confuse subjects more, or make it more difficult to answer. Possible rephrasings for consideration could include:

5.1 During the three years 2010 to 2012, did your enterprise engage in the following innovation activities:

Yes No In-house R&D Creative work undertaken within your enterprise to increase the stock of

knowledge for developing new and improved products and processes (include software development in-house that meets this requirement)

If so, please estimate the amount of expenditure aimed specifically at product and process innovation for 2012 only: __________€

If so, did your enterprise perform R&D during the three years 2008 to 2010:

Continuously (your enterprise has permanent R&D staff in-house) Occasionally (as needed only)

External R&D Same activities as above, but performed by other enterprises (including other

enterprises or subsidiaries within your group) or by public or private research organisations and purchased by your enterprise If so, please estimate the amount of expenditure aimed specifically at product and process innovation for 2012 only: __________€

Acquisition of machinery, equipment and software

Acquisition of advanced machinery, equipment (including computer hardware) or software to produce new or significantly improved products and processes If so, please estimate the amount of expenditure aimed specifically at product and process innovation for 2012 only: __________€

Acquisition of external knowledge

Purchase or licensing of patents and non-patented inventions, know-how, and other types of knowledge from other enterprises or organisations for the development of new or significantly improved products and processes If so, please estimate the amount of expenditure aimed specifically at product and process innovation for 2012 only: __________€

Training for innovative activities

Internal or external training for your personnel specifically for the development and/or introduction of new or significantly improved products and processes If so, please estimate the amount of expenditure aimed specifically at product and process innovation for 2012 only: __________€

5.2 Please estimate the amount of expenditure for each of the following four innovation activities in 2012 only. (Include personnel and related costs)

If your enterprise had no expenditures in 2010, please fill in ‘0’ In-house R&D (Include capital expenditures on buildings and equipment specifically for R&D) specifically aimed at product and process innovation

__________.000€

Purchase of external R&D specifically aimed at product and process innovation

__________.000€

Acquisition of machinery, equipment, and software specifically aimed at product and process innovation (exclude expenditures on equipment for R&D)

__________.000€

Acquisition of external knowledge specifically aimed at product and process innovation

__________.000€

Total of these four innovation expenditure categories __________.000€