Using survey data collection as a tool for improving the survey process Silvia Biffignandi, Antonio Laureti Giulio Perani University of Bergamo Istat Istat.

Using survey data collection as a tool for improving the survey

process

Silvia Biffignandi, Antonio Laureti Giulio Perani University of Bergamo Istat Istat

EESW11,European Establishment Statistics Workshop, 12-14 September 2011, Neuchatel, Swiss

Outline Outline


Introduction

Survey

Questionnaire

Preliminary results

Research in progress

Introduction: Paradata and web surveysIntroduction: Paradata and web surveys

Web surveys allow for the collection of paradata during web questionnaire completion;

paradata are data generated during the fieldwork of the survey.

Two types of paradata: server-side paradata are collected by software tools running at the server. They relate mainly to the questionnaire compilation process. This data are contained in the so called logfiles.

client-side paradata describe how the respondents are answering the questions (order, questions skipped, keys that have been pressed and so on). (Biffignandi and Bethlehem, 2012; Biffignandi, 2010)

Using survey data collection as a tool for improving the

survey process


Analysis of paradata for:

questionnaire improvement (identification of the most difficult questions in the survey form, of some missing (or too restrictive) checks )

quantification of the burden on respondents (from the degree of ready availability of R&D data in enterprises, to the time needed to fill in the questionnaire )

timeliness? costs? identification of the characteristics of late respondents.


survey process

Introduction: Tasks of paradata Introduction: Tasks of paradata analysis analysis


The surveyThe survey business R&D activities

yearly , census

target population: enterprises known, or assumed, to be R&D performers

many administrative sources to identify the target population (20000 “potential R&D performers” )(previous Istat R&D surveys, other Istat business surveys with R&D-related questions, ASIA business register, fiscal data, Italian Register of R&D performing institutions,data on national and EU funding to research projects, patent databases, private business reports)

no a cut-off point in term of size, i.e. considered all enterprises (including micro-enterprises, provided that they are employing at least one researcher)

reference year 2008; overall response rate 55%


survey process


Future R&D Future R&D performers performers

(who are just (who are just asked to asked to

report about report about their future their future investment investment

plans)plans)

Potential R&D performersPotential R&D performers

Questionnaire Questionnaire

AActual R&D ctual R&D performers performers (those who have (those who have to provide to provide extensive extensive information on information on the R&D the R&D activities activities undertaken in undertaken in the reference the reference year)year)


survey process


Non-R&D Non-R&D performers (no performers (no data requested)data requested)

R&D R&D personnelpersonnel

Actual R&D performersActual R&D performers

Questionnaire Questionnaire

R&D R&D expendituresexpenditures


survey process


Qualitative Qualitative questions on the questions on the R&D projectsR&D projects

Web survey administrationWeb survey administration

Advanced web questionnaire design

questionnaire mainly based on previous paper questionnaire

innovative software architecture : two physically distinct components

electronic form: delivered via Web on the remote PC of the respondent

checking tool; resident in a Web-server in the Istat’s premises. This is a tool for the identification of the respondent and of the correct sequencing of the questionnaire’s provision.


survey process


Available paradataAvailable paradata


survey process


•Timing of use:Entry and exit date.Days in which the tool has been accessed.Hours of use during the day.

•Intensity of use:Number and typology of actions (events).Processing time associated to each single action.Number of actions per unit of time (e.g. per day or fraction of day).Outcome of the compilation process.

•Compilation routes:Sequencing in accessing the questions.Changes to previously compiled questions.

•Generation of errors:Number and typology of errors.Questions (or groups of questions) mostly affected by errors.

DefinitionsDefinitions

““eventsevents”:

each time a respondents is interacting with the electronic form (basically, by typing a figure or a word, or even, by scrolling down the form itself).

We can identify each single “event” by its nature and by the time when it took place, as well as by its duration in time (usually, from a fraction to a few seconds).

Events could be either “correct” (according to the rationale behind the structure of the questionnaire) or leading to the generation of “errors”.

The questionnaire is “completed” only when all “errors” are properly fixed, the generation of “errors” will have as a result the need for new events aimed at correcting them. In the process of numbering the errors produced by each respondent, only the first appearance of an error type was taken into consideration, even in the event that the same error type would have been repeated in more than one session.


survey process


DefinitionsDefinitions

Each event – or a group of events – is associated to an “access”.

““access”:access”: itself to the questionnaire cannot be identified in a straightforward way as “log-ins” and “log-outs” are not recorded in the system.

We know that a respondents is (was) connected to the server because of the activity carried out on the questionnaire.

An access without any activity on the questionnaire will not be taken into consideration. An access has to be qualified in terms of time. Conventionally, all events taking place within one hour (or three, or six hours) could form an “access”.

Only “daily accesses”“daily accesses” are taken into consideration in this study, i.e. an “access” will be equivalent to the set of all events having taken place in a day.


survey process



survey process


Results. Results. Business R&D survey 2008 data collection. Paradata indicators: Number of respondents*, daily accesses and errors.

Paradata indicatorsActual R&D performers

Future R&D performers

Non R&D performers

Respondents who have accessed the

questionnaire without completing it

Total

N. of respondents* 2.601 268 1.086 324 4.279 Number of events 614.273 10.942 30.577 5.274 661.066

Average n. events per respondent

236,17 40,83 28,16 16,28 154,49

Number of daily accesses

4.471 371 1.359 391 6.592

Average n. accesses per respondent

1,72 1,38 1,25 1,21 1,54

Average number of events per access

137,39 29,49 22,50 13,49 100,28

Number of errors** 48.291 860 1.852 468 51.471 Average number of errors per respondent

18,57 3,21 1,71 1,44 12,03


survey process


ResultsResults

Type of errors.Type of errors.


survey process


Frequency of errors by type. Total number of errors=51,533

0

500

1000

1500

2000

2500

3000

3500

4000

E00

1

E01

1

E02

1

E03

1

E04

1

E05

1

E06

1

E07

1

E08

1

E09

1

E10

1

E11

1

E12

1

E13

1

E14

1

E15

1

E16

1

E17

1

E18

1

E19

1

E20

1

E21

1

E22

1

E23

1

E24

1

E25

1

E26

1

E27

1

E28

1

E29

1

E30

1

E31

1

E32

1

E33

1

E34

1

E35

1

E36

1

E37

1

E38

1

Daily accessesDaily accesses


Frequency of daily accesses by respondent. Actual R&D performersTotal number of accesses=4,478

1

10

100

1000

10000

1 2 3 4 5 6 7 8 9 10

Totale

Somma di Number of enterprises

Number of daily accesses

Final delivery of questionnaireFinal delivery of questionnaire


Final delivery of questionnaires (Actual R&D performers, 2,200 questionnaires). Observation period: March to October 2010

0

20

40

60

80

100

120

140

05

/03

/20

10

12

/03

/20

10

19

/03

/20

10

26

/03

/20

10

02

/04

/20

10

09

/04

/20

10

16

/04

/20

10

23

/04

/20

10

30

/04

/20

10

07

/05

/20

10

14

/05

/20

10

21

/05

/20

10

28

/05

/20

10

04

/06

/20

10

11

/06

/20

10

18

/06

/20

10

25

/06

/20

10

02

/07

/20

10

09

/07

/20

10

16

/07

/20

10

23

/07

/20

10

30

/07

/20

10

06

/08

/20

10

13

/08

/20

10

20

/08

/20

10

27

/08

/20

10

03

/09

/20

10

10

/09

/20

10

17

/09

/20

10

24

/09

/20

10

01

/10

/20

10

08

/10

/20

10

15

/10

/20

10

22

/10

/20

10

29

/10

/20

10

Actual R&D performers: events accesses and errorsActual R&D performers: events accesses and errors


survey process


Number of employees

Number of events for resp.

Number of daily accesses for resp.

Number of errors for resp.

A 1-9 215,7 1,7 18,2

B 10-49 232,2 1,6 17,9

C 50-249 234,3 1,6 18,5

D 250-499 209,9 1,8 17,9

E 500+ 291,0 2,1 21,8

Total 236,2 1,7 18,6

Research in progressResearch in progress


deeper analyses on delivering time and on sectoral break-down

more detailed cross checking of the errors and mapping

data processing of the recent R&D survey edition

conclusions on questionnaire improvement

conclusions/suggestions for more efficient follow up strategies


Suggestions? …..Suggestions? …..

Thank you for your attention!Thank you for your attention!

Using survey data collection as a tool for improving the survey process Silvia Biffignandi, Antonio Laureti Giulio Perani University of Bergamo Istat Istat.

Documents