Q2008 - Rome Italy, July 20081 Towards Producing and Using Response Burden Data for Establishment Surveys at Statistics Sweden Johan Erikson, Dan Hedlin,

Towards Producing and Using Response Burden Data for Establishment Surveys at Statistics Sweden

Johan Erikson, Dan Hedlin, Boris Lorenc

Statistics Sweden

Q2008 - Rome Italy, July 2008 2

Background

• A Statistics Sweden project in 2007 to establish measures and procedures for monitoring and reporting response burden in establishment surveys (May-October 2007)

• Coincided with publication of Eurostat’s Handbook for Monitoring and Evaluating Business Survey Response Burdens (August 2007)


Background




Background




Background



• Regulations, SCM, earlier Stats Sweden work


Agenda

• Background• Relation between Handbook and the results of the

Statistics Sweden response burden (RB) project– The process of measuring response burden according

to Handbook– Suggestions according to the Statistics Sweden RB

project

• Towards the implementation– the mini-measurement and its implementation– glance into the future

• Conclusions



The Handbook: Intro

• Basically discusses the following main topics:I. Monitoring the burden (Ch. 2)

• 9 core questions (q’s)

II.Evaluating analytically the burden, relating it to causal factors (Ch. 3)

• About 17 additional q’s

III.Sampling and data collection procedures (Ch. 4 & Ch. 5)

IV.Reporting (Ch. 6)


The Handbook: Intro

• Basically discusses the following main topics:I. Monitoring the burden (Ch. 2)

• 9 core questions (q’s)

II.Evaluating analytically the burden, relating it to causal factors (Ch. 3)

• About 17 additional q’s

III.Sampling and data collection procedures (Ch. 4 & Ch. 5)

IV.Reporting (Ch. 6)


The Handbook: Intro (cont’d)

• In discussing the topics I-II, identifies some key distinctions, amongst them:– actual vs. perceived response burden

– survey burden vs. time spent on survey

– response process steps: data collection vs. filling in the questionnaire (Q)


The Handbook: Actual/Perceived Distinction

• Actual burden = Perceived burden? No, that's why both actual & perceived needed!– “The aim of this handbook is to provide a tool for

measuring perceived and actual response burden in business surveys that will help statistical organisations and other parties carry out their own response burden (PRB) surveys.” [p.1]

– “This publication is a handbook written for statistical organisations and other institutions that carry out business surveys. Its main topic is how to monitor and evaluate perceived response burden (PRB) among business survey respondents.” [p.5]


The Handbook: Burden/Time Distinction

• Burden = Time? No, that's why both time & burden needed!– “...it is not the time use itself but the perception of time and

effort that is likely to affect response quality. Time passes quickly if the topic is interesting and the respondent feels that he or she is competent to answer the questions posed.” [p.6]

– “Actual burden is traditionally measured by time use. As we have mentioned earlier, we have chosen more detailed time measures than what has been traditionally used. We have designed two questions to measure time to collate information and one to measure time to fill in the questionnaire.” [p.11]


The Handbook: Steps of the Response Process Distinction

• The response process has these two steps (amongst others):– data collection (corresponds to step 5 of the Hybrid model

of Willimack, Nichols & Sudman)

– questionnaire fill in (corresponds to step 7 of the Hybrid model)


The Handbook: I. Monitoring Burden (Ch.2)• 4 dimensions, 8 indicators, 9 variables:Dimension Variable (question)

Perceived burden Perception of time to collect data - " - Perception of burden to complete questionnaire

Actual burden Time to collect information - self - " - Time to collect information - business - " - Time to complete questionnairePerceived causes of burden

Reason for being time-consuming

- " - Conditions for burdenMotivation Usefulness for own business - " - Usefulness for society

The Handbook: II. Analysing Causes of Burden (Ch.3)• 4 dimensions, 6 indicators, 17 variables:

Dimension Variable (question)Business context factors Available documentation - finding - " - Available documentation - accessDivision of labour Number of people involved - " - Roles - " - Number of Q’s to complete during the same

time period - other businesses - " - Number of Q’s to complete during the same

time period - other surveysPersonal characteristics Available time - finding - " - Available time - causes of problems - " - Available time - previous experience - " - CompetenceResponse problems Questionnaire properties [7 q’s]





Issues Discussed in the RB-Project• Given burden ≠ time, is Perceived length of time

(q.1 of Monitoring) properly an indicator of Perceived burden (dim.1)?

• Is Self-reported time (q.3 of Monitoring) properly an indicator of Actual burden (dim.2)?– from the premise of actual burden ≠ perceived burden– from the measurement perspective: a retrospective question

• Perceived time asked only regarding data collection (q.1), Perceived burden asked only regarding Q completion (q.2). Why? Why not the converse?


Issues Discussed in the RB-Project ( ’ )cont d• Is motivation to participate in surveys exhausted

by tackling the utility of participation for the business and the society?– Cooperation principles (mentioned in the Handbook)– Social exchange– Helpfulness, etc.

• Practicalities about the questionnaire:– Length– Content (e.g. Why causes in Monitoring [its dim.3]?)– Question formulation (e.g. Time to collect information -

business (q.4), does that include or exclude Time to collect information - self (q.3)?)

– Question order (e.g. the complex branching instructions pertaining to the first question)


Positions Taken in the RB-Project• Measure respondent burden or measure

respondent work?– Measure burden to begin with, but bring later on even

positive aspects of survey participation into focus

• Monitor or evaluate?– Monitor on a broad front, evaluate in conjunction with

changes or actions for improvement. (Same as in the Handbook.)

• Actual or perceived burden?– Perceived, as well as an (assumed) correlate of actual time:

the self-reported time (for oneself and for others) it took to participate in the current survey


Positions Taken in the RB-Project ( ’ )cont d• Breakdown of burden in the evaluation Q: along

questions or along steps of the response process?– Response process steps

• Motivation– More than utility: left out for the moment

• Explanatory variables– Number of people involved (q.3 of Evaluation)– Assessment of data availability (-)– Previous experience with the same survey (q.9 of Evaluation)– Respondent’s role in the current data collection (q.4 of Eval.) – [Data regarding simultaneous surveys can be taken out of our own

data base]


Basic Proposal of the RB-Project

• Perceived burden measured on three levels:– Monitoring on a broad level (level A; contains the mentioned

explanatory variables)

– Breakdown of burden along the steps of the response process (level B); measured in conjunction with carrying out an improvement project

– Identification of probable causes (level C); measured in

conjunction with carrying out an improvement project


RB-Project: Level A

• Variables– Perception regarding the current participation

• General level of burden

– A correlate of actual burden• self-reported time it took for own participation (retrospective)• reported time it took for other’s participation (retrospective)

– Explanatory variables• Number of people involved• Assessment of data availability• Previous experience with the same survey• Possibly, suggestions for improvement


RB-Project: Level B

• Variables– Perceptions regarding the current participation

• Burden to understand the request for data• Burden to distribute and coordinate the data collection• Burden to collect data• Burden to process data• Burden to provide data in the required form

– Explanatory variables• Respondent’s role in the current data collection


RB-Project: Level C

• Variables– Perceptions regarding the current participation

• Causes of the burden to understand the request for data• Causes of the burden to distribute and coordinate the data

collection• Causes of the burden to collect data• Causes of the burden to process data• Causes of the burden to provide data in the required form


RB-Project: Conclusion

• Level A measured systematically and continuously in all enterprise surveys– But, not each occurrence of an recurring survey: a sampling

procedure needed (not treated here)

• Levels B/C used in conjunction with evaluation or improvement of the data collection procedures– Provide input for simplification, questionnaire evaluation

work, etc. – Possibly followed by qualitative methods on an even smaller

scale


The Mini RB-measurement

• Implemented gradually since a year ago

• Measures self-reported time

• Consists of a single question, posed at the end of the survey– Time within the business it took to participate in the current

survey

• Routines to process, present and use these data are currently implemented– sampling– estimation– useful summary tables– input to other processes


Conclusion

• Do measure response burden

• But, only if the collected data are used for the benefit of the data provider

• Otherwise, no real justification to do this kind of survey


Thank You

Johan Erikson

Dan Hedlin

Boris Lorenc

[email protected]

Q2008 - Rome Italy, July 20081 Towards Producing and Using Response Burden Data for Establishment Surveys at Statistics Sweden Johan Erikson, Dan Hedlin,

Documents

reporting response burden

response burden data

actual response burden

perceived burden

q2008 rome italy

statistics sweden project

statistics sweden rb

handbook suggestions