Collecting online: How Statistics Netherlands moved its business Surveys onto the internet Ger Snijkers thanks to Deirdre Giesen, Hank Hermans, Myra Wieling.

Collecting online: How Statistics Netherlands moved its

business Surveys onto the internet

Ger Snijkers thanks to Deirdre Giesen, Hank Hermans, Myra Wieling

15 October 2013 2OSS Seminar, Statistics New Zealand, Wellington

Overview

1. Overview of developments at Statistics Netherlands:a. Electronic data collectionb. Organising data collection and data sharing

2. Case study: Moving the Structural Business Survey to the weba. Developing and testing the questionnaireb. Communication strategy to launch the survey and improving web-take up ratesc. Implementation of the survey

Lessons learned and challenges3. Looking into the future: future challenges


Brief history of electronic data collection at SN

Before 2000:• 1984 First PCs• Mid 1980s Blaise 1.0• 1993 E-reporting for International Trade: IRIS

- disk-by-mail (floppy disks, CDs), - now offline using downloadable software

• Mid 1990s Electronic Data Interchange: EDIsent

- software installed on business’ systems

• 1998 Simple establishment surveys - e-mail


E-mail questionnaire Business Tendency Survey 1999


Brief history of electronic data collection at SN (cont)

After 2000: 2000 Short Term Statistics (STS)

- first electronic forms (HTML)• 2003 Transportation Survey• 2006 STS – New HTML forms• 2006 Structural Business Surveys

- Pilot offline 2007 Structural Business Survey in the field

- offline downloadable software• 2009 Business Surveys online: Transportation

Survey• 2014 - New integrated system for online

surveys - Blaise 5

• Next EDI using XBRL integrated in questionnaires

• 2005 Experiments for household surveys (on-line)• 2008 Household surveys on the internet (on-line)


Transportation Survey 2003


Offline SBS Survey 2007 more to come


Online Transportation Survey 2009


XBRL-based system (future)


Health Survey 2010


Business Data Collection at the moment

A few numbers:

450 different questionnaires 400 mixed-mode: electronic & paper• > 500,000 business units every year• > 750,000 questionnaires on average each year 700,000 electronic questionnaires available


Drivers for using Web for Business Surveys

• From the data collection agency perspective:- data quality: cleaner (fewer edits) and more complete- cost-efficiency: cheaper and less editing- timeliness: faster

• From the business perspective:- internet penetration for businesses is high- much of the data held by businesses is already in digital form- many of the regulatory and administrative tasks they perform are already online- can be easier to do than paper (if designed well)

This only holds is surveys are designed well!- the whole communication process: from end to end


Conclusions so far

Until 2007: • Each survey for its own: stove-pipe approach• No coordinated/harmonised design• Have it in the field quickly• Focus on technology, not on

methodology/communication

No efficient questionnaire design and communication:

• Internally: every survey starts all over, no learning curve

• Externally: no professional questionnaire and survey communication:- various designs - respondents have to learn every questionnaire again


Lessons learned

• Analyse contextual factors:- Internal constraints and processes- Response processes within businesses

• Focus on usability = success factor:- internally AND externally - technology AND methodology perspective

“We suspect that many of the survey organisations that introduce web questionnaires forget that it is not the technology in itself, but how it is utilised that determines the result” ... both internally and within businesses

From: Haraldsen & Couper (2013)

• Apply Deming cycle – PDCA:- also C & A: Evaluate and Improve


Lessons learned (cont’d)

• Good design is to create benefits for both sides:

- Create internal conditions to be successful (under control)

> One Data Collection Division - Tailor to business conditions (not under control; try

to influence) > Moving SBS to the Web


Overview

1. Overview of developments at Statistics Netherlands:

a. Electronic data collection b. Organising data collection and data sharing

2. Case study:Moving the Structural Business Survey to the weba. Developing and testing the questionnaireb. Communication strategy to launch the survey and improving web-take up ratesc. Implementation of the survey



Data Collection over the years

Stove pipes Data Collection 2000

Partial Centralisation Data Collection Now

< 1994 1994 - 2000

2000 - 2007

Present


Organizing Data Collection in one DivisionProcess-and-knowledge Driven approach

Front desk:• Client relations• Survey design

Design:• Questionnaire• Sampling• Training

Survey deployment:

• CAPI / CATI Interviewing

• Planning & control• Fieldwork logistics• Support• Special services

Clustering of practices and knowledge• Focus on how knowledge is being used, for social and

business surveys• Pilot 2007; effectuated 2011

12 fte

35 fte

240 fte

• Fte’s:- 375 (2008)- 277 (2012)

• Budget:€ 18.5M

• 145 surveys

92 26 118 fte

122 fte


Drivers for centralizing data collection

Internal drivers• Achieving efficiency by: ∘ Abolishing redundant processes, workflows, and

activities for social and business surveys- Monitoring processes, workflows, and activities

∘ Abolishing redundant systems and tools - One data collection management system - Maintaining as less systems and tools as needed

• Implementation of Data Collection Strategy (2005)

External drivers:• Professional external focus• Reducing response burden


The 2011 Data Collection Strategy:Retrieving and returning (first version in 2005)

3 steps:1. Re-use of available data

- Data sharing & data warehousing

2. Use of new registers and other secondary sources - Traditional government-based registers - Big/organic data

- Data on the internet (web-crawlers)

3. Primary data collection:1. EDI technologies, like XBRL

2. Web surveys 3. traditional modes: paper, CATI, CAPI

• Using new technologies • Reciprocity: report back to respondents

Multi-source designs

Mixed-mode designs


Data collection: Developments over the yearsSummary and conclusions

• From stove-pipes to a coordinated system of data collection

• From uni-mode surveys to multi-source/mixed-mode data collection designs

• From single-survey managers to managers of integrated sets of statistics

• From local decision making to corporate decision making • From instrumental data collection to a communication

perspective

We are still in the middle of this process! A cultural shift needs years to change!But … are we moving too slow?


Overview





Moving to the Web: two basic approaches

• Copy all paper questionnaires to the web in a short period of time, in one or various ways:- PDF, offline electronic questionnaire, online questionnaire

Quick results, but how is the questionnaire utilised? High risks of: - web Qs not being used

- negative effects on public image Pushing problems and costs downstream Evaluate and improve, if you still get the chance ...• Develop one approach and apply to all surveys:

- investing in the organisation, technology, and methodology- investing in the ‘respondent experience’: survey communication - investing in PR: harmonised approach

Takes time and costs money Higher chances of good results: web Qs actually used


Moving the SBS questionnaire to the web: approach

Approach by Stats Neth.: 2nd approach• Develop one approach:

- Good results for SBS …

No systematic evaluations and improvements• that is applied to all surveys:

Some follow-up, but not consistent, due to:- organisation not yet ready (one of the main factors for project failure) - still existing stove-pipes

Needs a long-term vision that is followed up Assurance that investments are not lost

Do it right: create benefits for both sides (or

don’t do it all)! It is not going to be easy!


Moving the SBS questionnaire to the web: Project

Project goals and constraints:• Develop a web questionnaire

- same contents- mixed-mode design: coherent paper and web- support completion process- motivate respondents to use this mode- into the field: March 2006

• Start of project: June 2004- we had one year, until Sept. 2005


Dutch Annual Structural Business Survey

Some characteristics:• Annual survey of economic activity, three parts:

- revenues and costs- summary of business accounts: profits and losses- industry specific specifications

• Mandatory• 75.000 business each year:

- Sample of small businesses, bigger businesses each year

• Until 2006: paper questionnaire (A4 booklet):- Length may differ, 20 pages is typical- Items are grouped in sections- Completion process: complicated and difficult:

lots of details asked for, mismatch of definitions, various business departments involved, design issues


Vernieuwde huisstijl:2. Onderzoek naar de PS

–Q oud


Vernieuwde huisstijl:2. Onderzoek naar de PS

–Q oud


3. 4 projects3. Improving paper questionnaire

–Q nieuw


Developing the electronic questionnaire

Developing and testing in five stages:1. Develop and test the prototype (31-1-’05)

- pre-tests to test usability: 3 waves - field agents, business respondents

2. Revision of questionnaire (1-9-’05)- expert reviews

3. Testing of revised questionnaire (1-1-’06)- additional usability tests

4. Implementation of field pilot (1-3-’06)5. Implementation of survey (1-3-’07)


The electronic questionnaire: the prototype

Developing paper Q and testing this prototype helped in thinking about a good design for an eQ


Testing the prototype

Research issues:1. How does the e-Q work in practice?

- Completing the questionnaire- Question-and-answer process- Usability

2. What features should be included to make it easy to use?- Respondent friendly: ‘Computer-assisted’ tools- User demands

3. How should the eQ be designed in relation to the paper Q?- The same or a different design- ‘look-and-feel’ of paper and e-form


Testing the prototype: Research issues

1.How does the Q work?Laborious and complex process• Long, complex questionnaire (≥ 25 items)• Complex completion process:

- several sessions, several informants- kick-and-rush behaviour

• Imagine ... a respondent sitting behind his/her computer ...

Respondent got lost in the questionnaire


Illustration of usability

“Medieval tech support”

A new systemis introduced:The ‘book’ (“beek”), in stead ofrolls of paper.

The monk Ansgard has called the helpdesk and is waiting for someone to come …


Illustration of usability

“Medieval tech support”

It illustrates what we found in the pre-tests for the electronic questionnaire …


2. Features to make it easy to use?• What am I supposed to do (next)?

- Easy to download, install, complete, send data back- It is one process: going to the internet – submitting data, - Clear instructions and explanations (but not read)

• How is the questionnaire built up? - Show how the questionnaire is structured: overview- Help to find the way in the questionnaire- No hidden rules, no unexpected functionalities

• Where am I? What did I do so far?- Provide overview of the completion process- Clear navigation, no scrolling- Printing function

Usability, navigation and overview: design of Tax forms

Choose a design people know: don’t be original!



3. Design of paper and web Q?

The computer is different than paper• The web Q reacts to the respondent• Reading from the screen is different• Navigating and getting an overview works

differently• Kick-and-rush behaviour, even stronger than on

paper• The use of computer-assisted functionalities;• the respondent expects the computer to react


Start with paper, then add functionalities


Testing the prototype: Conclusions of pre-test waves

• Visual design Clear and logical: “Don’t make me think!” Simple, transparent, consistent No hidden and unexpected functionalities

• Support the completion process, end to end Other mode, other features, other visual design Use computer functionalities: automatic calculations Use layout and usability principles that are known to R (country specific)

• Tailor to kick-and-rush behaviour Split Q into small sections, small tasks Short and clear explanations


Web questionnaire design is communication design

Questionnaire designer=

Communication designer

Questionnaire communication design

Lessons learned: Web Design Guidelines


The revised questionnaire

Based on:• Pre-test results• Expert reviews

Iterative process with- Professional designer- Questionnaire designers- Methodologists

• A user-friendly design was put first, not the IT tool

• New prototypes designed in Power Point



The revised questionnaire: additional pretests

• 10 concurrent in-depth interviews

• Usability and user friendliness had been improved respondents enjoyed working with the questionnaire they could handle the task: they recognised the design and usability even though … the task had not changed


Field pilot

Goals:• Implementation of web questionnaire

- test usability and completion process in the field:

downloading – completing – sending-in data- test communication strategy to get high web take-up rates

• Test the whole process- logistics for paper and web flows

• Response rates and web take-up rates?• Data quality

- data editing, mode effects?


Field pilot

Set-up:• March-July 2006• 7200 businesses, 5 industries• Communication strategy:

- internet portal: www.cbs.nl/productiestatistiek- advance letter with user name and password- leaflet to introduce web questionnaire

and explain why the survey is conducted- paper questionnaire not mentioned- only 2e or 3e reminding letters include paper questionnaireIn the Netherlands businesses have to report electronically to the Tax Office.


Leaflet

• Informing

responden

ts about

new

questionn

aire

• Indicating

benefits

• Telling

what will

happen

with the

data

Showing some relevant

survey results


Field pilot

Results:• Web questionnaire worked well• Web take-up rate: 80%,

total response rate: comparable• No mode effects, but more research is needed• Internal processes worked well,

but we missed checking the link to data editing.

SBS questionnaire and communication design was implemented by the end of 2006, and run in 2007


Communication strategy: guidelines

How to increase web take-up rate:

• Easy access to the web version combined with restricted access to the paper version

• Offer real advantages• Making the respondents aware of the web option• Offer practical guidance

• Computer competence and motivation play a minor part(if designed well)


For more guidelines …


Implementation of SBS Survey

• Running from 2007 onward

• About 75.000 business receive this questionnaire every year

Web take-up rates:• Dutch Annual SBS, 2007: 84 %

Total response rate did not change (80%)• Dutch average 2008: 51% 3 largest Dutch business surveys (2008): 77

%(STS: 63%, SBS: 80%, Intrastat: 100%)


https://formulieren.cbs.nl/downloadsper survey, per period, per business unit the log on codes

Implementation of SBS Survey


The portal http://formulieren.cbs.nl


Welcome page


In the index items that are complete and correct

according to the person who is filling in the form are

ticked off


Click on the ? and then a more extensive explanation

appears


Print of the SBS questionnaire


The confirmation of receipt


Paper version SBS questionnaire


SBS design in production

Using SBS design as basis for other surveys• Visual design principles of new paper and

web questionnaires starting point for new style guide- for all 450 business questionnaires- for various modes: paper, offline, online

• Development of QUAT: Questionnaire Application Tool- project 2005-2006



Questionnaire Application Tool (QUAT): • Generic Tool for generating questionnaires:

- Editing of questionnaires, letters, and attachments - Uses data bases of populations and variables - Allows for tailored questionnaires, per stratum - Allows for generating questionnaires in mixed-mode

design: paper, online, offline

- Based on one set of visual design principles for each mode

• Uniform corporate identity concerning questionnaires and letters (one look-and-feel)

• Input coordination with regard to variables and classifications

• QUAT planner: for planning the dispatching of Qs



• SBS design still in production• But, offline system is out of date:

- runs only on Windows PC – lot of complaints from Apple users- Software to be downloaded – no open source software

• QUAT is still running• New system to be developed, on the same

principles


Overview





Future challenges for statistics

Global developments:

• Emergence of the information society• Globalization• Shift in the balance of power between NSIs and

businesses

• Erikson, Haraldsen & Snijkers, 2012, UNECE Seminar, Geneva

• Snijkers & Haraldsen, 2013, BLUE-ETS final Conference, Brussels


Challenges in modernising statistics

Based on global developments and data coll. strategy:

• Expanding usage of registers- data sharing and data warehousing- assessing quality of register-based statistics

• Blending designed and organic data - data integration theories

• Modernising current business survey designs- tailoring design to the business context, including

collecting data from globalised establishments- using paradata - adopting a communication perspective:

“Improving the Respondent Experience”

• Producing statistics in a globalized world- developing new statistical indicators- market position of an NSI?


Consequences …

• Challenges are too big for individual NSIs:- collaborate on international level- collaborate with universities

• Position in the information market: Our added value? - “slow” or/and “fast” statistics?

accurate or/and timely statistics?- NSIs have access to and can combine many data sources: new statistics- Bench marking of non-NSI statistics- National Data Archive? National Data Service Center?


Future challenges for statistics

Two final statements:

We are at the beginning of a new era in statistics production:

The era of data integration

Are we moving too slow?

The future is already here!


References

• Beukenhorst, D.J., and D. Giesen, 2010, Internet Surveys at Statistics Netherlands. Paper presented at the 2nd International Workshop on Internet Survey Methods, Statistics Korea, 8-9 september 2010, Deajon, South korea.

• Haraldsen, G., and M. Couper, 2013, How to Design Effective Business Web Surveys, Course at the Nordic Meeting, 14 August 2013.

• Haraldsen. G., and G. Snijkers, 2013, Modernisation and Quality of Business Statistics: The NSI Perspective. Paper presented at the 3rd European Establishment Statistics Workshop, 9-11 September 2013, Nuremberg, Germany.

• Groves, R.M. (2011), Three Eras of Survey Research. Public Opinion Quarterly, Vol. 75, pp. 861-871.

• Erikson, J., G. Haraldsen, and G. Snijkers, 2012, The Future of Statistical Data Collection? Challenges and Opportunities. Paper presented at the UNECE Seminar on New Frontiers for Statistical Data Collection, 31 October-2 November 2012, Geneva, Switzerland.


References

• Snijkers, G., H. Haraldsen, J. Jones, and D. Willimack, 2013, Designing and Conducting Business Surveys. Wiley, Hoboken.

• Snijkers, G., H. Hermans, and R. Göttgens, 2011, Data Collection and Sharing at Statistics Netherlands, Paper presented at the CES meeting, 14-16 June 2011, UNECE Geneva, Switzerland.

• Snijkers, G., E. Onat, E., and r. Vis-Visschers, 2007, The annual structural business survey: Developing and testing an electronic form, Proceedings of the 3rd International Conference on Establishment Surveys (ICES-III), Montreal, June 18–21, American Statistical Association, Alexandria, VA, pp. 456–463.

• Wieling, M., 2013, Electronic business questionnaires at Statistics Netherlands, Presentation at Statistics Netherlands for guests from Statistics Canada, 7 June 2013, Heerlen, Netherlands

Collecting online: How Statistics Netherlands moved its business Surveys onto the internet Ger Snijkers thanks to Deirdre Giesen, Hank Hermans, Myra Wieling.

Documents

statistics new zealand

statistics netherlands

mailoss seminar

comeoss seminar

lineoss seminar

future challenges oss

myra wielingoss seminar

short term statistics