Designing and Conducting Business Surveys for Official ... · Designing and Conducting Business Surveys for Official Statistics, Tbilisi, Georgia, November 2016 Business Survey Planning

Designing and Conducting Business Surveys for Official Statistics, Tbilisi, Georgia, November 2016

Business Survey Planning

Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016 1

Remember this: Process map: Business Survey Life Cycle

1. Specifying Target population

Sample frame (obs)

Sample

Indicators/q’stions

Questionnaire Data collection mode

Fieldwork

Survey communication

Analysis & estimation

Dissemination/Archive

Evaluation (PDCA)

Required output

Data cleaning

2. Designing Building, Testing survey components

3. Collection

4. Processing

5. Analysis, dissemination, archiving

6. Evaluation

Information needs

Reporting unit(s) Response

Testing

Constraints: org.&bus.



Sample frame (obs)

Sample



Fieldwork




Evaluation (PDCA)

Required output

Data cleaning


3. Collection

4. Processing


6. Evaluation

Information needs


Testing



Suppose, within your office, you are commissioned to do a business survey. Data are needed on a new topic, and those data are not available in secondary sources (like registers), so you need to design and conduct a business survey: What is the actual first thing you will do?

Remember this: Process map: Business Survey Life Cycle

Question:

Key Statement: survey planning

• The survey data are not achieved by itself

• All planning activities are aimed at

producing a pre-agreed level of

quality, at minimum costs and

response burden

• Plan for quality within constraints:

Data quality is not achieved by itself

Quality management

• Identify risks: what can go wrong?

Risk management


A Survey Project Plan is one of the survey components!

Overview & Take aways

• Planning a survey – basics: - What is a project?

- Project management basics

- Source of project failure

• Planning a business surveys within constraints: - Constraints to consider, specific for business surveys

- the planning process

• An example

• Planning and progress tool

Take aways:

• Issues to take into account when planning a business survey

• Detailed planning issues will be discussed with the various

survey components


What is a project?

• A project is a goal-oriented set of activities aimed at

delivering a pre-agreed, concrete and clearly specified

result

• Which has to be realized within constraints: money, time,

resources

• Which is produced by a temporary organization unit,

lead by a project manager

• While facing high levels of uncertainty

• A project has a clear stakeholder/client


Clear beginning and ending

In contrast to an ongoing process, but also applies to recurring surveys (tile structure of linked projects)

Examples of business survey projects

• (Re)designing and conducting a business survey

• (Re)designing a questionnaire

• Introducing a new mode of data collection

• (Re)designing a business survey communication strategy

• (Re)designing a new data collection system


Project management basics (§4.2)

1. Start with the planning the survey as a project, then

comes the “doing” • Plan people, resources, activities, and money in time, to make

sure that all survey components are ready in time, for the desired levels of data quality (e.g. response rates)

• Conduct risk assessment: an important part of the production process is outside the control of a project manager, and put in place mitigating plans and actions

• Deal with the project environment: balance between specific needs, established standards and production constraints

• Make expectations clear from the beginning


Relevant for

planning business surveys

Project management basics (§4.2)

2. Manage the project according to the plan: after the

planning comes the doing

3. Control and monitor the survey project

4. Communicate project progress

5. Improve the quality of processes:

apply the PDCA-cycle: Plan – Do – Evaluate – Adjust


The Project Manager role (§4.2)

• Make things happen: plan, start, and manage the project

• Control and monitor the survey project

• Focus on management tasks

This should be a separate role!


Three sources of project failure:

The project management principles are aimed at minimizing

project failure, caused by:

1. Organizational factors - Determine internal constraints - Conduct risk assessment and put in place mitigating plans and actions

2. Poorly identified customer needs and inadequately

specified project requirements

- Steps in the planning process

3. Poor planning and control

- SMART planning - Using paradata (process data) to monitor

progress and achieved levels of quality (using indicators)


“Quality is established through processes,

systems and procedures” (Biemer & Lyberg, 2003)

So, what does planning a survey involve?


Sample frame (obs)

Sample



Fieldwork




Evaluation (PDCA)

Required output

Data cleaning


3. Collection

4. Processing


6. Evaluation

Information needs


Testing


1. Project plan


Planning the survey, includes:

1. Specifying the survey

2. Planning the survey process in detail

3. Specifying required resources: money, people, materials

4. Specifying the timetable

5. Project management issues

6. Risk management issues


Structure of project plan

Step 1: Specifying the survey (§4.3.1)


Step 1: Specifying the survey


Sample frame (obs)

Sample



Fieldwork




Evaluation (PDCA)

Required output

Data cleaning


3. Collection

4. Processing


6. Evaluation

Information needs


Testing



Step 1: Specifying the survey (§4.3.1)

• Consult with stakeholder, determine: - Survey objectives, target population and information needs

- Deliverables and desired levels of quality (e.g. response rates)

- Budget, timeframe, and response burden

- Survey specifications

Signed tender

• Consult with survey organization: - Determine constraints

Establish business case: pre-plan go/no-go?

• Appoint project manager


Constraints to take into account

• Survey deliverables: the quality of the results, survey

specifications, delivery date Check if this can be achieved

• People, material resources, money, time Apply project

management principles

• Quality considerations: effects of methodologies on data

quality Have the quality framework in mind

• The business context: response process and response

burden Study the process that is outside of control and

reduce risks upfront

• Survey production environment Check and consider

upfront


Especially relevant when planning

business surveys

Study the business context

What do want to know upfront about the businesses you are

going to study?

• What data are available Questionnaire

• What terminology and definitions are used

• Where are the data (one/many departments)

• Format of the data (spread sheet, other format) & Mode

• Whom to contact Communication strategy

• Arguments for getting survey participation

• Other surveys they get (multi-surveyed businesses)

• When to inform businesses about the survey, so they have enough time to get prepared

• When are the data available Fielding survey & Frequency

Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016 18 Remember: business survey response process

Input for:

Study the business context

Example:

Redesign of Survey on Financial Position of businesses

• 2013-2014: new data model was developed, based on data

requirements

• End of Q1 in 2017: Fielding of new survey

• At the end of 2014: Planning of survey was discussed

- What is needed to reach this deadline?

- Including a feasibility study (business context)


This has to be planned in and carried out at an early stage, in order to reduce risks. Otherwise risks will be pushed downstream.

The survey production environment

(§4.3.2.2)

If working in a survey or statistical organization,

you will need to consider the survey production

environment that you are working in.



Constraints

Elements of a business

survey production

environment:

• These constraints have

to be checked upfront

• They are input for

(planning) the

survey process

• See pp. 139-140





This is the picture for one survey! However:

for business surveys, the designs are not independent of each other, but related!

Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016

The survey production environment

Investigate what other surveys your

businesses get and how these designs are

related:

• Data asked for (overlap in data requests)

• Timing of surveys

• Response Burden (workload) of surveys

26

Take into account that businesses may be multi-surveyed!



Plan for quality within constraints

Identify:

• Optimal survey design

• Trade-offs between quality and constraints (costs and

response burden):

- How do constraints affect the survey design?

- The consequences of these trade-offs regarding quality

• Decisions that need to be taken, and identify the risks for

the various options,

document

these decisions


Step 2: Plan the survey in detail (§4.3.2)


Step 2: Plan the survey in detail

• Plan all stages of the survey: Plan the entire survey process up to and including survey

deliverables and survey evaluation, including how it will be

monitored and managed.

• Write the project plan


Step 2: Project plan (Fig. 4.3)

Results of step 2: project plan, with the following sections: 1. Statement of the survey

- Objectives, deliverables, specifications, constraints

2. The survey in detail

- Work breakdown structure of activities

3. Required resources

- needed human and material resources,

money based on work breakdown structure

4. The timetable

- who is doing what and when core of the plan

5. Project management

- how the survey will be planned, managed, controlled and monitored

6. Risk management


Step 2: Plan the survey in detail (Fig. 4.2)

What the

business survey

project plan

needs to cover


Planned in step 2 (discussed in next sessions)

Desired quality Business context Response burden

Step 2: planning the survey


Sample frame (obs)

Sample



Fieldwork




Evaluation (PDCA)

Required output

Data cleaning


3. Collection

4. Processing


6. Evaluation

Information needs


Testing


1. Project plan


Step 2: Planning the survey in detail

Combine the planning for all survey components and stages:

1. List and number all activities in a work breakdown structure

table with activities

start with last activity: delivery of survey results, work backward

2. Determine critical path: no delays afforded

questionnaire design: start ASAP!

3. Prepare a timetable with milestones:

final delivery date of survey results

start and end of fieldwork

start and end of development of survey components

4. Plan the survey on a real-life calendar

5. Communicate the plan to those involved and revise, get sign-off

Remember: the plan is likely to change!


Three rules of thumb

• Do things right from the start, otherwise you are pushing

problems and costs downstream!

• Do not underestimate the time needed for DBT activities:

multiply estimated time by 1.5 or 2

• Plan for quality within constraints: quality is not achieved by

itself use the total survey error framework


Survey planning and Total Survey Framework


To conclude: planning a survey is …

• a complex process: planning and developments efforts rise

with survey complexity

• not a linear process: it is an iterative process, going back and

forth, making amendments to the plan. The project plan is

very likely going to change

• It requires communication skills, ingenuity, and creativity


After the planning comes the “doing”

Finalizing the project plan:

• Discuss with everybody involved

• Get commitment and approval

Getting started:

• Kick-off meeting

• The project is running

• Everybody is expected to do their jobs according to

the plan


Planning and progress tool (Fig.4.7)

To summarize:

A tool to assist the planning process and monitor progress

Progress Status Overview:

• Work breakdown structure for a business survey:

- are all activities planned?

• Overview of milestones:

- to monitor the survey progress: to be ticked off during the

various survey stages


Planning and progress tool (Fig.4.7)


Designing and Conducting Business Surveys for Official ... · Designing and Conducting Business Surveys for Official Statistics, Tbilisi, Georgia, November 2016 Business Survey Planning

Documents