Questionnaire Design - iea.org · Survey Steps Statement of Objectives Survey frame Sample design Questionnaire design Data collection Processing Data analysis Documentation/evaluation

Questionnaire Design

Mike Scrim

IEA Workshop, Indonesia

October 2013

Ask Yourself!

What is the problem?

What do I need to know?

How will the information be used?

How accurate & timely does the information have to be?

Means of Acquiring Information

Existing information – Administrative records

– Previous studies

– Government agencies

– Libraries

– Universities

Observation

Questioning

Survey Steps

Statement of Objectives

Survey frame

Sample design

Questionnaire design

Data collection

Processing

Data analysis

Documentation/evaluation

Data dissemination

Statement of Objectives

Identify Objectives and Information Needs

State the problem

Frame the problem

Concepts of interest

Items to be measured

Level of detail

1. State the problem

Purpose/objective Uses Users

2. Frame the problem

Who, what, when, where

Questionnaire Design

Definition of a Questionnaire

A questionnaire is a group or sequence of questions designed to obtain information on a subject from a respondent

Questionnaire Development

Identify objectives and information needs

Consultation

Review previous questionnaires

Draft questions

Review questionnaire/revise

Test questionnaire/revise

Finalize questionnaire

Previous Questionnaires

Examine questions used in similar surveys

Same questions may be used

– Useful for data comparability over time

– Can ensure that questions have worked in the past

Questionnnaire Testing

Are questions clear and easy to answer?

Does the question order affect responses?

Are instructions clear?

How do respondents feel about look/format of the questionnaire?

Methods of Data Collection

Why is data collection important?

High cost

Requires extensive human & physical resources

Time-consuming

Direct impact on data quality

Image of the survey organization

Methods of Data Collection

Face-to-face (personal) interview

Telephone interview

Self-completed

Face to face Telephone Self

Complete

Cost High Medium Low

Time Medium Fast Slow

Response High Medium –

High

Low (very)

Comparison of Data Collection Methods

Considerations in Questionnaire Design and Wording

Terminology

Frame of reference

Specificity

Applicability

Double-barreled

Wording

Considerations in Questionnaire Design and Wording

Are words simple, direct and familiar to all respondents?

Are the questions clear and specific as possible?

Are the questions applicable to all respondents?

Are the questions one dimensional?

Are the response categories mutually exclusive and exhaustive?

Are any questions leading or loaded?

Will any responses be influenced by response styles?

Do the questions read well?

Response Errors

Difference between the true answer to

question and respondent’s answer to it

Can occur anywhere during

– Questioning

– Answering

– Recording

Two types of response error

– Random errors

– Biases

Sources of Response Error

Questionnaire

– Wording

– Question order

– Complexity

– Length

Sources of Response Error

Respondent

– Understanding

– Recall

– Judgement

– Motivation

– Reporting