Questionnaire design & admin

Questionnaire Design &

Administration

Lecture 6

2

Chapter 9Questionnaire design

Learning Objectives• Discuss the process of questionnaire design• Draft, pre-test and revise questionnaires• Identify the potential sources of error that can

occur when conducting a survey• Explain the issues involved in the

administration of questionnaires using different survey methods

• Discuss the ethical issues involved in collecting data from respondents

Questionnaire Design & Administration

5

A survey is only as good as the questions it asks

• Questionnaire design is one of the most critical stages in the survey research process.– Importance of question wording is easily

overlooked.• Relevance and accuracy are the two basic

criteria for questionnaire design.

Questionnaires

• Standardise data collection for survey • Ensure comparability of the data, increase

speed and accuracy of recording and facilitate data processing.

Definition• Formalised set of questions for obtaining

information from respondents

7

Questionnaire design• Planning a questionnaire’s design involves

several decisions:– What should be asked?– How should questions be phrased?– In what sequence should the questions be

arranged?– What questionnaire layout will best serve

the research objectives?– How should the questionnaire be pre–

tested? Does the questionnaire need to be revised?

8

What should be asked?

• A questionnaire is relevant if no unnecessary information is collected and only the information needed to solve the marketing problem is obtained.

• A questionnaire is accurate if the information obtained is reliable and valid.

9

How should questions be phrased?

• Open–ended response questions pose some problem and ask the respondent to answer in his or her own words.– Most beneficial when conducting

exploratory research.• Fixed–alternative questions give the

respondent limited–alternative responses and ask him or her to choose one closest to his or her viewpoint.– Takes less time and easier to answer.

Objectives of a Questionnaire

• Translate the information needed into a set of specific questions that the respondents can and will answer

• To uplift, motivate, and encourage the respondent to become involved in the interview, to cooperate, and to complete the interview

• Questionnaires should minimise response error

Questionnaire Administration• Personal interviews

– Lengthy, complex and varied

• Telephone interviews

– Short and simple, conversational style

• Mail surveys

– Simple questions, detailed instructions

Questionnaire Design• Formulate questions to obtain the required

information• Question order and wording• Sequence and layout• Pre-testing

– correct and pre-test again

Suggest questions/statement you could

ask to measure:

•Satisfaction with event/ establishment

•Motivation to participate

•Age

•Determinant attributes

Formatting the Question

• Open-response questions• Closed-response questions

Open-ended Response Questions

• Open-response (unstructured) questions• Open-ended questions that respondents

answer in their own words• Express general attitude• Interviewer bias is high• Coding is costly and time consuming

What were you expecting by attending this event?

Open-ended Response Questions cont.

• Used for:

– introduction to a survey– when responses are unforeseen or

there are too many to list– when verbatim responses are

important– when behaviour being measured is

sensitive

Closed Response Questions

• Specify the set of response alternative and response format

• Two alternative formats:– choice from a list of responses– choice on a rating scale

Closed Response Questions cont.

• Advantages– Can be administered quickly– Coding and processing is less costly and

time consuming– Easy to answer and easier for interviewer– Tabulation and analysis easier– Less potential error– Responses are directly comparable

Closed Response Questions cont.

• Limitations– Effort required in designing alternatives

(decision on type of responses)– May require exploratory research for

determining choices (difficult to develop good questions)

– Bias response because people pick alternatives offered (less ability for self-expression)

– Potential order bias

Closed Response Questions cont.• Multiple-choice questions

– Respondents are asked to select one or more alternative

Which of the following people had an influence on your choice of university?

None Parents Friends Ex-Uni student Teachers at secondary school Careers teachers at secondary school Colleagues Other please specify………………….

Closed Response Questions cont.• Scales

– Comparative and non-comparative (covered last week)

• Dichotomous questions– Only two response alternatives– Mainly used for demographic and

behavioural data

Have you holidayed in Asia in the last 3 years?

Yes No

Question Wording• Vocabulary should be simple clear and direct• Words should not be vague or ambiguous • No ‘double-barreled’ questions • No leading or loaded questions • Instructions should be clear• Respondents should only answer applicable

questions• Test several alternatives of question wording• Questions should be appropriate length• Word sensitive questions carefully

Question Wording cont.

Is the question necessary?• Every question should contribute to the

information needed or serve some specific purpose.

• Question wording should address the research objectives in order to make better decisions

Question Wording cont.Is the respondent informed?• Filter questions are often used to ascertain

familiarity, product use and past experience

– Are you the main grocery buyer in the household?

– Have you seen the Holden Cruze ad?– Have you purchased car tyres in the last 3

months?

• If respondents answer “No”, they may not be suitable respondents (i.e., they have not met the criteria for inclusion in the study)

Question Wording cont.Can the respondent remember?• Inability to remember can lead to

– Errors of omission– Telescoping– Creation

• The ability to remember events are influenced by – the event itself– time elapsed since the event– presence/absence of other events that

would help memory


Can the respondent articulate? • Respondents may be unable to describe the

situation accurately

• Inability to articulate may lead to

– respondents ignoring questions

– refusing to respond to the rest of the questionnaire


Legitimate Purpose• Unwilling to divulge information which they do

not see as serving a legitimate purpose• Specify why the question is being asked

Sequence and Layout Decisions

Opening Questions• Important in gaining confidence and co-operation

of respondents• Questions should be interesting and non-

threatening• Could be qualifying question

Sequence and Layout Decisions cont.

Type of information

Basic

Classification

Identification


Difficult questions• Sensitive, embarrassing, complex or dull

questions should be placed late in the sequence

Effect on subsequent question• Questions asked early in a sequence can

influence the responses to subsequent questions • Use funnel or flowerpot approach


Logical order• All questions that deal with a particular issue/topic

should be asked before proceeding to a new topic• Use branching questions to direct respondents to

other questions

Form and Layout

• Format, spacing and position are important • A questionnaire can be divided into several

parts• All questions should be numbered• Questionnaires are sometimes pre-coded

where each possible response to a question is associated with a unique number (or letter)

• Questionnaire should be serially numbered

What sort of strategies do you use to ensure high response rates in

surveys?

Michael Sparks, Director

Intuitive Solutions

Reproduction of the Questionnaire

• Use good quality paper and have a professional appearance

• Consider using booklet if the questionnaire runs to several pages

• Avoid splitting a question and response categories over two pages

• Use single columns for response categories• Avoid overcrowding questions• Directions or instructions for individual questions

should be placed close to the questions as possible e.g., [PROBING]

Pre-testing and Correcting Problems

• Testing the questionnaire on a small sample of respondents to identify and eliminate potential problems

• Use protocol analysis or debriefing• After each significant revision of the questionnaire,

another pre-test should be performed with a different sample

Pre-testing

• Debriefing approach– respondent completes questionnaire with

interviewer observing– interviewer then debriefs respondent

• Protocol approach– respondent asked to think aloud as they

complete questionnaire– interviewer takes notes and asks for any

further clarification at the end

Pre-testing cont.

•Specific Questions

–Variation–Meaning–Task difficulty–Respondent interest and attention

•The Questionnaire

–Sequence

–Form and layout

–Skip patterns

–Length

–Instructions

Cover Letter and Openers

• Main role : to win over respondent’s co-operation and willingness to participate

• Respondents must be aware that a bonafide survey is taking place

• Identify the sponsor

• Purpose of survey should be described clearly and simply

Cover Letter and Openers cont.

• Prospective respondents must be made aware of how and why they were selected

• Assure respondent confidentiality• Specify the completion date• Cover letter should solicit the prospect

respondent’s help in the survey• Thank you statement for willingness to

participate

Observational Forms

• Specify the who, what when, where, why and way of behaviour to be observed

• Offers consistency, structure, completeness

Administering the Questionnaire:Personal Interviews

• Identification of location/venue• Recruitment of field staff

– Develop job specifications for the project– Decide on the characteristics the fieldworkers

should possess– Recruit appropriate individuals

• Training of field staff– Conducted at central location or by mail– To ensure questionnaire is administered in the

same manner

Training of Field Staff

Making the initial contact• To encourage respondents to participate• Convince potential respondents that their

participation is important

Training of Field Staff cont.

Asking the question• Be thoroughly familiar with the questionnaire• Ask the question in the order in which they

appear• Use exact wording given in the questionnaire• Read questions slowly• Repeat questions that are not understood• Ask every applicable question• Follow instructions


Probing• Motivate respondents to enlarge on, clarify, or

explain their answers• Should not introduce any bias• How to probe??

– Repeat the question– Repeating the respondent’s reply– Using a pause or silent probe– Boost or reassure respondents– Eliciting clarifications– Using objective / neutral questions or comments


Recording the answers• Same format should be used• Structured Tick box that reflects the respondent’s

answer• Unstructured Record the responses verbatim

– Record responses during the interview– Use the respondent’s own words– Do not summarise or paraphrase the respondent’s answers– Include everything that pertains to the question objectives– Include all probes and comments– Repeat the response as it is written down

Training of Field Workers cont.

Terminating the Interview• Terminate after all information is obtained• Record spontaneous comments • Interviewer should answer the respondent’s

questions about the project• Thank respondent and express appreciation

Field Editing and Checking

Quality control and editing • Check that questions have been completed,

do not accept incomplete answers, writing is legible

• Keep record of hours worked and expenses

Field Editing and Checking cont.

Sampling Control• Ensure interviews are strictly following

sampling plan.• Keep record of:

– number of calls made, – number of not-at-homes, – number of refusals, – number of completed interviews per interviewer.

Control of cheating• Falsifying part of a question or entire

questionnaire.• Cheating can be minimised through proper

training, supervision and validation of field work.


Validation• Check for authenticity.• 10 – 25% of respondents are contacted.• Ask about length and quality of the interview,

reaction to the interviewer, and basic demographic data.


Evaluation• Provide feedback to interviewers on:

– Cost and time– Response rates– Quality of interviewing– Quality of data

Telephone Interviews

• Similar to personal surveys plus:– selecting the telephone numbers– call outcomes– the introduction– when to call – [6-8pm week days, 10-8 weekends]– Call report

Mail Surveys• Identification of respondents• Mail interview package

– Outgoing envelope– Reply paid envelope– Postage– Method of addressing– Cover letter– Questionnaire length, content, layout etc– Method of notification– Incentives– Follow up

Online Surveys

– Similar to mail surveys– Design the cover letter/email– Technology issues should be addressed

Salaries and Wages

Classification Per hour rate*

Market Research Interviewer $21.72

Door-to Door $23.72

Editor/Coder/Key Punch Operator $22.12

Field Supervisor $26.32

Research Assistant $26.32

Research Officer $29.58

Research Manager $42.20

Source: NUW and AMRO (2005) Report on the Market Research Industry Agreement 2005 - 2008

* Casual rates (Australian dollars)

Non-response Issues

• Non-response error arises when potential respondents included in the sample do not respond.

• Major problem is whether non-respondents differ from respondents in terms of demographic, psychographic, personality, attitudinal, motivational and behavioural variables.

• To reduce non-response bias: improve the response rate and/or adjust for non-response.

Improving Response Rates

Sources of Error

• Inaccuracy in response– inability to respond– unwillingness to respond

• Interviewer error– lack of interviewing skills– fraud and deceit

Unethical Practices in Data Collection

• Using of surveys as a disguise for selling purposes• Using details in survey to develop a prospect list• Disguising the purpose of the measurement• Deceiving the respondent as to the true duration of

the interview• Misrepresenting the compensation in order to obtain

co-operation• Not informing respondents of follow-up interviews• Using hidden taping without respondent’s consent• Conducting simulated product test in which identical

products are given• Not debriefing the respondent

59

Types of fixed–alternative questions

• Simple–dichotomy: choose one of two alternatives.

• Determinant–choice: choose one and only one from among several alternatives.

• Frequency–determination: asks for general frequency of occurrence.

• Checklist: choose multiple answers to a single question.

60

Phrasing questions for surveys

• Means of data collection will influence the question format and phrasing.– Questions for mail, Internet, and telephone

surveys are less complex than those used in personal interviews.

– Questionnaires for telephone and personal interviews should be written in a conversational style.

61

The art of asking questions• Some guidelines to help prevent the most

common mistakes:– Avoid complexity: use simple,

conversational language.– Avoid leading and loaded questions.– Avoid ambiguity: be as specific as possible.– Avoid double–barrelled items.– Avoid making assumptions.– Avoid burdensome questions that may tax

the respondent’s memory.

62

What is the best question sequence?

• Order of questions serve several functions.– Opening questions influence respondents’

cooperation and involvement.– Demographic questions that may

embarrass or threaten respondents should be asked at the end of the questionnaire.

• Order bias may distort survey results.– Funnel technique: asking general

questions before specific questions.

63

What is the best question sequence?

• A filter question screens out respondents who are not qualified to answer a second question.– A pivot question is used to determine which

version of a second question will be asked.

64

What is the best layout?• Layout should be neat, attractive and

interviewer instructions easy to follow.– Decent margins, white space to separate

blocks of print, etc.,• Should appear as short as possible.

– Multiple–grid question• Layout is also important for Internet

questionnaires.– Discrepancies in screen configuration,

push button, status bar, radio button, etc.,

65

What is the best layout?

66

What is the best layout?

67

How much pre–testing and revising are necessary?

• Questionnaire is usually tried out on a group that is similar in make–up to the ultimate sample.

• A preliminary tabulation of pre–test results helps determine whether the questionnaire will meet the objectives of the research.

• Pre–tests provide means for the testing sampling procedure.

68

How much pre–testing and revising are necessary?

• Pre–tests typically are conducted to answer the following questions about the questionnaire:– Can the format be followed by interviewer?– Does it flow naturally and conversationally?– Are the questions clear and easy to

understand?– Can respondents answer the questions

clearly?– Which alternative forms of questions work

best?

69

Designing questionnaires for global markets

• Researchers must take cultural factors into account when designing questionnaires.

• The most common problem involves translating a questionnaire into other languages.– International marketing researchers often

have questionnaires back translated.

Questionnaire design & admin

Business