Chapter13

SURVEYS

© LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON

STRUCTURE OF THE CHAPTER

• What is a survey?• Some preliminary considerations• Planning a survey• Low response and non-response, and how to

reduce them• Survey sampling• Longitudinal, cross-sectional and trend studies• Strengths and weaknesses of longitudinal,

cohort and cross-sectional studies• Postal, interview and telephone surveys• Internet-based surveys• Comparing methods of data collection in surveys

KEY FEATURES OF SURVEYS• Gather data on a one-shot basis;• Represent a wide target population• Generate numerical, often large scale data;• Provide descriptive, inferential and explanatory data;• Manipulate key factors and variables;• Gather standardized information;• Ascertain correlations;• Remove contextual data;• Gather data from multiple choice, closed questions, test

scores or observation schedules;• Support or refute hypotheses about the target population;• They are piloted;• Make generalizations about, and observe patterns of

response in, the targets of focus;• Gather data which can be processed statistically.

SAME SAMPLE / COHORT OVER

TIME

TYPES OF SURVEY

LONGITUDINAL

CROSS-SECTIONAL

TREND/PREDICTION

STUDIES

SNAPSHOTS OF DIFFERENT

SAMPLES AT ONE OR MORE TIMES

SELECTED FACTORS

CONTINUOUSLY OVER TIME

PLANNING A SURVEY• Specify the exact purpose of the enquiry:

– Primary topics– Subsidiary topics– Specific information requirements

• Specify the population on which the survey is to focus;

• Specify the resources that are available:– Human (e.g. for administering and processing

survey)– Material– Financial– Administrative – Temporal– Geographical– Software

PLANNING A SURVEY• Data collection

– Researcher-administered– Self-administered– Postal survey– Telephone survey– Internet survey– Face-to-face interviews– Email

• Self-reporting– May lead to biased reporting

PLANNING A SURVEY• Problem definition• Sample selection • Design of measurements • Concern for participants

STAGES IN SURVEY DESIGN

DEFINE OBJECTIVES

DECIDE INFORMATION

SAMPLING INSTRUMENTATION

PILOT

DATA COLLECTION

DATA ANALYSIS

REPORTING

TRAINING

STAGES IN CONDUCTING A SURVEY1. Define the objectives;2. Decide the kind of survey required;3. Formulate research questions or hypotheses;4. Decide the issues on which to focus;5. Decide the information that is needed to address the

issues;6. Decide the sampling required;7. Decide the instrumentation and the metrics required;8. Generate the data collection instruments;9. Decide how the data will be collected;10. Pilot the instruments and refine them;11. Train the interviewers (if appropriate);12. Collect the data13. Analyze the data;14. Report the results.

PROBLEMS IN SURVEYS

• Poor sampling• Poor question design and wording (failure to

operationalize)• Incorrect or biased responses • Low response or non-response

HOW TO IMPROVE RESPONSE RATES• Identify where the problem lies:

– The survey never reached the intended people;– People refuse to answer ;– People may not be available;– People may not be able to answer the questions;– People may not actually have the information

requested;– People may overlook some items in error;– The survey was completed and posted but failed to

return.– The pressure of competing activities on the time of

the respondent;– Potential embarrassment at their own ignorance if

respondents feel unable to answer a question;

HOW TO IMPROVE RESPONSE RATES– Ignorance of the topic/no background in the topic;– Dislike of the contents or subject matter of the

interview;– Fear of possible consequences of the survey to

himself/herself or others;– Lack of clarity in the instructions;– Fear or dislike of being interviewed;– Sensitivity of the topic, or potentially insulting or

threatening topic;– Betrayal of confidences;– Losing the return envelope or return address;– The wrong person may open the mail, and fail to

pass it on to the most appropriate person.

HOW TO IMPROVE RESPONSE RATES• Ensure that the topic is interesting and motivating;• Ensure that it is easy to complete and layout is attractive;• Include a covering letter that explains the survey;• Choose a good time to conduct the survey;• Follow-ups and polite reminders;• Pre-paid return stamped addressed envelopes;• Institutional affiliation, survey sponsorship or support from

a high status agent;• Financial incentives and rewards for return;• Making instructions about responses and return very

clear;• Avoid open-ended questions unless these are really

important;• Avoid placing open-ended questions at the start of a

questionnaire;

HOW TO IMPROVE RESPONSE RATES• Have easy-to-follow instructions and spacing of

the text;• Flatter the participants without being seen to

flatter them;• Providing information about the research through

a covering letter and/or advance notification;• Making the survey look very unlike junk mail;• Consider asking the respondents for an interview

to complete the survey questionnaire;• Deliver the questionnaire personally rather than

through mail;• Ensure that the questions or items are non-

judgemental;• Assure confidentiality and anonymity.

HOW TO IMPROVE RESPONSE RATES• Follow a sequence:

– Send non-respondents a friendly reminder after ten days, stressing the importance of the research;

– Send a further friendly reminder ten days after the initial reminder, stressing the importance of the research;

– Make a telephone call to the respondents shortly after the second reminder, indicating the importance of the research.

CONSIDERATIONS IN SAMPLING• A probability and non-probability sample;• Confidence level and interval;• The desire to generalize, and to whom;• The sampling frame;• The sample size;• The representativeness of the sample;• Access to the sample;• Anticipated response rate.

LONGITUDINAL DATA ARE AFFECTED BY . . .

• History (events occurring may change the observations of a group under study);

• Maturation (participants mature at different speeds and in different ways);

• Testing (test sensitization may occur – participants learn from exposure to repeated testing/interviews);

• The timing of cause and effect: some causes may produce virtually instantaneous effects and others may take a long time for the effects to show;

• The direction of causality not always being clear or singular.

• Comparability of data over time;• Attrition (dropout);• Respondents’ memory and recall capabilities.

MEMORY IS AFFECTED BY . . . • The time that has elapsed since the event took

place;• The significance of the event for the participant;• The amount of information required for the study

– the greater the amount, the harder it is to provide;

• The contamination/interference effect of other memories of a similar event (i.e. the inability to separate similar events);

• The emotional content or the social desirability of the content;

• The psychological condition of the participant at interview.

• Hindsight.

ADVANTAGES OF POSTAL SURVEYS

• Reach many people• Comparatively cheap• Can be completed at respondents’ preferred

time• No risk of interviewer bias• Can reach scattered populations• Can gather sensitive data (as nobody else

present)

DISADVANTAGES OF POSTAL SURVEYS

• Low response rate• No check on understanding• No check whether all items are completed• Need to have a very simple format• Presentation matters affect completion

ADVANTAGES OF INTERVIEW SURVEYS

• Improves response rates• Can clarify queries from respondents • Can stimulate the respondent to give full

answers• Can be flexible (e.g. in item sequence)• Benefit from non-verbal communication• Can build in trust and rapport• Ensures that only the respondent answers

the questions

DISADVANTAGES OF INTERVIEW SURVEYS

• Interviewer characteristics affect responses• Conduct of interview affects responses• Flexibility can reduce standardization• Costly in time, travel and training

ADVANTAGES OF TELEPHONE SURVEYS

• Reduce bias in the researcher or the interviewee• Reduce costs of time and travel• Easy to find more people to contact

DISADVANTAGES OF TELEPHONE SURVEYS

• Not everyone has a telephone• Not everyone is available for interview• People are ex-directory• The person answering the call may not be the

most suitable person• Multiple choice and rating scale questions are

difficult• Order effects can be strong• People may lie or hang up if questions are

unwelcome, sensitive, too long, too many• Lack of non-verbal cues

ADVANTAGES OF INTERNET-BASED SURVEYS

• Many people can be reached• Cheap to administer• Quick to process results• Can reach difficult populations• Anonymity and non-traceability• Novelty value• Respondents can complete the questionnaire from

home;• Respondents can complete it at a time to suit

themselves;• Respondents can complete the survey over time;• Complex skip-patterns can be created and

organized by the computer, so that participants do not have to understand complicated instructions;

ADVANTAGES OF INTERNET-BASED SURVEYS

• The software can prompt respondents to complete missed items or to correct errors;

• Computer can check incomplete or inconsistent replies;

• For each screen, the computer can provide an on-screen indication of how much of the questionnaire has been completed;

• Reduction of researcher effects;• Human error is reduced in entering and processing

data;• Additional features may make the survey attractive;• Greater generalizability may be obtained as Internet

users come from a wide and diverse population;• Greater authenticity of responses may be obtained.

DISADVANTAGES OF INTERNET-BASED SURVEYS

• Not everyone has internet access• Limited control over sampling• Ethics (e.g. traceability)• Hardware and software problems• Expertise of respondents• Visual presentation takes on added signficance• Limited number of lines per screen• People give a minimal response• Order effects (early responses affect later ones)• People stop if it is too long or complicated