Survey Methodology EPID 626 Survey Instruments Manya Magnus, Ph.D. Fall 2001.

Survey MethodologyEPID 626

Survey InstrumentsManya Magnus, Ph.D.

Fall 2001

Lecture Outline

• Getting started• Types of questions• Group exercise

What is a survey instrument?

• For interviews: A script for interviewers, including introductions, instructions, and questions

• For self-administered surveys: A questionnaire

What is the worst thing you can do?

• Jump in without thinking about purpose of study, objectives, analysis plan!

• Think first, start slowly!

Essential steps in developing a survey instrument

• Statement of purposes– What do you want to accomplish with the

survey? Research questions, objectives

• List of the variables to be measured– Group into logical categories

• Draft analysis plan– What are your dependent, independent

variables? What are potential confounders?

Design/variable grid

Research question

Dependent variables

Independent variables

Potential confounders

Are clients satisfied with ancillary services?

Satisfaction with individual services (yes/no)

Gender

Race

Ethnicity

Age

Disease status

Pregnancy

Children

Need for services at baseline

CD4 count

Viral load

Medication status

Satisfaction with individual services (scale)

Etc.. Etc..

Preliminary question design steps

• Interdisciplinary research group– What are the research questions/

objectives/hypotheses?

• Focus groups

• Draft questions• Cognitive laboratory interviews

• Pilot/pre-test

Focus groups (1)

• Objective: To compare the reality about which respondents will be answering questions with the abstract concepts embedded in the study objectives

• Usually 6-8 people per group; have a couple of groups

Focus groups (2)

• Try to get participants who are similar to your survey population

• Talk about perceptions, experiences, and feelings related to survey subject

• Get people who are experts in focus group leadership

Drafting questions

• Look for standard questions from other surveys– Benefit from their experience– Enhance comparability of your survey with

previous surveys

• Write good questions

• Don’t remake the wheel!

Cognitive laboratory interviews

• Respondents are brought into a laboratory setting– May be videotaped– Interviews conducted by cognitive

psychologist or experienced investigator

• Goal: To get information about how the respondent understood the questions and about the way they answered them

Layout of survey instrument

• Make it clear and simple

• Usually begin with straightforward questions, put complicated or sensitive questions in the middle or at the end

Interviews

• Should have everything scripted, including introductions, instructions, transitions, definitions, and explanations

• Be very explicit about skip instructions

• Differentiate instructions and optional wording from questions– Usually done with all caps and

parentheses, respectively

Example

Interviewer instructions: … read aloud all sentences that are in bold. Italics indicate instructions for interviewer that are not to be read aloud…

Have you been pregnant in the last three (3) months?

� 1—yes [if yes, skip to question 4]� 2—no [if no, skip to question 5]

Self-administered

• Questionnaire should be self-explanatory (minimal instructions needed)

• Limit to closed questions

• Use short questions with consistent formats

• Minimize skips; make them very clear

Example

Instructions: Please make a check () in the box which best reflects your answer.

Have you been pregnant in the last three (3) months? � 1—yes Did you take a

prenatal vitamin? � 1—yes � 2—no

� 2—no Did you take a multi-vitamin? � 1—yes

� 2—no

*Note problems with these questions!

Field pre-testing

• Goal: Find out how well the data collection protocols and survey instruments work under realistic conditions

Interviews

• Usually interview 20-50 respondents who are similar to your survey population

• Interviewers note ways to improve survey instruments

Quantitative methods for pre-testing (1)

• Ask interviewers to fill out a rating for each question– Is it easy to read as worded?– Do respondents understand the question in

a consistent way?– Can respondents answer the question

accurately?

• Note problems possible in interpreting if that survey isn’t good…

• Taping and behavior coding– Trained coders review the recordings and

evaluate the question and answer process in a consistent way

• Do the interviewers read the question as worded?

• Do the respondents ask for clarification?• Do the respondents initially give an inadequate

answer that requires interviewer probing?

Quantitative methods for pre-testing (2)

Self-administered

• Have respondents (who are similar to your survey population) fill out the survey and then discuss– Were the instructions clear?– Were the questions clear?– Were there any problems understanding

what kind of answers were expected?

Validity and reliability

• If you are developing this tool, find ways to undertake testing of validity and reliability (crudesophisticated)

• Note difficulties with this, and benefits to using already-validated tool

Open-ended questions• Short, specific

– What is your current age?

• Long, narrative– Why did you choose to come to this clinic?

• Problems:– Illegible handwriting– Inappropriate detail

• Usually avoid-the rest of the lecture is devoted to closed-ended questions

Yes/No and checklist questions

• Yes/No

• Checklist– From a list of alternatives, check those that

apply– Problematic because you can’t distinguish

a “No” from a skip– Yes/No may be better because it forces

thought

Multiple-choice questions• Response alternatives should be mutually

exclusive and exhaustive• Think about whether you want respondents to

just check one response or to be able to check multiple responses– Multiple responses may complicate analysis– Think about presenting alternatives as a checklist

Multiple-choice questions (2)

• Include “Other” as an alternative?– What does it tell you?

• About the respondent?• About your question?

– Use open-ended “Specify”________– May be useful during pretesting or if you

are planning on revising the survey instrument

Semantic differential questions

• Two opposite adjectives at the ends

Best Worst

1 2 3 4 5 6 7

• Sometimes ask the question for two scenarios, ex: current situation and ideal situation, and look at the differential

Ranking questions

• Present alternatives and ask respondents to rank them– Ex: rank from the most important to the

least important

• Think about whether you want to allow tie rankings– How will that affect your analysis?

Two common formats

1

__ Eating fruit

__ Exercising

__ Meditating

__ Being happy

__ Sleeping enough

2

Eating fruit 1 2 3 4 5

Exercising 1 2 3 4 5

Meditating 1 2 3 4 5

Being happy 1 2 3 4 5

Sleep enough 1 2 3 4 5

Number 2 is usually preferable. Why?

Rating Scales• Present a respondent with a question or

statement and a range of responses

• Ex. How would you rate your relationship with your physician?

Excellent Very good Good Fair Poor

Rating scales (2)

• Likert scale:

Ex. Patients should have the right to sue Health Maintenance Organizations

Strongly Agree Unsure Disagree Strongly agree disagree

Rating scales (3)

• Psychological distance– Distance between alternatives should be

equal

• Number of response alternatives– Usually 3 to 7 are recommended– What might affect your choice?– Think about your research question– Think about the complexity of the issue

Rating scales (4)

• Order of response alternatives– Should be monotonically increasing or

decreasing– Should all be ordered in a column or row– Within a survey, may want to mix up

increasing and decreasing by section, but never within a section

Rating scales (5)

• Unipolar response alternatives– Range from “nothing” to “a great deal”

Excellent Very good Good Fair Poor

• Bipolar response alternatives– Range from “large negative” through “zero”

to “large positive”Strongly Disagree Unsure Agree Strongly

disagree agree

Rating scales (6)

• Odd or even number of alternatives?– Odd numbers create a midpoint

• This midpoint should be a neutral response• Many respondents really like midpoints and will

use them a lot!!!

– Even numbers force people to make a decision

– Depends on your research question

Rating scales (7)

– For very complex, emotional issues, you may want to have 2 middle points

• Ex. Slightly agree and Slightly disagree

• Balanced scales– Should have equal numbers on either side

of neutral – Unbalanced scales will lead to bias

Rating scales (8)

• “Don’t know” as an alternative– Usually necessary for knowledge questions– For attitude questions, “don’t know” usually

means “unsure”. Adding more middle categories may be a better solution

– Sometimes use a screen question then initiate a skip sequence

Rating scales (9)

• Example of a screening question:1. Are you familiar with the proposed legislation

regarding universal healthcare?

(circle one)

Yes Go to question 2.

No Go to question 17.

Rating scales (10)

• Behaviorally anchored scales– Objective, quantitative– Compare to subjective scales

Ex: How often do you exercise in a week?

• Subjective: Often Sometimes Never

• Behaviorally anchored: 5 times 3-4 times 1-2 times Never

• Which one to use?– Depends on your research question

General Recommendations• Write brief questions• Write clear questions

– Define ambiguous terms– Avoid jargon– Avoid double negatives– Avoid unclear pronouns– For open-ended questions, avoid adverbial

constructions such as how, why, when, where

General Recommendations (2)

• Be careful with:– Intentions– Hypotheticals– Assumptions: commonplace is not

universal

• Write unidimensional questions • Write mutually exclusive and exhaustive

response alternatives

General Recommendations (3)

• Generally avoid loaded questions– Special case for questions that may involve

social desirability– When might we want to ask loaded

questions?

• “Most people have times when they drink too

much and feel tipsy. How often has this happened to you in the last month?”

• “Most people feel that smoking marijuana is harmful. How do you feel?”

• “Marijuana has been shown to be an effective treatment for people with some symptoms of AIDS. How do you feel about legalizing marijuana?”