February 11, 2015 Survey Design and Administration Catherine R. Messina PhD.

February 11, 2015

Survey Design and Administration

Catherine R. Messina PhD

“The scientist is not a person who gives the right answers, he's one who asks the right questions.” ― Claude Lévi-Strauss (1908-2009), French anthropologist

No matter how good your study design, your sampling strategy, your response/participation rate, etc …………

The quality of your measurement instruments will directly impact your ability to draw meaningful conclusions about relationships studied

What is a survey or questionnaire?

…….. a systematic method for collecting information from a sample of individuals for the purpose of quantitatively describing the attributes of the larger population of which the sample are members.

Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau, 2004.

Benefits of surveysCan reach a large number of people relatively easilyProvide quantifiable answersRelatively easy to analyse

Drawbacks of surveysProvides only limited insight into problem

Information obtained is limited by questions asked Always possible that wrong questions are asked

Varying response – respondent may misunderstand or misinterpret questionsNeed to get it right first time

Can’t chase after missing data – or go back and try again

Making inferences from survey data

Respondent answers question(s)

You infer….

Characteristic(s) (attribute(s)) of Respondent

Characteristic(s) (attribute(s)) of Sample

You infer….

Characteristic(s) (attribute(s)) of Population

Measurement is a systematic and repeatable process that quantifies or classifies events or characteristics

Measurement tools (i.e., surveys/questionnaires) need to yield measurements that are reproducible (reliable) and accurate (valid)!

Reliability- the property of reproducability does the measure consistently yield a similar

finding? Validity- does the measure really assess (i.e.

measure) the concept/object that it is intended to? How accurate is the measure?


Measurement is represented by – 3 related concepts: observed value (aka the “measured value”) true value and measurement error

Observed value = true value + measurement error

Well designed measurement tools (i.e., surveys) minimize measurement error so that the observed value is very close to the true value


Creating a survey or questionnaire First, determine clearly what it is you want to

measure in order to answer your research question(s) or hypotheses Be very clear about what you want to know

Decisions about relevant variables are based on the research question (or hypothesis) Research question / hypotheses should be

stated in measureable terms

Creating a survey or questionnaire List all relevant variables!

Dependent variables Independent variables

Covariates - A secondary variable (and of interest to the researcher) that is observed and which can affect the relationship between the IVs and the DV E.g. age can effect the relationship between

CRC-screening reminders and the likelihood of having a colonoscopy

Identifying Existing Measures

Next, determine if questions that you want to include have already been developed (and tested) by other researchers

Review the literature!!!!… you do not have to reinvent the wheel!

Saves development time and can facilitate comparisons with other studies

Identifying Existing Measures

Use standardized measures as available Review the literature for standardized measures Look for validity and reliability information,

appropriate to your sample Usually not a good idea to “adapt” or change – but if

you do, describe this is your Methods section and note this as a potential limitation in Discussion section

Be cautious if using only a subset of questions from an existing survey as this may change the meaning of the summary scores. Check whether a shorter version of the instrument exists that has also been validated

Re: primary research articles – if questions not included look for on-line supplemental info

What if there are no existing measures on my topic, and I need to develop my own survey or questionnaire?

Developing Surveys – General Approach Determine the mode of survey administration:

self-administered or interviewer adminsitered Each has its own advantages and

disadvantages …… and will determine your survey format

Types of surveys

Self-administered•Hand out•Mailed•Email / web-based

Interviewer administered•Face to face•telephone

Self-administered surveys Advantages:

Relatively inexpensive and easy to administer Preserves confidentiality (can be anonymous) Can be completed at respondent's convenience No influence by interviewer

Disadvantages: Low response rate – email surveys sift to bottem of

inbox very quickly Questions can be misunderstood No control by interviewer

Interviewer-administered surveys Advantages:

Facilitates participation by people with low literacy Interviewer can clarify ambiguity

Disadvantages: May introduce interviewer bias Needs more resources – more expensive Short surveys - especially on telephone Difficult for sensitive issues

Developing Surveys – General Approach Create a pool of new questions or revise existing

questions – write more questions than will be included in the final draft

Put questions in sequence and format survey draft This includes all instructions for filling out survey

and possible skip patterns! Include appropriate white space to get a good

estimate of the length

Developing Surveys – General Approach Decided on operational definitions

How the researcher chooses to measure a particular variable E.g.: age – can be operationally defined as”

DOBYearsAge group

Use measure that offers most flexibility

Developing Surveys

If possible, ask experts to review for content Content validity: do the items in a questionnaire

adequately represent the universe of items relating to a specific construct? How accurately does a measurement instrument tap into the various aspects of a specific construct?

Developing Surveys Get feedback (e.g., advisor; peers) on your 1st draft Face validity: is the measurement logical on the

“face of it”? The degree to which the purpose of a measurement instrument is obvious to those using it or responding to it, i.e., “looks like” or appears to respondents as a measure of depression. Advantage of good face validity: respondent really

understands the point of the questions and interprets them accurately

Disadvantage of good face validity: respondent really understands the point of the questions and “tailors” their responses to “fake good” or “fake bad”

Developing Surveys Get feedback (e.g., advisor; peers) on your 1st draft –

con’t Which questions measure IV(s)? Which questions measure DV(s)? Which questions measure covariate(s)? Are questions clear and “answerable”? Are any questions potentially offensive? Does format make sense? Is it logical? Length of survey?

REVISE!! Get more feedback (e.g., advisor; peers) on your

revised draft

Developing Surveys Pilot-test on a small sample of respondents who are

similar to your study sample Are questions clear and “answerable”? Are any questions potentially offensive? Questions should be

culturally sensistive!! Does format make sense? Is it logical? Length of survey and how long to complete? (consider

participant burden) Cognitive interviewing and “think aloud”

Ask a few people in pilot sample to read the questions out loud to you and then explain to you, what each question means to him/her and how he/she thought about their answer

Test-drive your coding system – prelim analyses REVISE!

Developing Questions If possible, pilot-test the near final draft of the survey

with new pilot sample

Pilot-test all procedures and logistics for administering survey Self or interviewer administered? Who will administer? Who is responsible for collecting completed

surveys? Where will these be stored before pick-up?

Developing Surveys – Specifics

Your goal is to write survey questions that •Respondents will interpret the same way (that you interpret them) •Respondents are able to respond accurately •Respondents are willing to answer

Only include questions you plan to use Too many questions (especially if they seem

unrelated to the purpose of your survey) can irritate respondents (an IRB issue too)

Avoid lengthy or complex questions – keep it simple Write at a low reading level – understand the

literacy level of your respondents Avoid jargon and acronyms

Use words relevant to respondents - terms and concepts should be familiar to respondents and easy for them to understand

Writing good survey questions

Instructions need to be placed exactly where needed - not at the beginning of the entire survey.

Use italics or bolding or underline to emphasize instructions and directions


Use standard demographic questions Facilitates comparing your sample to others in the

literature Examples from national health surveys Look at the way these are asked in similar studies Only ask for necessary demo information

Race / ethnicity – comprised of two questions (NIH format):1. Are you Hispanic or Latino? (yes vs. no)2. What is your race? (allow for multiple answers)


Avoid “and” “or” (double-barreled) questions Trying to respond to a question measuring two or

more ideas is confusing to respondent and their response is difficult for the researcher to interpret E.g., Do you think that increasing physical activity and

(or) losing weight is beneficial to you child’s health? E.g., Do you think that eye contact and (or) using words

you understand improve communication between you and your child’s doctor

E.g., Were you satisfied with the timeliness and quality of the health care service your child received?


Avoid leading questions

Do you think that the food in the hospital made your child sick?

Do you agree that the hospital staff were overworked?


Avoid bias by using both positive and negative sides in the question stem

“To what extent do you agree….”

vs.

“To what extent do you agree or disagree with this statement…. “


Avoid vague quantifiers

How often did you use aspirin for pain relief in the last month?

Never

Rarely

Occasionally

Regularly

Vs……..


How often did you use aspirin for pain relief in the last month?

Not at all About once in the last month Two or three times in the last month About once a week More than once a week


Ask questions that people are willing to answer! Do not be too personal or risky unless absolutely

needed Be culturally sensitive

People are more likely to answer questions about behaviors and health compared to income or education Using income categories rather than asking for

exact $ amounts is a better strategy


Ordering questions Two key considerations:

Questions should be ordered such that the response to one question does not influence the response to the question that follows it

Questions should be ordered such that respondents are more likely to complete the survey Fact-based questions before opinion based questions Start with non-threatening or least sensitive questions Most important questions should be placed first Vary question format to reduce likelihood that

respondents will “auto-pilot” response sets


Good questions follow comfortably from the previous question. Requires good writing skills Smooth transitions between questions or groups

of questions Group similar questions together

Avoid non-response (i.e., missing data) by including response options for “Don’t Know” and “Refused” or “Prefer not to answer”


Recall of past events – people have limited recall of past events (this will vary with type of event)

Improve recall by providing reasonable prompts Past year Past 6 months Past 3 months Past 2 weeks

Use addition prompts, if needed: “What season was it when your child was last

vaccinated?” “Was it near a holiday or someone’s birthday?”


FORMAT OF QUESTIONS

Two main question formats Closed format forced choice

Yes Always No Sometimes Don’t know Never

Open format free text

Please describe your child’s most distressing symptom? ________________________________________________________________________________________________

Close-Ended Response Formats

the Likert-type response format “Indicate the strength of your agreement for each of the following statements, on a scale from 1 to 5, where a response of 1 indicates that you strongly disagree, a response of 5 indicates that you strongly agree, and 3 indicates that you are uncertain.” Odd number of categories is preferred – captures

respondents who are indifferent or undecided Reduces non-response No more than 5 response options

Close-Ended Response Formats Semantic Differential Scale - used to assess the

meaning of a variable to the respondent uses a pair of discrete descriptor words, which name

opposing positions, as anchors for the response scale Everything in between represents the continuum of

choices between the 2 anchors

Very satisfied __I__I__I__I__I__I__I__I__ Very dissatisfied

Very confident __I__I__I__I__I__I__I__I__ Not at all confident

Strongly agree __I__I__I__I__I__I__I__I__ Strongly disagree

NOTE: odd number of spaces

Advantages: Simple and quick Easy to code, record, analyse Easy to compare Easy to report results

Disadvantages: Restricted number of possible answers Loss of information

Close-Ended Response Formats

Open – ended response questions Advantages

Rich and detailed response – respondents are not constrained by researchers choices

Unexpected responses possible Good for exploring knowledge and attitudes

Disadvantages Cognitively demanding for respondent and

researcher (coding is challenging and time intensive) Interviewer bias Difficult to analyse Difficult to compare groups

Asking questions about: Knowledge Attitudes and beliefs Behavior

Developing Surveys

Measuring Knowledge True/False and multiple choice “fact” questions Open-ended response formats

E.g., list all of the problems related to X that come to mind

Close-ended response formats (response options are provided by the survey) Quicker, less burden compared to open-ended Less detail but easier to code and analyze

If you plan to intervene on knowledge, include questions at pre and post-test that measure knowledge specifically addressed by intervention

Measuring Attitudes and Beliefs

Close-ended response formats are very common when measuring attitudes / beliefs

Attitude questions can be stand-alone items but … Responses often combined (i.e., summed) as a

“Scale” which means that many items are used to assess a single attitude or belief

often using the same response format… e.g. strongly agree …… strongly disagree

Methods to measure behavior Self report: “I did not smoke any cigarettes today.”

By phone, mail, computer, interview Observation - Report of respondent’s behavior by

others (family member, health professional) “I saw him/her smoke cigarettes today.”

Survey design can go a long way in reducing non-response!

Consider using surveys that you think are attractive as a template Professional looking format - good use of “white space” Space response scales widely enough so that it is easy to circle or

check the correct answer without the mark accidentally including the answer above or below. Open-ended questions: the space for the response should be big enough to

allow respondents with large handwriting to write comfortably in the space. Closed-ended questions: line up answers vertically and precede them with

boxes or brackets to check, or by numbers to circle, rather than open blanks. Use larger font size (e.g., 14) and high contrast (black on white).

Proof read!! No typos! Get help if this is not your thing.

Always conclude your survey with THANK YOU!

February 11, 2015 Survey Design and Administration Catherine R. Messina PhD.

Documents

survey data slide

colonoscopy slide

observed value

true value measurement

research questions

right questions

measured value true

measurement instruments