Outline: Survey Research I. Sampling II. Survey Contents
Outline: Survey Research
I. Sampling
II. Survey Contents
I. Sampling from a Population
“Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.” Population - complete set of individuals having some
common characteristic – e.g., Australians
Sampling frame – subset of the population from which the sample is actually drawn – e.g., White pages
Sample – the set of people included in the study (i.e., selected from the sampling frame) – e.g., Every 1000th person in the white pages
Population and Sample
Population Sample
Use parameters to summarize features
Use statistics to summarize features
Inference on the population from the sample
Confidence Intervals
The problem is to use the sample to make inferences about the population
If we were to repeat the entire process of drawing a sample and computing the statistic many, many times, we would find that the statistic varies some
A confidence interval takes advantage of those variations and allows us to specify a range that probably contains the true value of the parameter Technically, for an X% confidence interval, the
parameter will fall inside the specified range X% of the time in repetitions of the study
Confidence intervals usually vary between 90% and 99.9%
Define a Population
The first step in choosing a sample is to define the population, or the overall set of cases that we’re interested in If we want to predict the results of an election,
then our population is “likely voters” If we want to understand who decides to vote,
then our population is “citizens eligible to vote” If we want to understand who decides to use
marijuana in the United States, our population is probably “everyone in the United States”
Nonprobability Sample Items included are chosen without
regard to their probability of occurrence
Probability Sample Items in the sample are chosen on the
basis of known probabilities
Types of Samples Used
Types of Sampling Methods
Samples
Non-Probability Samples
Convenience
Probability Samples
Simple Random Stratified Cluster
Quota
Probability Sampling
Members of the sample are chosen based on known probabilities
Probability Samples
Simple Random
Stratified Cluster
Simple Random Samples
Every individual or item from the frame has an equal chance of being selected
Selection may be with replacement or without replacement
Samples obtained from table of random numbers or computer random number generators
Stratified Samples
Population divided into two or more groups according to some common characteristic
Simple random sample selected from each group
The two or more samples are combined into one
Stratified Sampling Example
PopulationPopulation
Cash holdings of All Financial
Institutions in the United States
Large Institutions
Medium Size Institutions
Small Institutions
Stratified PopulationStratified Population
Stratum 1
Stratum 2
Stratum 3
Select n1
Select n2
Select n3
Financial Institutions
Cluster Samples
Population divided into several “clusters,” each representative of the population
Randomly select certain clusters The samples are combined into one
Population divided into 4 clusters.
Cluster Sampling Example
42 22 52
Illinois Scotland Florida
25 105 20 36 152 76 37
Algeria California Alaska New York Idaho Mexico Australia
Mid-Level Managers by Location for Morrison-Knudsen Construction Company
Cluster Sampling Example
42 22 52
Illinois Scotland Florida
Mid-Level Managers by Location for Morrison-Knudsen Construction Company
All members selected from these clusters
Types of Survey Errors
Coverage error
Non response error
Sampling error
Measurement error
Excluded from frame.
Follow up on non responses.
Chance differences from sample to sample.
Bad Question!
II. Survey Contents
Remember that your respondents will be “Lazy Thinkers.”
Survey Contents
When creating questions, consider the effects of the following elements: Type
Wording
Order
Content
Survey Contents: Type
Open-ended
A survey question to which the respondent replies in his or her own words, either by writing or by talking
difficult to analyze, subjective analysis
time consuming rich information useful for descriptive,
exploratory work
Closed-ended or fixed-choice
A survey question that provides preformatted response choices for the respondent to circle, check, mark, etc.
easier to analyze efficient useful for hypothesis
testing important info may be
lost forever
Closed-ended Questions
Closed-ended or fixed-choice Response options are limited
Yes/no True/false Multiple choice with an “other” option
Race: White, Black, Hispanic, Asian, Other Likert Scales 1 2 3 4 5
Strongly Agree Neither Disagree Strongly Agree A nor D Disagree
Rating Scales Ask respondents to rate something like a person, place, thing, idea,
attitude, etc. on a numbered scale, often Likert scales Semantic Differential a fifth grader is Bad GoodAwful ⊲ ⊲ ⊲ ⊲ Nice⊳ ⊳ ⊳ ⊳
Survey Contents: Wording
Guidelines for good questions:1. Be direct
2. Maintain simplicity
3. Be specific
4. Take the role of your respondent
The following points elaborate on these four main themes.
Survey Contents: Wording Respondents should understand your questions. No
complex rhetoric, syntax, or disciplinary slang or jargon. Pre-testing is a very effective way to see if they do
Do not expect them to learn new information just answer a question. (Sometimes, you may establish context with a short paragraph then ask a series of short, specific questions.)
Avoid ambiguous questions. “Do you teach your children to effectively function?” “Does your
boss engage you in interactive dialogue?” “Effectively function” and “interactive dialogue” are subject to interpretation.
Words such as "usually" or "normally" mean different things to different people. “Do your customers normally complain?” has a variety of interpretations.
Survey Contents: Wording
Avoid Double-barreled questions. They contain two questions in one. For example, “Do you think that students and Professors should be given discounts on sports tickets?”
Avoid Double-negative questions. For example, “Do you disagree that professors should not be required to help students outside of class?”
Avoid loaded questions. Avoid making one response option look more suitable than the other, using emotionally loaded terms, or using unbalanced response categories. “Don’t you think that suffering terminal cancer patients should be allowed
to be released from their pain by choosing death?” Virginia pays teachers more than similar states. Virginia should: spend
more, keep spending the same, reduce spending little, reduce spending some, reduce spending a lot, dramatically reduce spending
Survey Contents: Order
Relative placement of questions in the instrument
Opening questions should be simple and introduce the topic of the survey. Also put most important questions early.
Put closed-ended questions early in the survey and open-ended questions later.
Avoid a lot of “skipping” or “go to next section”. R’s will get confused/frustrated.
Survey Contents: Order
Try not to mix topics. Put like things into sets of questions. Clearly transition between types of questions.
Sensitive questions should never be at the beginning. Put in middle. May need opening paragraph for them.
Avoid framing later questions with topics that can be linked to them in previous questions. For example, one should not ask about attitudes toward crack use right before asking about attitudes toward the urban poor. This may invoke stereotypes about the poor.
Survey Contents: Content
Topics the questions cover
You should make the topic of the survey clear to R’s. Do not surprise them with questions about unrelated topics.
If including sensitive questions, provide a rationale for asking them. Explain how honest answers will be helpful to others and reassure confidentiality.
Questions relevant to deviance should include normalizing statements. For example, “Many people use drugs for a variety of reasons. Have you ever used ecstasy (or x, e, MDMA) to feel closer to other people?”
Questions that require lots of specific details or a good memory are typically useless. If they are necessary, employ techniques to prompt recollection.
Survey Contents General Advice
Always consult other surveys first
Always pre-test your instrument Consider these issues:
Consider “no opinion” as an option.
People want to appear to agree. Consider negative and positive statements.
R’s may lose track and choose salient options (such as first in series). Keep questions simple.
Response set problem
Survey Contents
Consider these issues: Bad memory leads to:
Forward telescoping (reporting that events occurred more recently than in reality)
Backward telescoping (reporting events further back than in reality).
Salient events are overreported Mundane events are underreported “Habitual” events will fill in for lost information.
Provide aides to recall such as reference points, landmark events, etc. Use limited time frames in questions.
Class Exercise
Split into groups
Select a topic for a survey
Develop 2 closed-ended and 2 open-ended questions. For each question, write a ‘poor’ version and a ‘good’ version
Administer the survey to member of another group. Critique each other’s questions