Survey Methods - OIISDP 2015

survey methods

Why?

1. Measure behaviors, attitudes, or a construct

2. Collect relatively large amounts of data

3. Efficiency (cost vs. output)

Steps in survey process• Step 1: Selecting/developing questions

• Step 2: Select method of administration

• Step 3: Pilot testing

• Step 4: Sampling

• Step 5: Writing up results

Selecting / Developing Questions

What to look for in a good measure

•Published

•Used in a similar study

•Good psychometric properties

•Internal consistency estimates (𝛂 >.80)

•Good factor structure

•Evidence of validity

What is Factor Analysis?•Factor analysis is all about simplification

•Allows us to understand large quantities of observable variables in terms of a smaller number of unobservable variables

•These unobservable variables are called “latent variables”

What is Homer Simpson like?

Homer likes to stick his hand in beehives

Homer encourages his daughter to smoke

Homer is distracted during a reactor

meltdown

• These are all directly observable phenomena

• It might be easier just to say he’s stupid

• Stupidity is a latent variable.

Factor StructureFactor Loadings for Exploratory Factor Analysis with Varimax Rotation of Facebook Activities

Items Factor 1 Communicating

Factor 2 Self-presentation

Factor 3 Social Information Seeking

Commenting 0.796

Liking 0.649

Sharing links 0.635

Status updates 0.628

Private messages 0.444

Chat 0.422

Tagging photos 0.826

Posting photos 0.811

Viewing photos 0.798

Checking up on 0.626

Creating or RSVP’ing to events

0.310

Developing Questions

Qualitative Research

Survey Mode1.Mail 2.Phone 3.In Person 4.Online

Survey ModePros Cons

Online

Lower cost Quickly collect data

Easy to modify Review results

Branching

Response rates Digital Inequalities

Contacting participants Data loss

PhoneHigh response rates

Interact with real person Diverse sampling

High cost Take longer

Cell phone numbers

Mail Convenient

Detailed responses Lower cost

Efficient

Take much longer Higher non completion

Compete w/mail Project management

Levels of Measurement•Nominal: Names. Has no order. Assignment is arbitrary

(1=East, 2=North, 3=South, etc.)

•Ordinal: Has order, but interval between scale points may be uneven (1st place vs 2nd place runners compared to 50th and 51st place). Arithmetic operations are impossible with ordinal data. Can count and order.

•Interval: Has order and equal intervals with an arbitrary zero point (years: year 1 AD is arbitrary - not when time began). Can add and subtract but not multiply and divide.

•Ratio: Same as interval data with a true zero point (income: 0 income is truly no income). Can conduct all operations.

Measurement Level Operations Examples

Nominal No Ordering Sex (Male, Female)

Ordinal Ordering, but not distance

Student Class Standing (Freshman,

etc.)

Interval Distance, but not ratios GPA

Ratio Ratios Number of Credit Hours

Clarity and Usability

Clarity• Provide clear instructions

• Emphasize rating scale

• Never use compound questions

• Will you allow “don’t know” option?

QuestionsOpen Ended Closed Ended

Responses Greater variety of responses

InterpretationRespondents

interpret question the same way

Missing Data More likely to skip

Data Analyses No coding involved

Writing Questions• Avoid leading words

• The government should force you to pay higher taxes.

• Give mutually exclusive choices

• What is your age?

• 0-10

• 10-20

• 20-30

Writing Questions• Be direct

• How do you use the Internet?

• Cover all possible answer choices

• You indicated that you no longer use Facebook. Why not?

• My parents joined Facebook

• I like Instagram better

• I lost my password

Likert Scales• Make sure they are ordinal and perhaps even

interval

• 5 or 7? Doesn’t matter

• Balanced # of positives and negatives

• Use scales with similar anchors

• Reduces cognitive load but…

• Spurious covariance

Question Order• Easy/fun/engaging questions first

• Sensitive & demographics last

• Counterbalance

Counterbalancing

Question Order• Easy/fun/engaging questions first

• Sensitive & demographics last

• Counterbalance

• Branch when you can

Pilot Testing

Populations & Samples•Population: Entire collection of all of the data of

interest

•Sample: A subset of the population

Population

Sample

Choosing a Sample•Random:

1. Each person is chosen entirely by chance

2. Each member of the population has an equal chance of being included in the sample

•Representative: The characteristics of the sample should match the characteristics of the population

Population: All college students who use Facebook

Sample: All students in an introductory psychology course who use Facebook

Sample Size•Larger sample sizes generally lead to increased

precision when estimating parameters.

•Sample must be large enough to detect differences in significance testing.

•Calculate the sample size required to yield a certain power for a test, given a predetermined Type I error rate (ử).

•Power: The probability that you will conclude there is no relationship when in fact there is.

Statistical Significance•Types of Error

•Type I Error (significance): the chance you will conclude there is a relationship when there is not.

•The chance that another random sample from the same population would result in a relationship as strong or stronger than the observed one, just by chance of sampling. Typically set at 5% (p < .05).

•Type II Error (power): the chance you will conclude there is no relationship when in fact there is. Typically set at 20%, corresponding to a power level of .80.

•Low power: where relationships which are real cannot be found to be significant (usually because sample size is too small).

•High power: where even trivially small relationships are found significant (because sample size is excessive).

Choosing your sample•Calculate the sample size required to yield a

certain power for a test, given a predetermined Type I error rate (ử).

•Figure out how to obtain a sample that is representative of your population of interest.

•Randomly sample your population.

•Simple random sampling: when you have a list which approximates all members of the population, then you draw from that list using a random number generator.

Size Doesn’t Matter

Representativeness Does

and so do effect sizes

Effect Size•A measure of how strongly the independent

variables affect the dependent variables.

• p-value is not effect size!

•Cohen suggested:

•Small: 0.01

•Medium: 0.059

•Large: 0.138

“These results were highly significant”

Convenience Samples

•Understand your population

•Offer congruent incentives

•Appeal to intrinsic altruism

•Personalize

Data Analyses

Outliers

168 hours/week

Contradicting Results

15% of respondents engage in sporadic

careless responding

Use data quality indicators?

What does act of giving survey do to future evaluations?

Writing up results

I’m done