Survey Design and Measurement Jeremy Kees, Ph.D..

Survey Design and Measurement

Jeremy Kees, Ph.D.

Some practical issues….

• Qualtrics Research Platform– Free you under VSB’s “site license”– Extremely user friendly, but also very

robust– www.qualtrics.com

Some practical issues….

• Amazon Mechanical Turk– The most inexpensive way to collect

consumer data– Extremely user friendly, but also very

robust– www.mturk.com

Online Survey (created by you and housed on Qualtrics’ server)

Create HIT (Human Intelligence Task) on Mturk•Description of your study and a (Qualtrics) link to it

Mturk workers (survey responders) “work” on your HIT (i.e., they take your survey)

Data is recorded by Qualtrics. Participants who complete the survey are given a code to input into Mturk. Those that enter a valid code, get paid.

Everyone is happy

The Mturk data I collected today….

• N=200

• Cost = $100

• Data collected in less than 1 hour

• Demographics– Mean age = 36

– 56% male

– 76% Caucasian

– 80% at least some college• 41% are college grads

– Median income = $35-50k

– Highly engaged!

Stages in theResearch Process

Determine Research Design

Analyze and Interpret the Data

Design Sample and Collect Data

Formulate Problem

Design Data Collection Method and Forms

Prepare the Research Report

Surveys / Questionnaires

• The most common measurement instrument when quantitative data is sought– Descriptive research– Experiments– Modeling– Etc….

Developing Surveys

• Good, well-specified research objectives lead to good surveys

• Research design dictates what types of questions should be used– Exploratory research = unstructured

script– Confirmatory research = structured

survey

Desirable Characteristics

• Brief

• Objective

• Specific

• Relevant

Survey Methods

• Usually should determine administration method prior to developing items– Can dictate what types of questions you

should ask

• Internet panels have become the most efficient and versatile method to collect data– Phone is still a viable option

– Mall intercepts can still be useful

– Mail/fax makes little sense anymore

Mall Intercepts vs. E-Panels

mall internet paneltests tests members

household size 2.8 2.9 3.0average age 40.5 39.2 37.2employed 71% 72% 69%white 86% 88% 89%male 20% 21% 15%college 40% 43% 46%

Correlation between Responses:

mall vs. internetinternet test/retest reliability

purchase intent .86 .94frequency .94 .97liking .85 .91price / value .90 .99

Jeff Miller and Alan Hogg “Internet vs. Telephone Data Collection” Burke White Paper series 2 (4) (www.burke.com). Also see Ashok Ranchhod and Fan Zhou “Comparing Respondents of E-Mail and Mail Surveys,” Marketing Intelligence & Planning 19 (2001), 254.

Internet PhoneTime survey took to administer 12.5 19.4 minutes

Upon completion, would respondent participate in future studies? 35% yes 26% yes

More experienced Internet Users x

Used rating scale extreme “endpoints”more frequently x

E-Panels vs. Phone

Types of Questions

• Screening Variables

• Independent Variables

• Dependent Variables

• Classification Variables– Segmentation– Moderators

• Attention Filters

Primary Data: Overview

• Types of Primary Data– Demographic / Socioeconomic Characteristics– Psychological / Lifestyle Characteristics– Attitudes / Opinions– Awareness / Knowledge– Intentions– Motivation– Behavior

• What, how much, where, when, how, who• Purchase behavior vs. use behavior

• E.g., --- basic hierarchy of effects models• Example (CWL Study)

QUESTION WORDING - General Guidelines • Use simple words and questions• Avoid ambiguous words and questions• Avoid leading questions---be objective• Avoid implicit alternatives• Avoid generalizations and estimates ---

Be specific • Avoid double-barreled questions

1. What is your income?

$10,000 or less………………….1

$10,000 to $25,000……………..2

$25,000 to $50,000……………..3

$50,000 to $75,000………..…….4

$75,000 to $100,000..……..…….5

$100,000 or more…………..……6

What is the problem and how would you revise the question?

4. Is the speed and efficiency of the drive-in teller services at your regular bank…..(READ CATEGORIES)

Very Satisfactory………………………4

Somewhat satisfactory…………………3

Somewhat unsatisfactory………………2

Very unsatisfactory……………….……1

Question Wording

• It is good practice to use scales whenever possible– Likert or semantic differential– Multi-item

Itemized Rating Scales

• The respondents are provided with a scale that has a number or brief description associated with each category.

• The categories are ordered in terms of scale position, and the respondents are required to select the specified category that best describes the object being rated.

• The commonly used itemized rating scales are the Likert and semantic differential

Types of Scales

• Nominal scales: those that use only labels

• Ordinal scales: those with which the researcher can rank-order the respondents or responses

• Interval scales: those in which the distance between each descriptor is equal

• Ratio scales: ones in which a true zero exists

21

IntervalPlease indicate how much you like each soft drink by checking the appropriate position on thescale: dislike like

a lot dislike like a lotCoke ____ ____ ____ ___Dr. Pepper ____ ____ ____ ___Mountain Dew ____ ____ ____ ___Pepsi ____ ____ ____ ___Seven Up ____ ____ ____ ___Sprite ____ ____ ____ ___

NominalWhich of the soft drinks in the following list do you like? (Check ALL that apply):

___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite

OrdinalRank the soft drinks according to how much you like each (most preferred drink = 1, and least preferred drink = 6):

___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite

RatioPlease divide 100 points among these soft drinksTo represent how much you like each:

___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite100

Examples…

Itemized Rating Scales

• Likert Scales– requires the respondents to indicate a degree of

agreement or disagreement with each of a series of statements about the stimulus objects

 Strongly Disagree Neither Agree Strongly

disagree agree nor agree

disagree

1. Wal-Mart sells high quality merchandise. 1 2X 3 4 5

2. Wal-Mart has poor in-store service.1 2X 3 4 5

3. I like to shop at Wal-Mart . 1 2 3X 4 5

4. Wal-Mart has low prices . 1 2 3X 4 5

Itemized Rating Scales

• Semantic Differential Scales– End points associated with bipolar labels that have

semantic meaning

SEARS IS:

Powerful --:--:--:--:-X-:--:--: Weak

Unreliable --:--:--:--:--:-X-:--: Reliable

Modern --:--:--:--:--:--:-X-: Old-fashioned

Decisions for Itemized Scales

• Number of scale items• More is better, but there is a diminishing return

around 11 points (Nunnally 1978)• 7-point scales are customary

• Enough to discriminate• Allows for a scale midpoint• Manageable

• Odd/even number of categories

• Forced vs. non-forced

Why Multi-Item Scales??

Construct Abstract Concept

“Unobservable”

“Latent”

“Psychological”**Single items are typically not

sufficient to assess unobservable constructs

Multi-Item Scales are More “Reliable”• True Score Test Theory

– All measures have1. “True” Score2. “Error” (Random and Systematic)

• Good measures minimize the systematic error component of the score

• Types of Reliability– Inter-Rater– Test-Retest– Internal Consistency (Cronbach’s Alpha)

Developing Sound Measures

Step 2:

Specify Domainof the Construct

Generate Sampleof Items

Collect Data

Purify Measure

Assess Validity

Step 1:

Step 3:

Step 4:

Step 5:

Question Sequencing

• After you have developed your measures, think about the order in which they should be asked

QUESTION SEQUENCING - General Guidelines

• Use (more) simple, interesting opening questions

• Use the funnel approach, asking broad questions first, and follow with more specific questions

• Carefully design branching questions– Skip/display logic

– Ask for classification information last• Place more difficult or sensitive

questions near the end

Question ordering #1

1 – EVALUATION OF FAT LEVEL OF A PRODUCT

2 – EVALUATION OF OVERALL PROD. NUTRITIOUSNESS

3 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE

Question ordering #2

1 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE

2 – EVALUATION OF LEVEL OF PROD. NUTRITIOUSNESS

3 – EVALUATION OF FAT LEVEL OF PRODUCT

QUESTION SEQUENCING - General Guidelines

FOP Labeling Study

• We were interested in consumer evaluations of:1. Facts Up Front

2. All On-Package Labeling

3. Front-of-Package Nutrition Info

Why was question sequencing critical??

Tips for Maximizing Participation

• Offer an incentive ($$$)• Importance/relevance of the research

project and its purpose• Completing the questionnaire will take

only a short time• Answers are anonymous or

confidential• Reminder 2-3 days after the initial ask

Attention Filters

• Always include an attention filter to ensure that you are getting “quality” respondents– Eliminate “click throughs”

Attention Filters (Case Study)

• Advertising Experiment– Very stringent screening criteria

• Total # that started the study = 15,458• Number that qualified = 870

– Incidence Rate (IR) = 5.6%

• Number that qualified and passed the attention screener = 451– 48% failed the attention filter!!!

• NOT GOOD, criticalmix!

“Easy” Attention Filter

“Difficult” Attention Filter

And finally, remember the golden rule….

Do unto your respondents as you would have them do unto you!!

Team Assignment #2• Refine your research questions

– Need to be clear, concise, and “testable”

• Based on your research questions1. Design 2 potential studies that could

address your research questions1. Explain the benefits and weaknesses of each

approach

2. Pick the “best” design and explain your decision

(Note: Don’t worry about measurement or sampling too much---you’ll have your chance to do that later)

Team Assignment #3

** Don’t start on this assignment until you’ve read Fowler (CH 6-7)

• Based on your research design1. Write a paragraph about what your

measurement instrument is supposed to accomplish

2. Make a list of what should be measured to accomplish the goals of the study

3. Develop your measurement instrument

Team Assignment #3

• Deliverables include:1. A very clean, polished version that you

could use to actually collect data• This means you will need to carefully think

through all of the issues we covered tonight (e.g., set-up, ordering, length, multi-item scales, etc.)

2. Intro paragraph and variable list (see previous slide)

(Note: Don’t worry about defining your sample--you’ll have your chance to do that next week)