What Your Customers Really Think About You (Relate Live Sydney)

#RelateLive

Lori Gauthier, Ph.D.Zendesk Director of Marketing Research

@datadocgauthier

https://twitter.com/datadocgauthier

#RelateLive

What Your Customers Really Think About You

Let’s Start with the Don’ts!

What’s wrong with this question?Measuring Customer Satisfaction

How much do you agree with the following statement? I am happy with the customer support I received today.

Strongly disagree

Strongly agreeDisagree AgreeSomewhat

disagreeSomewhat

agree

construct not specified in scale

unbalanced question

non-modified response optionsagree/disagree

scale

question as a statement

missing ambivalent midpoint

incorrectly defined construct

Question source: The Effortless Experience

How much effort did you personally have to put forth to get your issue resolved?

Very low effort Very high effortNeutral High effortLow effort

missing correct midpointmissing “no effort”

end pointconfusing

scale


awkward question

Measuring Customer EffortWhat’s wrong with this question?

Measurement ErrorThe survey itself impacts responses

specification errorrandom error

systematic error

largest source of

error

controlled by

surveyor

“I know you think you understand what you thought I said but I'm not sure you realize that what you heard is not what I meant”

- Unknown

Wait. What? I Thought You Meant…

Specification ErrorEven well designed surveys can yield bad data when the wrong constructs are measured or the right constructs aren’t measured completely.



Strongly disagree


disagreeSomewhat

agree


(specification error)

What’s wrong with this question?Measuring Customer Effort





(specification error)

Stewie DataLook at him go!

Random ErrorBad survey design can introduce data-destroying random error, making your data — and decisions — bounce all over the place.



Strongly disagree


disagreeSomewhat

agree


(random error)

non-modified response options

(random error)agree/disagree

scale(random and systematic error)




awkward question

(random error)

confusing scale

(random error)

missing correct midpoint

(random error)


Rooting Out Random ErrorSo long, Stewie!

no!nooo!

noo!

double barreled questionunexpected scale direction

insensitive scaleoverly sensitive scale

scale without midpointscale without verbal labels

overlapping scale labelsnon construct-specific scale

confusing question or scale

true|false, yes|no, agree|disagree scale

Tower of Pisa DataOne way or another, it’s gonna getcha!

Systematic ErrorBad survey design can introduce data-destroying systematic error, leading you to make biased decisions.



Strongly disagree


disagreeSomewhat

agree

unbalanced question

(systematic error)

agree/disagree scale

(random and systematic error)


(systematic error)

missing ambivalent midpoint

(systematic error)




missing “no effort” end point

(systematic error)


Banishing BiasArrivederci, Pisa!

worst

ever!!!thingunbalanced scale

leading question

true|false, yes|no, agree|disagree scale

missing extreme endpoints bipolar scale without neither/nor midpoint

order effectscontext effects

unbalanced question

question formatted as statement

Done with the Don’ts. Let’s Review the Do’s!

Know What You Need from Your Data

answer construct question scale

Start with your destination.

What Are You Measuring?Are You Sure?

Define What You Need to Measure

Words Mean ThingsSearch definitions, synonyms, antonyms.

Use the language and tone appropriate for your population.

Result: Respondents answer the question you think you’re asking.

Source: snappywords.com

http://www.snappywords.com/

What Questions and Scales Should You Use?Understanding Construct Polarity and Scale Sensitivity

Which Way Do We Go?Construct polarity

Unipolar Construct Bipolar ConstructVery common; typically specific; often descriptive Very rare; typically global; occasionally comparative

Measures absence to maximum: not at all likely to extremely likely

Measures maximum negative to maximum positive: disapprove a great deal to approve a great deal

Midpoint represents half of construct Midpoint represents ambiguity or no opinion

5-point scale is ideal 7- or 9-point scale is ideal

How likely are you to vote in a primary this year? Do you approve or disapprove of negative campaigning?

Examples: likelihood, frequency, duration, intensity Examples: bad/good, dis/satisfied, dis/like, worse/better

common labels: not at all, slightly, moderately, very, extremely

none, a little, a moderate amount, a lot, a great deal

common labels (mirrored sides): extremely, very, moderately, slightly, neither/nor … a great deal, a lot, a moderate amount, a little, neither/nor …

zero????

Ideal scale sensitivity (example 1)How Many Scale Points Should You Use?

unipolar

not at a

ll

extre

mely

moderatel

y

sligh

tly very

1000 5025 75

bipolar

neither/

nor

extre

mely

moderatel

y

sligh

tly very

1000 5025 75

sligh

tlyvery

extre

mely

moderatel

y-25-75-100 -50

Ideal scale sensitivity (example 2)How Many Scale Points Should You Use?

unipolar

not at a

ll

a grea

t dea

l

a modera

te am

ount

a litt

lea l

ot1000 5025 75

bipolar

neither/

nor

a grea

t dea

l

a modera

te am

ount

a litt

lea l

ot1000 5025 75

a litt

lea l

ot

a grea

t dea

l

a modera

te am

ount-25-75-100 -50

How Many Scale Points Should You Use?Sensitivity reduced as scale points removed

unipolar

not at all likely

extremely likely

moderately likely

slightly likely

very likely

1000 5025 75

????not likely likely


bipolar

1000 5025 75-25-75-100 -50

neither like nor dislike

like a great

deal

like a moderate

amount

like a little

like a lot

dislike a little

dislike a lot

dislike a great

deal

dislike a moderate

amount



like a great

deal

like a moderate

amount

like a little

dislike a little

dislike a great

deal

dislike a moderate

amount

1000 33 67-33-67-100

bipolar

1000 5025 75-25-75-100 -50


like a great

deal

like a moderate

amount

like a little

like a lot

dislike a little

dislike a lot

dislike a great

deal

dislike a moderate

amount


bipolar

1000 5025 75-25-75-100 -50


like a great

deal

like a moderate

amount

like a little

like a lot

dislike a little

dislike a lot

dislike a great

deal

dislike a moderate

amount

1000 50-100 -50


like a great

deal

like a moderate

amount

dislike a great

deal

dislike a moderate

amount


bipolar

1000 5025 75-25-75-100 -50


like a great

deal

like a moderate

amount

like a little

like a lot

dislike a little

dislike a lot

dislike a great

deal

dislike a moderate

amount

1000-100


like a great

deal

dislike a great

deal

Measuring Customer Satisfaction

How satisfied are you with Acme’s customer support?

1 3 42



How satisfied are you with Acme’s customer support?

1 3 42

unbalanced question

(systematic error)

missing construct- specific verbal labels

(random error)missing negative half of scale

(systematic error)

missing midpoint on positive half of scale

(random error)

missing zero scale point

(systematic error)

A methodologically sound questionMeasuring Customer Satisfaction

Overall, how satisfied or dissatisfied are you with Acme’s customer support?

moderately dissatisfied

slightly dissatisfied

neither satisfied nor dissatisfied

slightly satisfied

moderately satisfied

extremely dissatisfied

extremely satisfied

7-point, fully labeled, construct-specific,

bipolar scale

measures what we want to measure: satisfaction with customer support

“overall” appropriate for global-level measure

balanced question

ambivalent midpoint

Measuring Customer Effort

To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.

Strongly disagree

Strongly agree

Neither agree nor disagree

Disagree AgreeSomewhat disagree

Somewhat agree



To what extent do you agree or disagree with the following statement? The company made it easy for me to handle my issue.

Strongly disagree

Strongly agree

Neither agree nor disagree

Disagree AgreeSomewhat disagree

Somewhat agree



(random error)

non-modified response options

(random error)agree/disagree scale

(random and systematic error)


(systematic error)


Measuring Customer EffortA methodologically sound question

How easy was it to get the help you needed from us today?

not at all easy

extremely easy

moderately easy

very easy

slightly easy

measures what we want to measure: effort needed to get company’s help “today” appropriate for

transaction-level measure

5-point, fully labeled, construct-specific,

unipolar scale

Measuring Customer EffortWhat is driving customer effort?

Content source for drivers of effort: The Effortless Experience

How did we make it difficult? (Check all that apply)

You didn’t solve the problem I had to contact the company multiple times I felt like I was talking to a robot I had to repeat myself I had to use a channel I don’t like (phone, web form, chat, email, FAQ) I was transferred from person to person Some other reason (Please specify)

don’t assume resolution

pick list Q measures frequency of known drivers

open-ended option captures unknown drivers

limit list to 7-9 options

random rotate pick list

Sound design. Accurate data. Better relationships.A Step-by-Step Approach to Survey Design

start at your destination

define your construct

draft question + scale

check for random error

check for systematic error

collect accurate data

bing!bing!

bing!determine polarity

Thank You!Questions? Contact me at [email protected] or @datadocgauthier.

mailto:[email protected]?subject=

https://twitter.com/datadocgauthier

#RelateLive

What Your Customers Really Think About You (Relate Live Sydney)

Business