Using Surveys in Design

Sam Ladner, PhDOctober 14, 2010

Key Themes

• When to use surveys in design• Qual vs quant• Sampling• From interviews to instruments• Survey basics• Should we do a survey?• Asking good questions• The limits of surveys

The Four Cs of ResearchDesign

• Creativity• Complexity• Compromise• Choice

Borja De Mozota, Brigitte. 2003. Design Management: Using Design to Build Value. New York: All Worth Press .

Make your choice:

What research design would you choose?

Quantitative Research in Design

I want to know what causes something else.

What do quant researchers worry about?

I really spend a lot of time wondering how to measure things.

I wonder how small patterns generalize to big patterns.

I want to make sure others can repeat my findings.

I want to see the world through the eyes of my respondents.

What do qual researchers worry about?

I want to describe the context in a lot of detail.

I want to show how social change occurs. I’m interested in how things come to be.

I really want my research approach to be flexible and able to change.

Qualitative vs. Quantitative

Quantitative Qualitative

Role of theory in research

Deductive, testing of theory

Inductive, generating theory

Ontological orientation

Objectivism Constructionism

Epistemological orientation

Natural science model

Interpretivism

What is your orientation?

A famous sampling mistake

Even with proper sampling…beware!

“…predicting behavior on the basis of knowledge of attitude is a very hazardous

venture.”

What exactly IS a “sample”?

What exactly IS a “sample”?

A subset of the population, selected by either “probability” or “non-probability”

methods. If you have a “probability sample” you simply know the likelihood of any member of the population being

included (not necessarily that it is “random”).

Assumptions of quantitative sampling

We want to generalize to the population.

Random events are predictable.

Therefore…We can compare random events to our results.

Probability sampling is the best approach.

Assumptions of qualitative sampling

Social actors are not predictable like objects.

Randomized events are irrelevant to social life.

Probability sampling is expensive and inefficient.

Therefore…

Non-probability sampling is the best approach.

What kind of sample would you use for a design project?

From interviews to instruments…

“concepts are…categories for the organization of ideas and observations” (Bulmer, 1984: 43)

A “variable” varies…

What is varying?

Concepts and variables

What is a measure?

Something that can be counted.

Age

Height

Number of books

What is an indicator?

Something that “indicates” the presence of something else.

GDP

IQ

Number of hospital visits

What is survey research?

Print or Online surveys

Telephone surveysStructured face-to-face interviews

Asking structured questions

Surveys versus other qualitative methodsWhen a you want to know… …you need

What are users’ opinions of the design?

What is the demographic profile of our users?

What kinds of beliefs or attitudes do our users have about our…

-brand-customer service

How can we be sure that design changes changed users opinions of the product or experience?

A survey

How usable is our Web site?

What do our users generally dislike about our product?

What kinds of features should we include in our new product?

Qualitative

research

method

Is a survey the right research method?

• I know a lot about the topic• I need to summarize the

findings in numerical format• I don’t need a lot of detail and

nuance• I need to generalize the results

to the population• I need to predict the likelihood

of a certain thing happening• I want to measure incremental

change

• I don’t really know much about the topic

• I don’t need to summarize the findings in numerical format

• I need to communicate a lot of detail and nuance

• I’m interested in case study information

• I’m not planning on predicting the likelihood of anything

• I don’t need to measure any change

Yes No

Is it time to do a survey?

Yes

No

Discovered questions and

answers throughQual research?

Clear idea of all possible questions

and answers?

YesNo

Formulated all questions and

possible answers?Do a survey

Do more Qual research

Yes

Pilot test the questions

No

Asking good questions

• Mutually exclusive and exhaustive– No overlap between answers– All possible answers

• No double-barreled questions– Wrong: “What do you think about our response times and customer

service?”– Right: “What is your opinion of our response time?”

• Focus on opinions or beliefs of participants– Wrong: “How usable is our Web site?”– Right: “What are users’ opinions of the site’s look and feel?”

• Close-ended is best– Cuts down on analysis time

• Demographic questions at the end

The lay of the land: descriptive statistics• Summarizes the responses• What is the most “typical” response?

– Mean– Mode– Median

• How much variation is there between responses?– Standard deviation– Inter-quartile range

Predicting behavior: inferential statistics• Infers or predicts future behavior• Which respondents are more likely to…

– Cross-tabulation

• Which factors most influence– Behavior– Satisfaction– etc

The tools you need

Online Survey Tool

Statistical Analysis Software

The skills you need

• Knowledge of the topic• Ability to write good questions• Familiarity with the survey tool• Some basic knowledge of statistics• A framework for writing the report

– Important to write the outline of the final report before you start!

The payoff you’ll get• “More men are dissatisfied with the

experience.” • “Younger users are much more unhappy with

customer service that older users.”• “Higher-income individuals are the most

unlikely to recommend this product to friends and family.”

• “Men are more likely to prefer a two-step process than women.”

They always seem to measure artificial things

and say they’re really precise.

Yeah and they treat people like they’re test tubes or

something. It’s the way they do things…it makes it hard for people to see their research as relevant to

them.

Well their research is just so static. Real life actually

changes.

What’s wrong with quant research?

What won’t you learn from your survey?

What you won’t learn…

• Why aren’t people using our product?• How can we innovate on our current

design?• Why don’t people finish the checkout

process?• What are the main characteristics of the

people who don’t use our product?

Copernicus ConsultingDesign research and strategy

http://www.copernicusconsulting.net