Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau Rebecca L. Morrison.

Developing Establishment Survey Questionnaire Design Guidelines at the U.S. Census Bureau

Rebecca L. Morrison

2

Outline

Introduction & Background

Development of the Guidelines

Preliminary Guidelines

Application of Guidelines

Conclusion & Next Steps

Introduction & Background

4

Introduction & Background

U.S. Census Bureau:– “…leading source of quality data…nation’s

people and economy”– Tries to minimize burden of data collection

Burden = time, level of effort

Reduce cognitive burden through visual design, structure of data request

5

Introduction & Background (cont’d)

Paper does not focus on specific question wording and order issues.

Paper presents preliminary guidelines in questionnaire design – formatting, layout, navigation, instructions, and data request phrasing and style.

6

Development of the Guidelines

7

Development of the Guidelines: Census Bureau

Agency standards for various survey quality issues

No standard for design of questionnaires

Economic surveys moving toward consistent “look and feel”

8

Development of the Guidelines: ESMS

Establishment Survey Methods Staff– Group of survey methodologists– In-house consultants to economic

programs– Noticed differences across questionnaires– Survey programs work with ESMS on

questionnaire design and pretesting– Catalyst towards consistency

9

Development of the Guidelines: Methods

Questionnaire design principles from standard texts

Special attention to literature on visual design

Based on pretesting a variety of questionnaires with respondents

10

Development of the Guidelines: Considerations

List of design elements to consider

Used to assess tradeoffs within specific survey conditions

Not a cookbook

Preliminary Guidelines

12

Guideline A: Text styles

Be aware of text styles, and how they are used for emphasis within a survey instrument.

13

Guideline A: Example

2004 Annual Survey of Local Government Finances

14

Guideline A: Example

15

Guideline A: Text styles

These are favored by respondents:– Print item numbers in reverse-print

bubbles, e.g., , , – Print questions in bold black text.– Print instructions in plain text or italics.– Use an 8-point font or larger.

16

Guideline A: Text styles

Answer spaces:– Open, non-delineated vs. delineated

– Respondents do not seem to have a strong preference

– Be consistent

17

Guideline A: Text styles

Key codes / Punch codes:– De-emphasize processing codes for

respondents– Print in a darker shade of background

color– Example:

18

Guideline B: Reduce visual clutter. Use natural navigational paths.

Reduce clutter on the page. Use navigational paths and layouts that are natural and readable for respondents.

19

Guideline B: Reduce visual clutter. Use natural navigational paths.

20

Guideline B: Reduce visual clutter. Use natural navigational paths.Format in 1997 Economic Census

confusing:– Lines were “speedbumps” to navigation– Not clear how respondent was to navigate

two adjoined columns

Format changed for 2002– Single column of questions– Generally, a single column of response

options

21

Guideline B: Reduce visual clutter. Use natural navigational paths.

Two columns of questions– Occasionally done when questions:

• Are shorter• Do not involve extensive instructions• Do not ask for numerical information

– Survey of Business Owners (SBO)• Information collected is categorical or ordinal,

not interval or ratio

22

Guideline B: Reduce visual clutter. Use natural navigational paths.

Spread out the questions across more pages

Use bulleted lists

More open space = more “user-friendly”

23

Guideline C: Instructions

Place instructions close to questions, or incorporate them into questions. Where possible, convert instructions into questions.

24

Guideline C: Instructions

Instructions convey specifications, intent of question

Respondents tend not to pay attention to instructions, or only look when they think they need them

25

Guideline C: Instructions

2 Goals for instructions:– Eliminate, or reduce, amount of

instructions located separate from question

– Place instructions/information where it is most needed

26

Guideline C: Instructions

Convert instructions into questions so respondents attend to them– When content critical to correct

interpretation of later questions– When it helps clarify/correct reported data

Example (2004 Medical Expenditure Panel Survey):

27

Guideline D: Avoid sentence fragments and key words

Phrase data requests as questions or imperative statements, not as sentence fragments or key words.

28

Guideline D: Avoid sentence fragments and key words

Types of data requests– Question (question word, question mark)– Imperative statement (report, enter, add)– Sentence fragment (key words, no verb)

29

Guideline D: Avoid sentence fragments and key words

Example, 2002 Economic Census:– Is this establishment physically located

inside the legal boundaries of the city, town, village, etc.?

Versus:– Type of municipality where this

establishment is physically located

30

Guideline D: Avoid sentence fragments and key words

Cognitive interview study: respondents prefer questions over sentence fragments

Survey methodology grad students: questions more effective, imply respondent has to do something

31

Guideline E: Simple vs. complicated questions

Ask additional, simple questions, rather than fewer, more complicated ones.

32

Guideline E: Simple vs. complicated questions

Length and complexity of question affects how long it takes for respondent to understand it

More complicated questions might also be more likely to be double-barreled

May be easier for respondents to answer series of shorter, simpler questions

33

Guideline E: Simple vs. complicated questions

Example: 2002 Industrial Research & Development Survey

34

Guideline E: Simple vs. complicated questions

2006 R&D Survey

35

Guideline F: Use matrices judiciously

Use matrices judiciously, and consider the likely respondents’ background when deciding whether or not to use them.

36

Guideline G: Be consistent!

Items should be numbered in the same way

Bold and italicized print should always mean the same thing

Navigation path should remain constant

Data requests should be in the same form

Variation can be confusing for respondents

37

Application of Guidelines

38

Guideline A: Text Styles

Work in progress among estab surveys at Census Bureau

Economic Census automation required consistent design features– Plain text: questions– Italics: instructions, definitions,

include/exclude lists– Bold: Separating items into categories, sums

of added lines, emphasis within questions

39

Guideline A: Text Styles

Survey of Business Owners:

– Questions: plain– Instructions: bold and/or italics– Emphasis withinquestions: bold and/or italics

40

Guideline B: Reduce visual clutter. Use natural navigational paths.

Survey of Residential Alterations and Repairs (SORAR)– Two columns to one column– Removed or lightened lines– Use of bulleted lists– More open space

41

Guideline B: Reduce visual clutter. Use natural navigational paths.

Old

SORAR

42

Guideline B: Reduce visual clutter. Use natural navigational paths.New SORAR

43

Guideline B: Reduce visual clutter. Use natural navigational paths.Two columns:

Survey of

Business

Owners

44

Guideline B: Reduce visual clutter. Use natural navigational paths.BEA quarterly foreign direct investment (FDI)

Complex navigational path

Crowded text

45

BEA form

revised

Guideline B: Reduce visual clutter. Use natural navigational paths.

46

Guideline C: Instructions

Economic census continues to use separate instruction sheets and booklets

Two questionnaires moved many instructions to questionnaire:– 2007 Commodity Flow Survey– BEA quarterly FDI form

47

Guideline C: Instructions

48

Guideline D: Avoid sentence fragments and key words

Mixed success in application across surveys

49

Guideline D: Avoid sentence fragments and key words

Commodity Flow Survey uses mix of questions and imperative statements

50

Guideline E: Simple vs. complicated questions

Example:

2002 Survey

of Business

Owners

51

Guideline E: Simple vs. complicated questions

Example: 2007

Survey of Business

Owners

(still in cognitive

pretesting)

52

Guideline F: Use matrices judiciously

Example:

BEA

quarterly

FDI

53

Guideline F: Use matrices judiciously

54

Conclusion & Next Steps

55

Conclusion & Next Steps

Guidelines are preliminary, based on principles of questionnaire design

Further refinement is necessary before they can be applied to establishment surveys at Census Bureau

56

Conclusion & Next Steps

Other agencies/organizations may develop guidelines that work better for their environment

There is no cookbook for questionnaire design

57

Thank you!

Rebecca L. Morrison

301-763-7595

Rebecca.L.Morrison@census.gov