From Start to Pilot: Comprehensive Redesign of a Questionnaire Measuring Foreign Direct Investment Alfred D. Tuttle Rebecca L. Morrison United States Census Bureau David H. Galler United States Bureau of Economic Analysis
Jan 04, 2016
From Start to Pilot: Comprehensive Redesign of a
Questionnaire Measuring Foreign Direct Investment
Alfred D. Tuttle
Rebecca L. Morrison
United States Census Bureau
David H. Galler
United States Bureau of
Economic Analysis
2
Overview – Redesign of BEA Form BE-605
• Reimbursable project with BEA
• Multiple, intensive stages of research, design, testing, revision
• Qualitative and quantitative evaluations
• New BE-605 form mailed out in March 2007
3
Outline
• Background – BEA Form BE-605
• Summary of stages of research
• Findings
• Next steps
4
Background
• BE-605 (quarterly) and BE-15 (annual)
• Foreign Direct Investment (FDI) – >10% of voting rights in an incorporated entity (or equivalent interest in unincorporated entity)
• Sample – foreign-owned businesses in US
5
Research stages and methods
• Background investigations– Expert review of form– Focus groups with survey analysts– Observations of callbacks– Respondent debriefings
6
Research stages and methods
• Cognitive testing of mockups
7
Research stages and methods
• Pilot test – Respondent debriefings – Evaluation questions
8
Findings from background research
Data collection and design issues:
• Dense formatting
• Economic accounting concepts differ from US GAAP
• Separate instructions
9
Recommendations
• Formatting recommendations– Letter size pages (not legal)– More “open” visual design– Incorporate instructions into form:
• within questions• adjacent to questions on facing page• key instructions become questions
– Diagrams to augment questions
10
Original form
11
Recommendation: More “open” design
12
Recommendation: Embedded instructions
13
Recommendation: Adjacent instructions
14
Recommendations: Questions from instructions, and diagrams
15
Cognitive testing
• Problematic sections mocked-up and tested (not entire forms)
• 5 rounds of cognitive interviews
• Iterative process: – mock-ups revised between rounds – design features modified & re-tested
16
Findings from cognitive testing
• New design features were effective:– Majority of respondents liked the more
open format – Embedded instructions would likely
improve data quality– Some “InstructionsQuestions” and
diagrams required extensive modification, though most did not
17
Developing 605 pilot
• Redesigned entire BE-605 form
• Close collaboration with BEA survey staff
18
Pilot test
• Mailed 2nd quarter 2006
• Sent to sub-sample (n = 653; survey sample ~4,000)
• Completing pilot form optional
• Response rate: 53% (350 returned forms) during initial 8-week processing period
19
Pilot form evaluation
• Respondent debriefings
• Evaluation questions in form
20
Findings – R debriefings
• Favorable impressions of new form:– More pleasing to the eye– White reporting spaces easier to see
against colored background – Larger font, bold text, more space– Letter size pages easier to handle– Diagrams were helpful– Embedded instructions helpful
21
Findings – R debriefings
• Drawbacks:– More pages– Booklet format makes it harder to fax– Took longer to complete than original form
(disruption of reporting routine)
22
Limitations
• Stymied by experienced Rs, reporting routines…
• …though even experienced Rs were helped by new formatting and instructions.
23
Findings – Evaluation questions
• Four questions appended to survey
• Each item received between 279 and 290 responses
24
Findings – Evaluation questions
Instruction placement?
79.3% - Near questions
18.3% - In separate booklet
2.4% - Not applicable – I haven’t completed
the usual BE-605
25
Findings – Evaluation questions
Open space and number of pages?
48.7% - less open space, fewer pages
47% - more open space, more pages
4.3% - Not applicable – I haven’t completed the usual BE-605
26
Findings – Evaluation questions
Use of diagrams?
65.4% - very helpful or somewhat helpful
11.7% - only a little helpful
18% - did not need them
3.9% - confusing
27
Findings – Evaluation questions
Easy/difficult compared to usual form?
50.5% - harder
42.5% - easier
7% - Not applicable – I haven’t completed the usual BE-605
28
Findings – Evaluation questions
• Instructions preferred near questions
• Organization charts helpful
• Easier or more difficult?
• Preference for more/fewer pages?
29
Summary
• Redesigned BE-605 went into production in March 2007– Recommended analysis of edit checks
during form processing
30
Summary
• Multi-method approach to comprehensive survey redevelopment