GET WHAT YOU NEED FROM ONLINE SURVEYS Janelle Commins, MSPH Research Associate EMT Associates, Inc. 15720 Ventura Blvd., Penthouse Encino, California 91436
GET WHAT YOU NEED FROM
ONLINE SURVEYSJanelle Commins, MSPH
Research AssociateEMT Associates, Inc.
15720 Ventura Blvd., PenthouseEncino, California 91436
Topic Overview
2
When is an online survey appropriate? What do I have to choose from?
Commercial or custom-built How do I get people to take MY survey? Case Study: LA County Service Provider
Survey.
When are Online Surveys Appropriate?
3
Respondent factors. Internet access, geographic spread.
Questionnaire factors. Type of question, length of questionnaire.
Evaluator factors. Time frame, budget, technological
expertise.
What Are My Options?
4
Commercial
Software
Custom-Built
Web-hosted or off-the-
shelf
JAVA
Convenience (for evaluator)
X
Convenience (for user) X X
PC Compatibility X
Complex Skip Logic X
Foreign Languages X
Real-time Data Feeds X X
Look and Feel X
Endless Possibilities X
How Do I Get a Good Response?
5
What IS a good response? Depends on population and relevance. For service providers: 2-45%; median 13%. For market research < 2%.
Recruiting Strategies
Get executive buy-in Send a personal letter to the Executive or
Research Director Provide summary sheet with quick facts.
Present at collaborative meetings. Cold-calling.
Response Strategies
Distribute survey invitations through a trusted intermediary.
Send a pre-announcement email. Emphasize relevance and simplicity
Avoid using the word “survey” in the Subject line.
Case Study: The Los Angeles County Service Provider Online Survey
What is the Los Angeles County Service Provider Online Survey?
9
• First 5 LA-sponsored.• Countywide survey of agency decision-
makers.• Custom-built (JAVA).• Evidence-based recruitment,
distribution, and formatting.
Recruitment Strategy
10
Executive buy-in at public institutions. Presentation at collaborative meetings. Cold calling.
Outcome: Distributed to 1,800 service providers
(150 agencies)
Response Strategy
11
Subject line asked for “help” and “assistance.”
7 Day Contact Day 0: Pre-announcement email. Day 2: Invitation email. Day 7: Reminder email.
Outcome: 23% response rate (N=461)
Item Response Strategy
Survey loaded upfront allowing users to “glide” through questions.
Uncluttered appearance. Invisible drop-downs appeared depending
on users’ selection.
Potential Obstacles
#1: Coordinating Revisions
14
Development Problem: Unpredictable turnaround time.
Solution: Pilot survey and bundle revisions for programmer, understand complexity of revisions, set a realistic deadline.
#2: Limited incentive
Incentive: What do agencies stand to gain from participating?
Solution: Offer agency administrators the opportunity
to include a question of interest. Provide a summary of findings.
#3: Technical incompatibility
16
User Problem: Slow loading time for those operating old PC’s or early web browsers.
Solution: Know the user IT environment! What kind of
browsers are they operating? Use graphics conservatively.
Scenario
17
Very important stakeholder. 600+ candidates Problem: Dated web browsers took 1-2
minutes to load survey. Solution: Eventually loaded survey in
department’s commercial software. Resolved lag time Lost smooth look and feel Required more data cleaning 25% response rate (N=165 respondents)
Typically institution receives a 2-12% response to internal surveys.
Comparison of Labor Time
• 26 developer hours
• 461 responses
• 0.06 developer hours per response
• 8 developer hours• 165 responses
• 0.05 developer hours per response*
18
Custom built Off-the-shelf software
Lessons Learned
19
Facilitating Recruitment. Get executive buy-in.
Boosting Response. Use trusted intermediary. Send pre-announcement. Emphasize relevance and simplicity. Know the IT environment EARLY.
Take-Home Message
Unless you are conducting a survey with unique questionnaire options, special formatting requirements, or complex data analysis, commercial software will probably suit your needs.
Questions?
Contact information:Janelle Commins, [email protected]