Running Successful User Surveys Surprises and Lessons Learned April 6, 2005 Doug Stark Sr. Director, Product Management [email protected] 510.647.1300
May 11, 2015
Running Successful User SurveysSurprises and Lessons Learned
April 6, 2005
Doug StarkSr. Director, Product Management
1©2005 MedeFinance. Confidential & Proprietary /
Today’s Discussion
Fast facts about MedeFinance
Our survey process
Key survey questions
Surprises & Next time…
2©2005 MedeFinance. Confidential & Proprietary /
Fast Facts About MedeFinance
Founded By Jim Quist in 1994
Who We Are Web analytic software and client services provider
What We Do Help financial executives improve performance
How We Do It Our Performance On Demand Service1) Adaptive Web Analytics – personalized
2) Instant, on demand IT environment and
3) Client services – deep healthcare domain experience
Headquartered Emeryville, CA
Ownership Private, Profitable, 100% Growth Each of Last 3 Years
Clients Over 400 hospitals…100% Healthcare Focused…
3©2005 MedeFinance. Confidential & Proprietary /
Service MedeFinance Offers
MedeFinance Performance OnDemandTM
On DemandIT
Environment
ClientServices
AdaptiveWeb
Analytics
4©2005 MedeFinance. Confidential & Proprietary /
A Few MedeFinance Clients
5©2005 MedeFinance. Confidential & Proprietary /
Today’s Discussion
Fast facts about MedeFinance
Our survey process
Key survey questions
Surprises & Next time…
6©2005 MedeFinance. Confidential & Proprietary /
Why survey?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
1) Determine the ObjectivesMeasure end-user satisfaction with services & product Measure the company’s perceptionGather input on future product enhancementsDeepen knowledge about our user baseGain experience with direct user surveys
7©2005 MedeFinance. Confidential & Proprietary /
What defines success?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
2) Establish Survey Goals10 – 15 minutes to complete
Response rate of 20 – 25%
Complete in time for roadmap planning
8©2005 MedeFinance. Confidential & Proprietary /
What’s important to know?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
3) Create Survey Questions50 initial questions created for:
User Profile
Services
Product Functionality
Future Functionality
Open Feedback & Optional Self Identification
~75% Quantitative vs. ~25% Qualitative
Cross-functional team wrote questionsProduct Management / Client Services / Account Management / Executives
50 Qs
35 Qs
9©2005 MedeFinance. Confidential & Proprietary /
How should we survey?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
4) Select Survey EngineThere are many options on the market
Key considerations:Advanced AnalysisAnonymous ResponsesBranchingBranding & Easy to UseCost
Email DistributionExport FunctionalityGraphicsResponse TrackingUnsubscribe/Remove
10©2005 MedeFinance. Confidential & Proprietary /
Is it easy to read and use?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
The Survey:
11©2005 MedeFinance. Confidential & Proprietary /
Who should we survey?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
5) Target UsersFrequent to infrequent users
Recent to past users
CFOs, Directors, Managers, Consultants
Gain customer stakeholders’ permission!
Daily50%
Weekly25%
Monthly15%
Quarterly10%
Targeted users logged in: Number of times targeted users logged in:
12©2005 MedeFinance. Confidential & Proprietary /
Why is it important?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
6) Conduct Survey - InvitationPersonalize email invitation from CEO
“As a valued friend and partner of MedeFinance, I trust you have experienced the commitment of our Company to excellence and our goal to make you successful…
The survey starts at:http://www.surveymonkey.com/a...
I look forward to your response by…
Sincerely,Jim Quist, CEO
13©2005 MedeFinance. Confidential & Proprietary /
Did you forget?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
7) Conduct Survey - ReminderAfter 7 days, we experienced a response rate of 25%
Reminder invitation to those who had not responded boosted response rate to 35%!“Last week, Jim Quist, sent an online survey invitation…I want to follow up personally to ensure you receive this invitation…
We truly are shaping the product based on our customer's direct feedback…
I look forward to your response by…
Sincerely,Doug StarkSr. Director, Product Management
14©2005 MedeFinance. Confidential & Proprietary /
What does it all mean?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
8) Analyze ResultsDetermine validity
Compare averages/means
Observe distribution
Correlate results with conjoint analysis
Use common sense…
15©2005 MedeFinance. Confidential & Proprietary /
What is important?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
9) Report ResultsShare key results early!
Don’t make it a science project…
Present results in actionable manner
Publish by account, function, etc…
Provide supporting detail for outliers
Correlate key questions with supporting and/or opposing questions
16©2005 MedeFinance. Confidential & Proprietary /
How to publish the results?
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
Example Results Graphs
0%
1%
14%
58%
27%
Strongly Disagree
Disagree
Neutral
Agree
Strongly Agree
StronglyDisagree
Disagree Neutral StronglyAgree
Not
at
All
Som
eS
uppo
rt W
ell
Eno
ugh
Ext
rem
ely
Wel
l
10. I feel I was ADEQUATELY TRAINED on how to use...
9. H
ow w
ell d
o yo
u K
NO
W H
OW
TO
USE
...?
Agree
ExtremelyDissatisfied
(1)
Dissatisfied(2)
Neutral(3)
Satisfied(4)
ExtremelySatisfied
(5)
Response Rate = n%Response Rate = n%
ExtremelyDissatisfied
(1)
Dissatisfied(2)
Neutral(3)
Satisfied(4)
ExtremelySatisfied
(5)
Response Rate = n%Response Rate = n%
“It would be helpful if…”
“It would be helpful if…”
“We really like…”“We really like…”
n
17©2005 MedeFinance. Confidential & Proprietary /
Get the value!
Week 8Week 7Week 6Week 5Week 4Week 3Week 2Week 1
10) Take ActionRespondents expect action!
Assign functional leads to create action plans for:Immediate, Tactical, Strategic
Product vs. Service
By Functional Area
Earn a strong future response rate by responding to respondents’ feedback!
18©2005 MedeFinance. Confidential & Proprietary /
Today’s Discussion
Fast facts about MedeFinance
Our survey process
Key survey questions
Surprises & Next time…
19©2005 MedeFinance. Confidential & Proprietary /
Satisfaction
20©2005 MedeFinance. Confidential & Proprietary /
Benefits
21©2005 MedeFinance. Confidential & Proprietary /
Credible Results
22©2005 MedeFinance. Confidential & Proprietary /
Usability & Training
23©2005 MedeFinance. Confidential & Proprietary /
MVQ - Most Valuable Questions
24©2005 MedeFinance. Confidential & Proprietary /
Today’s Discussion
Fast facts about MedeFinance
Our survey process
Key survey questions
Surprises & Next time…
25©2005 MedeFinance. Confidential & Proprietary /
Surprises
The survey was a success!Validation and reality check -significant value can be achieved from a small group n<300Response rate - reminders really do drive response ratesCustomer affinity countsHow much work it really takes
26©2005 MedeFinance. Confidential & Proprietary /
Next time…
Now that we have experience, next time we will…Segment the survey - ask the right questions to the right people
Quality over quantity – perform more analysis with fewer questions. How will the answer be used?
Increase survey group sizeSeriously consider outsourcing!
27©2005 MedeFinance. Confidential & Proprietary /
Go survey!
This was a helpful presentation.Strongly DisagreeDisagreeAverageAgreeStrongly Agree