Managing Participant Satisfaction
Elisabeth LangIntangible Asset Manager, Senior Consultant, CFI Group
+46-8-562 800 00, Box 70373,107 24 Stockholm, Sweden Thursday October 9th, 2008
© 2008 CFI Group - 2
Content
– The point of satisfaction
– The CFI Group method
– Describing IFL from the participants’ perspective
– Analyzing results
– Identifying focus areas
– How to leverage the information to achieve IFL’s goals
– The IFL perspective
Feel free to interrupt with questions whenever you likeNB! Results have been altered, data is not real
© 2008 CFI Group - 3
Is there a point to focusing on quality?
IFL Overall
80
77
81
80
91
0 10 20 30 40 50 60 70 80 90 100
Investment
Activities
Personal development
Loyalty
RecommendExam
ple
Increased satisfaction, loyalty and willingness to recommend
Increased participation in alumni activities, strengthening the alumni network
Stronger brand Higher revenue
© 2008 CFI Group - 4
What’s the effect of satisfied customers?
The cost of retaining customers is much smaller than the cost of acquiring new ones
Satisfied customers are much more likely to consume more and again, and are also much more likely to recommend the company, generating new business at a lower cost
Source: The Satisfied Customer,Claes Fornell
© 2008 CFI Group - 5
Satisfaction drives profitability and shareholder value! ACSI Fund vs. S&P 500: April 2000 (start) – May 26th, 2008
Source: S&P 500 from msn.com-100%
-50%
0%
50%
100%
150%
200%
Cu
mu
lati
ve P
erfo
rman
ce
S&P 500
ACSI Fund
© 2008 CFI Group - 6
The reasons why IFL focuses on quality and satisfaction
1. Identify relevant areas from the customers’, i.e. participants’, perspective
2. Identify the present situation, i.e. the participants’ assessment of IFL as a supplier. This is achieved by:
– Asking questions in relevant areas
– Calculating the scores for the individual questions that comprise the areas, as well as for the areas as a whole
3. Identify key areas that affect participant satisfaction and desired behaviors. This is achieved by:
– Statistical analyses, e.g. factor analyses and structural equations modeling (PLS regression)
– Combining scores and impacts derived from the statistical analysis
4. Initiate actions for improvement
– Maintain current position on local and international market
– Improve position on local and international market
© 2008 CFI Group - 7
Content
– The point of satisfaction
– The CFI Group method
– Describing IFL from the participants’ perspective
– Analyzing results
– Identifying focus areas
– How to leverage the information to achieve IFL’s goals
– The IFL perspective
© 2008 CFI Group - 8
The method captures cause and effect relationships, and identifies the driving factors
Example:
PerceptionGut
feelingAction
Cause and effect
Faculty? Satisfied? Recommend?
© 2008 CFI Group - 9
The model is a conceptual version of the participants’ relationship with IFL
The model includes relevant questions and performance areas (e.g. Image) that describe the participants’ opinion on IFL
Each performance area consists of one or more questions
The questions are weighted together to form the performance area scores, thus the results are not percentages
PSI consists of three separate questions, also weighted together
If a respondent answers less than 2/3 of the questionnaire, the answer is not included in the analysis
If a respondent leaves a question unanswered, it is left blank
Information
Effects
Applicationprocedure
Facilities &accommodation
Program director & program admin.
Faculty's inspiration
Faculty’s pedagogy
Faculty's competence
Literature & materials
Interaction
Participant group
Program structure
Investment
Activities
Personal development
Loyalty
Recommend
PSI
Exam
ple
© 2008 CFI Group - 10
A value is calculated for each performance area on both the left hand side…
IFL Overall
78
88
86
85
83
81
86
77
84
85
76
85
81
0 10 20 30 40 50 60 70 80 90 100
Information
Application procedure
Facilities & accommodation
Program director & program admin.
Faculty's inspiration
Faculty’s pedagogy
Faculty's competence
Literature & materials
Interaction
Participant group
Program structure
Effects
PSI
Exam
ple
© 2008 CFI Group - 11
…and the right hand side of the model
IFL Overall
80
78
81
79
88
0 10 20 30 40 50 60 70 80 90 100
Investment
Activities
Personal development
Loyalty
Recommend
Exam
ple
© 2008 CFI Group - 12
Each performance area consists of one or more questions, weighted together to minimize the error term
Literature &Materials
PSI
Litterature & materials consists of four questions, for example:– The format of the material that was distributed (compendia, hand-
outs etc.) was user friendly– The literature and material used was up-to-date– The literature and material used complimented the teaching
process– The literature and material used will be helpful to me in my work
PSI (Participant Satisfaction Index) is not an average score of all other questions, PSI is an index based on three separate questions:
– How satisfied are you with the program?– To what extent has the program met your expectations?– Imagine a program which is perfect in every sense. How satisfied
are you with the program compared to this ideal program?
It is necessary to have a separate PSI to be able to do the cause and effect analysis
Example:
Literature &Materials
PSI
© 2008 CFI Group - 13
The questions are answered on a scale from 1-10, in the analysis, the answers are converted to a score on a scale from 0-100
In the questionnaire, the questions are answered on a scale from 1-10
“Don’t know” is an option
To make it easier to interpret results and detect differences between segments and/or changes between years, the answers are converted to a score on a scale from 0-100
If the respondent answers “7” on a question, this becomes a score of “67”
Please note that this is a score, not a percentage
If a respondent answers less than 2/3 of the questionnaire, the answer is not included in the analysis
1 02 113 224 335 446 567 678 789 89
10 100
1 02 113 224 335 446 567 678 789 89
10 100
© 2008 CFI Group - 14
Scores can be more or less precise, depending on the method used
Using optimal weights all the available information is used, including the relationship to all other questions and performance areas
Calculating a performance area value based on several questions increases the reliability of the value, decreasing the width of the confidence interval
A scale with ten points allows for more response alternatives and increases the possibility of detecting actual changes
In a “top box” approach only a small part of the information is put to use (typically responses 4 and 5)
Top Box approach: % 4 and 5
Single item, five point scale
Single item, ten point scale
Multiple item scale, equal weights
Multiple item scale,optimal weights
PRECISION: Width of Score Confidence Interval
PO
WE
R:
Ab
ility
to
det
ect
chan
ge
Pred
ictio
n Er
ror
inte
rval
© 2008 CFI Group - 15
Content
– The point of satisfaction
– The CFI Group method
– Describing IFL from the participants’ perspective
– Analyzing results
– Identifying focus areas
– How to leverage the information to achieve IFL’s goals
– The IFL perspective
© 2008 CFI Group - 16
External benchmark indicates performance compared to other industries, but internal benchmark will prove most useful in day-to-day management
Source: SKI 2007
Satisfaction benchmark
8278
71 7168 67 67 65 63 61
0
10
20
30
40
50
60
70
80
90
100
IFL O
vera
ll
US MBA P
rogra
m (F
T Top
30)
Banki
ng, B
2C-m
arke
tM
ortg
ages
, B2C
Insu
rance
com
panie
s, B
2CTel
ecom
/bro
adba
nd
Retai
l cha
ins
College/
univ
ersi
ty e
ducatio
n
Trans
ports
(airl
ines
/trai
n/bus
)Pow
er d
istri
butio
n, B2C
* Full time MBA
*
Exam
ple
© 2008 CFI Group - 17
21%
56%
23%
Up to 35 years old
Between 36 and 45years
Older than 45years old
There is no significant difference in PSI scores between age groups
PSI
80 81 82
0
10
20
30
40
50
60
70
80
90
100
Up to 3
5 ye
ars
old
Betwee
n 36
and 4
5 ye
ars
Older
than
45
year
s old
PSI
PSIscore
Exam
ple
© 2008 CFI Group - 18
42%
58%
Female
Male
PSI
83 80
0
10
20
30
40
50
60
70
80
90
100
Femal
e
Mal
e
There is only a slight difference in PSI scores between genders
PSI
Exam
ple
© 2008 CFI Group - 19
69%
31%
Private sector
Public sector
PSI
87
73
0
10
20
30
40
50
60
70
80
90
100
Privat
e se
ctor
Public s
ecto
r
The private sector is significantly more satisfied
Exam
ple
PSI
© 2008 CFI Group - 20
Identify programs with abnormally low or irregular scores, rather than just focusing on low scores
EGMP-FLHT07
88 (+3)
83 (+6)
88 (+3)
89 (+5)
75 (-12)
93 (+4)
84 (+3)
70 (0)
96 (+9)
88 (+2)
98 (+15)
85 (+7)
86 (+5)
0 20 40 60 80 100
PSI
Information
Application procedure
Facilities & accommodation
Program director & program admin.
Faculty's inspiration
Faculty’s pedagogy
Faculty's competence
Litterature & materials
Interaction
Participant group
Program structure
Effects
(* Difference compared to IFL Overall indicated in parentheses)
Exam
ple
Ask yourself: – Why is this program so much better/worse in
these areas?– Is there some key learning that can be
applied when comparing to other programs?
© 2008 CFI Group - 21
Natural score variations occur between types of questions
When analyzing score levels it is unwise to compare scores across performance areas
Performance scores should be compared with relevant benchmark instead
1 10
© 2008 CFI Group - 22
Compare PSI scores across programs and participant groups
PSI
8284
68 67
86 87
7680
82 8277
86
0
10
20
30
40
50
60
70
80
90
100
IFL O
veral
l 1 2 3 4 5 6 7 8 9 10 11
© 2008 CFI Group - 23
Focus on important performance areas and compare to benchmark
Program structure
7983
6157
83 82
69
7470
81
70
82
0
10
20
30
40
50
60
70
80
90
IFL O
veral
l 1 2 3 4 5 6 7 8 9 10 11
© 2008 CFI Group - 24
Identify programs which deviate from the overall pattern
0 10 20 30 40 50 60 70 80 90 100
Information
Litterature & materials
Faculty's inspiration
Faculty’s pedagogy
Interaction
Effects
Program structure
Application procedure
Participant group
Facilities & accommodation
Program director & program admin.
Faculty's competence
PSI
EXEXVT07 IFL Overall
Performance areas will always have varying levels
Identify “normal” pattern Identify programs that deviate from this
pattern
© 2008 CFI Group - 25
New programs could have uneven performance – it is important to identify weaknesses quickly and take action!
New Program (HTXP 06/07)
68
68
94
84
86
70
63
70
72
75
84
55
74
0 10 20 30 40 50 60 70 80 90 100
PSI
Information
Application procedure
Facilities & accommodation
Program director & program admin.
Faculty's inspiration
Faculty’s pedagogy
Faculty's competence
Litterature & materials
Interaction
Participant group
Program structure
Effects
© 2008 CFI Group - 26
Established programs that run smoothly should have relatively even scores – if not, what is the quality issue?
Program on 50th run (LY 1-07)
87
78
88
87
92
88
88
90
85
91
88
82
91
0 10 20 30 40 50 60 70 80 90 100
PSI
Information
Application procedure
Facilities & accommodation
Program director & program admin.
Faculty's inspiration
Faculty’s pedagogy
Faculty's competence
Litterature & materials
Interaction
Participant group
Program structure
Effects
© 2008 CFI Group - 27
Find patterns and identify underlying reasons
Private vs Public sector
87
90
82
86
93
83
73
82
74
78
86
76
0 10 20 30 40 50 60 70 80 90 100
PSI
Effects
Program structure
Participant group
Interaction
Literature & materials
Private sector Public sector
Exam
ple
Participants from the private sector give notably higher scores
What is the reason (background, prior knowledge, levels of expectation, quality issues on targeted courses)?
If found, can something be done to affect participants from the public sector in the same way?
© 2008 CFI Group - 28
Find changed patterns and identify underlying reasons
Exam
ple Private vs Public sector
94
87
60
87
89
93
85
76
64
92
87
68
72
71
0 10 20 30 40 50 60 70 80 90 100
Faculty's competence
Faculty’s pedagogy
Faculty's inspiration
Program director & programadmin.
Facilities & accommodation
Application procedure
Information
Private sector Public sector
Why is the pattern broken for two performance areas?
Which steps need to be taken to prevent quality issues in the future?
What is the cost of implementing these steps – and what is the cost of refraining from implementation?
© 2008 CFI Group - 29
Content
– The point of satisfaction
– The CFI Group method
– Describing IFL from the participants’ perspective
– Analyzing results
– Identifying focus areas
– How to leverage the information to achieve IFL’s goals
– The IFL perspective
© 2008 CFI Group - 30
Where should improvement efforts be focused?
Resources (e.g. time, money) are always limited
Improvements should be focused on areas that will have a higher impact on PSI and the desired behaviors
These areas are identified in the cause and effect analysis
Information
Effects
Applicationprocedure
Facilities &accommodation
Program director & program admin.
Faculty's inspiration
Faculty’s pedagogy
Faculty's competence
Literature & materials
Interaction
Participant group
Program structure
Investment
Activities
Personal development
Loyalty
Recommend
PSI
© 2008 CFI Group - 31
How to identify the drivers of Satisfaction
The graphs show the impact of Information and Faculty’s competence on Satisfaction– The slope of the line represents the impact– The steeper the line, the higher the impact on Satisfaction
In this example:– If the score for Information increases with five (5) units, the score for PSI will increase with 0,2– If the score for Faculty’s competence increases with five (5) units, the score for PSI will increase with
1,5
The impacts are calculated using PLS-analysis (an advanced type of regression analysis) using PSI as the dependent variable
Scores for Information
Scores for PSI
Low impact
+5
+0,2
Scores for Faculty’s competence
Scores for PSI
High impact
+5
+1,5
Illustration
© 2008 CFI Group - 32
To be able to rely on the results we need a significant amount of answers
The analysis can in theory be carried out with a few answers, but…– The results would be very unreliable as an individual respondent would influence the results
heavily– The results might be due to chance, rather than certainty
As the number of responses increases so does the stability of the results
n=2 n=20 n=100 n=250
Examples
© 2008 CFI Group - 33
Example comparing high impact vs. low impact performance areas
© 2008 CFI Group - 34
Working with the PSI results – prioritizing actions
To prioritize actions, focus on those areas that will have the greatest impact on PSI and recommendation
– Performance areas with high impacts will affect PSI and recommendation more than performance areas with low impacts
– It is easier to improve performance areas with lower scores
The Priority Matrix combines these two aspects:
– Focus primarily on performance areas within the Improve-quadrant – these have higher impact and lower scores
High
Low
Impact on PSI
4. Exploit/Save 2. Improve/Maintain
3. Monitor 1. Improve
1
2
Score
© 2008 CFI Group - 35
Example of a priority matrix
Exam
ple
High
Low
Score
Impact on PSI
4. Exploit/Save 2. Improve/Maintain
Application procedure Faculty's competence
3. Monitor 1. Improve
Information Room and food
© 2008 CFI Group - 36
Content
– The point of satisfaction
– The CFI Group method
– Describing IFL from the participants’ perspective
– Analyzing results
– Identifying focus areas
– How to leverage the information to achieve IFL’s goals
– The IFL perspective
© 2008 CFI Group - 37
Working with PSI results to achieve the goals
PSI results
Get all co-workers involved
Action plansExecution
Follow-up
Planning:Internal
coordination groupRevision
© 2008 CFI Group - 38
How do other companies do it?Example from a real estate firm
A strong vision dictates all actions Actions are value-based and firmly
established on all levels– Strong endorsement from top
management Humor
– Welcome to a better world (landlord)!
Exam
ple
© 2008 CFI Group - 39
Consistency is key
The vision to become Sweden's premier real estate company – and the conviction that this is achieved through customers – guides all actions
The company wants to be number one…– Employer– Landlord– Investment
© 2008 CFI Group - 40
Symbolic and systematic
The customer is always in focus All processes have been…
– Analyzed– Documented– Implemented as a part of daily lives
© 2008 CFI Group - 41
An example taken from a telecom firm
Symbolism – King Customer! Clear, attainable goals Champions are appointed to each area Everybody is involved in quality assurance
and customer delivery (e.g. billing, complaint handling, technicians, sales personnel, PR, HR…)
Endorsed by management– Management actively ensures that the whole
company sees King Customer as important
Exam
ple
© 2008 CFI Group - 42
The ultimate goal is clear – and customer satisfaction is just a stepping stone
The goal with ”Kung Kund” is a larger customer base and increased revenue!
More paying customers
More paying customers
More satisfied customers
More satisfied customers
© 2008 CFI Group - 43
Company culture focuses on the customers
Use of humor to focus on customers, e.g. using known sayings and proverbs, “adapted” to the situation
– For Customer and country– The best with this company is the customer– Don’t wait until tomorrow what you can give
to the customer today– A good customer makes your life longer
Priorities are set straight by management
© 2008 CFI Group - 44
Everybody influences customer satisfaction
HR: potential employees are also customers
Billing: how can we best meet customer needs?
PR: marketing influences customer expectations
Product development: pricing solutions have a great impact on customers
© 2008 CFI Group - 45
Specific actions and deliberate work leads to positive results
Structure is everything– Without proper organization results will not
be achieved
SMART actions and goals– (Specific, Measurable, Attainable, Realistic,
Timely)
Use technology!– E.g. a database to keep track of actions and
that sends out automatic reminders
© 2008 CFI Group - 46
Working with the PSI results – setting goals and increasing scores
When results are low it is easier to increase them and they will have a higher effect on both CSI and Loyalty
CSI
Loyalty
High returns oninvestments in CSI
Low returns oninvestments in CSI
40%
60%
80%
100%
50 100807060 90
Example
© 2008 CFI Group - 47
Working with the PSI results – Positive and hygiene factors
The priority matrix consists of both positive factors of satisfaction as well as hygiene factors. Normally, the positive factors have a high impact on the participants’ satisfaction with IFL. The hygiene factors don’t.
However, hygiene factors have a greater impact on PSI when the scores are low – once the scores have reached the hygiene level the impact on PSI decreases. If IFL would underperform in an area that is considered hygiene it may start to become relatively more important for the overall satisfaction.
Positive factors are qualifiers – you need to maintain a certain level to even be considered. Once this level is achieved increases will lead to higher CSI scores.
Score
Hygienefactors
Positivefactors
CS
I le
ve
l
© 2008 CFI Group - 48
Living, breathing and thinking PSI is a key factor
PSI results
Get all co-workersinvolved
Action plansExecution
Follow-up
Planning:Internal
coordination groupRevision
PSI results
Get all co-workersinvolved
Action plansExecution
Follow-up
Planning:Internal
coordination groupRevision
Set goals!E.g. today PSI is 81 – next year we aim for 84
Include the co-workers in the creation of action plans
Use internal benchmarks and best practice – show success cases
Inspire through workshops and seminars
Integrate with internal systems and processes
Follow up!
© 2008 CFI Group - 49
Focus on key areas
High
Low
Score
Impact on PSI
4. Exploit/Save 2. Improve/Maintain
3. Monitor 1. Improve
© 2008 CFI Group - 50
Based on key areas, plan actions and set targets
Activity/actionPerformance areaTarget score
Score today
DoneResponsible Target date
© 2008 CFI Group - 51
Take the drivers seat and control the process!
PSI results2009!
Get all co-workers involved
Action plansExecution
Follow-up
Planning:Internal
coordination groupRevision
© 2008 CFI Group - 52
Content
– The point of satisfaction
– The CFI Group method
– Describing IFL from the participants’ perspective
– Analyzing results
– Identifying focus areas
– How to leverage the information to achieve IFL’s goals
– The IFL perspective
Managing Participant Satisfaction
Elisabeth LangIntangible Asset Manager, Senior Consultant, CFI Group
+46-8-562 800 00, Box 70373,107 24 Stockholm, Sweden Thursday October 9th, 2008