The current position
• Importance of health and well-being established
• Lots of investment (time, money and action) to promote
and protect health and well-being
• Still lacking good quality evidence on whether and/or how
interventions work (or not)
Why should we bother?
• Quality assurance
• Understanding of intervention
• Impact on staff and service
• Duty of care
• Sharing learning
QUIDS
Checklist
1. Aims of the evaluation
2. Gathering information
3. Formulating key questions
4. Designing the evaluation
5. Project resources
6. Organisational context
7. Communication strategy
8. Reflecting on practice
Evaluating PhysioPlus
• Team keen to demonstrate the broader value of the
intervention for Trust staff
• Sickness absence seen as a key driver/statistic
against which to assess success or value
• General statistics, e.g. number of patients seen and
patient experience, necessary but not sufficient
Evaluation process
• Longitudinal design – repeated assessments
• Surveys and some interviews where necessary
• Strongest design for assessing interventions
FIRST BASELINE
PRE TREATMENT SURVEY
POST SESSION SURVEYS
FOLLOW UP POST
TREATMENT SURVEY
Importance of pain confidence (PC)
PAIN CONFIDENCE
Burnout Sub Scales Low High Significant
Emotional Exhaustion 16.93 14.79 Sig. <0.05
Cynicism 12.07 10.25 Sig. <0.05
Professional Efficacy 32.03 33.52 Sig. <0.05
• Staff with higher levels of pain confidence lower
exhaustion and cynicism and higher professional
confidence
What’s happening: PC over time?
30
32
34
36
38
40
42
44
Baseline P1 P2 P3 P4 P5
Scal
e 0
-6
0
CHRONIC CASES –ADDITIONAL SUPPORT ?
Time of measurement
RECALIBRATION EFFECT
Staff quotes
“Without the level of treatment and support that I received from the therapists, I’m sure it would
have taken me a great deal longer to return to work.”
“I found it gives you a positive approach to actually managing the pain and that acute episode
really, It could have led to a couple of days off sick but it helped to prevent that”
Treatment confidence and anxiety
• Levels of confidence increase over time and across/between sessions
• Levels of treatment anxiety reduce over time and across/between sessions
Manager quotes
“They’re teaching staff how to manage their symptoms. They’re also
treating the symptoms and are reducing pain levels. This enables staff
to be at work rather than off sick.”
“It has kept staff at work and able to do their job effectively. As a result of the
advice and guidance about how to cope or deal with their condition, It has kept them
feeling psychologically very strong. ”
PhysioPlusSheffield Teaching Hospitals
Staff Physiotherapy Service
Service evaluation
David Craig April 2015
Brief history2007 First outline business case written with proposal to start staff physiotherapy
service. The main drivers were:
• concerns over level of staff absence due to musculoskeletal disorders
• recent publication by Dame Carol Black and Dr Steve Boorman
• increasing awareness of the importance of staff wellbeing.
2009 Service started in Sheffield PCT (3000 staff based in community settings
across Sheffield).
2011 Sheffield PCT Provider Services, including PhysioPlus transfer to Sheffield
Teaching Hospitals as part of Transforming Community Services (17,000
staff based mostly based in 2 campuses).
2011 Pilot service started in Obstetrics and Gynaecology Care Group.
2012 Pilot extended to Specialised Medicine Care Group.
2014 Service expanded and open to whole of STH.
Service principles:
• Self referral
• Fully confidential service, not a management reporting service
• Targeted at staff off sick with MSD, but not restricted to this group
• Responsive – target of 48 response time
• Paperless
• Separate from, but working closely with Occupational Health Services.
Evaluation – why?
• Various different patient experience questionnaires.
• Different outcome measures, e.g. EQ-5D, Oswestry.
• Analysis of sickness absence data.
Were not confident that any of these gave a true indication of the ‘range’ value of
the service.
Agreed to work with NHS Employers and Zeal Solutions to develop a
comprehensive evaluation tool, dimensions include:
• managing pain
• degree to which intervention has affected capacity to work
• confidence in clinician
• degree to which intervention has affected work attendance/absence
• burnout
• perceived level of organisational support
• the patient experience.
Evaluation – how?
• Need to know what you want to find out.
• Need a lot of data sets – over 100 matched sets collected for our evaluation.
• Significant implications for clinical team. Patients completed questionnaires prior
to seeing a physiotherapist at the start of treatment, after each treatment session
and three months after discharge.
• Needs to have a longitudinal element three month follow up in this case, an
element of attrition is inevitable.
• Needs to include different sampling questionnaire, semi structured interviews
etc.
• Needs to target different groups – patients, managers, staff working in service.
Evaluation - the benefits of doing it well
• Traditionally staff services are measured solely by their perceived impact on the
overall level of absence in the trust. This is too crude to be of real value, too
many external factors, some known some do not affect absence.
• Gaining a knowledge and understanding of our own service.
• Applying the knowledge gained to promote and attract further investment into the
service
• Informs future development in promotion and management of health and
wellbeing.
Principles of best practice
Design of an effective evaluation
1. Ensure the purpose of the evaluation is determined
2. Establish your evaluation criteria
3. Plan, prepare and document
4. Look for change
5. Long-term impact
Principles of best practice
Context of an effective evaluation
6. Consider the bigger picture
7. Senior management engagement
8. Building an evaluation capability
9. Dual focus on process and outcome
10.Good communication