How to robustly evaluate health and wellbeing interventions

@

How to robustly evaluate health
and wellbeing interventions

Welcome

23rd April 2015
Dr Tony Zarola

The current position

• Importance of health and well-being established

• Lots of investment (time, money and action) to promote
and protect health and well-being

• Still lacking good quality evidence on whether and/or how
interventions work (or not)

Why should we bother?

• Quality assurance QUIDS
• Understanding of intervention

• Impact on staff and service
• Duty of care

• Sharing learning

Challenge: Time

Challenge: Criteria

Challenge: Process

Challenge: Technical

Checklist

1. Aims of the evaluation
2. Gathering information
3. Formulating key questions
4. Designing the evaluation
5. Project resources
6. Organisational context
7. Communication strategy
8. Reflecting on practice

Evaluating PhysioPlus

• Team keen to demonstrate the broader value of the
intervention for Trust staff

• Sickness absence seen as a key driver/statistic
against which to assess success or value

• General statistics, e.g. number of patients seen and
patient experience, necessary but not sufficient

Evaluation process

• Longitudinal design – repeated assessments

FIRST POST SESSION FOLLOW UP
BASELINE SURVEYS POST
PRE TREATMENT
SURVEY TREATMENT
SURVEY

• Surveys and some interviews where necessary

• Strongest design for assessing interventions

What’s happening over time?

14 Trend 1
12 Trend 2
10 Trend 3

8 Time 2 Time 3
6
4
2
0

Time 1

Sample of evaluation criteria

Importance of pain confidence (PC)

• Staff with higher levels of pain confidence lower
exhaustion and cynicism and higher professional
confidence

PAIN CONFIDENCE

Burnout Sub Scales Low High Significant
Emotional Exhaustion Sig. <0.05
Cynicism 16.93 14.79 Sig. <0.05
Professional Efficacy Sig. <0.05
12.07 10.25

32.03 33.52

What’s happening: PC over time?

44 CHRONIC CASES –
42 ADDITIONAL SUPPORT ?

Scale 0 - 60 40 RECALIBRATION EFFECT
38

36

34

32

30 P1 P2 P3 P4 P5
Baseline Time of measurement

Overall

PhysioPlus Prevented The Need For Time Off

No
21%

Yes
79%

Staff quotes

“I found it gives you a positive “Without the level of treatment
approach to actually managing and support that I received from
the pain and that acute episode the therapists, I’m sure it would

really, It could have led to a have taken me a great deal
couple of days off sick but it longer to return to work.”
helped to prevent that”

Treatment confidence and anxiety

• Levels of confidence increase over time and
across/between sessions

• Levels of treatment anxiety reduce over time
and across/between sessions

Manager quotes

“It has kept staff at work and able to do
their job effectively. As a result of the

advice and guidance about how to cope or
deal with their condition, It has kept them
feeling psychologically very strong. ”

“They’re teaching staff how to
manage their symptoms. They’re also

treating the symptoms and are
reducing pain levels. This enables staff

to be at work rather than off sick.”

PhysioPlus

Sheffield Teaching Hospitals
Staff Physiotherapy Service

Service evaluation

David Craig April 2015

Brief history

2007 First outline business case written with proposal to start staff physiotherapy
service. The main drivers were:
• concerns over level of staff absence due to musculoskeletal disorders
• recent publication by Dame Carol Black and Dr Steve Boorman
• increasing awareness of the importance of staff wellbeing.

2009 Service started in Sheffield PCT (3000 staff based in community settings
across Sheffield).

2011 Sheffield PCT Provider Services, including PhysioPlus transfer to Sheffield
Teaching Hospitals as part of Transforming Community Services (17,000
staff based mostly based in 2 campuses).

2011 Pilot service started in Obstetrics and Gynaecology Care Group.

2012 Pilot extended to Specialised Medicine Care Group.

2014 Service expanded and open to whole of STH.

Service principles:

• Self referral
• Fully confidential service, not a management reporting service
• Targeted at staff off sick with MSD, but not restricted to this group
• Responsive – target of 48 response time
• Paperless
• Separate from, but working closely with Occupational Health Services.

Evaluation – why?

• Various different patient experience questionnaires.
• Different outcome measures, e.g. EQ-5D, Oswestry.
• Analysis of sickness absence data.

Were not confident that any of these gave a true indication of the ‘range’ value of
the service.

Agreed to work with NHS Employers and Zeal Solutions to develop a
comprehensive evaluation tool, dimensions include:

• managing pain
• degree to which intervention has affected capacity to work
• confidence in clinician
• degree to which intervention has affected work attendance/absence
• burnout
• perceived level of organisational support
• the patient experience.

Evaluation – how?

• Need to know what you want to find out.

• Need a lot of data sets – over 100 matched sets collected for our evaluation.

• Significant implications for clinical team. Patients completed questionnaires prior
to seeing a physiotherapist at the start of treatment, after each treatment session
and three months after discharge.

• Needs to have a longitudinal element three month follow up in this case, an
element of attrition is inevitable.

• Needs to include different sampling questionnaire, semi structured interviews
etc.

• Needs to target different groups – patients, managers, staff working in service.

Evaluation - the benefits of doing it well

• Traditionally staff services are measured solely by their perceived impact on the
overall level of absence in the trust. This is too crude to be of real value, too
many external factors, some known some do not affect absence.

• Gaining a knowledge and understanding of our own service.

• Applying the knowledge gained to promote and attract further investment into the
service

• Informs future development in promotion and management of health and
wellbeing.

Principles of best practice

Design of an effective evaluation

1. Ensure the purpose of the evaluation is determined
2. Establish your evaluation criteria
3. Plan, prepare and document
4. Look for change
5. Long-term impact

Principles of best practice

Context of an effective evaluation

6. Consider the bigger picture
7. Senior management engagement
8. Building an evaluation capability
9. Dual focus on process and outcome
10. Good communication

Be realistic

Right tools for the job