African Partnerships for Patient Safety (APPS): APPS ... · The initial raw framework Partnership strength: ... conceptual framework and a data capture tool created. A number of challenges

African Partnerships for Patient Safety (APPS): APPS Evaluation Synthesis

Webinar 6 of 6

APPS Webinar Series

Julie Storr

March 6, 2014

Review of Webinar 5

1. Analyzed the critical importance of

community and patient engagement for

patient safety improvement;

2. Described an overview of the ACE

approach and associated

“Implementation Pack”;

3. Outlined the current collaborative

activities on patient and community

engagement.

http://www.who.int/patientsafety/implementation/apps/webinars/en/index1.html

Webinar 6 - Objectives

1. Review the APPS Approach to Evaluation

2. Outline key findings of the summative evaluation

(including reflections on all learning to date)

3. Examine transferable lessons on utilizing the partnership-

based approach, patient safety improvement and national

spread of patient safety change

INTRODUCTION The journey to evaluation – a review

Source: Jha AK, Larizgoitia I, Audera-Lopez C, et al.

BMJ Qual Saf 2013; 22:809-815

"Using a conservative

approach, we estimated that

there are at least 43 million

injuries each year due to

medical care, and that nearly

23 million DALYs are lost as a

consequence."

Magnitude of the problem

Source: Jha AK, Larizgoitia I, Audera-Lopez C, et al.

BMJ Qual Saf 2013; 22:809-815

"Using a conservative

approach, we estimated that

there are at least 43 million

injuries each year due to

medical care, and that nearly

23 million DALYs are lost as a

consequence."

Magnitude of the problem

Q. How do we demonstrate impact of APPS ?

Program Theory of Change

1. North-South hospital-to-hospital partnerships add value

to conventional approaches to improve the safety of

healthcare

2. They facilitate improvement via shared learning and the

opportunity to co-develop approaches and resources

3. Local, hospital activity drives action

4. Supported by parallel advocacy and engagement at the

national and regional policy level

Demonstrating impact – recap on the three APPS objectives

1.

Pati

en

t safe

ty i

mp

rove

men

t

2. Partnership strength

- Strong

partnerships

- Improved

patient safety

- Spread

• Replicable

• Sustainable

• Implementation

driving policy

3. SPREAD

Program theory of

change

Purpose of APPS evaluation

• Ultimately - demonstrate

impact (program theory

of change)

• And:

• Motivate and stimulate

• Drive improvement

• Aid sustainability

Methods of evaluation

• 2009: initial framework

structured around the

three core objectives

• African-based and

European-based clinicians

and experts in patient

safety closely involved in

developing the

evaluation framework, in

a multi-step process.

The initial raw framework

Partnership strength:

• Process defined six domains of partnership strength,

each with measurable sub-domains (questionnaire).

Hospital patient safety improvement:

• Six indicators from a short-list of 22 based on relevance,

sensitivity to intervention and feasibility of measurement.

Spread:

• 20 measures of spread proposed - refined into a two-part

conceptual framework and a data capture tool created.

A number of challenges

• APPS interventions involve socio-adaptive strategies

• Engaging local leaders

• Securing commitment

• Implementing tools and resources

• Evaluation, like the program itself, has evolved over its

lifetime and undergone iterative refinement (planning,

learning and doing – simultaneously!)

• A fine balance between rigorous evaluation design and

optimization of program interventions Informed by Shojania K G (2013) Conventional evaluations of improvement

interventions: more trials or just more tribulations? BMJ Quality and Safety 22:

881-884

SUMMATIVE EVALUATION

Key findings from the first wave

• Two-stage evaluation

• Synthesize evidence on

impact, focused on first

wave

• Summarize barriers

• Explore how partnerships

contribute to continuous

improvement

• Mixed methods evaluation • http://www.who.int/patientsafety/implementation/apps/

events/evaluation-briefing-apps.pdf?ua=1

Situational analysis synthesis

Patient safety partnership findings

Patient safety improvement

Patient safety spread

Summary - 1 Partnerships

■ A pooled knowledge resource, enabling bidirectional flow

of expertise and solutions. They enhance individual and

institutional capacity and leadership development.

Patient safety improvement

■ The partnership approach directly impacts on knowledge,

understanding and behavior aiding development of patient

safety skills and expertise

.

Summary - 2

Spread

■ Partnerships act as a vehicle for advocacy and are

beginning to influence policy-level action. Community

engagement is a catalyst for spread

Challenges

• Resource constraints and high employee turnover –

impact morale and motivation

• Leadership capacity, teamwork and succession planning

present a barrier to success

• Communication channels impact partnership effectiveness

CURRENT APPROACH TO EVALUATION

A new framework and approach

The refined APPS Framework

• First wave lessons learned influenced a

refined Second Wave framework

• Addressed program and participatory

(partnership-specific) evaluation - the two

being complementary

• In summary – a simplified approach was

developed to take account of context

Programme Evaluation Framework

Evolution of the current approach to evaluation

Evolution

Programme Evaluation Framework

1. Situational Analysis

2. Partnership Strength

3. Patient Safety Spread

4. Hand Hygiene Self Assessment Framework

Participatory (partnership-specific) Evaluation Framework

The 6 step cycle

• Partnership focus and

ownership

• Broad range of local

stakeholders participate

• Evaluation developed by

partners based on Partnership

Plan

• Emphasis on learning

SUMMATIVE EVALUATION

Key findings from second wave program evaluation

Situational Analysis – 100 pieces of data

Example results – Beira, Mozambique (Ipswich, UK, - Beira partnership)

The Situational Analysis – a stimulus for improvement

Partnership Strength Evaluation Questions

Emerging findings - 1

Example 1: Zambia (Ndola) – England (Guys and St

Thomas’s)

• Common vision

• Decision making

• Respect

• Use of skills

• Celebrate success

Strongly agree at

baseline & repeat

(both partners)

Emerging findings - 2

Example 1: Zambia (Ndola) – England (Guys and St

Thomas’s)

• Up to date information

• Effective communication

• Institutional ownership

• Stakeholder engagement

Shift from agree to

strongly agree,

baseline and repeat

(both partners)

Key benefits of partnership work (Ndola-Guys)

Baseline:

• Opportunity to review medical practice [Ndola]

• Highlights resources for implementation [Ndola]

• Peer review of our service from the outside [Ndola]

• Using resources more effectively [Guys]

• Development of innovation in training [Guys]

Repeat:

• Access to resources [Ndola]

• Access to training [Ndola]

• How to use resources, human and physical - much better [Guys]

• Joint problem solving - both organisations have benefited - we share so many

problems such as challenges improving hand hygiene [Guys]

• Cross cultural learning [Guys]

Additional comments for learning

“..this has matured, the

group has got bigger

and pulled in people

with appropriate skills

across Zambia and UK”

On use of skills

(Guys & Ndola)

“Very positive -

partnership plan has

been reviewed and

both sides pleased with

rapid progress. The

partnership has

reached a level of

maturity.”

On a common

vision

“Good discussions with

ideas coming from both

sides and decisions

made jointly…[we]

have a regular monthly

steering group meeting

across the

partnerships.”

On decision making

Patient Safety Spread Evaluation Questions

Emerging findings Example 1: Ghana (KATH),

England (St Georges)

• Patient safety champions

• Patient safety networks

• National events

• National hospital spread

• Connected with Ministry of Health

Involvement in spread events

• Patient safety campaign

• Patient safety publications

• Securement of funding

• Diaspora support

Yes (baseline), Yes (repeat)

No (baseline), Yes (repeat)

No (baseline), No (repeat)

Yes (baseline), No (repeat)

Spread comments (summary)

“This is a timing issue -

as we continue to

institutionalize these

processes we can at

some point bring in

other figures to

campaign”

On campaigning

“Our IPC lead in close

contact with Ministry of

Health”

On national

engagement

“We applied for THET

medium grants for

biomedical

engineering”

On funding

Hand Hygiene Self Assessment Framework

• A validated tool to identify

the level a facility occupies

on a hand hygiene

improvement continuum

• 5 parts – addressing each

component of the WHO

Multimodal Improvement

Stratgey

How it works

Example results – KATH, Ghana (St Georges, UK – KATH partnership)

Baseline: 187.5

Repeat: 327.5

Example results – Beira, Mozambique (Ipswich, UK – Beira partnership)

Baseline: 112

Repeat: 310

TRANSFERABLE LESSONS

From the second wave partners

Activity and impact (1)

• Cross-partnership newsletter to

support advocacy

• Awareness raising (hand

hygiene) to build skills

• Capacity building – training of

workers in manufacture of WHO

hand sanitizer

• Empowerment of staff through

implementation of WHO

Surgical Checklist

• National recognition (Africa) as

patient safety leaders

Activity and impact (2)

•Strengthen structures and processes

establishment of audit programme and

baseline epidemiology for healthcare

associated infection

•Train the trainer –

use a cascade model for e.g. hand hygiene

improvement

Key challenges

•Inability to source raw materials/hardware for e.g. hand

hygiene consumables

•Human resource constraints

•Initial non-existence of national policies

•Partnership communication mechanisms (internet,

telephone)

Snapshot of advice from the second wave (1)

“Accept that progress

will be slow and avoid

being over-ambitious in

initial planning”

On speed of

progress

“Focus on one or two

priorities that are

agreed upon and see

them through to the

end”

On prioritizing

action

“Understand that

resources may not just

be limited but may be

non existent”

On resources

Snapshot of advice from the second wave (2)

• “Surgical checklist

saves lives”

• “Surgical checklist

aids surveillance of

surgical infection”

On why this is

important

“The realization that

many changes can be

achieved with little/no

financial burden has

been a great boost for

change”

On resources

• “The surgical

checklist was

adapted to meet the

requirements of local

teams”

On context

Snapshot of advice from the second wave (3)

• “Support from

hospital

management”

(African hospital)

• “Exchange visits”

• “Local enthusiasm”

On what catalyzes

change

“Train administrative

managers to gain

support”

On how to influence

• “Try and meet and

involve as many

people as possible”

• “Train champions”

On winning hearts

and minds

SUMMARY RECOMMENDATIONS

Based on all findings to date

Recommendations - POLICY

1. Build capacity to leverage action on patient safety at the

WHO Regional level;

2. Spread the word - disseminate findings from evaluation

to ministries of health in Africa & key organizations

involved in patient safety;

3. Advocate for change – demonstrate the power that

partnerships bring – the sum is greater than its parts.

Recommendations - PARTNERSHIP

1. Networks – use WHO APPS community, webinars,

resources to address knowledge gaps highlighted

through the evaluation, for learning & to promote south-

south collaboration;

2. Resources/Tools - review and simplify APPS resources;

3. Technical expertise – broaden the pool of technical

expertise available e.g. engineers, behaviorists,

anthropologists, resource mobilization experts;

4. POPS – strengthen role of WHO Private Organization’s

for Patient Safety.

Recommendations - STAKEHOLDERS

1. Strengthen existing collaborations:

1. to promote the importance of funding future improvement work that

builds on and consolidates APPS;

2. Work with THET-Engineers without borders collaboration to

address issues around maintenance and repair of patient safety

related equipment;

2. Patient and community engagement - strengthen

patient and community engagement through active

collaboration with relevant organizations.

WHAT NEXT? Analysis, finalization, dissemination

Regional commitment – Yaounde Ministerial Support 2008

On-the-ground implementation and evaluation – programmatic and participatory

Influencing National Policy & spread – Harare 2013

Implementation

driving policy

Regional commitment – Yaounde Ministerial Support 2008

On-the-ground implementation and evaluation – programmatic and participatory

Influencing National Policy & spread – Harare 2013, West Africa 2014

In summary -

evaluation plays a

critical role in what

happens next

This concludes the

current APPS

Webinar Series –

please share

relevant recordings

with colleagues.

And watch this

space for a new

series later in

2014.

THANK YOU

Please visit our website to find out more about

the programme and access our resources

www.who.int/patientsafety/implementation/apps