Replication analysis of the validity and comparability of ... · Replication analysis of the validity and comparability of Patient Safety Indicators (PSI): ... Add case to d en o

Replication analysis

of the validity and comparability

of Patient Safety Indicators (PSI):

the impact of AHRQ exclusions

by Vladimir Stevanovic and Lihan Wei

The OECD HCQI Expert Group Meeting

Paris, 3 June 2010

Background HCQI Expert Group meeting in June 2009

• Concerns were raised that the data may reflect more

coding and registration practices than actual differences

in patient safety

• Several countries expressed reservations about the

publication of PSIs in Health at a Glance 2009 due to

perceived risk of misinterpretation

Background PSI Subgroup meeting in October 2009

• The Secretariat presented a technical analysis on

the impact of the AHRQ exclusions

• The findings implied that the exclusions may have

varying impacts across all indicators apart from the

obstetric ones

• It was proposed further analysis to be undertaken

through a voluntary subsample of countries

Objectives

To improve the comparability of PSIs by:

• Assessing the impact of AHRQ exclusions on the

PSI rates, and

• Exploring whether these exclusions account for any

undesired or increased variation across countries

The scope of this analysis did NOT include exclusions

that are inherent to the concept of an indicator by

their nature (e.g. children, pregnant women)

Patient safety indicators

• PSI05 - Foreign body left in during procedure

• PSI07 - Catheter-related bloodstream infection

• PSI12 - Post-operative pulmonary embolism (PE)

or deep vein thrombosis (DVT)

• PSI13 - Postoperative sepsis

• PSI15 - Accidental puncture or laceration

Countries

• Canada*

• Denmark

• Finland

• Israel

• New Zealand+

• Norway

• Singapore

• Spain

• Sweden

• Switzerland

• United States*

* both POA and non-POA data + previous analysis

Methods

The assessment of impact of each individual

exclusion in the AHRQ algorithm

• Rate ratio of the difference between post- and

pre-exclusion rate and the pre-exclusion rate

• Allows the code criteria to be met at any dg field

• Negative value = the rate-lowering effect

Positive value = the rate-increasing effect

Example PSI05 LOS exclusion

PreExcl

88 / 865,955 = 0.0102

LOS 0 LOS 1+

Excl

27 / 321,953 = 0.0084

PostExcl

61 / 544,002 = 0.0112

Impact = (PostExcl - PreExcl) / PreExcl = (0.0112 - 0.0102) / 0.0102 * 100= +11%

Example PSI05 LOS exclusion

yes

no Excludeno

yes

Add case to

denominator

population

Excludeyes

Count denominator

population and report to

OECD secretariat

Add case to

numerator

population

yes

Case is a surgical or amedical discharge

Case is

assigned to MDC 14 or the PDx

is listed in table M3?

PDx is

identical to the numerator

definition?

no

SDx is

identical to the numerator

definition?Count numerator

population and report to

OECD secretariat

Foreign body left in during procedure

Age=18 y or>18 y?

LOS < 24hours or 0

days?

no

yes Exclude

the impact of

each individual

exclusion

Data collection

Results

• The results indicate that the exclusions within the

AHRQ algorithms have varying impact

• While clinical exclusions are considered inherent

to the construction of PSIs, length of stay and

non-elective admission type exclusions are

believed to introduce bias

PSI12 DVT/PE – length of stay <2 days

29.7

49.5

74.2

30.9

110.3 111.4

19.8

175.7

19.7

66.7

0

20

40

60

80

100

120

140

160

180

200

% im

pact

PSI12 DVT/PE – NZL

2

14 13 13

22

24

22

29

0

5

10

15

20

25

30

35

0 1 2 3 4 5 6 7

Eve

nts

(n

um

)

LOS (days)

LOS excl.

PSI12 DVT/PE – NZL

Initial rate

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0 1 2 3 4 5 6 7

LO

S e

xc

lus

ion

ra

te

LOS (days)

rate

increasing

effect

rate

lowering

effect

PSI13 Postop.sepsis – acute adm.types

-63.0

-17.7

-47.8

-69.4 -67.5 -69.7

-90.9

-54.8

123.6

-36.9 -37.8

-100

-50

0

50

100

150

% im

pact

Non-elective admission types

PSI13 Postop.sepsis – SPA & SWE

Non-elective

admission

Exclusion

rate

Impact

SPAIN 39.5% 1.91 -55%

SWEDEN 62.5% 0.28 124%

PSI13 Postop.sepsis – initial vs. final rate

0.0000

0.0020

0.0040

0.0060

0.0080

0.0100

0.0120

SPA SWE

Initial

Final

Results – cont.

• As a result of multiple exclusions, some indicators

are calculated from very small samples and are

therefore especially sensitive to variations

PSI13 Postoperative sepsis

Numerator sample sizes

Initial 2dx Final sample size

Canada 3,421 479 14.0

Denmark 2,524 78 3.1

Finland 698 92 13.2

Israel 1273 54 4.2

New Zealand 1,141 91 8.0

Norway 930 25 2.7

Singapore 2,865 10 8.0

Spain 12,794 1,038 8.1

Sweden 3,555 1,449 40.8

United States 86,892 4,894 5.6

Total 116,093 8,210 7.1

-100.0

-50.0

0.0

50.0

100.0

150.0

200.0

250.0

300.0

% im

pact

Non-elective admission types Length of stay <4 days

0.0000

1.0000

2.0000

3.0000

4.0000

5.0000

6.0000

rate

per

100

dis

ch

arg

es

Results – cont.

• Comparisons between the patient and discharge

level data show relatively consistent ratios for the

PSI05 - Foreign body left in during procedure

PSI15 - Accidental puncture or laceration

across the following rates:

- patient (qualifying event in any dg field based on patient level data),

- discharge (qualifying event in any dg field based on discharge data)

- discharge sdx (qualifying event in secondary dg field based on

discharge data as calculated from the AHRQ algorithms)

PSI05 Foreign body left in during proc

0.00000

0.00002

0.00004

0.00006

0.00008

0.00010

0.00012

0.00014

0.00016

0.00018

Canada Denmark Finland Israel New Zealand

Singapore Spain Sweden United States

patient discharge discharge sdx

PSI15 Accidental puncture or laceration

0.0000

0.0010

0.0020

0.0030

0.0040

0.0050

0.0060

0.0070

Canada Denmark Finland Israel New Zealand

Spain Sweden United States

patient discharge discharge sdx

Present on admission flag

• Data provided by Canada and the United States

show considerable differences in terms of percent

change due to POA coding for PSI12 DVT/PE and

PSI13 Postoperative sepsis

• The effect is larger in general for the US than

Canada across PSIs

Canada – POA vs. non-POA

United States – POA vs. non-POA

PSI Subgroup’s recommendations

Length of stay

• The AHRQ algorithms are built on several exclusions that are

intended to affect the bias in comparison across US states

• Patient safety indicators with short length of stay (< 24h)

exclusion have an effect of reducing bias, while longer length

of stay exclusions tend to increase bias

Recommendation 1:

Collecting data for events by day breakdowns for length

of stay may give greater understanding of the effect of this

exclusion and inform possible revisions

PSI Subgroup’s recommendations

Non-elective admission type

• The post-operative sepsis indicator shows large and varying

effects from the non-elective admission type exclusion

• This is due to varying definitions and coding practices across

countries

Recommendation 2:

Collect additional information to understand how acute and

elective admissions are defined in each country and consider

possible revision of this exclusion

PSI Subgroup’s recommendations

Patient level data

• Patient safety indicators rely heavily on the quality of principal

and secondary diagnoses data coded in hospital records

• Discharge level data does not provide the necessary information

to detect under-reporting, hence those countries with UPI may be

able to provide additional data for patient-level events

Recommendation 3:

Collect additional information on qualifying events in PDX

field only based on UPI

PSI Subgroup’s recommendations

Coding practice

• Coding may be performed by various healthcare professionals

or dedicated clinical coders

• Clinical coding practice could be also affected if the medical

records determine financial reimbursement

Recommendation 4:

Collect additional information on coding and registry

practices in each country

Members of the HCQI Expert Group are invited to:

• Comment on the findings of this analysis

• Decide on whether a limited number of PSIs

- PSI05 Foreign body left in during procedure

- PSI15 Accidental puncture or laceration

- PSI18 Obstetric trauma due to vaginal delivery with instrument

- PSI19 Obstetric trauma due to vaginal delivery without instrument

is mature enough for the publication in the OECD Quality of

Care document in preparation for the Ministerial meeting in

October 2010

• Decide on whether the collection of additional data in the year

2010/2011 is warranted to inform the future development work