Validation of HAI-Net ICU data. Jacqui Reilly (UK)

The case for validation in ICU surveillance

Professor Jacqui Reilly Health Protection Scotland

UK

Why does it matter? 3Cs: Consistency, comparisons and

confidence Low sensitivity (false negatives, or underreporting) of HAIs is a

frequently encountered problem in HAI surveillance systems. Low specificity (false positives, or over reporting) is usually less of a

problem Both may be related to one or more of following factors:

• Difficulty in confirming the case definition of an infection if signs and symptoms were not well documented in the patient’s records

• If diagnostic tests included in the case definition of a particular HAI type were not done

• Non compliance with the definition of the key term ‘healthcare-associated’: even if the case definition of an infection is matched due to cultural or financial/ political incentives and disincentives at a hospital or country level

The case for validation in multi-country ICU surveillance

In order to investigate variation between countries the first question to ask is: is it the data? Validity? Reliability?

Reference: European Centre for Disease Prevention and Control. Annual Epidemiological Report 2013. Reporting on 2011 surveillance data and 2012 epidemic intelligence data. Stockholm: ECDC; 2014

ICU HAI surveillance validation

Validation findings by HAI type

Reference: http://www.cdph.ca.gov/programs/hai/Documents/BuildingConfidenceInReportedHAIDataSuccessAndChallengesFromState-basedValidationEffortsInCAandBeyond102012.pdf

Reasons for errors in reporting

Reference: http://www.cdph.ca.gov/programs/hai/Documents/BuildingConfidenceInReportedHAIDataSuccessAndChallengesFromState-basedValidationEffortsInCAandBeyond102012.pdf

Validity of automated surveillance-ICU

Manual ward surveillance (MS) and electronic surveillance (ES) were performed

ES was found to be more effective than MS

Validity of denominator data

1988 ICU patient charts from 23 hospitals reviewed by DPH external team 74% of hospitals collected data manually Over reporting of 300 PD and 200 CLD PD manual collection methods were more accurate than

electronic methods (P < .01) For central line days, there was no significant difference in

collection method (P > .05)

Other potential reasons for errors.....

– national targets with financial penalties – the fear of creating a negative image of clinical areas

or hospitals – lack of diagnostic testing and strict case definitions in

the protocol – the consequences of these “underreporting is

probable,” “there will be less cases” and “the most common consequence is that some HAI will not have met the criteria”

Ref: Price L et al (2014) A Cross-Sectional Survey of the acceptability of data collection processes for

validation of an European Point Prevalence Study of Healthcare-Associated Infections and Antimicrobial Use (ECDC Pilot study of PPS validation)

.

Summary

Validation is a key component of surveillance for comparisons, consistency and confidence Without it we do not know the true incidence of HAI in

ICU Without it we cannot investigate reasons for variation

in HAI incidence between hospitals and/ or countries

Knowing the true incidence of HAI makes the case for infection prevention and control measures and enables improvement in ICU

Key issues for discussion today

European surveillance of Healthcare-Associated Infections in intensive care units (HAI-Net ICU): Validation of ICU surveillance data

Carl Suetens Surveillance and Response Support Unit European Centre for Disease Prevention and Control

HAI validation in ECDC PPS

0102030405060708090

100

0 2 4 6 8 10

Primary PPS HAI%

Valid

atio

n - S

ensi

tivity

(%)

Validation in surveillance vs PPS

• Validation is crucial for reliable burden estimates and

interpretation of inter-country variations

• Unlike validation of PPS data, validation of surveillance data needs to be performed after the primary surveillance (retrospective surveillance). (hospital staff to prepare patient files of the selected surveillance period).

• Blind data collection: the validation team member(s) is/are not allowed looking at the primary ICU surveillance forms during the data collection (except for identifying the patient number in the primary surveillance database).

Selection

• Selection of intensive care units: Validated ICUs should be selected randomly from the list of ICUs participating to the primary ICU surveillance, using systematic sampling after sorting the ICU list by number of patients included in the surveillance. For each selected hospital, select the next one as reserve hospital. Should be proportional to N of pts in surveillance

Selection of ICUs: include all ICUs included in the surveillance Selection of surveillance period: depends on the number of patients

included per surveillance period; from 2009 to 2011, an ICU contributed on average 155 patients (median 126 patients) per surveillance-year and 21 patients (median 18 patients) per surveillance-month.

Selection of patients: – include all patients staying more than 2 days in the selected ICUs, at

least until the required number of validation records per hospital is obtained.

– Random selection of patients (only possible if standard protocol is followed)/select all HAI pos – Random selection of HAI negatives

Variation of the 95% confidence interval around a sensitivity of 80% according to the number of patients included in the validation sample

40%

50%

60%

70%

80%

90%

100%

250 500 750 1000 1250 1500 1750 2000

Sens

itivi

ty

SeLL (Pr 7%)UL (Pr 7%)LL (Pr 2%)UL (Pr 2%)

Validation of HAI-Net ICU data

Ideally: 750 patients, 30 ICUs (or all if less) Pragmatic solution:

– Min 250 patients, 5 ICUs Support contracts with ECDC (PPS: 10 000 EUR per contract) Interrater reliability of national validation team members Minimum data:

– Validation of Infection data – Additional validation data: validation method, primary

patient ID, reason for discordance (if any) – Optional:

Denominator data (exhaustiveness)

Data forms: infection data

Patient Counter: Date of admission in ICU: ___ / ___ / _____

Age in years: ____ yrs Gender: M F UNK Date of ICU discharge: ___ / ___ / _____

Patient ICU outcome: O discharged alive O death, HAI definitely contributed to death O death, HAI possibly contributed to death O death, no relation to HAI O death, relationship to HAI unknown

Case definition codeRelevant device in situ before onset*Date of onset**

BSI: source of BSI***

Micro-organism 1

Micro-organism 2

Micro-organism 3

*** C-CVC, C-PER, C-ART, S-PUL, S-UTI, S-DIG, S-SSI, S-SST, S-OTH, UNK

European Surveillance of ICU-acquired infectionsHAI and AMR form, light protocol

*relevant device use (intubation for PN, CVC for BSI, urinary catheter for UTI) in 48 hours before onset of infection (even intermittent use), 7 days for UTI **Only for infections not present/active at admission

MO-codeMO-code MO-code

___ / ___ / ______

ICU-acquired infections

HAI 1 HAI 2 HAI 3

O Yes O No O Unknown

O Yes O No O Unknown

___ / ___ / ______

O Yes O No O Unknown

___ / ___ / ______

Additional validation data

ICU level: validation survey date, protocol primary surveillance, method for selection of patients

note: if the primary surveillance is LIGHT, all patients should be selected Patient level: Primary Patient Counter (in primary surveillance) VT results checked with primary PPS results after data collection? O No O Yes IF YES: Discordant results discussed: O No O Yes O NA VT decision changed: O No O Yes O NA Reasons for discordance / VT comments for this patient/HAI: (Free text)

HAI-Net ICU validation protocol part of Helicswin.Net

Discussion…

National experiences Feasibility of validation and financial support Validation of HAI data, including mortality data

Strategies for extension of HAI surveillance “Extension”:

– Increase N of participating countries – Increase N of participating ICUs – Increase duration of surveillance in ICUs – Validation of ICU surveillance

Tools:

– Light protocol – Free hospital software (Helicswin.Net) – Infection prevention indicators increase added value – Financial support for validation