Controlling Control Charts Interpreting p -values Intermediate Statistics for ICPs

Controlling Control Charts Interpreting p-valuesIntermediate Statistics for ICPs

Ona Montgomery RN, BSN, MSHA, CICAnne Denison, RN, BSN, MSTexas Society of Infection Control PractitionersOctober 2006

Objectives• Discuss the basic principles of

epidemiology as they apply to monitoring of infections in a health care setting.

• Differentiate between common cause variation and special cause variation.

• Interpret statistical significance through the use of control charts and p-values.

• Relate the use of statistical analysis to action plans and intervention strategies.

Epidemiology Basic Principles

Why Epidemiology?

• Systematic way to look at health problems

• The focus is on defining a problem in order to focus on prevention– Understand the cause of disease– Plan interventions– Evaluate preventive measures

• Applies to diverse situations, flexible

Epidemiology Key Points

• ‘distribution’ – frequency of disease in a specific population

• ‘determinants’ – factors or events associated with changes in health outcomes

• ‘population’ – a group of people rather than an individual

Epidemiological Perspective• Prevention of osteoporotic hip

fracture– Medical / Clinical model

• Target persons at risk based on low BMD• Intervention: pharmacotherapy

– Epidemiological model• Target populations at risk• Intervention: broad based changes in

health behaviors of populations,

exercise, smoking

Infectious Disease Process

• The chain of infection– Causative agent– Reservoir– Portal of exit– Mode of transmission– Portal of entry– Susceptible host

Why determine causality?

• Identify points where disease process can be interrupted

• Develop prevention and control efforts that decrease the outcome measured

• Identify the natural history of disease

Numerators and Denominators

Numerator - the outcome or process you will “count”

Denominator - “out of how many”

Must be clearly defined (and written down)

Case Definitions• Internally created

– Especially useful in outbreak situation where the organism has not been identified – Eg. Gastroenteritis or respiratory illness in a nursing home

• Surveillance definitions ≠ clinical diagnoses

• Clearly define both the numerator AND denominator

• Choose carefully, difficult to modify mid-stream

Case Definition• National Nosocomial Infection Surveillance (NNIS)• http://www.cdc.gov/ncidod/dhqp/nnis_pubs.html• Standardized criteria and case definitions

– Clinical signs and symptoms– Laboratory results– Physician actions/diagnoses

• Applies to inpatient hospitals• McGeer et al.

– Long term care setting– Rehabilitation setting

CDC Acute Care Definition: Symptomatic Urinary Tract Infection

Criterion 1: Patient must have

– 1 of the following: • fever, urgency, frequency, dysuria or

suprapubic tenderness - AND -– Positive urine culture with > 100,000

col/ml with < 2 species

UTI definition continued

Criterion 2: Patient must have– 2 of the following:

• fever, urgency, frequency, dysuria or suprapubic tenderness

- AND -– 1 of the following:

• positive dipstick test • pyuria (>10 WBC/cc) • organisms seen on gram stain of unspun

urine• 2 urine cultures, same organism, > 10,000

col/ml, in non voided specimens• Physician’s diagnosis/ initiation of

antimicrobial therapy

Device Associated Rates

• Numerator = events• Denominator = device days

– Catheter associated UTIs per 1000 Foley days

• 4 UTIs / 235 days x 1000 = 17 CUTI per 1000 Foley days

• CL-BSI per 1000 CL days• VAP per 1000 ventilator days

http://www.cdc.gov/ncidod/hip/nhsn/members/PSProtocolsMay06.pdf

Apply Risk Stratification Methods

Risk stratification simply means subdividing (stratifying) your surveillance population into groups at similar levels of infection risk prior to performing any analyses or comparisons.

To ensure comparing “apples to apples”

CDC NNIS Risk Index for SSI Surveillance

Patient-specific Risk Score Total 0-3 points

Wound class class III or IV 1 point

ASA score 3, 4, 5 1 point

Duration of surgery > cutpoint 1 point

SSI – Wound Class vs NNIS Class

Wound Class All NNIS 0 NNIS 1 NNIS 2 NNIS 3

Clean 2.1% 1.0% 2.3% 5.4% N/A Cl /Contam 3.3% 2.1% 4.0% 9.5% N/AContaminated 6.4% N/A 3.4% 6.8% 13.2%Dirty infected 7.1% N/A 3.1% 8.1% 12.8%

All 2.8% 1.5% 2.9% 6.8% 13.0%

NNIS. CDC. Am J Infect Control. 2001;29:404-421.

CENTERS FOR DISEASE CONTROL

AND PREVENTION

Determining the NNIS Risk IndexCategory in 3 Patients

Determining the NNIS Risk IndexCategory in 3 Patients

Elements of theNNIS Risk Index

Operation >t hours

Wound class

ASA class

NNIS Risk Category

Patient 1

Yes

Dirty

4

Patient 2

No

Clean

2

Patient 3

Yes

Clean-contaminated

2

Infection Rate And Timing of Prophylactic Antibiotics

0

0.5

1

1.5

2

2.5

3

3.5

4

<3 -2 -1 0 1 2 3 4 5

14/369

5/699

5/1009

2/180

1/81

1/41 1/47

14/441

Classen DC et al, NEJM, 1992

CDC NNIS Risk Stratification for High Risk Nursery (HRN)

Surveillance

Stratification by Birthweight Categories:

• </= 1000 grams

• 1001-1500 grams

• 1501-2500 grams

• >2500 grams

Interpretation of this surveillance data

• Compare to NISS report risk class by strata

• example

On the UP side, you are the healthiest patient in ICU

BSI Rates in MICU and SICU

0

2

4

6

8

10

12

14

16

18

MICUNNIS MICU MedianSICUNNIS SICU Median

See: Am J Infect Control 2002;30:458-75.

NNIS 90th percentile

NNIS 10th percentile

BSI Rates in MICU and SICU

0

2

4

6

8

10

12

14

16

18

MICUNNIS MICU MedianSICUNNIS SICU Median

See: Am J Infect Control 2002;30:458-75.

NNIS 90th percentile

NNIS 10th percentile

Surgical site rate analysis

EXAMPLE OF CALCULATING RISK-ADJUSTED RATES

Colon surgery

# Risk Factors #SSI #Operations Rate %

0 2 48 4.17%1 5 77 6.49%2 4 39 10.26%3 1 5 20.00%TOTAL 12 169 --------

Pro

cedu

re

Risk In

dex

Categ

ory

10% 25% 50% 75% 90%

Ho

spital X

YZ

Spinal Fusion 0 0 0 0.7 1.4 2.5 1.2

Spinal Fusion 1 0 0.8 2.2 3.5 4.7 2.6

Spinal Fusion 2,3 0 2.3 4.8 7.3 10.2 18.8

• Ratios• Proportions• Crude Rates• Adjusted

Rates• Incidence• Prevalence• Attack Rates• Mean • Median

Control Chart Theory

• Looking at a system / population

• Individual measurements are unpredictable

• BUT, if all observations are from a stable common system, as an aggregate they will follow a predictable pattern of distribution

Gaussian Distribution Function‘Normal curve’

Normal pattern of common cause variation

Standard Deviations

Variation

Types of variation

• Common cause variation– Part of the natural process– Always present– Partially unknown– Difficult to control

Types of variation

• Special cause variation– Larger variation– Special or non-typical event– Easier to pinpoint in time– Sentinel event

Sources of Variation

• People

• Machines

• Materials

• Methods

• Bias

Both kinds of variation are important in a health care setting

• Monitor common cause variation – Look for non random patterns that

may indicate positive or negative trends

• Monitor special cause variation – Identify critical system errors and

analyze to prevent recurrence

Control Charts, Components

• Data in the form of a line graph

• Statistical parameters– Mean– Upper Control Limit– Lower Control Limit

Control Chart

UCL

MEAN

LCL

Standard deviation= square root of the variance

Or let Excel do it for you!!Insert

Function ‘Select a function’ = STDEV

Art and Science

• Surveillance, tracking and plotting the data = Science

• Interpreting the data for effective and appropriate response = Art

• Achieve a balance / flexibility– Avoid tampering – React appropriately

Tampering • Identify a ‘trend’ where there is none

• Try to explain natural variation as a special event

• Blame or credit people for processes they have no control over

• Makes it difficult to understand past processes

• Makes it difficult to plan future priorities and interventions

Spotting Special Cause Variation

• Non-random patterns – Points more that 3 SDs from mean– Two of 3 successive points more than 2

SDs from mean– Four of 5 successive points more than 1

SDs from mean– Eight successive points on one side of the

center line – Six successive points increasing or

decreasing (trend)

P-Charts, G- Charts

• P-Charts track percentage data– Surgical infection rates into control chart

• G-Charts track days between events– Number of days between on the job

accidents – Good for tracking rare occurrences

(like VRE infections – I hope)– Real time – ‘Early warning system’

0

50

100

150

200

250

300

350

03/0

1/01

06/0

6/01

08/2

1/01

12/2

0/01

05/0

6/02

07/1

8/02

08/2

3/02

10/2

1/02

11/0

5/02

02/1

9/03

04/0

2/03

06/1

0/03

09/0

4/03

04/2

1/04

09/2

7/04

11/1

7/04

12/0

2/04

12/0

8/04

01/1

1/05

03/0

9/05

08/0

9/05

08/1

6/05

08/2

4/05

09/1

4/05

01/0

4/06

01/1

1/06

02/2

2/06

Date of Surgery

# Ca

ses

betw

een

Mean = 86

Post-op PneumoniaTrend Analysis

mean calc based on FY01 - FY04

GOOD

• Data are the raw materials

• Statistical analysis provides the tools

• Use the data analysis to drive intervention

Relate the use of statistical analysis to action plans and intervention strategies

Only You Can Prevent Pneumonia

Walk, Walk, Walk Use your incentive spirometry (deep

breathing device)

Breathe deeply, turn and cough

P-Values

• P stands for PROBABILITY• It is a proportion of times a particular

event will occur in a series of repeated trials

• Quantifies degree of uncertainty about our data

• Range is from 0 to 1– 0 is no probability– 1 is 100% probability

Hypothesis testing

• Looking for a difference between 2 populations

• Question: Does eating ice cream cause heart attacks/– Null hypothesis = Any difference between ice

cream eater and non-ice cream eaters is a result of chance

– Alternative hypothesis = People who eat ice cream have more heart attacks

Significance

• If the p-value is very small you have strong evidence against the null hypothesis

• A result that is statistically significant is one that has a very small probability of happening just by chance or coincidence

Significance ParametersHow much difference is enough?

• Types of Error – Type I error (α) (significance level)

• ‘False Alarm’ • Found a difference but there really was none

– Type II error (β) • ‘Missed detection’ • Found no difference where one really existed• May be a result of too small of a sample size

Interpretation of the p-value

• Determine that the difference is statistically significant (reject the null hypothesis) if the p-value is smaller or equal to the significance level (α) – [Usually 0.05]

• If statistic is greater than the significance level, conclude there is no statistically significant difference between the two populations

Significance ≠ Proof

• Take any statistically significant results with a grain of salt

• How was the sample selected?

• How big was the sample?

• Were there confounding effects

Table A.4 Health behavior determinants of osteoporotic hip fracture: Univariate comparison of cases and controls

Health Behavior Characteristics

Sample N

Sample %

Control %

Case %

p-valu

e

Smoke now? 0.147

Yes 45 13.04 11.21 16.81

No 300 86.96 88.79 83.19

Exercise more than 2 times per week? <0.001

Yes 143 42.31 50.88 24.55

No 195 57.69 49.12 75.45

Drink any alcohol? 0.012

Yes 86 25.44 29.65 16.96

No 252 74.56 70.35 83.04

Confidence Interval• Margin of error in your data

• Usually set at 95%

• 95% likelihood that the true value of a statistic lies within the upper and lower 95%CI

• Wide is bad, narrow is good

• Factors that influence CI– Sample size– Amount of variability in the data

95% Confidence interval

The Odds Ratio (OR) for an infant dying because of Sudden Infant Death Syndrome was found to be 2.5 for households where the mother smoked cigarettes. The p value was < .05. Three studies found same OR but different CI.

• Study #1 showed an OR of 2.5

–95% CI from 2.4-2.6

• Study #2 showed an OR of 2.5

–95% CI from 1.4-3.6

• Study #3 showed an OR of 2.5

–95% CI from 0.4-5.6

Table A.8 Multiple logistic regression analysis of health behavior determinants of osteoporotic hip fracture (n=315)

Health Behavior Characteristics

Adjusted OR* (95%CI)

Current smoker

No 1.00

Yes2.99 (1.30,

6.86)

Exercise more than 2 times per week?

No 1.00

Yes0.64 (0.35,

1.20)

Drink any alcohol

No 1.00

Yes0.71 (0.35,

1.44)

* adjusted for age/fragility

P-values are just tools

• They help you assess the ‘merchandise’– You would not buy an avocado without

squeezing it to see if it is ripe

• Not all data are collected and analyzed correctly and fairly

• Look beyond P-values– What was the confidence interval?– What was the N?– Random sample or Convenience sample

Good Data

• Reliable– Reproducible results

• Unbiased– No systematic errors in study design and

sample selection

• Valid– The data measure what they are supposed

to measure

Top Ten Statistical Mistakes• #1 Misleading Graphs

84

85

86

87

88

89

90

Unit A Unit B Unit C Unit D

per

cen

t

0

20

40

60

80

100

Unit A Unit B Unit C Unit D

per

cen

t

Percent of Patient Care Employees with Influenza Vaccination, 2006

• #2 Biased Data– Instrument not calibrated properly– Participants influenced by the way questions

are asked– Sample does not represent the total

population of interest– The researcher is not objective

Top Ten Statistical Mistakes

Top Ten Statistical Mistakes

• #3 No margin of error reported– 10 + 1.2

• Range 8.8 to 11.2

– 10 + 6.4• Range 3.6 to 16.4

• #4 Non-Random Samples

• #5 Missing Sample sizes– What’s the N?

Top Ten Statistical Mistakes

• #6 Misinterpreted correlation– Correlation does not automatically

mean cause and effect

• #7 Confounding variables• #8 Botched Numbers• #9 Selectively reporting results

– Data fishing

• #10 Anecdote– Show me the data!’