Quality Assessment of Minnesota Newborn Hearing Screening Data: A Pilot Study National EHDI Conference March 26, 2007 Salt Lake City Penny Hatcher, Supervisor.

Quality Assessment of Minnesota Newborn Hearing Screening Data: A Pilot Study

National EHDI ConferenceMarch 26, 2007 Salt Lake City

Penny Hatcher, Supervisor and Grant DirectorYaoli Li, CDC EHDI Coordinator

Nicole Brown, HRSA UNHSI CoordinatorKatie James, UNHSI Student WorkerSarah Solarz, EHDI Student WorkerJudy Punyko, MDH Epidemiologist

Minnesota Department of Health (MDH)Community & Family Health

Newborn & Child Screening Unit

2

Faculty Disclosure Information

In the past 12 months, I have not had a significant financial interest or other relationship with the manufacturer(s)of the product(s) or provider(s) of the service(s) that will be discussed in my presentation.

This presentation will not include discussion of pharmaceutical or devices that have not been approved by the FDA or if you will be discussing unapproved or “off-label” uses of pharmaceuticals or devices.

3

Study Purpose

In compliance with the CDC guidelines for evaluating public health surveillance systems…

Assess the quality of Minnesota newborn hearing screening data

-Validity

-Reliability

4

Methodology – Planning Phase

Develop partnership with– Vital Records (CHS)– Birth Defects Information System (EH)– Newborn Bloodspot Screening (PHL)– Newborn Hearing Screening

Data fieldsMedical record abstraction form

5

Medical Record Abstraction Form (Infants)

(5) BIRTHDATE __ __ /__ __ /__ __ __ __ Month Day Year

(7) SEX

1 = Male 2 = Female 9 = Undetermined, not stated

(9) BIRTH WEIGHT

grams

pounds ounces *Fill appropriate boxes according to units specified in record

(10) MULTIPLE BIRTHS / BIRTH ORDER (a, b, etc.) 1 = Single Birth 2 = Twin ___ (a or b) 3 = Triplet ___ (a, b, or c) 4 = Quadruplet ___ (a, b, c, or d) 5 = Quintuplet ___ (a, b, c, d, or e) 6 = Other 9 = Unknown, not stated, unclassifiable

(13) DATE MOST RECENT HEARING SCREEN

From BS: mm dd yyyy mm dd yyyy *9-fill fields if missing

(14) LEFT EAR SCREEN RESULTS, 1 MONTH 1 = Pass 2 = Fail 9 = Not screened (missing)

(15) RIGHT EAR SCREEN RESULTS, 1 MONTH 1 = Pass 2 = Fail 9 = Not screened (missing)

(16) SCREEN METHOD 1 = ABR 2 = OAE 3 = Other 9 = Unknown, not stated

(17) REASON FOR NO SCREEN

1 = Missed 4 = Equipment problem 2 = Refused 5 = Transferred 3 = Delayed 9 = No reason given

(11) ANTIBIOTICS ADMINISTERED? 1 = Yes 2 = No 9 = Unknown, not stated

(12) TRANSFUSIONS GIVEN? 1 = Yes 2 = No 9 = Unknown, not stated

FOLLOW-UP PHYSICIAN NAME From BS: Last Name First Name (18) Last Name (19) First Name

9 = Unknown, not stated 9 = Unknown, not stated

(20) TRANSFERRED? 1 = Yes 2 = No 9 = Unknown, not stated (21) Location of transfer if yes

(1) Hospital ID: (2) Study ID: (3) Medical Record #:

NAME

From BC:

Last Name First Name (4) Last Name (5) First Name

(6) BIRTHDATE From BC:

mm dd yyyy *9-fill fields if missing

BC DATA

BC DATA

BC DATA

BS DATA

BS DATA

BS DATA

BS DATA

BS DATA

BS DATA

BC DATA

(8) FAS

1 = Yes 2 = No 9 = Unknown

6

Medical record abstraction form

(7) BIRTH DATE

From BC:

01/10/2005

mm dd yyyy

*9-fill fields if missing

(11) MULTIPLE BIRTHS / BIRTH ORDER (a, b, etc.) 1 = Single Birth

2 = Twin ___ (a or b)

3 = Triplet ___ (a, b, or c)

4 = Quadruplet ___ (a, b, c, or d)

5 = Quintuplet ___ (a, b, c, d, or e) 6 = Other 9 = Unknown, not stated, unclassifiable

BC DATA

2

a

(15) LEFT EAR SCREEN RESULTS, 1 MONTH

1 = Pass2 = Fail9 = Not screened/missing

BS DATA

1

7

Methodology – Planning Phase

Select 20 MN hospitals with ≥15 births in 2005– Hospitals rank-ordered by size (i.e. # births)

– Every 5th hospital chosen

e.g. Hospital

SizeA

100start B 93

C 90D 86E 82F 79G 70

8

Methodology – Implementation

Two graduate student workers oriented by BDIS staff

Vital Records randomly selects

mothers (n = 200) and

their infants (n = 200) from birth certificates (Total N = 400)

9

Methodology – Implementation

Letters sent to hospitalsPhone calls made to confirm appointmentsList of 10 infant and 10 mother records

faxed to each hospital

10

Students hit the road…

11

Methodology – Data Collection

At each hospital…– Collect and record information from medical

recordsUpon return to MDH…

– Students double enter data

12

Methodology – Data Analysis

Assess inter-rater reliability– How well do students’ data agree?– If discrepancies, determine which “answer”

is correct– Create final (corrected) database

13

Methodology – Data Analysis

Merge medical record (MR) data with hearing screening (HS) data

14

Methodology – Data Analysis

Clean the merged dataset– e.g. Duplicate records

• Use only records with “lab” or “loose” (i.e. from birth hospital) designation as data source.

NAME BC NUMBER DOB SOURCE

Baby, Girl 2005-MN-0123456 1/1/2005 labBaby, Girl 2005-MN-0123456 1/1/2005 C1 referBaby, Girl 2005-MN-0123456 1/1/2005 refer

15

Methodology – Data Analysis

Clean the merged dataset– e.g. Missing records

• 5 infants– 2 died (no hearing screen done)

– 1 transferred to NICU (no record of HS at birth hospital)

– 2 with evidence of HS in medical record but were not in HS database

16

Methodology – Data Analysis

Missing values– Recoded into a ‘no/unknown’ category

Weighting scheme

17

Methodology – Data Analysis

Analysis of categorical (yes/no) variables included calculations of:– Sensitivity– Specificity– Positive predictive value

18

Sensitivity

Medical Record – Left Ear Results

HS data – Left Ear Results

Yes Pass

No Pass

Total

Yes Pass 160 13 173

No Pass 7 20 27

Total 167 33 200

Sensitivity = 160 / 167 = 95.8%

19

Specificity

Medical Record – Left Ear Results

HS data – Left Ear Results

Yes Pass

No Pass

Total

Yes Pass 160 13 173

No Pass 7 20 27

Total 167 33 200

Specificity = 20 / 33 = 60.6%

20

Positive Predictive Value

Medical Record – Left Ear Results

HS data – Left Ear Results

Yes Pass

No Pass

Total

Yes Pass 160 13 173

No Pass 7 20 27

Total 167 33 200

PVP = 160/173 = 92.5

21

Results – Left Ear/Right Ear Screens

Characteristics

HS MR

% Missing In

HS MR

Sensitivity (%)

(SE)

Specificity (%)

(SE)

PVP (%)

(SE)

% left ear pass

98.3 97.1 12.0 14.0 99.0

(0.6)

48.0

(15.2)

94.6

(2.2)%

right ear pass

96.6 97.7 12.0 14.0 98.0

(1.1)

52.5

(17.4)

95.4

(2.0)

22

Results – Left Ear/Right Ear Screens

% of infants with pass results in L and R ears:– ↑ sensitivity and ↓ specificity.

-9/200 infants: L/R ear results in MR but missing in HS database.

– 13/200 infants: L/R ear results in HS database but missing in MR.

23

Results – Reasons For No Screen

Reason for no screen

Character-istics

HS MR

% Missing In

HS MR

Sensitivity (%)

(SE)

Specificity (%)

(SE)

PVP (%)

(SE)

% refused 2.0 1.5 0 1.0 94.9

(6.3)

99.8

(0.2)

86.5

(15.1)% delayed 1.5 0 0 1.0 NA 98.9

(0.8)

NA

% equipment problem

1.5 0.5 0 1.0 100 99.6

(0.3)

46.5

(4.2)% No reason given

6.5 8.6 0 1.0 1.4

(1.2)

97.9

(1.1)

3.2

(2.5)

24

Results – Reasons for no screen

↑ specificity Variable sensitivityVariable positive predictive valueIn HS database, of 24/200 infants with

missing L/R ear results…– Only 11 out of 24 with reason for why screen

was missing or not done.

25

Results – Birth and Screen Dates

Discrepancies between dates

Infant Birth Dates

Hearing Screen Dates

0 days 194 129

1-3 days 1 22

4-7 days 0 0

8-14 days 0 2

15-21 days 0 2

22+ days 0 3

Overall range -1 to 0 -63 to 366

26

Results – Birth and Screen Dates

MR and HS:3% of infant birth dates did not agree 29% of HS dates did not agree

– Of the 58/200 discrepant screen dates:• 22 missing in MR but available in the HS database

• 5 missing in the HS database but available in MR

27

Noteworthy Findings

Data fields with less frequent outcomes

have lower validity:– Antibiotic use– Failed hearing screen results (left or right ear)

28

Noteworthy Findings

Low to moderate agreement among the various reasons for no hearing screen– But… small numbers = reduced precision

Hospital-specific results

29

Moderate agreement and high % missing among hearing screening dates– 29% overall disagreement– 17% missing in medical records– 11% missing in hearing screening database.– Some screen dates in HS database recorded as

being prior to birth date

Noteworthy Findings

30

Errors in abstraction process

- Information recorded as “missing” by abstractors

- Misinterpretation of language or results in medical record

Medical record as “gold standard” assumption

Study Limitations:

31

Implement safeguards in hearing screening database

Meet with hospitals Modify study design

Where do we go from here?

32

Minnesota Contact Information:

Penny Hatcher, MSN, [email protected]

651-201-3744

Yaoli Li, MD, MS, [email protected]

651-201-3750

Thanks You!