Validation of Results Leveraging Navy Medicine’s Analytic Resources
Dec 16, 2015
Objectives
Compare analyst’s information to real world knowledge to assess reasonableness.
Review an annotated query panel to confirm analyst understanding and implementation.
Perform or review decompositions for reasonableness in appropriate dimensions.
Selectively exploit the M2 Data Dictionary and data caveat “blasters” for impact on validity.
22
Introduction
Critical Thinking – required of all producers and recipients of information.
Analysts’ powerful tools don’t substitute for managers’ assessment of validity.
In this block:Define validityMethods and illustrations of assessing validityStewardship of a scarce resource: analysts!
33
VALIDITY
Face Validity – the measuring method seems OK.
Construct Validity – either measuring the right thing, or something highly correlated with it.
External Validity – findings are likely to apply to “the bigger world”.
Reliability – Repeated measurements get similar answers.
44
Healthcare Analytics Newsletter
ILLUSTRATION
HOW MANY ADMISSIONS LAST YEAR AT NAVY MTFs WERE RELATED TO OBSTETRICS?
6
ANALYST’S ANSWER: 286,727
ILLUSTRATION
8
ANALYST’S ANSWER: 286,727
HAZARD A ROUGH GUESS FOR NAVY OB?
THE ANALYST ANSWER SEEMS ABOUT 7 TIMES TOO BIG!
2. REVIEWING ANNOTATED QUERY PANEL
99
• The analyst tells the computer what to extract via a query panel
• M2 then converts the display into a program which retrieves the data from the M2 database.
• Management can ask the analyst to provide both the query panel, and an explanation for it.
• BUT must also check for filters applied AFTER the data were extracted!
ILLUSTRATION
HOW MANY ACTIVE DUTY NAVY LIVE AROUND NAVAL HOSPITAL, JACKSONVILLE?
10
ANALYST’S ANSWER: 20,206
Used December, although January was available, as most recent month is usually low.Included both Navy, and Navy Afloat
Used Relationship Summary since all AD in it and didn’t need details
Included only the active duty
Residence zip in Jacksonville catchment area
OR enrolled to Jacksonville
Used December, although January was available, as most recent month is usually low.Included both Navy, and Navy Afloat
Used Relationship Summary since all AD in it and didn’t need details
FILTERING THE EXTRACTED DATA?
12
The analyst chose to
omit anyone
enrolled to a non-Navy,
non-contractor site that was not
near Jacksonville
RELIABILITY?
The analyst chose December because the most recent month is often immature and unreliable.
14
3. DECOMPOSING RESULTS
1616
• Big sums can hide obvious errors!
• Manager or analyst can decompose, if stratifiers were or are retrieved with the data.
• Just “TLAR”, but at a more granular level.
ILLUSTRATION
HOW DO THE THREE SERVICES COMPARE ON AVERAGE LOS FOR SIMPLE PNEUMONIA?
17
ANALYST’S ANSWER:
Army: 2.36 days
Navy: 2.70 days
AF: 4.41 days
ILLUSTRATION
ANY SURPRISES IF DECOMPOSED BY GENDER?
18
F M
A 2.32 2.37
F 2.18 4.41
N 2.39 2.83
What about Air Force males would cause unusually long LOS?
F
1
F
2
F
3
F
4
M
1
M
2
M
3
M
4
A 2.17 3.00 2.24 2.58 2.14 2.23 2.30 2.47
F 1.85 1.80 2.19 2.64 2.00 1.95 2.66 10.48
N 1.83 1.67 2.32 4.40 4.00 2.17 2.44 3.11
ILLUSTRATION
ANY SURPRISES IF DECOMPOSED BY BENCAT?
19
What about Air Force active duty males would cause unusually long LOS?
Tmt DMIS ID Tmt DMIS ID Name Average LOS
0006 673RD MED GRP-JB ELMENDORF-RICHARDSON 2.00
0014 60TH MED GRP-TRAVIS 3.00
0073 81ST MED GRP-KEESLER 6.00
0079 99TH MED GRP-O'CALLAGHAN HOSP 1.33
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 15.59
0120 633RD MED GRP-JB LANGLEY-EUSTIS 2.00
0638 51ST MED GRP-OSAN AB 3.00
0639 35TH MED GRP-MISAWA 1.33
0808 31ST MED GRP-AVIANO 2.00
ILLUSTRATION
SURPRISES IF DECOMPOSED BY MTF (AF only)?
20
Why would Wilford Hall (59th Med Wing) have so much longer stays than any other MTF?
Tmt DMIS ID Tmt DMIS ID Name Record ID Admission Date Discharge Date Average LOS
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6250055 10/09/2010 10/10/2010 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6250332 10/18/2010 10/21/2010 3
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6250769 10/29/2010 10/30/2010 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6251754 11/30/2010 12/01/2010 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6251941 12/05/2010 12/09/2010 4
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6252152 12/12/2010 12/13/2010 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6252322 12/16/2010 12/19/2010 3
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6252563 12/24/2010 12/28/2010 4
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6253357 01/19/2011 01/21/2011 2
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6253437 01/20/2011 01/22/2011 2
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6253461 01/14/2010 01/15/2011 366
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6253701 01/27/2011 01/29/2011 2
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6253881 02/01/2011 02/02/2011 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6254703 02/26/2011 02/27/2011 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6254931 03/05/2011 03/06/2011 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6254960 03/06/2011 03/08/2011 2
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6255017 03/07/2011 03/11/2011 4
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6255282 03/15/2011 03/16/2011 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6255446 03/20/2011 03/21/2011 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6255937 04/03/2011 04/05/2011 2
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6255956 04/04/2011 04/04/2011 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6256448 04/20/2011 04/22/2011 2
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6256702 05/01/2011 05/02/2011 1
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6256705 05/01/2011 05/03/2011 2
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6257077 05/18/2011 05/22/2011 4
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6257109 05/19/2011 05/23/2011 4
0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 7002253 01/14/2011 01/18/2011 4
ILLUSTRATIONDECOMPOSED BY PATIENT (WH ONLY)?
21
EXPLOITING DOCUMENTATION
23
• AKIN TO HUNTING WHERE THE LIGHT IS, BECAUSE NOTHING CAN BE FOUND IN THE DARK.
• NEITHER THE DICTIONARY NOR THE BLASTERS ARE NEAR PERFECT. . .
• BUT THEY ARE THE BEST WE HAVE.
• THERE IS NO LIST ANYWHERE OF ALL THE PROBLEMS IN M2 DATA
ILLUSTRATION
24
Both are M2, both report on the same events, why do they differ by a million encounters?
ILLUSTRATION
25
• The M2 Data Dictionary shows that:o CAPERS have more procedure fields and so
will get different estimates of costs.o SADRs are not updated on the same
frequency as CAPERs, so the data are not equally fresh.
• Worldwide blasters were sent:• Cautioning on using the CAPERS, especially
costs.• But subsequent blasters have said that is
fixed.
ILLUSTRATION
26
• Neither Dictionary nor Blasters explain that:o Edits on CAPERS prevent many events from
being reported that are found in the SADRs.o SADRs use an imperfect key so that there are
duplicate records, where these duplicates are not in CAPERs.
Probably good to caveat any M2 answer as no one knows the extent and effect of validity problems in the data.
“According to the data in M2. . .”