EVIDENCE: IS IT ALL ABOUT INTERPRETATION?mini-university.com/wp-content/uploads/2017/09/... · Data Capture Method •Precision of data •Accuracy of data •Level of detail Cost

EVIDENCE: IS IT ALL ABOUT

INTERPRETATION?GH Mini USeptember 2017

A. Chafetz, K. Gichuhi, R. Godbole, C. NicholsUSAID | Global Health Bureau | Office of HIV/AIDS

Be a Critical Consumer of Information

Do you….

• Make ‘informed’ decisions when you buy food, buy a car, pay for a medical procedure?

• Read the newspaper or watch the news?• Analyze data?• Visualize data?

• Make policy decisions?

• Perform research?

A Critical Consumer of Information Is:

Curious

Clear about the Question they Want Answered

Maintains Data Integrity

According to recent research…“A Rebounding Economy Remains Fragile for Many” Sept. 14, 2016

“HHS Issues New rules to Open Up Data From Clinical Trials” Sept. 16, 2016

“Sugar Industry Secretly Paid for Favorable Harvard Research” Sept. 12, 2016

Select appropriate data visualization when presenting your analysis. “Same data, different messages.”

When presenting data, be clear about what the data does and does not tell you. “Data may not answer your question, but should help you ask the right questions.”

Same data, different messages

Jim L

egan

s, Jr

.

Why does data viz matter?

I II

III

mean x = 9mean y = 7.5var x = 10var y = 3.75corr = 0.82

mean x = 9mean y = 7.5var x = 10var y = 3.75corr = 0.82

mean x = 9mean y = 7.5var x = 10var y = 3.75corr = 0.82

mean x = 9mean y = 7.5var x = 10var y = 3.75corr = 0.82

IV

Anscombe’s Quartet

Exploring the data can help you better understand it

site id indicator period value

fx3ddmd2 X1 fy2008q1 66

fx3ddmd2 X1 fy2008q2 79

fx3ddmd2 X1 fy2008q3 13

fx3ddmd2 X1 fy2008q4 52

fx3ddmd2 X1 fy2009q1 75

fx3ddmd2 X1 fy2009q2 81

fx3ddmd2 X1 fy2009q3 10

fx3ddmd2 X1 fy2009q4 30

fx3ddmd2 X1 fy2010q1 79

Raw Data Communication Tool

Visualizing data can reveal trends and relationships

Fina

ncia

l Tim

es V

isua

l Jou

rnal

ism

team

Lies, damn lies, and statistics

- Benjamin Disraeli

● Are the data accurately represented?

● What is the data source and is it clearly identified?

● Are there any reasons to question the data quality?

● What is the message the author is trying to convey?

Are the visualizations trustworthy?

Common Data Viz “Lies”1 2 3

1

1

2

2

3

xkcd.c

om

3

Informing a Policymaker1 2 3 4 5

1

Start with a question or purpose

wallp

aperc

ave.c

om

2

Collect and tidy data

3

Visualize Series 1

4

Simplify and use font, color, and SIZE to

your advantageC3C1

C2

x

y

Further readingAlbert Cairo’s The Functional Art

Stephanie Evergreen’s Effective Data VisualizationCole Nussbaumer Knaflic’s Storytelling with Data

Edward Tufte’s The Visual Display of Quantitative InformationAndy Kirk’s Data Visualization

5

C3C1C2

x

y

Have a clear message, state it, and provide necessary context

Source: PEPFAR FY17Q2 Dataset

PEPFAR should focus on the 3 countries that have HIGH testing volume and LOW positivity.

Only you can prevent the

perpetuation of data visualization

lies.

Think critically about global health metrics

Examples from PEPFAR Dashboards

Showing your data in a clear and intentional format is important because…

WE HAVE ACCESS TO A LOT OF HEALTH DATA!

Global Health topics:

● Maternal and

reproductive health

● Child health

● HIV/AIDS

● Tuberculosis

● Malaria

● Neglected tropical

diseases

● Cholera

● Influenza

● Meningitis

● STIs

● Noncommunicable

diseases

● Mental health

● Substance abuse

● Road safety

● Environmental

health

● Vaccines

● And many more...

Sources of Global Health

data:

● UN data

● World Health

Organization Data

Repository

● PEPFAR

Dashboards

● World Bank Open

Data

● Sustainable

Development Global

Indicators and

Database

● Millennium

Development

Indicators

● Humanitarian Data

Exchange

● World Population

Data

● Demographic and

Health Surveys

● And many more...

Sources of Global

Health Data

and Global Health

Topics

...on the following Global

Health topics:

● Maternal and

reproductive health

● Child health

● HIV/AIDS

● Tuberculosis

● Malaria

● Neglected tropical

diseases

● Cholera

● Influenza

● Meningitis

● STIs

● Noncommunicable

diseases

● Mental health

● Substance abuse

● Road safety

● Environmental

health

● Vaccines

● And many more...

Open Sources of Global

Health data:

● UN data

● World Health

Organization Data

Repository

● PEPFAR Dashboards

● World Bank Open

Data

● Sustainable

Development Global

Indicators and

Database

● Millennium

Development

Indicators

● Humanitarian Data

Exchange

● World Population Data

● Demographic and

Health Surveys

● And many more...

What we will

focus on

What is PEPFAR?

Launched in 2003 by President George W. Bush and strengthened by President Barack Obama, the program

represents America’s commitment to achieving an AIDS-free generation. The U.S. government has committed more

than $70 billion to HIV/AIDS programs to date.www.pepfar.gov

Monitoring

Is the routine process of systematic data collection and the measurement of progress toward program

objectives

Why is monitoring and reporting important to PEPFAR?

Drives greater impact, transparencyand accountability.

Improves partner performance and increased program efficiency and effectiveness.

pepfar.gov/documents/organization/264887.pdf

You can see PEPFAR results data for yourself on…the publically-available PEPFAR Dashboards!

You can view lots of information and data here, including results and targets for PEPFAR’s monitoring indicators by

country, and sub-national units.

data.pepfar.net/global

Tanzania Example: first, a few definitions... One PEPFAR indicator used to monitor treatment coverage is TX_CURR.

results

----------

targets

= % achievement

ONE way that PEPFAR monitors performance is by examining achievement towards targets or ‘% achievement’

90

-----

100= 90% achievement

DEFINITION of % achievement EXAMPLE of % achievement

TX_CURR: The number of adults and children currently receiving ART.

Tanzania Example

Q: What was the % achievement for TX_CURR for Superior Health Services in Tanzania for FY 2016?

78,499 results

--------------------

84,710 targets= 93% achievement

Setting: There is a PEPFAR implementing partner in Tanzania called Superior Health Services* that provides direct service delivery of ART for adults and children in 1,500 sites.

*fictitious example

Tanzania Example

----------------------------------------------------------------------------1. What factors impact % achievement (results/targets)?

For example, what happens when a target is “low”?2. Is the % achievement useful as a metric in isolation?3. In what ways is this metric helpful to monitoring

progress?4. What other metrics would you want to consider to assess

performance?

DiscussionSuperior Health Services in Tanzania, TX_CURR, FY 2016:

78,499 results

--------------------

84,710 targets= 93% achievement

Indicator 2016 Q1 2016 Q2 2016 Q3 2016 Q4 2017 Q1 2017 Q2 2017 Q3

Average Quarterly Growth

Rate

Projected 2017 Q4 Results

APR Achieved of 2016 Target

Q3 Achieved of 2017 Target

Projected APR

Achieved of 2017 Target

2016 APRResults

2016 Targets

2017 APR Results

(Projected)2017

Targets

Partner A

Afya Jijini HTS_TST 28,654 57,145 72,307 84,866 77,706 67,603 73,545 17% 86,056 213% 99% 138% 242,972 114,071 304,910 221,151

Afya Jijini HTS_TST_POS 1,726 3,097 3,066 3,001 2,179 2,507 2,192 4% 2,281 317% 69% 92% 10,890 3,436 9,159 9,933

Afya Jijini TX_NEW 926 1,185 1,408 2,434 1,591 1,630 1,673 10% 1,846 106% 54% 74% 5,953 5,607 6,740 9,068

Afya Jijini TX_CURR - 34,353 - 32,170 32,864 35,436 37,000 1% 37,553 91% 85% 86% 32,170 35,238 37,553 43,716

Afya Jijini TX_NET_NEW - 1,509 - (2,183) 694 2,572 1,564 553 -28% 42% 47% (674) 2,394 5,383 11,546

Look at TX_CURR within a ‘Cascade’

Why Context & Methodology Matter

Illustrative Examples from PEPFAR Expenditure Analysis

Health economic datapresents a unique set

of interpretation challenges as context

and methodologymatter a lot

Why are context andmethodology so

important?

IT’S ALL ABOUT THE

Which health programs

get $

How much $ gets

budgeted

How $ gets allocated across

geographies, interventions,

etc.

Health

Economic

Data

Informs…

$

Without context and methodology we run the risk of misinterpretation and misuse of data – which can

have significant health & financial impacts

Elements of Health Cost Analyses:

What kinds of things should you consider when you’re reviewing and interpreting data?

Type of economic

evaluation

•Costing •Cost Effectiveness•Cost Benefit•Cost utility

Perspective•Donor•Patient•Society•Insurer/Payer

Purpose

•Advocacy•Budgeting •Resource Tracking •Financial Management •Program Planning •Strategic Planning

Data Capture Method

•Precision of data•Accuracy of data•Level of detail

Cost Methodology

•Categorization of costs•Actual cost vs charges vs prices•Cash vs accruals•Discounting over time•Inflation adjustments

Time Horizon & Perspective

•Time frame•Retrospective vs Prospective

Analysis •Other data used plus associated methodology•Nuances and other considerations

What is Expenditure Analysis (EA)?

Annual retrospective expenditure tracking

PEPFAR investments as reported by implementing

partners

Reported using a cash-basis of accounting from partner perspective via a specific methodology that includes disaggregation by site vs above-site, program

area, geography, & cost category

Linked to indicator data when available &

applicable to estimate a historical “unit expenditure”

How is EA data used for decision-making within PEPFAR?

Understand “Costs” to PEPFAR of achieving

results

Demonstrate Accountability of

PEPFAR ProgrammingPromote Transparency Among Stakeholders

Estimate Future Resource Requirements

Inform Resource Allocation Across

Locations, Program Areas, Interventions,

Partners

Monitor Partner Performance Alongside

other Data

primary uses

secondary uses

EA Data can and does inform all of the above – but interpretation with the correct caveats is important

Interpreting and using expenditure data considering context and

methodologyExamples

Making Decisions with Expenditure

Data• Could I make a budget for September

using my August spending? • Does this reflect the full cost to

support my family (two incomes, one child)?

• Should I reallocate the way I spend money?

• The cost per trip to Whole Foods varies significantly per trip – should I only buy the things that were on my list on August 5th?

• Would you expect other families to have similar spending patterns? Should you benchmark my spending against others?

Ramona’s

Spend by

Category –

August 2017

Using Historical EA Data for Budget ProjectionsPEPFAR Program Area Historical Unit

Expenditure

(FY16)

Target Projected PEPFAR COP17

Budget

(Implemented FY18)

Contextual Information

Treatment $150 500,000 $75 million Changing guidance & implementation test

and start and optimal service delivery

Testing $10 1,000,000 $10 million Change in ratio of test kits procured by

PEPFAR vs Global Fund

Prevention $25 1,000,000 $25 milion Fairly static programming, but

implementation transitioning to MoH

KP Prevention $60 200,000 $12 million FY16 high start up costs relative results

OVC $35 100,000 $3.5 million Anticipated increase in intensity of services

• Are the projected budgets appropriate given the context? • Can you tell if they are over- or under- estimates? • For prevention, should MoH consider $25/person the “cost” of the program?• What types of other data would you want to have? • Would you feel comfortable making assumptions to “adjust” the projected budget?

Discussion

Allocating Resources Based on EA and Other Data

• How does this graph help (or not) us to have a discussion around aligning PEPFAR resources investments with the epidemic profile?

• What additional information is missing from this graph that would allow for better interpretation?

• What methodological flaws may further exacerbate misinterpretation?

Discussion

Assessing Efficiency with EA Data

Hospital based testing in which PEPFAR supports limited HRH

costs, but rent, most salary, etc. is supported by MOH.

Home based testing almost completely funded by

PEPFAR (including staff, transportation,

management, etc.).

A Critical Consumer of Information Is:

Curious

Clear about the Question they Want Answered

Maintains Data Integrity

Notes and Attribution● Prepared for the GH Mini U Conference, September 2017 (Washington)● Image sources

○ The Sun front page - https://twitter.com/bettinanissen/status/872944714944794625○ Grocery store aisle, Jim Legans, Jr -

https://www.flickr.com/photos/9575673@N08/2295486674/in/photolist-TXXz9x-4P85a1-deK43s-VeM1iz-axmd55-9H9WcQ-5eEni2-e6m5rf-e7eJcE-isQDkN-5JeDbP-b11on8-4ifngb-TzKDcz-VeLYLM-4pwrYK-4Bpt5K-4uQYds-feUnYz-nvQFWP-axiw5e-e6fp5H-TgFhQu-d3Vwth-9Em9tU

○ Cover of How to Lie with Statistics -https://upload.wikimedia.org/wikipedia/en/2/2a/How_to_Lie_with_Statistics.jpg

○ Pie Charts, adapted - https://commons.wikimedia.org/wiki/File:Piecharts.svg○ Map comic - https://xkcd.com/1138/○ Icons from The Noun Project: Bull horn - Johannes Hirsekom, Microscope - IYIKON○ Batman - http://wallpapercave.com/wp/4cBCyPL.jpg○ Benjamin Disraeli - http://static.guim.co.uk/sys-

images/Guardian/Pix/pictures/2011/5/14/1305390365594/Benjamin-Disraeli-c1878-007.jpg

○ Smokey the Bear - https://en.wikipedia.org/wiki/Smokey_Bear#/media/File:Smokey3.jpg