Melinda Karp MHQP Director of Programs July 12, 2007

Measuring and Reporting Patients’ Experiences with

Their Doctors Process, Politics and Public Reports in Massachusetts

Melinda KarpMHQP Director of Programs

July 12, 2007

Today’s Objectives

Describe evolution of MHQP agenda for measuring and reporting patient experiences-- key methods questions in moving from research to large scale implementation

Describe stakeholder perspectives and decision points around key reporting issues

Discuss how MHQP data is being used and key ongoing challenges

The Headlines from October, 1994…

…Led to the Creation of MHQP in 1995

• Provider Organizations– MA Hospital Association– MA Medical Society– 2 MHQP Physician Council

representatives

• Government Agencies– MA EOHHS– CMS Region 1

• Employers– Analogue Devices

• Health Plans– Blue Cross Blue Shield of

Massachusetts

– Fallon Community Health Plan

– Harvard Pilgrim Health Care

– Health New England

– Neighborhood Health Plan

– Tufts Health Plan

• Consumers– Exec. Director HCFA

– Exec. Director NE Serve

• Academic– Harris Berman, MD, Board Chair

2002-2003 Demonstration project in partnership with The

Health Institute (Funded by Commonwealth and RWJF)Gained interest and acceptance of survey among

key stakeholdersDemonstrated relevance of physician level data

and feasibility of collaborative approachAmbulatory Care Experiences Survey (ACES)

developed for the project has figured importantly in development of C-G CAHPS

The Evolution of MHQP’s Patient Experience Measurement Agenda

2004-2005 Development of viable business model for

implementing statewide patient experience survey Approved cost sharing methodology for health plans and

physician organizations to finance survey.

2005-2006 Fielding and reporting of first statewide survey

Survey field period, July-September 2005 Focus on primary care practice sites

497 practices 92% of registered primary care physicians (over 4000 MDs)

50 item instrument covering 8 domains Internal release of results to physician practices, November 2005 Public release of results, March 2006

The Evolution of MHQP’s Patient Experience Measurement Agenda

Next Steps for the MHQP Patient Experience Measurement Agenda

2007-2008Fielding of Specialist Care Survey (2007)Repeat Primary Care Survey (2007)Assess the impact of reporting efforts for physician

and health plan stakeholdersEngagement around QI activities

– Participation in Commonwealth Fund grant to study highest performing practices

– Physician Foundation Grant to develop and pilot integrated clinical-patient experience QI curriculum

Pilot Medicaid Survey (2008)

“1st Generation” Questions: Moving MD-Level Measurement into Practice

• What sample size is needed for highly reliable estimate of patients’ experiences with a physician?

• Is there enough performance variability to justify measurement?

• How much of the measurement variance is accounted for by physicians as opposed to other elements of the system (practice site, network organization, plan)?

• What is the risk of misclassification under varying reporting frameworks?

Allocation of Explainable Variance: Allocation of Explainable Variance: Doctor-Patient InteractionsDoctor-Patient Interactions

3825 22

29

6274 77 84

70

160

20

40

60

80

100

Doctor

Site

Network

Plan

Comm

unicat

ion

Whole

-per

son o

rienta

tion

Health

pro

motio

n

Inte

rper

sonal

trea

tmen

t

Patie

nt tru

st

Source: Safran et al. JGIM 2006.

45 56 77

39 3623

8160

20

40

60

80

100

OrganizationalAccess

Visit-basedContinuity

Integration

Doctor

Site

Network

Plan

Allocation of Explainable Variance: Allocation of Explainable Variance: Organizational/Structural Features of CareOrganizational/Structural Features of Care

Source: Safran et al. JGIM 2006.

Summary and ImplicationsSummary and Implications

• Reliable information can be obtained with sufficient sample size– Data obtained using C/G CAHPS approach yields data with MD- and

site-level reliability >0.70– For site-level reliability, number of MDs per site influences required

sample sizes

• There is enough variability to justify physician-level reporting

• Risk of misclassification of performance can be minimized– Can be held to <5% by:

• Limiting number of performance categories • Creating buffer (“zone of uncertainty”) around performance cutpoints

• Trade-offs are likely around data quality standards (e.g., acceptable “risk”) vs. data completeness

Setting the Stage for Public Reporting: Key Issues for Physicians

• What measures get reported– Criteria for reporting measures publicly– Use of “super” composites to rate overall performance – Level of detail presented to consumers—composites vs. item

level

• How measures get reported– Absolute vs. relative scoring– Use of “subjective” performance labels—Excellent, Good, etc.– Determining performance categories – Minimizing the risk of misclassification– Recognizing high achievers

• Setting the context for the user

Consumer Perspectives:Focus Group Feedback On…

• Labeling measure composites

• Providing item level detail

• Describing how care is delivered in MA and where the MHQP data fits

• Creating trust for the user– Transparency about project funding, methods– Endorsement from the physician/health care

community--AMA, MMS or State Health Department

Consumer Perspectives: Focus Group Feedback On…

• Using quality information– Picking a new doctor– Evaluating a current doctor– Recommending a doctor for family and friends– Seeing how the doctors in their area were performing overall

• Useful information to support use of the performance data– Useful tools to help pick a doctor– Information about the doctor and the practice– Links to other websites

Consumer Perspectives: Focus Group Feedback On…

• Talking about quality with doctors– Skeptical – Worried“Wouldn’t have any effect or worse, might be negative”

• Benefits of the information– Information = Empowerment

“Having options and the ability to make a choice when finding a doctor”“Taking charge of my health because now I can make decisions based on

information that I didn’t have before”

– Improving the quality of care“I would hope they [doctors] would use this as feedback to improve their

practices”

Integrating Stakeholder Perspectives

“We must be willing to learn the lesson that cooperation may imply compromise, but if it brings a world advance it is a gain for each individual nation”.

Eleanor Roosevelt (1884 – 1962)

“All government -- indeed, every human benefit and enjoyment, every virtue and every prudent act -- is founded on compromise and barter”.

Edmund Burke (1729 - 1797)

Key Decisions for Public Reports

• Search Approaches:– search by Physician Name– proximity search using region or zip code– Search from list of practice sites, medical groups

• Front-end presentation of how care is delivered and context for report

• Umbrella categories for measures but no data roll-up

Key Decisions for Public Reporting

• 4 categories of relative performance– ½ star “buffer zones” rounded to next performance category for

public reporting

• Criteria for including a measure– 50% of practices needed “A” level reliability for measure to be

included– A practice needed 3 eligible measures to be included

• No “subjective” labels attached to performance• Drill down to item level results• Measure specific messages about quality

Visit the MHQP website at www.mhqp.org

Visit the MHQP website at www.mhqp.org

One More Stakeholder: The Media

What the headlines could have been…

The Headlines from March 9, 2006

The Headlines from March 9, 2006

Worcester Telegram & Gazette

How is MHQP Data Being Used?

• Current uses – Reporting to physicians for quality improvement, compensation– Direct to consumer online reporting

• Links from MA state website to MHQP reports

• Links from health plan websites to MHQP reports

• Links from several provider organization websites to MHQP reports

• Emerging uses– Support MA transparency agenda in State Health Care Reform

Law (Section 16)– Physician certification– Links to MHQP reports by employers – Health plan recognition programs, P4P, and product design

Ongoing Challenges

• Creating a sustainable financing model for continued measurement and reporting– Collaboration alone is not enough—aligning stakeholder

agendas and incentives is critical

• Continuing to meet the rapidly evolving information needs of the marketplace while maintaining the collaborative– physician needs for quality improvement

– health plan/employer needs to develop innovative insurance products/incentives

– Consumer needs to guide decision making

For more information about MHQP…

Melinda Karp, Director of Programs [email protected]

617-972-9056

www.mhqp.org