Performance Improvement Project Validation Process Outcome Focused Scoring Methodology and Critical Analysis

Performance Improvement Project Validation Process

Outcome Focused Scoring Methodologyand Critical Analysis

Presenter:Christi Melendez, RN, CPHQAssociate Director, PIP Review Team

2

CMS PIP Protocol Changes

Activities III, IV, VII, and VIII have been reversed in order.Activity III: Use a Representative and Generalizable Study PopulationActivity IV: Select the Study Indicator(s)Activity VII: Data Analysis and Interpretation of ResultsActivity VIII: Improvement Strategies

3

Activity I: Choose the Study Topic

HSAG Evaluation Elements: The Study Topic

1. Is selected following collection and analysis of data (critical element)

2.Has the potential to affect member health, outcomes of care, functional status, or satisfaction

4

Activity II: State the Study Question

HSAG Evaluation Elements: The Study Question

1. States the problem to be studied in simple terms and is in the recommended X/Y format (critical element)

5

HSAG Evaluation Elements: The Study Population1. Is accurately and completely defined and

captures all members to whom the study question applies (critical element)

Activity III: Identify the Study Population

6

Activity IV: Select the Study Indicator

HSAG Evaluation Elements: The Study Indicator1.Is well-defined, objective, and measures changes

in health or functional status, consumer satisfaction, or valid process alternatives (critical element)

2.Includes the basis on which the indicator was adopted, if internally developed

3.Allows for the study question to be answered (critical element)

7

HSAG Evaluation Elements: Sampling Techniques1. Specify the measurement period for the sampling

methods used2. Provide the title of the applicable study indicator3. Identify the population size4. Identify the sample size (critical element)5. Specify the margin of error and confidence level6. Describe in detail the methods used to select the sample

* Activity V is only scored if sampling techniques were used.

Activity V: Use Valid Sampling Techniques*

8

HSAG Evaluation Elements: Data Collection

The data collection procedures:

• Identify the data elements to be collected• Include a defined and systematic process for

collecting baseline and remeasurement data

Activity VI: Define Data Collection

9

HSAG Evaluation Elements: Data Collection

The manual data collection procedures:3. Include the qualifications of staff member(s)

collecting manual data4. Include a manual data collection tool that

ensures consistent and accurate collection of data according to indicator specifications (critical element)

Activity VI: Define Data Collection

10

Activity VI: Define Data Collection

HSAG Evaluation Elements: Data Collection

The administrative data collection procedures:5. Include an estimated degree of administrative

data completeness

6. Describe the data analysis plan

11

HSAG Evaluation Elements: Data Analysis1. Is conducted according to the data analysis plan

in the study design2. Allows for the generalization of results to the

study population if a sample was selected (critical element)

3. Identifies factors that threaten internal or external validity of findings

4. Includes an interpretation of findings*Evaluation Elements 1-5, in Activity VII, are scored for PIPs that provide baseline

data

Activity VII: Analyze Data and Interpret Study Results

12

HSAG Evaluation Elements: Interpretation of Study Results5. Is presented in a way that provides accurate, clear, easily

understood information (critical element)6. Identifies the initial measurement and the remeasurement

of study indicators7. Identifies statistical differences between the initial

measurement and the remeasurement8. Identifies factors that affect the ability to compare the initial

measurement with the remeasurement9. Includes an interpretation of the extent to which

the study was successful

Activity VII: Analyze Data and Interpret Study Results

13

Activity VIII:Implementing Interventions and

Improvement StrategiesHSAG Evaluation Elements: Improvement Strategies

1. Are related to causes/barriers identified through data analysis and quality improvement processes (critical element)

2. Are system changes that are likely to induce permanent change

3. Are revised if the original interventions are not successful

4. Are standardized and monitored if interventions are successful

14

Activity IX: Real Improvement*HSAG Evaluation Elements: Report Improvement

1. The remeasurement methodology is the same as the baseline methodology

2. There is documented improvement in processes or outcomes of care

3. There is statistical evidence that observed improvement is true improvement over baseline (critical element)

4. The improvement appears to be the result of planned intervention(s)

* Activity IX is scored when the PIP has progressed to Remeasurement 1 and will be scored on an annual basis until statistically significant improvement is achieved from baseline to a

subsequent remeasurement for all study indicators. Once Evaluation Element 3 receives a Met score, it will remain Met for the duration of the PIP.

15

HSAG Evaluation Elements: Sustained Improvement1. Repeated measurements over comparable time

periods demonstrate sustained improvement, or that a decline in improvement is not statistically significant (critical element)

*HSAG will not validate Activity X until statistically significant improvement has been achieved across all study indicators. Once statistically significant improvement is achieved, the MCO will need to demonstrate a subsequent remeasurement period demonstrating that they sustained that improvement to receive an overall Met validation status. 

Activity X: Sustained Improvement*

16

Old Tool Format 10 Activities 53 Evaluation Elements Activity VII Interventions Activity VIII Data Analysis 13 Critical Elements

New Tool Format 10 Activities 37 Evaluation Elements Activity III Study Population Activity IV Study Indicator(s) Activity VII Data Analysis Activity VIII Interventions 12 Critical Elements

PIP Tool Format

17

Outcome Focused PIP Scoring

HSAG Evaluation Tool 37 Evaluation Elements Total 12 Critical Elements (CE)

Activity I: Activity II: Activity III: Activity IV: Activity V:

Activity VI: Activity VII: Activity VIII: Activity IX: Activity X:

1 CE1 CE1 CE2 CE1 CE

1 CE2 CE1 CE1 CE 1 CE

18

Outcome Focused PIP ScoringChangesActivity VII

- Evaluation Element 5 is critical- MCOs should ensure that data reported in all PIPs are accurate

and align with what has been reported in its IDSS.

Activity IX- Evaluation Elements 3 and 4 have been reversed- New criteria for scoring Activity IX

Activity X- New criteria for scoring Activity X

19

HSAG Evaluation Elements: Assessing for Real Improvement

1. The remeasurement methodology is the same as the baseline methodology

2. There is documented improvement in processes or outcomes of care

3. There is statistical evidence that observed improvement is true improvement over baseline and across all study indicators

4. The improvement appears to be the result of planned intervention(s)

Activity IX:Outcome Focused PIP Scoring

20

HSAG Evaluation Elements: Assessing for Sustained Improvement

1. Repeated measurements over comparable time periods demonstrate sustained improvement, or that a decline in improvement is not statistically significant across all study indicators

Activity X:Outcome Focused PIP Scoring

21

Outcome Focused PIP Scoring Activity IX

– Repeated measurement of the indicators demonstrates meaningful change in performance

– Improvement must be statistically significant for all study indicators to receive an overall Met validation status

– Is scored on an annual basis until statistically significant improvement over baseline has been achieved for all study indicators

– Once Evaluation Element 3 receives a Met score, it will remain Met for the duration of the PIP

– Evaluation elements 3 and 4 are linked

22

Outcome Focused PIP Scoring

Activity X

– Repeated measurement of the indicators demonstrates sustained improvement

– HSAG will not validate Activity X until Evaluation Element 3 of Activity IX is Met

– Once statistically significant improvement has been achieved for all indicators, the MCO will need to document a subsequent measurement period demonstrating sustained improvement in order to receive a Met in Activity X

23

Outcome Focused PIP Rationale

Overall Met Validation Status

– The changes align the actual outcomes of the project with the overall validation status

– Emphasis on statistically significant, sustained improvement in outcomes

24

Critical Analysis

HSAG will be evaluating whether or not…

A current causal/barrier analysis was completed-MCOs should conduct an annual causal/barrier and drill-down analysis in addition to periodic analyses of their most recent data. MCOs should include the updated causal/barrier analysis outcomes in its PIPs.

25

Critical Analysis

HSAG will be evaluating whether or not…

Barriers and interventions were relevant to the focus of the study and can impact the study indicator(s) outcomes

26

Critical Analysis

For any intervention implemented, the MCO should have a process in place to evaluate the efficacy of the intervention to determine if it is having the desired effect. This evaluation process should be detailed in the PIP documentation. If the interventions are not having the desired effect, the MCO should discuss how it will be addressing these deficiencies and what changes will be made to its improvement strategies.

27

Critical AnalysisThe MCO should ensure that the intervention(s) implemented will impact the study indicator(s) outcomes.

Member-focused interventions will not impact a study indicator measuring the quality of service provided by a PCP- WCC HEDIS Measure (Childhood Obesity PIP) Interventions focused on educating MCO staff on HEDIS measures will not impact members accessing care and seeking well-child visits

28

Critical Analysis

The MCO should be cognizant of the timing of interventions. Interventions implemented in the last few months of the year will not have been in place long enough to have an impact on the results.

29

Questions and Answers