1 Survival Skills: Learn How to Measure What Matters January 7, 2008 Presented by: Yvonne M. Watson and Britta Johnson Evaluation Support Division National.

1

Survival Skills: Learn How to Measure What Matters

January 7, 2008

Presented by: Yvonne M. Watson and Britta JohnsonEvaluation Support DivisionNational Center for Environmental InnovationOffice of Policy, Economics and InnovationU.S. Environmental Protection Agency

2

Presentation Goals

Enable participants to:

• Use a logic model to help understand how program resources and activities produce specific outputs and how they connect to outcomes to achieve program/project goals and objectives

• Understand the different types of performance measures (resource, outputs, outcomes, efficiency, productivity, etc.)

• Use a step-by-step approach to develop, assess, and choose the appropriate measures for their organization, program, or project

3

Steps to Developing, Implementing and Reporting Performance Measurement Information

I. Identify Team/Develop Performance Measurement Plan

II. Describe the Program

III. Develop Performance Measurement Questions

IV. Develop Measures

VI. Analyze and Interpret Information

VII. Develop the Report

V. Collect Information

4

Performance Measurement: Session Agenda

Module 1: Planning for Performance Measurement

Module 2: Identifying and Developing Performance Measures

5

PERFORMANCE MANAGEMENTPerformance management includes activities to ensure that goals are consistently being met in an effective and efficient manner. Performance management tools include logic models, performance measurement and program evaluation.

Logic Model

Tool/framework that helps identify the program/project

resources, activities, outputs customers, and

outcomes.

Performance Measurement

Helps you understand what

level of performance is achieved by the program/project.

Program Evaluation

Helps you understand and

explain why you’re seeing the

program/project results.

PERFORMANCE MANAGEMENT TOOLS

6

Drivers for Performance Measurement Good Program Management

Government Performance and Results Act (GPRA) of 1993

• Requires EPA to report schedules for and summaries of program evaluations that have been or will be conducted and identify those that influence development of the Agency’s Strategic Plan.

OMB’s Program Assessment Rating Tool (PART)

• Tool designed to assess and evaluate programs across the federal government.

Environmental Results Order 5700.7

• Requires EPA grant officers and grant recipients to identify outputs and outcomes from grants and connect them to EPA’s strategic plan.

Regional Priorities

7

Orientation Exercise

In small groups, discuss and complete the following incomplete sentences:

• A program is … • Performance Measurement is …

Performance Measurement leads to …

• How do we use measurement in our everyday lives?

One thing we want to learn from this workshop is…

After completing the sentences, select a reporter.

8

Module 1:

Planning for Performance Measurement

9









10

Part A. Identify Team Members

Individuals responsible for the design, collection and reporting of performance information

Individuals with intimate knowledge of the program/project

Individuals who have a vested interest in the conduct/impact of the program

Individuals with some knowledge of the measurement planning process

Identify a Skeptic!

11

Part B. Develop the Measurement Plan: Things to Consider

Purpose of the performance measurement system

Project/ program mission

Primary audience

Scope (including program description/ logic model)

Context (organizational, management, political)

Role, expectations for program staff, participants, and key stakeholders

12

Part B. Measurement Plan Outline

Performance measurement questions

Data collection/analysis

Reporting (including feedback loop)

Resources (staff and budget)

Timeline

Communication

Steps to monitor implementation of the plan

13









14

Elements of the Logic Model

Inter-mediate

Changes in behavior, practice or decisions.

Behavior

Inter-mediate

Changes in behavior, practice or decisions.

Behavior

Customer

User of the products/ services. Target audience the program is designed to reach.

Customer

User of the products/ services. Target audience the program is designed to reach.

Activities

Things you do– activities you plan to conduct in your program.

Activities

Things you do– activities you plan to conduct in your program.

Outputs

Product or service delivery/ implementation targets you aim to produce.

Outputs

Product or service delivery/ implementation targets you aim to produce.

Resources/ Inputs:

Programmatic investments available to support the program.

Resources/ Inputs:

Programmatic investments available to support the program.

Short-term

Changes in learning, knowledge, attitude, skills, understanding.

Attitudes

Short-term

Changes in learning, knowledge, attitude, skills, understanding.

Attitudes

Long-term

Change in condition.

Condition

Long-term

Change in condition.

Condition

External Influences

Factors outside of your control (positive or negative) that may influence the outcome and impact of your program/project.

External Influences

Factors outside of your control (positive or negative) that may influence the outcome and impact of your program/project.

Outcomes

WHYHOW

PROGRAM RESULTS FROM PROGRAM

15

Resources Activities OutputsFor

CustomersShort-term outcomes

Intermediate outcomes

Long-term outcomes

Fitness Plan

Strength Training

Plan

Strength Training

Plan

External Influences: Injuries, Broken Equipment

Source: Britta Johnson, 2007

Britta,Yoga Mat,

Cardio Equipment,

Weights

Britta,Yoga Mat,

Cardio Equipment,

Weights

Build Muscles

Build Muscles

Physically Fit (Increased Flexibility, Increased Cardiac

Capacity, Increased Strength)

Physically Fit (Increased Flexibility, Increased Cardiac

Capacity, Increased Strength)

BrittaBritta

Practice Yoga

Practice Yoga

Muscles Stretched

Muscles Stretched

Cardio Workout

Cardio Workout

Calories Burned,

Heart Rate Elevated

Calories Burned,

Heart Rate Elevated

Work Out on a

Regular Basis

Work Out on a

Regular Basis

Increased Drive to

Work Out

Increased Drive to

Work Out

16

Develop and design PE, PM, IA and Logic Model curriculum and exercises.

Deliver PE, PM, IA and Logic Model training.

• PE skills are used by customers in the work environment

• # of evaluations conducted and managed increased.

Resources

Outcomes

Short-term

Intermediate Long-term

OutputsActivities Customers

Knowledge of PE increased/ improved.

Customers equipped with skills to manage and conduct evaluations.

ESD Staff:

Y. Watson

M. Mandolia

J. Heffelfinger

D. Bend

C. Kakoyannis

Access to: John McLaughlin

NCEI Staff

IAC Staff

PEN

PEC Winners

HQ/ Regional managers & staff

Partners

OCFO

OW

OSWER

ORD

OARM

Knowledge of PM increased/ improved.

Customers equipped with skills to develop measures.

Technical assistance delivered.

Strategic Plan

ESD TRAINING LOGIC MODEL

Knowledge of Logic modeling increased/ improved.

Customers equipped with skills to develop logic models of their programs.

• Customers understanding of their programs is improved.

• PM skills are used by customers in the work environment.

• # of staff developing measures is increased.

Customers use program evaluation regularly and systematically to improve environmental programs in terms of: - environmental & health outcomes- reduced costs- cost effective-ness- EJ Benefits-Public Involvement- Efficiency

Environ-m

ental programs m

ore effectively

achieve their strategic goals.

Quality of evaluations managed and conducted is improved.

Quality of measures developed and reported is improved.

Provide technical assistance for workshop/ training attendees.

PM training materials.

Customers complete training.

PE training materials.


NCEI Staff

SIG Recipients

HQ/ Regional managers & staff

States/Tribes

SBAP

CARE

Customers use logic models to help conduct evaluations and develop measures.

Logic Model training materials.


NCEI Staff

SIG Recipients

States

SBAP

CARE

Provide guidance for Environmental Results Grants Training.

Facilitate Train the trainer sessions for PE, PM and Logic Modeling.

Environmental Results Grants Training materials.

Partners complete training.

IA training materials.


OCFO, OW, OSWER, ORD, OARM

EPA Project Officers & Grant Managers

Partners deliver PE, PM and Logic model training to their clients/ customers.

EPA POs & GMs recognize outputs/ outcomes in grant proposals

ESD Training Goal: To provide training to enable our EPA partners to more effectively conduct and manage program evaluations and analyses and develop performance measures that can be used to improve their programs and demonstrate environmental results.

17

Two Important Rules to Follow

For every action identified in the Logic Model, the must be an output that connects to an outcome through a specific customer.

OR

An action must produce an output that becomes a key input to another activity.

THINK CONNECTIONS!

18

Exercise 1: Logic Modeling

Developing a logic model

19

Outcomes

Shorter-term Intermediate Longer-termOutputsActivities Customers

Figure 2-1. EPA Region 1 New England Marina Initiative Logic Model

Program Goal: Improve marina environmental performance through implementing an effective regional education and outreach campaign that includes: improving awareness and compliance, increasing implementation of best management practices, and enhancing the current assistance provider network to help achieve sustained industry environmental support.

PLANNING• Establish goals, identify resources &

gaps• Design strategic projects to achieve

goals• Identify Marina owners in Region 1• Monitor progress

EDUCATION/TECHNICAL ASSISTANCE• Conduct environmental workshops • Conduct special demonstration training

events• Provide technical assistance w/

Regional Marina Workgroup• Develop technical assistance tools • Develop & maintain Marina website• Provide regulatory guidance

OUTREACH/NETWORKING• Facilitate Regional Marina Workgroup• Conduct internal/external meetings• Prepare written updates• Establish partnerships

Pro

jec

t P

lan

nin

g a

nd

Nee

ds

Ass

essm

ent

Sta

ge

Pro

jec

t Im

ple

men

tati

on

S

tag

e

• State Environmental Agencies (Regional Marina Workgroup)

• State & Regional Marine Trades Associations

• NEWMOA• NEIWPCC• NGOs

• Workshops• Technical

assistance• Marina Guidance

Documents• Fact sheets• Marina

Environmental Management Plan Workbook

• Marina Website• Regulatory

interpretations

• Baseline data of Marina performance

• Presentations on aggregated results

• Meetings• Written updates• Communication

letters• Network of

Partners

• Strategic plans

• Marina owners & staff

• Boaters

Enforcement actions are taken

Region 1& Partners’awareness & understanding of marina’s environmental impact improves

Enhanced support for marinas

Region 1 & Partners capabilities to assist marina owners is enhanced

Emerging issues are identified and studied

Collaboration to provide assistance

Strategic projects implemented

Marina owners & boaters have increased knowledge of regulations and best practices

Marina owners & boaters adopt marina best practices

Improved regulatory compliance from marina owners

Assistance provided to marina owners

Region 1& Partners explore ways to address issues

Partners agree on need for assistance in this sector

Trades Associations are more proactive in addressing Marina environmental issues

Region 1 & Partners decide to provide resources to address issues

Marina owners & boaters take corrective action to comply with regulations

Communication among Partners improves

Additional compliance data collected

Effective relationships established through a regional Marina stakeholder network

Improved environmental performance from marina owners

Healthier marina communities

February 19, 2007

MEASUREMENT • Develop measurement process using

statistical principles, random sampling techniques

• Develop environmental indicators• Develop measurement tool (checklist of

indicators)• Conduct baseline on-site visits• Aggregate results

Note: Gray boxes in the “Project Planning and Needs Assessment Stage” reflect activities in the “Project Implementation Stage.”

• Inventory of marina owners

20

Module 2:

Identifying and Developing Performance Measures

21

Definitions:

Performance Measurement: The ongoing monitoring and reporting of program progress and accomplishments, using pre-selected performance measures.

Performance Measure: A metric used to gauge program or project performance.

Indicators: Measures, usually quantitative, that provide information on program performance and evidence of a change in the “state or condition” in the system.

22

Differences Between an Indicator and Performance Measure

Terms used to describe data associated with outcomes

Difference between terms is degree of control we have over them

Performance measures help assess the effect of the program

Indicators measure the “state of” something typically in the natural environment

23

Uses of Performance Measurement

Monitoring and reporting

Strategic planning

Budgeting and financial management

Program management

Process improvement

Contract management

Communication with publicSource: Chapel, T., Center for Disease Control and Prevention, Power Point Presentation, Program Alignment, Performance Measurement, and Program Improvement

24

Limitations and Pitfalls of Performance Measures

Provide descriptive data, not rigorously evaluative.

Can encourage undesirable behavior.

May require too much time and effort.

Can be ignored, not automatically used.

Performance measurement is time-bound, and context-bound.

25









26

Performance Questions Across the Performance Spectrum

PROGRAM ELEMENTS:

Resources

(We use these)

Activities/

Outputs

(To do these things)

Target Customer

(For these people)

Short term Outcome

(To change them in these ways)

Intermediate Outcome

(So they can do these things)

Long-Term Outcome

(Which leads to these outcomes)

PERFORMANCE QUESTIONS:

Do we have enough,

The right,

The necessary level,

The consistency?

Are we doing things the way we say we should?

Are we producing products and services at the levels anticipated?

According to anticipated quality indicators measures?

Are we reaching the customers targeted?

Are we reaching the anticipated numbers?

Are they satisfied?

Did the customer’s attitude, knowledge, skills or understanding change?

Are customers using the change as expected? With what results?

Are customers served changing in the expected direction and level?

If so, what did we (others) do to cause the change?

What changes in condition have occurred?

Did the program achieve its goals and objectives?

EXTERNAL INFLUENCES:

What factors might influence my program’s success?

27

Measures Across the Logic Model SpectrumElement Definition Example Measure

Resources/ Inputs

Measure of resources consumed by the organization.

Amount of funds, # of FTE, materials, equipment, supplies (etc.).

Activities Measure of work performed that directly produces the core products and services.

# of training classes offered as designed; Hours of technical assistance training for staff.

Outputs Measure of products and services provided as a direct result of program activities.

# of technical assistance requests responded to; # of compliance workbooks developed/delivered.

Customer Reached

Measure of target population receiving outputs.

% of target population trained; # of target population receiving technical assistance.

Customer Satisfaction

Measure of satisfaction with outputs. % of customers dissatisfied with training; % of customers “very satisfied” with assistance received.

Outcomes Accomplishment of program goals and objectives (short-term and intermediate outcomes, long-term outcomes--impacts).

% increase in industry’s understanding of regulatory recycling exclusion; # of sectors that adopt regulatory recycling exclusion; % increase in materials recycled.

28

Work Quality Measures

Category Definition Examples

Efficiency Measure that relates outputs to costs.

Cost per workbook produced; cost per inspection conducted.

Productivity Measure of the rate of production per some specific unit of resource (e.g., staff or employee). The focus is on labor productivity.

Number of enforcement cases investigated per inspector.

Cost

Effectiveness

Measure that relates outcomes to costs.

Cost per pounds of pollutants reduced; cost per mile of beach cleaned.

Service

Quality

Measure of the quality of products and services produced.

Percent of technical assistance requests responded to within one week.

29

Types of Performance Measures

1. Number of technical assistance requests responded to.

2. Number of workshops conducted

3. Gallons of waste gasoline emitted into marinas

4. Number and percent of marina owners & boaters reporting increased knowledge of regulations and best practices

5. Number and percent of marina’s contacted

6. Percent increase in targeted marina owners/boaters adopting BMPs

7. Total combined in-kind hours contributed by partner organizations

1. Output

2. Activity

3. Long-term outcome

4. Short-term Outcome

5. Customer reached

6. Intermediate Outcome

7. Resource

Example Type of Measure

30









31

Steps for Developing Measures

Step 1: Identify Potential Measures

Step 2: Assess Each Measure

Step 3: Choose the Best Measures

Step 4: Identify Baseline, Target, Timeline and Reporting Schedule

32

Key Steps in Identifying Potential Measures

Identify the information needed and the audience

Identify measures in existing documents

Review the logic model and select the appropriate logic model element

Express the logic model element as a performance measure

Determine if the measure clearly relates to the program/project goal or objective

33

Identify the Information Needed and the Audience

Review the performance measurement questions developed earlier

Consider what information is needed to assess whether your program/project is meeting its goals and objectives.

Ask yourself:

Who needs to know what about the program, why, and in what format?

34

Identify Measures in Existing Documents

Review measures specified in:

• Program/Project Mission, Goals, Objectives, Service standards

• Legislation, Strategic plans (GPRA), Court Orders, PART, Regional Plans, National Program Management Guidance, Regional Priority Commitments

• Previous evaluations and research reports

• Consider other sources

35

Review the Logic Model

Review the logic model –

• Identify the aspects of performance that are most important to measure (resources, activities, outputs, outcomes)

• Identify contextual factors that could influence the program either positively or negatively and generate measures for them as appropriate

36

Outcomes

Shorter-term Intermediate Longer-termOutputsActivities Customers

Figure 2-1. EPA Region 1 New England Marina Initiative Logic Model

Program Goal: Improve marina environmental performance through implementing an effective regional education and outreach campaign that includes: improving awareness and compliance, increasing implementation of best management practices, and enhancing the current assistance provider network to help achieve sustained industry environmental support.

PLANNING• Establish goals, identify resources &

gaps• Design strategic projects to achieve

goals• Identify Marina owners in Region 1• Monitor progress

EDUCATION/TECHNICAL ASSISTANCE• Conduct environmental workshops • Conduct special demonstration training

events• Provide technical assistance w/

Regional Marina Workgroup• Develop technical assistance tools • Develop & maintain Marina website• Provide regulatory guidance

OUTREACH/NETWORKING• Facilitate Regional Marina Workgroup• Conduct internal/external meetings• Prepare written updates• Establish partnerships

Pro

jec

t P

lan

nin

g a

nd

Nee

ds

Ass

essm

ent

Sta

ge

Pro

jec

t Im

ple

men

tati

on

S

tag

e

• State Environmental Agencies (Regional Marina Workgroup)

• State & Regional Marine Trades Associations

• NEWMOA• NEIWPCC• NGOs

• Workshops• Technical

assistance• Marina Guidance

Documents• Fact sheets• Marina

Environmental Management Plan Workbook

• Marina Website• Regulatory

interpretations

• Baseline data of Marina performance

• Presentations on aggregated results

• Meetings• Written updates• Communication

letters• Network of

Partners

• Strategic plans

• Marina owners & staff

• Boaters

Enforcement actions are taken

Region 1& Partners’awareness & understanding of marina’s environmental impact improves

Enhanced support for marinas

Region 1 & Partners capabilities to assist marina owners is enhanced

Emerging issues are identified and studied

Collaboration to provide assistance

Strategic projects implemented

Marina owners & boaters have increased knowledge of regulations and best practices

Marina owners & boaters adopt marina best practices

Improved regulatory compliance from marina owners

Assistance provided to marina owners

Region 1& Partners explore ways to address issues

Partners agree on need for assistance in this sector

Trades Associations are more proactive in addressing Marina environmental issues

Region 1 & Partners decide to provide resources to address issues

Marina owners & boaters take corrective action to comply with regulations

Communication among Partners improves

Additional compliance data collected

Effective relationships established through a regional Marina stakeholder network

Improved environmental performance from marina owners

Healthier marina communities

February 19, 2007

MEASUREMENT • Develop measurement process using

statistical principles, random sampling techniques

• Develop environmental indicators• Develop measurement tool (checklist of

indicators)• Conduct baseline on-site visits• Aggregate results

Note: Gray boxes in the “Project Planning and Needs Assessment Stage” reflect activities in the “Project Implementation Stage.”

• Inventory of marina owners

37

Express the Logic Model element as a performance measure

Consider how to express the measure in terms of:

• Data:– Raw Numbers (number of marinas in compliance)– Averages (mean number of compliance issues) – Percentages (% of marinas in compliance)– Ratios (Cost per marina in compliance)– Rates (number of compliance violations per 100

marinas)

• Unit of Measure:– Is it appropriate to the measure?

38

New England Marina Initiative Measures

Activities Outputs Customer reached Short-term Outcome

Intermediate Outcome

Long-term Outcomes

Establish goals, identify resources, & gaps

Facilitate Regional Marina Workgroup

Develop measurement process using statistical principles, random sampling techniques

Conduct environmental workshops

Inventory of marina owners

Strategic Plans

Network of Partners

Baseline Data

Technical assistance provided

Brochures & Fact sheets

State Environmental Agencies (Regional Marina Workgroup)

State & Regional Marine Trade Associations

NGOs

Marina Owners & Staff

Boaters

Region 1 & Partners’ Awareness and Understanding of Marinas’ Environmental Impact Improves

Region 1 & Partners Agree to Address Issues

Marina Owners & Boaters have Increased Knowledge of Regulations and Best Practices

Emerging Issues are Identified

Marina Owners & Boaters take Corrective Action to Comply with Regulations

Marina Owners & Boaters Adopt Marina Best Practices

Trade Associations are More Proactive in Addressing Marina Environmental Issues

Healthier Marina Communities

# of measurement processes developed

# of workshops conducted

# of guidance documents and fact sheets distributed

Number of technical assistance requests responded to

#/% of marinas contacted

Level of satisfaction with technical assistance

#/% of marina owners & boaters reporting increase in knowledge of regulations and best practices

% increase in targeted marina owners/ boaters adopting best practices

Gallons of waste gasoline emitted into marinas

Efficiency: Cost per training workshop

Productivity: Hours per technical assistance visit per FTE

Ex

am

ple

Me

as

ure

sL

og

ic M

od

el

Ele

me

nts

39

Determine whether the measures clearly relate to the mission/goal

Review the program/project mission and or goal

• What key activities, outputs or outcomes are specified in the mission or goal?

Review the list of potential measures developed

• Will the data collected from the measures developed clearly demonstrate that the mission and or goal was accomplished?

40

Determine whether the measures clearly relate to the mission/goal

New England Marina Initiative Purpose:

The New England Marinas Initiative is a regionally coordinated assistance initiative to improve marina environmental performance through implementing an effective regional education and outreach campaign that includes: improving awareness and compliance, increasing implementation of best management practices, and enhancing the current assistance provider network to help achieve sustained industry environmental support.

Performance Measures

# of measurement processes developed

# of workshops conducted

# of guidance documents and fact sheets distributed

Number of technical assistance requests responded to

#/% of marinas contacted

Level of satisfaction with technical assistance

#/% of marina owner & boaters reporting increase in knowledge of regulations and best practices

% of targeted marina owners/boaters adopting best practices

Gallons of waste gasoline emitted into marinas

41

Exercise 2: Performance Measures

Developing Your Own Measures

42

Step 2: Assess Each Measure

Assess the feasibility of the measures for:

data collection

data quality

analysis

reporting

43

Assess Each Measure for Data Collection

Availability

• Existing data vs. new data

Frequency

• One time collection, continued data collection

Utility

• Data available for use

• Supports an acceptable baseline

Cost

• Overall implementation cost

44

Assess Each Measure for Data Quality

Reliability

• Provides consistent readings

Validity

• Measures what it is supposed to measure

Objectivity

• Free from bias and represents reality

45

Assess Each Measure for Analysis:

Statistically reliable

• Population data

• Sample data

Type of analysis

• Trends over time

• Actual performance against targets or standards

• Variation across units (internal benchmarking)

• Against benchmarks (external benchmarking)

External factors

• Direct or near direct control over the measure• Account for impact of external factors on the measure

46

Step 3: Choose the Best Measures

Assess the value of the measures in relation to the goals and objectives of the program.

• Required

• Important

• Interesting

Select final list of measures – you won’t be able to collect data for all measures.

Check in with managers and stakeholders.

Identify a priority list of measures

47

Step 4: Identify a Standard

For each performance measure develop a:

1. Baseline – current state

2. Target – desired level of performance

3. Timeline – date when performance will be achieved

48

Criteria for Useful Performance Measures Is each measure: If so, then it will be:

Objective-linked

Directly related to clearly stated objectives for your program.

Responsibility-linked

Matched to specific organizational units and people that are responsible for AND capable of taking action to improve performance.

Organizationally acceptable

Valued by all levels in the organization, used as a management tool, and viewed as being “owned” by those accountable for performance.

Comprehensive Inclusive of all relevant aspects of the program performance; e.g., measuring quality and quantity.

Credible Based on accurate and reliable data sources and methods, and to the extent possible, not open to manipulation or distortion.

Cost-effective

Acceptable in terms of data collection, processing, and reporting.

Compatible

Integrated with existing information systems.

Comparable with other data

Useful in making comparisons; e.g., performance can be compared from period to period, with peers, to other programs.

Easy to interpret and report

Presented graphically and accompanied by commentary!

[1] Adapted from Price Waterhouse – Center for Performance Measurement

49

Tips for Choosing the Best Measures

For each measure ask…

Does the measure clearly relate to the project goal and objective?

Is the measure important to management and stakeholders?

Is it possible to collect accurate and reliable data for the measure?

Taken together, do the measures accurately reflect the key results of the program, activity or service?

Is there more than one measure for each goal or objective?

Are your measures primarily outcome, efficiency, or quality measures?

50

Muddy Waters

Describe three different types of performance measures (slide 36-37)

Why is it important to consider your audience when developing performance measures (slide 41)?

How can a logic model help you develop performance measures (slide 45)?

51

Contacts:

Yvonne M. Watson

[email protected]

(202) 566-2239

Britta Johnson

[email protected]

(202) 566-1465

1 Survival Skills: Learn How to Measure What Matters January 7, 2008 Presented by: Yvonne M. Watson and Britta Johnson Evaluation Support Division National.

Documents

performance measurementhelps

performance measurementsteps

performance measurementmodule

level of performance

program resources

program evaluationhelps

outputs customers

specific outputs