You Gives Your Money and You Takes Your Chances Grants QA Can Be: A. Fun B. Exasperating C. Rewarding Marcus E. Kantz USEPA Region 2 Air and Water QA Team.

You Gives Your Money and You Takes Your Chances

Grants QA Can Be:A. Fun B. Exasperating C. Rewarding

Marcus E. KantzUSEPA Region 2Air and Water QA Team [email protected]

PART

Quiz

Who out there knows what the acronym PART stands for in Governmentese?

PART• It stands for:

• Program Assessment Rating Tool

• It was developed by OMB to help assess the management and performance of government programs.

PART• It is used to evaluate a program’s

purpose, design, planning, management, results, and accountability

• to determine its overall effectiveness, and • even to weed out government programs

that are not producing tangible, relevant results.

PART

• Used in EPA’s 2008 Performance and Accountability Report

• To rate every individual program

• Red light – Green light (and Blue for “Data not yet available”)

Assessing Program and Project Performance is Here to Stay

• Equally Clear: Assessing performance and acting on the results is a very “QA Thing to do”.

• Fits perfectly in EPA’s

Plan – Do – Check – Act

Assessing Program and Project Performance is Here to Stay

• Same as EPA Region 2’s PIE

Planning – Implementation – Evaluation

How does this fit into a Grants Talk?

OMB says it used PART to evaluate 98% of all US Government Programs.

• Must include some Grants Programs

• The great Karnak picks the EPA Pollution Prevention, or P2 Program

What did OMB find out about the P2 grants program?

• It was “Moderately Effective”

– The second highest rating

– One of the highest of all EPA ratings

• Yippee !

• Soooooooo… ?

What is P2 about, and why do we care?

• P2 gives grants to states who help businesses find ways to reduce use and discharge of hazardous chemicals.

• We care because our P2 POs asked about QA for Environmental Performance Measures for their Grants.

New Requirements(Always Bring Trouble, Right?)

• P2 Program has new requirements: Assess project effectiveness in terms of Actual Environmental Outcomes.

• Most P2 “training grants” become “environmental operations” grants

• So who knew? Practically no one.

• Some PO’s asked us. We pondered.

It Happened to Fit Into Our Audit Schedule

• Donna Ringel, Kathryn Seaver and I in the Region 2 QA Office were planning an Assessment of the Water Grants Program.

• The P2 Program in in one of the Water Branches.

• We looked at grants in same time period as OMB’s assessment (not intentionally).

What did we do in the Audit?

• Looked at all grants from 4 Water Branches

• Were the most basic QA measures taken?

– Did the PO properly determine if grant involved generation and/or use of Environmental Information?

– If so, did PO ensure that QMP or QAPP is in place?

Then what?

• The POs who did it right got Gold Stars.

• For POs who did not, we:

– Tried to figure out why not.

– Wanted to help POs do it better in the future.

What did we find,and why does it matter here?

• Of the 202 grants, 149 involved collection and/or use of environmental information.

• Of those 149, for 47 the POs had NOT ensured an appropriate QMP or QAPP approved.

• That is, One Third of environmental data grants occurred outside the QA System.

How could this happen in the Greatest QA Program in the World?

Four Recurring Causes:

1. PO Uncertain about need to apply QA to P2 grants using data ONLY for assessing environmental effectiveness of the grant.

2. PO Uncertain about QA for multi-project grants

3. PO Uncertain about QA for “repeating” grants

4. PO Failed to follow up once determined that QAPP needed

Types of Grants Having Inadequate QA

Other19%

Recurring Program Grants

19%

Both Multi-Project and Recurring

Program Grants19%

Multi-Project/Partner

Grants15%

P2 Grants28%

PO Uncertain about QA for P2 Grants using data Only for

assessing environmental effectiveness

• 13 P2 grants: 29% of the 47 grants that “failed” our audit.

• Pointed to situation under the QA umbrella

How does this become a QA issue?• P2 grantees who really do just training,

now need to demonstrate how much pollution, sickness, or death this work is project preventing

• For Credibility: need to show how they figured it out and how reliably

• Clearly fits in QA Program, whether the grantees are actually measuring something, or estimating, or guessing

The Light Went On

• We realized that these “naked” performance measures:

– Are very real

– Are important

– Require QA

– AND We Had Missed ‘em

• But we couldn’t do it for them

QA Geek Tip-of-the-Week• QA Guys Don’t Decide How Good is

Good Enough. The Data Users Decide.• QA Guys help them do it.• For P2, the Home Office decides. We

help them figure out how best to report it.

• Same is true for ARRA. Programs specify the measures, with influence from above.

The Key: the Graded Aproach

• Program Managers figure out what they need, and what level of quality they require…

• QA Practitioners guide them in developing their systematic planning.

• Using the Graded Approach.

Provide Reassurance• Show Project Managers and QA

Practitioners the Graded Approach

• The Level of Sophistication & Rigor proportional to the Complexity & Importance of the Project AND

• The importance of the QUALITY of the measurements.

If an Order of Magnitude is OK,

It’s OK

For P2: a mini-QAPP

• Just the facts

• And not many of them

• Tell the POs What You’ll Report, and How You’ll Come Up With It.

• If the POs (Regions & HQ) say it’s OK, it’s OK

What About the Issues with the Water Grants?

• Multi-Project Grants:34% of the 47 grants that “failed the Audit

• Have several interlocking but separate projects on the same “water”

• Often separate sub-contractors, etc.• POs (and grantees) forget to get QAPPs

for all of them.• Result: projects with no QA Planning or

Oversight

The “Repeating Grants”

• Issued Year after Year, usually with evolving focus and projects.

• In 38% of the failed audits the POs failed to require review and/or revision of the Old QAPPS or development of new QAPPs

• Not bureaucratic.

• Major projects, major evolution, but major planning not happening

PO Knows It But Doesn’t Do It• On 10 of the 47 failed grants, PO

checked box in IGMS “needs QAPP” but failed to make it happen.

• IGMS software even sends out reminders and ticklers, but they’re not ticklish.

• QA and Grants Offices had a plan but didn’t make it sufficiently fail safe.

We Learned a Lot (1)

• Confirmed that some POs don’t know their jobs, or don’t pay attention

• Confirmed that isolated environmental perrormance measure projects like P2 are important and need attention (like mini-QAPP)

• Learned we need to deliver much more focused training.

We Learned a Lot (2)

• That training needs on QA roles and responsibility needs to be combined with carefully targeting tools (like better utilized IGMS “tickling” capabilities)

QAOs - Wake Up• IT IS OUR JOB to make sure that our

Project Officers and Project Managers for Grants, Interoffice Agreements and Contracts understand that any and all environmental performance measures are covered by our QA requirements and all require an approved QAPP, utilizing the Graded Approach.

• Even if they have no other data operations.

Communicate withFunds Recipients

• If they’re delegated QA responsibility, it’s all their job.

• But they are unlikely to realize their role

• Unlikely to understand importance.

• We need “Terms and Conditions” explaining this to them

Summary• Our audit finding woke us up• Hope this talk wakes you up.• We need to apply Graded Approach to

any program or project attempt to demonstrate environmental effectiveness

• We in QA need more vigilance in making sure our Program Community partners know their responsibilities and perform them properly

Bye Bye