1 AUTOPSY OF A FAILED EVALUATION Examination Against The Government Auditing Standards Daniel L. Stufflebeam.

11

AUTOPSY OF A FAILED AUTOPSY OF A FAILED EVALUATIONEVALUATION

Examination Against Examination Against The Government The Government Auditing StandardsAuditing Standards

Daniel L. StufflebeamDaniel L. Stufflebeam

22

FOCUS:FOCUS:

The Government Auditing The Government Auditing StandardsStandards

Commonly known as the Commonly known as the GAO Standards or the GAO Standards or the Yellow Book StandardsYellow Book Standards

33

THE SESSION’S PARTS THE SESSION’S PARTS AREARE

1.1. A rationale for evaluation A rationale for evaluation standardsstandards

2.2. A case showing the utility of A case showing the utility of The GAO StandardsThe GAO Standards

3.3. The contents of The contents of The GAO The GAO StandardsStandards

4.4. Recommendations for Recommendations for applying applying The GAO StandardsThe GAO Standards

44

DISCUSSION PROTOCOLDISCUSSION PROTOCOL

Record your questions on a piece Record your questions on a piece of paper during the presentation .of paper during the presentation .

At appropriate times you may pose At appropriate times you may pose your questions so that the class can your questions so that the class can hear and discuss them. hear and discuss them.

55

PART 1PART 1

A RATIONALE FOR A RATIONALE FOR EVALUATION EVALUATION STANDARDSSTANDARDS

66

STANDARDS FOR STANDARDS FOR EVALUATIONS AREEVALUATIONS ARE

Widely shared principles Widely shared principles for guiding & judging for guiding & judging the conduct of the conduct of evaluationsevaluations

Developed & approved by Developed & approved by experts in the conduct & experts in the conduct & use of evaluationuse of evaluation

77

THE EVALUATION FIELD’S THE EVALUATION FIELD’S STANDARDS INCLUDESTANDARDS INCLUDE

2004 AEA2004 AEA Guiding Principles for Guiding Principles for EvaluatorsEvaluators

20112011 Program Evaluation Standards Program Evaluation Standards

2007 2007 Government Auditing Government Auditing Standards Standards (GAO Standards)(GAO Standards)

88

STANDARDS FOR STANDARDS FOR EVALUATIONS PROVIDEEVALUATIONS PROVIDE

Principled DirectionPrincipled Direction

Technical GuidanceTechnical Guidance

A Basis forA Basis for Professional Professional CredibilityCredibility

A Basis forA Basis for Evaluating Evaluations Evaluating Evaluations

A Basis forA Basis for Public Accountability Public Accountability

99

EVALUATORS IGNORE OR EVALUATORS IGNORE OR FAIL TO MEET STANDARDS FAIL TO MEET STANDARDS TOTO

Their professional perilTheir professional peril

The detriment of their The detriment of their clientsclients

1010

FAILURE TO MEET STANDARDS FAILURE TO MEET STANDARDS CAN CONTRIBUTE TOCAN CONTRIBUTE TO

Conflict of interest & biasConflict of interest & bias Erroneous conclusionsErroneous conclusions Cover-up of findingsCover-up of findings Misguided decisionsMisguided decisions Breakdown of trustBreakdown of trust Organizational repercussionsOrganizational repercussions Personal losses & tragediesPersonal losses & tragedies Lowered credibility of evaluators, Lowered credibility of evaluators,

their organizations, & their their organizations, & their professionprofession

Increased government controls Increased government controls

1111

QUESTIONS RE PART 1QUESTIONS RE PART 1

What are What are positive reasonspositive reasons for for learning & applying learning & applying evaluation standards?evaluation standards?

What are possible What are possible drawbacksdrawbacks to the use of standards?to the use of standards?

Have you seen Have you seen bad bad consequencesconsequences for evaluations for evaluations that didn’t meet standards?that didn’t meet standards?

1212

What are your What are your questions regarding questions regarding Part 1?Part 1?

1313

PART 2: A CASEPART 2: A CASE

A UNIVERSITY’S REVIEW OF A UNIVERSITY’S REVIEW OF ITS GRADUATE PROGRAMSITS GRADUATE PROGRAMS

(A university group should have (A university group should have followed evaluation standards followed evaluation standards but didn’t.)but didn’t.)

1414

CONTEXT WAS CONTEXT WAS PROBLEMATICPROBLEMATIC

Board had Board had voted voted confidence in confidence in the president the president (12/06)(12/06)

Faculty gave Faculty gave president & president & provost low provost low ratings (2/07)ratings (2/07)

Enrollment was Enrollment was decliningdeclining

U. faced a U. faced a fiscal crisisfiscal crisis

Review focused Review focused on resource on resource allocationallocation

Morale was lowMorale was low

1515

REVIEW’S STATED REVIEW’S STATED PURPOSES:PURPOSES:

Address a fiscal crisisAddress a fiscal crisis over the over the university’s inability to support all university’s inability to support all of its programs & maintain of its programs & maintain excellenceexcellence

Determine which programs are Determine which programs are highest strategic priorities highest strategic priorities based on qualitybased on quality

Identify Identify programs for programs for increased fundsincreased funds

1616

SCOPE OF THE REVIEWSCOPE OF THE REVIEW

To be completed within 1 yearTo be completed within 1 year

All masters & doctoral programsAll masters & doctoral programs

Launched on 7/19/06Launched on 7/19/06

114 programs were reviewed114 programs were reviewed

1717

THE REVIEW’S PLANTHE REVIEW’S PLAN

Keyed to Dickeson book (chapter 5)Keyed to Dickeson book (chapter 5) Data bookData book Program’s reportProgram’s report Dean’s reportDean’s report Review team’s reportReview team’s report Appeals of review team’s reportAppeals of review team’s report Provost’s final reportProvost’s final report Board’s decisionsBoard’s decisions

1818

THE REVIEW PLAN’S THE REVIEW PLAN’S OMISSIONSOMISSIONS

NoNo update of the university’s mission update of the university’s mission No No appeals of provost’s conclusionsappeals of provost’s conclusions No No adoption of standards for reviewsadoption of standards for reviews Minimal Minimal participation of outside participation of outside

evaluatorsevaluators No No external metaevaluationexternal metaevaluation

1919

GENERAL REVIEW GENERAL REVIEW CRITERIACRITERIA External External demanddemand Quality of student & program Quality of student & program

outcomesoutcomes Quality of program Quality of program administration & administration &

planningplanning Program Program size, scope, & productivitysize, scope, & productivity Program Program impact, justification, & impact, justification, &

essentialityessentiality Opportunity analysisOpportunity analysis Compelling program factor Compelling program factor (features (features

that make it unique & excellent)that make it unique & excellent)

2020

DEFINITION OF SUB-DEFINITION OF SUB-CRITERIACRITERIA

ManyMany

Evolved throughout the Evolved throughout the reviewreview

Caused confusion & Caused confusion & controversycontroversy

2121

CRITERIA OMITTED FROM CRITERIA OMITTED FROM DICKESON’S LISTDICKESON’S LIST

History, development, & History, development, & expectations of the programexpectations of the program

Internal demand for the programInternal demand for the program Quality of program inputs & Quality of program inputs &

processesprocesses Revenue & other resources Revenue & other resources

generatedgenerated Program costs & associated Program costs & associated

costscosts

2222

EVALUATION EVALUATION PROCEDURESPROCEDURES

Program’s Program’s self-reportself-report Document & data book reviewDocument & data book review Group & individualGroup & individual interviews interviews Variable Variable protocols for ratings protocols for ratings (1-5)(1-5) Training Training of review team leadersof review team leaders Rating Rating of each program by department, of each program by department,

dean, review team, & provostdean, review team, & provost Synthesis Synthesis by provost & staffby provost & staff

2323

REVIEW PERSONNELREVIEW PERSONNEL

Essentially Essentially internalinternal PProvost rovost was both the was both the primary decision primary decision

maker & de facto lead evaluatormaker & de facto lead evaluator Provost’s staff Provost’s staff assisted the processassisted the process A programA program representative representative wrote the wrote the

program’s report &sent it to deptartment program’s report &sent it to deptartment faculty, dean, & review teamfaculty, dean, & review team

Faculty input variedFaculty input varied across programs across programs The The deandean rated the college’s programs & rated the college’s programs &

sent reports to the department chairs & sent reports to the department chairs & review team (not in original plan)review team (not in original plan)

2424

REVIEW PERSONNEL REVIEW PERSONNEL (continued)(continued)

Seven 7-person Seven 7-person review teamsreview teams rated rated designated programs & on the same day designated programs & on the same day e-mailed all reports to the provost & e-mailed all reports to the provost & pertinent deans & department chairspertinent deans & department chairs

Review team members were Review team members were mainly from mainly from outsideoutside the program’s college the program’s college

PProvost met with deansrovost met with deans before before finalizing decisionsfinalizing decisions

Provost met with team leadersProvost met with team leaders before before releasing final reportreleasing final report

An internal An internal evaluation expertevaluation expert assisted assisted

2525

FINAL REPORTFINAL REPORT

Issued on May 11, 2007Issued on May 11, 2007 Gave Gave priorities for fundingpriorities for funding in in

each collegeeach college Announced plans to Announced plans to maintain maintain 56, 56,

increaseincrease 16, 16, merge merge 6, 6, maintain/merge maintain/merge 17 subject to 17 subject to review, review, transfer transfer 8, 8, closeclose 26, & 26, & createcreate 6 new degrees 6 new degrees

2626

FINAL REPORT LIMITATIONSFINAL REPORT LIMITATIONS

Gave Gave no evidentiary basisno evidentiary basis for for decisionsdecisions

Referenced Referenced no technical no technical appendixappendix

Referenced Referenced no accessible files no accessible files of supporting data, analyses, of supporting data, analyses, & data collection tools& data collection tools

Gave Gave no rating of each no rating of each program on each criterion & program on each criterion & overalloverall

2727

POSITIVE OUTCOMEPOSITIVE OUTCOME

Local paper Local paper applauded applauded the the report (5/12/07)report (5/12/07)

2828

NEGATIVE OUTCOMESNEGATIVE OUTCOMES

Review Review evidence & link to conclusions evidence & link to conclusions were inaccessiblewere inaccessible to many interested to many interested partiesparties

Professors, alumni, & others Professors, alumni, & others protestedprotested President announced an President announced an appeal process appeal process

(5/18/07)(5/18/07) Faculty voted to call for a Faculty voted to call for a censure censure of the of the

provost (5/18/07)provost (5/18/07) Provost Provost resigned resigned (5/20/07)(5/20/07) Appeals overturned 10 planned cuts Appeals overturned 10 planned cuts

(7/14/07)(7/14/07) Potential Potential savings from cuts were reducedsavings from cuts were reduced

2929

NEGATIVE OUTCOMES NEGATIVE OUTCOMES (continued)(continued)

CCommunity watched ommunity watched a contentious a contentious processprocess

Board fired the presidentBoard fired the president President threatened to sue President threatened to sue Board awarded the ex-president Board awarded the ex-president

$530,000 in severance pay$530,000 in severance pay Projected Projected review of undergraduate review of undergraduate

programs was canceledprograms was canceled, ceding that , ceding that area priority by default area priority by default

Reviews to resume in 2010Reviews to resume in 2010

3030

CLEARLY, THIS CLEARLY, THIS EVALUATION FAILEDEVALUATION FAILED

No standards were required No standards were required to to reach this conclusion.reach this conclusion.

However, However, adherence to adherence to approved standards might approved standards might have prevented the review’s have prevented the review’s failurefailure..

3131

QUESTIONS RE PART 2QUESTIONS RE PART 2

Are you surprised that this Are you surprised that this evaluation failed? Why or why evaluation failed? Why or why not?not?

What were this evaluationWhat were this evaluation’s strengths?

Where did the evaluation go wrong?

3232

MORE QUESTIONS RE PART MORE QUESTIONS RE PART 22

When planning a similar When planning a similar evaluation how would you avoid a evaluation how would you avoid a failure?failure?

What roles should standards play What roles should standards play in assessing a program in assessing a program evaluation?evaluation?

What are your questions?What are your questions?

3333

MY TAKE-MY TAKE-ON THE ON THE PLUS SIDEPLUS SIDE::

Review was keyed to an Review was keyed to an important need important need to restructure to restructure programs.programs.

There was significant There was significant faculty faculty involvement involvement in studying in studying programs.programs.

General General criteria criteria were were established.established.

3434

HOWEVER, THERE WERE HOWEVER, THERE WERE SERIOUS SERIOUS DEFICIENCIESDEFICIENCIES

No No independent perspectivesindependent perspectives Evaluator & decision maker Evaluator & decision maker were were

the samethe same Evidence to support conclusions Evidence to support conclusions

was not reportedwas not reported Political viability was not Political viability was not

maintainedmaintained No independent evaluation of the No independent evaluation of the

reviewreview

3535

10 MINUTE BREAK10 MINUTE BREAK

3636

PART 3PART 3

THE GOVERNMENT AUDITING THE GOVERNMENT AUDITING STANDARDSSTANDARDS

3737

FOR A MORE SYSTEMATIC FOR A MORE SYSTEMATIC EXAMINATION OF THE EXAMINATION OF THE CASECASE Let’s see if use of the GAO Let’s see if use of the GAO

Standards might have helped Standards might have helped ensure the study’s success.ensure the study’s success.

Let’s also use the case to Let’s also use the case to develop a working knowledge develop a working knowledge of of the GAO Standards.the GAO Standards.

3838

THE PARTICULAR USE OF THE GAO THE PARTICULAR USE OF THE GAO STANDARDS I HAVE IN MIND STANDARDS I HAVE IN MIND INVOLVESINVOLVES

Metaevaluation (the evaluation of Metaevaluation (the evaluation of evaluations)evaluations)

Formative metaevaluation is Formative metaevaluation is proactive evaluation of an evaluation proactive evaluation of an evaluation (in process) to help ensure its (in process) to help ensure its success.success.

Summative metaevaluation is Summative metaevaluation is retroactive evaluation of a retroactive evaluation of a completed evaluation to judge its completed evaluation to judge its integrity & soundness.integrity & soundness.

A Metaevaluation A Metaevaluation FrameworkFramework

Formative Metaevaluation

Summative Metaevaluation

Internal Evaluator assesses & strengthens the evaluation

Evaluator reports an attestation of the evaluation’s adherence to standards for evaluations

External Independent metaevaluator provides feedback for strengthening the ongoing evaluation

Independent metaevaluator reports on the completed evaluation’s adherence to professional standards

3939

4040

SOME BACKGROUND SOME BACKGROUND INFORMATION ON THE INFORMATION ON THE GAO STANDARDSGAO STANDARDS

4141

THE GAO STANDARDSTHE GAO STANDARDS

Are issued by the U.S. Government Are issued by the U.S. Government Accountability Office.Accountability Office.

Provide a framework for performing Provide a framework for performing high-quality audit work with high-quality audit work with competencecompetence, , integrityintegrity, , objectivityobjectivity, & , & independenceindependence..

Foster Foster transparencytransparency, , performanceperformance, , accountabilityaccountability, & , & qualityquality through the through the audit process. audit process.

Focused on U.S. government Focused on U.S. government agenciesagencies

Are used throughout the world.Are used throughout the world.

4242

BACKGROUND OF THE GAO BACKGROUND OF THE GAO STANDARDSSTANDARDS

Since 1974, the US Comptroller Since 1974, the US Comptroller General has been issuing the General has been issuing the Government Auditing Standards Government Auditing Standards (Yellow Book)(Yellow Book)

The GAO standards cover The GAO standards cover government-funded departments & government-funded departments & programs at federal, state, & local programs at federal, state, & local levelslevels

Public universities are among the Public universities are among the intended usersintended users

4343

THE GAO STANDARDSTHE GAO STANDARDSFOCUS ONFOCUS ON

FINANCIAL AUDITSFINANCIAL AUDITS

ATTESTATION ENGAGEMENTSATTESTATION ENGAGEMENTS ((conducting agreed-upon procedures on a subject matter or conducting agreed-upon procedures on a subject matter or an assertion about a subject matter & reporting on the an assertion about a subject matter & reporting on the results)results)

PERFORMANCE AUDITSPERFORMANCE AUDITS (including (including program evaluations)program evaluations)

4444

THE GAO STANDARDS THE GAO STANDARDS includeinclude

General StandardsGeneral Standards

Field Work StandardsField Work Standards

Reporting StandardsReporting Standards

4545

NOW, LET’S LOOK ATNOW, LET’S LOOK AT

THE CONTENTS OF THE GAO THE CONTENTS OF THE GAO STANDARDS &STANDARDS &

DISCUSS THEIR RELEVANCE DISCUSS THEIR RELEVANCE TO THE PROGRAM REVIEW TO THE PROGRAM REVIEW CASECASE

4646

THE GAO STANDARDS THE GAO STANDARDS INCLUDE FOUR GENERAL INCLUDE FOUR GENERAL STANDARDS:STANDARDS:

IndependenceIndependence Professional JudgmentProfessional Judgment CompetenceCompetence Quality Control & Quality Control &

AssuranceAssurance (including (including external peer review)external peer review)

4747

General Standard 1: General Standard 1: INDEPENDENCEINDEPENDENCE

The audit organization & auditor The audit organization & auditor must be must be free from personal, external, free from personal, external, & organizational impairments& organizational impairments to to independence & must independence & must avoid the avoid the appearanceappearance of such impairments. of such impairments.

Opinions, findings, conclusions, Opinions, findings, conclusions, judgments, & recommendations must judgments, & recommendations must be be impartialimpartial & viewed as impartial by & viewed as impartial by knowledgeable, objective third knowledgeable, objective third parties.parties.

4848

QUESTION REGARDING QUESTION REGARDING INDEPENDENCEINDEPENDENCE::

How well did the program How well did the program review meet the requirements review meet the requirements for independence & impartiality?for independence & impartiality?

How would you have designed How would you have designed the review differently to meet the review differently to meet the requirement for the requirement for independence? independence?

4949

General Standard 2: General Standard 2: PROFESSIONAL PROFESSIONAL JUDGMENTJUDGMENT Auditors must use professional Auditors must use professional

judgment in planning & judgment in planning & performing evaluations & in performing evaluations & in reporting the results. reporting the results.

Auditors must diligently Auditors must diligently perform, in good faith & with perform, in good faith & with integrity, the gathering & integrity, the gathering & assessment of evidence.assessment of evidence.

5050

QUESTION REGARDING QUESTION REGARDING PROFESSIONAL JUDGMENTPROFESSIONAL JUDGMENT

In the program review case, in what In the program review case, in what respects did the provost exercise respects did the provost exercise questionable judgment in gathering questionable judgment in gathering & controlling evidence & reporting & controlling evidence & reporting findings & conclusions?findings & conclusions?

How would you have conducted How would you have conducted the review differently to meet the review differently to meet the requirement for the requirement for professional judgment? professional judgment?

5151

General Standard 3: General Standard 3: COMPETENCECOMPETENCE The evaluation staff must The evaluation staff must

collectively possess adequate collectively possess adequate professional competence for professional competence for the tasks required.the tasks required.

Competence is derived from a Competence is derived from a blending of education & blending of education & experience.experience.

5252

QUESTION REGARDING QUESTION REGARDING COMPETENCECOMPETENCE

Would you agree or disagree Would you agree or disagree that competence likely was that competence likely was not the cause of the program not the cause of the program review’s failure?review’s failure?

Why or why not?Why or why not?

5353

General Standard 4: General Standard 4: QUALITY CONTROL & QUALITY CONTROL & ASSURANCEASSURANCE Each audit organization . . . must Each audit organization . . . must

establish a system of quality establish a system of quality control that provides reasonable control that provides reasonable assurance of compliance with assurance of compliance with standards & legal & regulatory standards & legal & regulatory requirements.requirements.

It must secure an external peer It must secure an external peer review at least once every 3 review at least once every 3 yearsyears

5454

QUESTIONS REGARDING QUESTIONS REGARDING QUALITY CONTROL & QUALITY CONTROL & ASSURANCEASSURANCE

Despite this standard’s Despite this standard’s reference to an reference to an organization’sorganization’s quality control system, is its quality control system, is its message applicable to individual message applicable to individual studies?studies?

What might the provost have What might the provost have done differently to assure the done differently to assure the program review’s quality & program review’s quality & acceptability?acceptability?

QUESTIONS REGARDING QUALITY QUESTIONS REGARDING QUALITY CONTROL & ASSURANCE (continued)CONTROL & ASSURANCE (continued)

Do you think the GAO Standards’ Do you think the GAO Standards’ provision for peer reviews at 3-provision for peer reviews at 3-year intervals is sufficient?year intervals is sufficient?

How might this provision be How might this provision be strengthened?strengthened?

5555

5656

What questions do What questions do you have concerning you have concerning the GAO General the GAO General Standards?Standards?

5757

NEXT, WE LOOK AT NEXT, WE LOOK AT THE GAO FIELD WORK THE GAO FIELD WORK STANDARDS FOR STANDARDS FOR PERFORMANCE AUDITSPERFORMANCE AUDITS

5858

THE FIELD WORK THE FIELD WORK STANDARDS INCLUDESTANDARDS INCLUDE

OVERARCHING OVERARCHING CONCEPTSCONCEPTS::

- ReasonablReasonable e assuranceassurance

- SignificancSignificancee

- Audit risk- Audit risk

FOUR STANDARDSFOUR STANDARDS

1.1. PlanningPlanning

2.2. SupervisionSupervision

3.3. Obtaining Obtaining Sufficient, Sufficient, Appropriate Appropriate EvidenceEvidence

4.4. Audit Audit DocumentationDocumentation

5959

FRAMEWORKFRAMEWORK FOR THE FOR THEFIELD WORK STANDARDS:FIELD WORK STANDARDS:

PlanninPlanningg

SupervisioSupervisionn

Obtaining Obtaining Sufficient, Sufficient, Appropriate Appropriate EvidenceEvidence

AuditAudit

DocumentationDocumentation

ReasonablReasonablee

assuranceassurance

SignificancSignificancee

Audit riskAudit risk

6060

OVERALL, EVALUATORS OVERALL, EVALUATORS SHOULDSHOULD

Provide Provide reasonable assurance reasonable assurance that evidence that evidence is sufficient & appropriate to support the is sufficient & appropriate to support the findings & conclusions.findings & conclusions.

ConsiderConsider the the relative significance relative significance of of relevant matters when deciding the type & relevant matters when deciding the type & extent of audit work, evaluating audit results, extent of audit work, evaluating audit results, & reporting.& reporting.

Assess & address Assess & address audit risk audit risk to ensure that to ensure that findings, conclusions, & recommendations are findings, conclusions, & recommendations are proper & complete, re. such factors as proper & complete, re. such factors as sufficient, appropriate evidence, an adequate sufficient, appropriate evidence, an adequate audit process, & avoidance of omissions, audit process, & avoidance of omissions, misrepresentation, & fraud.misrepresentation, & fraud.

6161

QUESTION REGARDING QUESTION REGARDING REASONABLE REASONABLE ASSURANCEASSURANCE Clearly, the provost did not Clearly, the provost did not

give give reasonable assurancereasonable assurance that his conclusions were that his conclusions were grounded in solid, grounded in solid, transparent evidence.transparent evidence.

What would you have done to What would you have done to provide such reasonable provide such reasonable assurance?assurance?

6262

QUESTION REGARDING QUESTION REGARDING SIGNIFICANCESIGNIFICANCE

Do you think the provost Do you think the provost accorded sufficient accorded sufficient significancesignificance to the matter of to the matter of abrogating his commitment to abrogating his commitment to make the program review’s make the program review’s evidence available for review?evidence available for review?

Why or why not?Why or why not?

6363

QUESTIONS QUESTIONS REGARDING REGARDING AUDIT AUDIT RISKRISK Clearly, the provost did not Clearly, the provost did not

exercise sufficient safeguards to exercise sufficient safeguards to avoid risksavoid risks to the success of this to the success of this program review.program review.

What were important audit risk What were important audit risk factors?factors?

What steps were needed to assure What steps were needed to assure the review’s success and promote the review’s success and promote acceptance by the faculty?acceptance by the faculty?

6464

Field Work Standard 1: Field Work Standard 1: PLANNINGPLANNING Auditors must adequately Auditors must adequately

plan & document the planning plan & document the planning of required audit work.of required audit work.

They should consider They should consider significance & audit risk in significance & audit risk in defining the audit objectives defining the audit objectives & the required scope & & the required scope & methodology.methodology.

6565

QUESTION REGARDING QUESTION REGARDING PLANNINGPLANNING Do you think the problem in Do you think the problem in

the program review was more the program review was more in deficient planning or in deficient planning or deficient execution of plans?deficient execution of plans?

Please explain.Please explain.

6666

Field Work Standard 2: Field Work Standard 2: SUPERVISIONSUPERVISION Auditors must supervise staff.Auditors must supervise staff.

Audit supervision involves Audit supervision involves providing sufficient guidance providing sufficient guidance & direction to staff, & direction to staff, monitoring & reviewing their monitoring & reviewing their performance, & providing on-performance, & providing on-the-job training.the-job training.

6767

QUESTION REGARDING QUESTION REGARDING SUPERVISIONSUPERVISION

Considering that the program review Considering that the program review was required to be completed within was required to be completed within 1 year & that 114 programs were 1 year & that 114 programs were reviewed, was the standard of reviewed, was the standard of quality likely high across all program quality likely high across all program reviews?reviews?

How would you have assured that all How would you have assured that all 114 reviews were well supervised?114 reviews were well supervised?

6868

Field Work Standard 3: Field Work Standard 3: SUFFICIENT, APPROPRIATE SUFFICIENT, APPROPRIATE EVIDENCEEVIDENCE Sufficient, appropriate evidence Sufficient, appropriate evidence

must be obtained to provide a must be obtained to provide a reasonable basis for findings & reasonable basis for findings & conclusions.conclusions.

Evidence should be assessed for its Evidence should be assessed for its relevance, validity, reliability, & relevance, validity, reliability, & sufficiency.sufficiency.

It should be adequate to persuade a It should be adequate to persuade a knowledgeable person that the knowledgeable person that the findings are reasonable.findings are reasonable.

6969

QUESTION REGARDING QUESTION REGARDING SUFFICIENT, APPROPRIATE SUFFICIENT, APPROPRIATE EVIDENCEEVIDENCE

Assume the university engaged Assume the university engaged you to conduct an independent you to conduct an independent evaluation of the program evaluation of the program review’s evidence.review’s evidence.

What information would you What information would you request?request?

How would you assess it?How would you assess it?

7070

Field Work Standard 4: Field Work Standard 4: AUDIT AUDIT DOCUMENTATIONDOCUMENTATION Auditors must prepare audit Auditors must prepare audit

documentation related to documentation related to planning, conducting, & planning, conducting, & reporting before issuing their reporting before issuing their report.report.

The documentation should The documentation should contain support for findings, contain support for findings, conclusions, & recommendations. conclusions, & recommendations.

7171

QUESTIONS REGARDING QUESTIONS REGARDING AUDIT DOCUMENTATIONAUDIT DOCUMENTATION

How important is it to How important is it to document evaluation studies?document evaluation studies?

Should such documentation Should such documentation be made available to all be made available to all interested persons?interested persons?

What are the pros and cons of What are the pros and cons of releasing such releasing such documentation?documentation?

7272

THIS CONCLUDES OUR THIS CONCLUDES OUR DISCUSSION OF THE GAO DISCUSSION OF THE GAO FIELD WORK STANDARDSFIELD WORK STANDARDS

What questions do you What questions do you have?have?

7373

5 MINUTE BREAK5 MINUTE BREAK

7474

NEW TOPICNEW TOPIC

Reporting StandardsReporting Standards

7575

REPORTING STANDARDSREPORTING STANDARDS for Performance Audits arefor Performance Audits are

1.1. ReportingReporting

2.2. Report ContentsReport Contents

3.3. Report Issuance & Report Issuance & DistributionDistribution

7676

Reporting Standard 1: Reporting Standard 1: REPORTINGREPORTING Auditors must issue Auditors must issue

reports communicating reports communicating the audit results.the audit results.

Reports should be in Reports should be in writing or some other writing or some other retrievable form.retrievable form.

7777

QUESTION REGARDING QUESTION REGARDING REPORTINGREPORTING The provost issued a written The provost issued a written

report & gave it to faculty & report & gave it to faculty & the media. the media.

Why, then, did the faculty Why, then, did the faculty object & secure the provost’s object & secure the provost’s dismissal? dismissal?

7878

Reporting Standard 2: Reporting Standard 2: REPORT CONTENTSREPORT CONTENTS Auditors’ reports should containAuditors’ reports should contain

(1) the audit’s objectives, scope, & (1) the audit’s objectives, scope, & methodsmethods

(2) findings, conclusions, & (2) findings, conclusions, & recommendations, as appropriaterecommendations, as appropriate

(3) stated compliance with standards(3) stated compliance with standards

(4) views of responsible officials(4) views of responsible officials

(5) if applicable, the nature of any omitted (5) if applicable, the nature of any omitted privileged & confidential informationprivileged & confidential information

7979

QUESTION REGARDING QUESTION REGARDING REPORT CONTENTSREPORT CONTENTS Provost’s report Provost’s report

– did not reference compliance did not reference compliance with standardswith standards

– did not disclose evidence that did not disclose evidence that supported his conclusionssupported his conclusions

How important were these How important were these omissions to the review’s omissions to the review’s success?success?

8080

Reporting Standard 3: Reporting Standard 3: DISTRIBUTING DISTRIBUTING REPORTSREPORTS Usually, reports should be given Usually, reports should be given

to governance personnel, to governance personnel, officials of the audited entity, officials of the audited entity, appropriate oversight bodies, appropriate oversight bodies, other right-to-know parties, & other right-to-know parties, & the public, as applicable.the public, as applicable.

Auditors should document any Auditors should document any limitation on report distribution.limitation on report distribution.

8181

QUESTIONS REGARDING QUESTIONS REGARDING DISTRIBUTING REPORTSDISTRIBUTING REPORTS

The provost distributed his The provost distributed his report to all interested report to all interested parties, including the media.parties, including the media.

Why, then, did the faculty Why, then, did the faculty object & cause the provost to object & cause the provost to be fired?be fired?

Had you been the provost, Had you been the provost, how would you have how would you have prevented this dire outcome?prevented this dire outcome?

8282

RATE THE PROGRAM RATE THE PROGRAM REVIEW ON EACH GAO REVIEW ON EACH GAO STANDARDSTANDARDSTANDARDSTANDARD METMET PARTIALLY PARTIALLY

METMETUNMETUNMET

IndependenceIndependence

Professional JudgmentProfessional Judgment

CompetenceCompetence

Quality Control & Quality Control & AssuranceAssurance

PlanningPlanning

SupervisionSupervision

Obtaining Sufficient, Obtaining Sufficient, Appropriate EvidenceAppropriate Evidence

Audit DocumentationAudit Documentation

ReportingReporting

Report ContentsReport Contents

Distributing ReportsDistributing Reports

MY RATING OF THE PROGRAM MY RATING OF THE PROGRAM REVIEW ON EACH GAO REVIEW ON EACH GAO STANDARDSTANDARDSTANDARDSTANDARD METMET Partially Partially

MetMetUNMETUNMET

IndependenceIndependence XX

Professional JudgmentProfessional Judgment XX

CompetenceCompetence XX

Quality Control & Quality Control & AssuranceAssurance

XX

PlanningPlanning XX

SupervisionSupervision XX

Obtaining Sufficient, Obtaining Sufficient, Appropriate EvidenceAppropriate Evidence

XX

Audit DocumentationAudit Documentation XX

ReportingReporting XX

Report ContentsReport Contents XX

Distributing ReportsDistributing Reports XX

8383

8484

IN SUMMARYIN SUMMARY

The program review failed.The program review failed.

The review’s leader did not invoke The review’s leader did not invoke approved standards for evaluations.approved standards for evaluations.

The GAO standards are useful for The GAO standards are useful for examining the review’s strengths & examining the review’s strengths & weaknessesweaknesses

Likely, this review would have succeeded Likely, this review would have succeeded had it met the GAO Standards.had it met the GAO Standards.

8585

YOUR WORDYOUR WORD

ON THE GAO STANDARDS’ ON THE GAO STANDARDS’ APPROPRIATENESSAPPROPRIATENESS FOR FOR GUIDING & ASSESSING GUIDING & ASSESSING PROGRAM REVIEWSPROGRAM REVIEWS

8686

APPLICABILITY OF THEAPPLICABILITY OF THEGAO STANDARDSGAO STANDARDS

Do the GAO Standards address Do the GAO Standards address important important issues & problemsissues & problems involved in your involved in your evaluations?evaluations?

How would the GAO Standards have to be How would the GAO Standards have to be changedchanged to make them useful in your fields? to make them useful in your fields?

What What steps would you need to take steps would you need to take before before you could confidently use these standards? you could confidently use these standards?

8787

10 MINUTE BREAK10 MINUTE BREAK

8888

PART 4PART 4

RECOMMENDATIONS FOR RECOMMENDATIONS FOR APPLYING THE GAO APPLYING THE GAO

STANDARDSSTANDARDS

8989

1. REMEMBER THE 1. REMEMBER THE BASICS OF BASICS OF METAEVALUATIONMETAEVALUATION The most important purpose of The most important purpose of

evaluation OR METAEVALUATION is evaluation OR METAEVALUATION is not only to PROVE but to IMPROVE.not only to PROVE but to IMPROVE.

Formative metaevaluations are Formative metaevaluations are needed to improve ongoing needed to improve ongoing evaluation work.evaluation work.

Summative metaevaluations are Summative metaevaluations are needed to prove the evaluation’s needed to prove the evaluation’s merit.merit.

9090

2. TRY OUT THE GAO 2. TRY OUT THE GAO STANDARDS FOR GUIDING & STANDARDS FOR GUIDING & JUDGING ONE OF YOUR JUDGING ONE OF YOUR NEXT EVALUATIONS.NEXT EVALUATIONS.

9191

3. TRAIN EVALUATION STAFF IN 3. TRAIN EVALUATION STAFF IN THE CONTENT & APPLICATION THE CONTENT & APPLICATION OF THE GAO STANDARDS.OF THE GAO STANDARDS.

Obtain copies of the GAO Obtain copies of the GAO Standards at no cost Standards at no cost through GAO’s Web site (through GAO’s Web site (www.gao.gov).).

9292

4. CONDUCT & USE FORMATIVE 4. CONDUCT & USE FORMATIVE METAEVALUATIONS OF YOUR METAEVALUATIONS OF YOUR EVALUATIONSEVALUATIONS

Base metaevaluations on the Base metaevaluations on the GAO Standards.GAO Standards.

Use formative feedback to Use formative feedback to ensure quality in your ensure quality in your evaluations.evaluations.

9393

5. APPLY THE DETAILS OF 5. APPLY THE DETAILS OF EACH STANDARDEACH STANDARD

Carefully review and apply Carefully review and apply the details of the GAO the details of the GAO Standards.Standards.

Make detailed notes on how Make detailed notes on how your evaluation responds to your evaluation responds to relevant points in the GAO relevant points in the GAO Standards.Standards.

9494

6. USE THE RESULTS OF YOUR 6. USE THE RESULTS OF YOUR APPLICATION OF THE GAO APPLICATION OF THE GAO STANDARDS TOSTANDARDS TO

Improve your evaluation as Improve your evaluation as you go.you go.

Document issues in the Document issues in the evaluation related to the GAO evaluation related to the GAO Standards & how they were Standards & how they were addressed.addressed.

9595

7. ADVISE YOUR SPONSORS TO 7. ADVISE YOUR SPONSORS TO FUND INDEPENDENT SUMMATIVE FUND INDEPENDENT SUMMATIVE METAEVALUATIONS OF YOUR METAEVALUATIONS OF YOUR EVALUATIONS.EVALUATIONS.

Recommend that the Recommend that the independent metaevaluator independent metaevaluator use the GAO Standards to use the GAO Standards to assess your evaluation.assess your evaluation.

Provide the independent Provide the independent metaevaluator with full metaevaluator with full documentation of your documentation of your formative metaevaluation.formative metaevaluation.

9696

8. SUPPORT RELEASE OF 8. SUPPORT RELEASE OF SUMMATIVE SUMMATIVE METAEVALUATION METAEVALUATION FINDINGS TO ALL RIGHT-FINDINGS TO ALL RIGHT-TO-KNOW AUDIENCES.TO-KNOW AUDIENCES.

9797

QUESTIONS QUESTIONS REGARDING PART 4REGARDING PART 4 Do you concur that both Do you concur that both

formative & summative formative & summative metaevaluations are metaevaluations are important?important?

Do you see value in using the Do you see value in using the GAO Standards for both kinds GAO Standards for both kinds of metaevaluation?of metaevaluation?

9898

DO YOU HAVE ANY DO YOU HAVE ANY FINAL QUESTIONS?FINAL QUESTIONS?

9999

IN CONCLUSION,IN CONCLUSION,

Standards have a crucial role in Standards have a crucial role in assuring the quality, credibility, & assuring the quality, credibility, & value of evaluations.value of evaluations.

Unprincipled evaluations can have Unprincipled evaluations can have dire consequences.dire consequences.

The GAO Standards are valuable The GAO Standards are valuable for guiding & judging evaluations.for guiding & judging evaluations.

100100

SOURCES OF STANDARDS SOURCES OF STANDARDS DOCUMENTS & RELATED DOCUMENTS & RELATED

AIDSAIDS

101101

US GOVERNMENT US GOVERNMENT AUDITING STANDARDSAUDITING STANDARDS

By the Comptroller General of the By the Comptroller General of the United States (July 2007)United States (July 2007)

GAO-03-731G Government Auditing GAO-03-731G Government Auditing StandardsStandards

United States General Accounting United States General Accounting OfficeOffice

Washington, DC 20548-0001Washington, DC 20548-0001 www.gao.gov/govaud/ybk01.htmwww.gao.gov/govaud/ybk01.htm

102102

2004 AEA GUIDING PRINCIPLES2004 AEA GUIDING PRINCIPLES

Printed on inside cover of Printed on inside cover of issues of the issues of the American American Journal of EvaluationJournal of Evaluation

Also see Also see

http://www.eval.org/Publications/GuidingPrinciples.asp

103103

THE PROGRAM THE PROGRAM EVALUATION EVALUATION STANDARDSSTANDARDS

Joint Committee on Standards for Joint Committee on Standards for Educational Evaluation (2011). Educational Evaluation (2011). The Program Evaluation The Program Evaluation Standards. Standards. Thousand Oaks, CA: Thousand Oaks, CA: Sage.Sage.

ISBN 978-1-4129-8908-4 ISBN 978-1-4129-8908-4 (paperback)(paperback)

104104

DICKESON BOOKDICKESON BOOK

Dickeson, Robert C. (1999). Dickeson, Robert C. (1999). Prioritizing Prioritizing

Academic Programs and Services: Academic Programs and Services:

Reallocating Resources to Achieve Reallocating Resources to Achieve

Strategic Balance.Strategic Balance.San Francisco: Jossey-San Francisco: Jossey-

Bass.Bass.

105105

Stufflebeam, D. L., & Shinkfield, A. J. Stufflebeam, D. L., & Shinkfield, A. J. (2007). (2007). Evaluation Theory, Models, & Evaluation Theory, Models, &

Applications. Applications. San Francisco: Jossey-BassSan Francisco: Jossey-Bass

See Chapter 3: Standards for See Chapter 3: Standards for Program EvaluationsProgram Evaluations

ISBN-13: 978-0-7879-7765-8 ISBN-13: 978-0-7879-7765-8 (cloth)(cloth)

ISBN-10: 0-7879-7765-9 (cloth)ISBN-10: 0-7879-7765-9 (cloth)

106106

POWER POINT POWER POINT ADDRESSADDRESS MetaevalCOURSE2011 MetaevalCOURSE2011

GRADREVREGAOSTNDS10GRADREVREGAOSTNDS10