EVALUATOR TIPS FOR REVIEW AND COMMENT WRITING The following slides were excerpted from an evaluator training session presented as part of the June 2011.

EVALUATOR TIPS FOR REVIEW AND COMMENT WRITINGThe following slides were excerpted from an evaluator training session presented as part of the June 2011 CBFA Conference.

Special thanks to Dr. Annette Craven for providing resources she received after participating in a recent national Baldrige examiner training.

Use of Standards & Criteria

The Standards and Criteria serve as the basis for:Providing feedback to applicants in the form of

Process & Results Strength commentsOFI comments

Process Maturity

Individual Review

Individual Review Process

Read Overview

Read Standards One - Six

Evaluation Dimensions

ApproachDeploymentResultsImprovement (Learning)

Approach

Appropriateness of methodsEffectiveness of the methodsThe degree to which the approach is repeatable and based on reliable data and information (i.e., systematic)

Systematic

DocumentedRepeatedEvaluatedImproved

Deployment (Implementation)

the approach applied in addressing the requirements is relevant and important to the organizationthe approach is applied consistentlythe approach is used by all appropriate work units

Results• Current performance• Rate and breadth of your performance

improvements• Performance relative to appropriate comparisons

and/or benchmarks• Linking results measures to important customer,

market, process, and action plan performance requirements identified in the Overview and in Approach-Deployment Items.

Where to Look for Results• Results are found throughout the ACBSP Standards, for

example:• Student/Stakeholder Satisfaction (Standard 3)• Student Learning Outcomes (Standard 4)• Faculty Deployment & Qualifications (Standard 5)• Business Process Results (Standard 6)

• Performance is generally presented in one of three waysCharts/GraphsTablesNarrative

What Should You Look For?

Expected ResultsAppropriate ComparisonsImportant Segmentation

Appropriate Comparisons

“Performance is examined relative to competitors and/or other organizations providing similar products and services”

“Include appropriate comparative data”

Improvement (Learning)

Refining the approach through cycles of evaluation and improvementEncouraging breakthrough change to the approach through innovationSharing refinements and innovations with other academic departments or disciplines and processes in your school of business or institution

Writing Evaluator Observations

Self-studyStandards and Criteria BookletACBSP handouts including evaluator workbook

Observation Guidelines

Observations should be:Standards and Criterion basedRelevantClearConciseActionable

Observation Guidelines (Cont.)

Observations should not:“Parrot” the self-studyBe prescriptiveBe judgmentalConflict with one another

Strength Observations: 3 ComponentsNugget

what the business unit does to merit the score you assign to them

Example(s)are the things the organization does with some level of proficiency

RelevanceWhy the observation is important for the business unit (e.g. mission related, etc.)

Strength Observation Example

• “The development of the budget is an integral part of the strategic planning process. This ensures that the budget reflects the business unit’s strategic priorities. As a result, both long-term plans and shorter range operational activities are focused on the organization’s mission, vision, and values.”

OFI’s

Is an opportunity for improvementIs not necessarily a weaknessIs what prevents an applicant from scoring at a higher levelIs tied to the Standards and Criteria

Example of an OFI Observation• the business unit, through its strategic

planning process has identified four key initiatives with associated goals. It is not clear, however, how these goals are being measured and the lack of an established timeframe make it difficult to gauge progress toward achieving them.

So What’s

Explains the significance of the observation so that the business unit doesn’t have to ask, “So what?”Can be used with both Strengths and OFI’s though more commonly used with OFI’s

Where’s the So What?• the business unit, through its strategic

planning process has identified four key initiatives with associated goals. It is not clear, however, how these goals are being measured and the lack of an established timeframe make it difficult to gauge progress toward achieving them.

Where’s the So What?• the business unit, through its strategic

planning process has identified four key initiatives with associated goals. It is not clear, however, how these goals are being measured and the lack of an established timeframe make it difficult to gauge progress toward achieving them.

Guidelines for Results

Start with a subject from the self-study or the CriteriaInclude the time frame you are writing about -- such as “in 2009” or “from 2005 to 2010”Include the actual numbers observed in the levels or trendsInclude figure references

Results Strength

• the business unit operates in a fiscally sound manner as evidenced by its Aaa and AAA bond ratings (Figure 7.3-1) and debt capacity (Figures 7.3-8 and 7.3-11). These results suggest that the business unit makes financial decisions with its long-term financial viability and sustainability in mind.

Results OFI

•Only 3 of the 13 figures in Standard 4, Criterion 4.3 contain comparative data. Without comparative data it is difficult to evaluate the business unit’s current level of performance or the trends they report in these figures.

What’s Wrong with this Observation?

• “The standard being addressed is in need of improvement.”

The OFI does not provide enough information – how does the business unit respond to the comment and how do the commissioners identify the weak area.

What’s Wrong with this Observation?

• An opportunity exists to broaden the advanced educational experience of the faculty by hiring and/or sending current faculty to a broader range of institutions than is currently the case. This may result in a broader base of knowledge for students in the Department of Business.

• This is a prescriptive comment – the Standards and Criteria do not prescribe which institutions faculty members should be hired from.

What’s Wrong with this Observation?• “Although the team was delighted with the depth

of the understanding and the commitment to assessment, there were numerous closing the loop issues based on the self study report review. Upon examination and audit during the site visit, numerous closing the loop evidences were discovered.”

• “Team was delighted” is judgmental and the issues discussed in the OFI were resolved while on site. This is not an OFI – it can actually be a strength.

Critique vs. information “It is not clear” Not addressing both feedback report customers: Commissioner and school/program

Too much brevity Lack of clarity; “So what”? Program Evaluator Self-Study Review not converted to Feedback Report format

Comments limited to “they have a plan…”

Preliminary comments not prepared prior to site visit

Common Mistakes

• QUESTIONS TO ASK YOURSELF • WHEN REVIEWING THE SELF-STUDY:

1. What approach is described? Is it systematic?

2. How is the approach deployed and to what extent for applicable stakeholder groups?

3. Are the results presented?

•4. Is there evidence of learning or improvements?

• QUESTIONS TO ASK YOURSELF • WHEN REVIEWING THE SELF-STUDY:

1. Are the results presented?

2. Are there three to five data points?

3. Are the results segmented?

3. Are comparisons provided (e.g.,peer, national norm, etc.) ?

•4. Are targets/benchmarks identified?