1 What Gets Measured Gets Undone Dr. Jim Mirabella Director of Institutional Research Professor of Statistics Florida Community College - Jacksonville.

1

What Gets Measured Gets Undone

Dr. Jim Mirabella

Director of Institutional Research

Professor of Statistics

Florida Community College - Jacksonville (FCCJ)

J. Michael Adams

Corporate Manager, Quality Services

Florida Power & Light/ FPL Group, Inc.

Florida Sterling Conference

Wednesday, May 31, 2000

Disney World, Orlando, Florida

2

What’s getting “undone” ?

• The desired outcome and supporting processes that the measure(s) intended to describe

• Why????– Poor measures drive poor performance!

3

Objectives of the workshop• Know good measurements from poor

measurements

• Distinguish the implications of good and poor measures on performance

• Balance an array of measures that favorably tracks a process’s outcome for planning, assessing performance, and analysis

• Assess attendees’ measures

• Any other expectation?

4

“Why” Measure at All?

• Measurement is the language of progress and comparison

• Provides a sense of where we are AND where we are going (planning)

• Can guide a steady advancement toward established goals (tracking)

• Can identify goal shortfalls, or over-achievement (analysis)

• Communicates to the work force what is important to the organization (behavioral)

5

Good measures are born from SMART goals

• Specific

• Measurable

• Agreed upon

• Realistic

• Time-Bound

6

The Challenge of Measuring• Workers might perceive it as a threat• Workers might disregard organizational goals,

customers, products and services• Workers might focus on obtaining favorable

measurements• Measuring items with no influence on organizational

success = waste of time – result is a bean-counting approach that focuses on irrelevant

details

• Expected to do something!

7

The Challenges- examples

• “Apples and oranges”

• Impact on comparisons, benchmarking and performance

• turnover

• consistency

• defects vs. defectives

• commutes (measured in time or miles)

• school achievement

8

“Where” are these measures

• Organizational

• Process

• Department/work unit

• Individual– Exercise

9

Your measures Jot down your various

performance measures for you, your department, unit and organization. Indicate the purpose of the measure ( 1= planning, 2= tracking and analysis; 3: behavioral changes)

• Organization

• Process

• Department/ Unit

• Individual

10

“What” Do We Measure?

• Quality• Delivery• Cycle Time• Waste• Cost

• Defects• Satisfaction• Complaints• Financials• Price

11

Measurement Examples

Operations-related measures- reliability- timeliness of delivery- order processing time- errors / defects- product lead time- inventory turnover- cost of quality- employee

12

Measurement Examples

Customer-related measures- customer satisfaction- customer complaints- customer retention

Financial measures- market share- sales per employee- return on assets- return on sales

13

Baldrige/ Sterling: Results CategoryPrivate, Education, Health Care7.1: Customer Focus Results; Student

Performance, Patient and other Customer Focus

7.2: Financial and Market; Student and Stakeholder Focused Results,

7.3: Human Resource Result; Budgetary and Financial Results, Staff and Work

System7.4: Supplier and Partner Results; Faculty

and Staff Results7.5: Organizational Effectiveness

14

Determining and Reviewing Measurements for Balance

BaldrigeResults 7.0

•Customer

•Financial

•Human Resources

•Supplier/ Partner

•Organizationalsupplier

dept

process

15

Desired Outcomes- Airlinesoverall organization balance scorecard

• “Assumed Quality”- arrive safely

• Loyal Customers– Market differentiation, key value attributes:

• on-time arrival• baggage handling• customer complaints

• Profitability and Market share– Satisfied stakeholders (shareholders, partners)

• Employee growth and retention

16

Airline Measurement System

• Individual measures for each flight process• Group measures for overall airline• If problems occur with a flight, who do you

blame -- the flight crew or the airline?• What measures do you affiliate with the

flight crew?• What measures do you affiliate with the

airline?• What is important to you for a satisfactory

flying experience?

17

Review your measures

• Are the organization measures reflective of all Baldrige results items?

• Do the measures reward favorable behaviors?

• Is there alignment with the contributing departments and suppliers?

• Are the overall results managed as an outcome of a process?

18

What’s the opportunity?• Many organizational measurement systems are

too short, too rigid, or used like a strict teacher’s ruler ... to whack rather than to motivate

• Need to replace these outdated measurement systems with more dynamic measurement system that motivates continuous improvement in customer satisfaction, flexibility, and productivity … SIMULTANEOUSLY

19

Mis-measurement Systems

• Unless specifically tuned to flight plan, measurement systems may:– yield irrelevant / misleading information– provoke behavior not conducive to strategy

• Traditional measures ignore requirements & perspectives of customers (internal/external)

• Bottom-line measures (profitability) too late for mid-course correction / remedial action

20

Mis-measurement Systems

• Many measurement systems overlook key non-financial performance indicators

• Measures often used for punishment rather than to promote learning

• Many measurement systems are inflexible and limited in what they can do

21

What Should a Measurement System Do?

• Measures must link operations to strategic goals– departments should know how they contribute

separately and together toward strategic mission

• System has to integrate financial / non-financial info in a way usable by managers– managers need right info at right time

• Measurement system’s real value lies in its ability to focus all business activities on customer requirements

22

What Should a Measurement System Do?

• Measure what is important to the customers

• Motivate operations to continually improve against customer expectations

• Identify and eliminate waste -- of both time and resources

• Help to accelerate organizational learning and build a consensus for change when customer expectations shift

23

Measurement Mismanagement

In an effort to increase market share, a computer service bureau strategizes to improve the timeliness of its voicemail service. The goal is to provide new customer with service w/in 24 hours. To help speed the process, a program is developed to accept verbal phone orders. Most new customers were online w/in a day as promised, causing the company to celebrate their success. A later audit revealed 70% error rate in order entry, with 30% of customers disputing their bill and eventually canceling service..

24

Measurement Mismanagement A Hi-Tech company sets objective to become highly

profitable by being a product leader. To measure the performance of its marketing and R&D functions, the number of new products developed is the most watched barometer. An internal review reveals that in a sample of 20 new product introductions, 80% were delivered over a month late, and significant waste piles up in production. Management cannot understand why the accountant’s ink is red – after all, their yardstick tells them they are developing new products at a record rate.

25

Beware of ……..

Seemingly Simple Measures

• How satisfied are your customers?

• What is your employee turnover?• What is your client retention

rate?• Is the value of our service worth

the price?

26

Beware of ……..

Treating customer perceptions as objective measures• Customer satisfaction product quality

• Customer satisfaction is a complex phenomenon

• A well-handled complaint results in higher customer satisfaction than does no complaint

• More reasons for complaint more dissatisfaction

27

Beware of ……..

Non-specific measurements• Results are actionable

• Hard to improve what you cannot assess

Failing to measure adequately• Think BALANCED SCORECARD

• Don’t give employees an outlet for gaming success

• Identify all areas important to customers

28

Beware of ……..

Using results incorrectly• Don’t tie results to employee pay

unless employees can directly influence results

• Don’t base employee pay on results that cannot be measured

29

Recap of Objectives and Expectations

• Know good measurements from poor measurements

• Distinguish the implications of good and poor measures on performance

• Balance an array of measures that favorably tracks a process’s outcome for planning, assessing performance, and analysis

• Assess attendees measures

30

Thank you!!!!!!

Jim Mirabella

Mike Adams