ERP success presentation in Shanghai, China, 1999

Measuring the Effectiveness of ERP systems

Peter Seddon, PhD

Senior Lecturer

Department of Information Systems

The University of Melbourne

[email protected]

http://www.dis.unimelb.edu.au/staff/peter


Chinese version translated by

Bin Hu



[email protected]

3


1.An example calculation of ROI2.Why is measuring IT Effectiveness so hard?3.What do we know about IT

Effectiveness measurement in practice?

4.Individuals: IT effectiveness5. Senior Managers: ERP

effectiveness6. Summary and Lessons

4






5

1. Example calculation of ROI

Report sponsored by SAP Analysis of The Boston Beer Company’s

investment in an SAP R/3 system Implementation, January, 1996 Standard IRR calculation: ROI of 83%

Data from “The ROI Report”, Scalea and Company, 346 Beacon Street, Boston MA, August, 1997

6

Boston Beer in 1997 Leading “craft” brewer in the US contract brewer - using excess

capacity of other brewers - plus one brewery in Cincinnati

380 employees: 190 in sales, 120 in brewing, and 70 in administration

1991-1996 net sales growth: 46% compound, from $30M to $191M

7

Boston Beer’s SAP R/3 Modules

FI Financials

SD Sales and Distribution

AM Asset Management

MM Material Management

CO Controlling

PP Production Planning

PA Profit Analysis

BS Basis (middleware)

8

The Investment, 1995-1996

Initial implementation & licence fees 700,000

ABAP Development and reporting 300,000

Further implementation and training 293,000

Hardware (2 x HP 9000) 462,000

Internal costs 88,000

Total $1,843,000

1995: $1,103,0001996: $ 740,000

$1,843,000(Consultant fees over 2 years: $1,043,000)

9

Estimated Benefits, 1996-2000 Reduction in Gen. & Admin. Expense: 5% of $12.5m

= $300,000 p.a. (rising to $900,000 by year 2000) Interest Savings on A/R reduction: 6% of reduction of

$4m in Working Cap = $240,000 (rising to $500,000 by year 2000)

Improved Budget & Control of Sales Expenses: $300,000 (rising to $800,000 by year 2000)

Improved Control of POS Expenses: $240,000 Increased A/P Discounts: $100,000 p.a

Gains ($'000)

0

500

1000

1500

2000

2500

3000

1996 1997 1998 1999 2000Reduction in General and Administrative ExpenseInterest Savings on A/R reductionImproved Budget & Control of Sales ExpensesImproved Control of POS ExpensesIncreased A/P Discounts

740

1520

1990

23402540

Calculating ROI (=IRR), 1996?

Year 1995 1996 1997 1998 1999 2000

Cash in 740 1520 1990 2340 2540

Cash out -1103 -740

Net cashflow

-1103 0 1520 1990 2340 2540

IRR 83% =IRR(B4:G4,0.1)

Boston Beer Share Price 1996-99

Source: http://www.etrade.com, 27 Nov 1999

13

Measuring ERP Effectiveness

Boston Beer’s sales fell 3% during 1996-98.

Most of the expected cost savings (which assumed sales growth) were not realized.

The expected ROI was not achieved.

Does this mean the investment in the ERP system was a failure?

How should we measure ERP effectiveness?

14






15

2. Why is measuring IT Effectiveness so hard? (not just for ERP systems) Often hard to link benefits to any specific IT investment

(e.g., benefits of a LAN). Cost savings are the easier to quantify, but these are not

the only benefits. There are many different goals for evaluation (e.g.,

feasibility, audit) Different stakeholders evaluate the same system

differently Evaluations change over time

16

Willcocks and Lester list six types of goals for IT investment

Infrastructure building (can’t cost justify) Cost efficiency (e.g., Boston Beer) Service to the business (e.g., better

information, better decision-making) Enable business improvement (processes) Differentiating the business competitively Revenue generation

Willcocks and Lester, Beyond the IT Productivity Paradox Wiley 1999

17

Five Key Questions to get clear about IT Effectiveness evaluation

Q1. From whose perspective is effectiveness being judged?

Q2. What is the system being evaluated?

Q3. What is the purpose of evaluation?

Q.4 What time frame is employed? (short, long)

Q.5 How is effectiveness to be judged?

Based on: Seddon, Staples, Patnayakuni, and Bowtell, Dimensions of IS Success, Communications of the AIS October 1999

18

Q1. From whose perspective is effectiveness being judged?

independent stakeholder individual user a group of people project manager IT management owner/senior management society

19

Q2. What is ‘the system’ being evaluated?

part of an information system (e.g., GUI) one particular system (e.g., the HR module) a type of system (e.g., ERP) all IT applications in an organization an inter-organizational information system an IT implementation process (e.g., ASAP) the IT function in an organization

20

Q.3 What is the purpose of the evaluation?

Performance appraisal to learn how to do better feasibility-study evaluations in future expect different levels of cooperation from staff depending on

purpose of evaluation some evaluations may be much more comprehensive than others

21

short term (e.g., “on time and within budget” for implementation projects)

long term

Q.4 What is the time frame is employed?

22

Q.5 How is effectiveness of the IT system to be judged?

Compared to: some other organization (benchmarking) some ideal level of performance stated goals of the organization (e.g., the

feasibility study for an IT project) past performance of the organization other desirable characteristics

23


1.An example calculation of ROI

2.Why is measuring IT Effectiveness so hard?3.What do we know about IT


4.Individuals: IT effectiveness

5. Senior managers: ERP effectiveness

6. Summary and Lessons

24

Firms evaluate effectiveness at 3 points in the system’s lifecycle

Perceived Organizational Performance

Implementation

Design

Continuous Improvement

Stabilisation TransformationTime

J.W. Ross, “The ERP revolution: Surviving versus Thriving,” Working Paper, CISR, MIT, 1998



Implementation

Design




Feasibility Study



Implementation

Design




Feasibility Study

Development Stage



Implementation

Design




Feasibility Study

Development Stage

Post-implementation

28

What percentage of firms evaluate their investments in IT?

(formal reviews; not just ERP)

Source: Seddon, Graeser, and Willcocks (1999): data from IT managers in 80 large European and US firms in 1998. Average annual turnover US$4.5B. Average annual IT budget was US$38 million.

IT Evaluation N %Feasibility 54 68%During IT development Project 55* 69%Post-implementation 40 50%

* 48 firms had abandoned some projects, 10% of projects

1. Feasibility Stage Evaluation problems(survey of 97 UK firms, Ballantine 1993)

Nature of evaluation problems for IS/IT projects Problem (%)Information requirements1. Quantifying relevant benefits2. Identification of relevant benefits3. Quantifying relevant opportunity costs4. Identification of relevant opportunity costs5. Identification of relevant costs6. Quantifying relevant costs

816536353127

Knowledge related7. Difficulty with interpretation of results8. Unfamiliarity with project evaluation techniques9. Calculation of discount rate

17123

Organizational problems10. Lack of time11. Lack of data/information12. Lack of interest13. Others

3719158

1. Feasibility Stage Evaluation Methods (80 respondents in US, UK, A & NZ, Bacon 1992)

Criteria% of companies

using the criterion(n=80)

Ranking byimportance

Financial CriteriaDiscounted Cash Flow (DCF)1. Net Present Value2. Internal Rate of Return3. Profitability IndexOther Financial4. Average/Accounting Rate of Return5. Payback Method6. Budgetary Constraint

49548

166168

4214

1058

Management Criteria7. Support Explicit Business Objectives8. Support Implicit Business Objectives9. Response to Competitive Systems10. Support Management Decision Making11. Probability of Achieving Benefits12. Legal/Government Requirements

886961884671

1367913

31

2. Development-stage Evaluation


Implementation

Design



Development Stage

32

2. Development-stage Evaluation

For project managers, the key success measure is often “on time and within budget”.

This has little to do with the economic value of the project.

Some projects are abandoned, which means the entire cost of the project is wasted.

Studying these abandoned projects gives insight into development-stage problems and evaluation

33

2. Development-stage Evaluation(Willcocks and Lester 1996, n=50)

80% of the 50 organizations had abandoned projects because of negative development-stage evaluations.

The most important reasons for abandonment were: project over budget (17/50 respondents) organizational needs changed (23/50 respondents) user requirements changed (22/50 respondents).

34

2. Development Stage Evaluation: Reasons for project abandonment

(Norris 1996)

Unacknowledged divisions between users on mandatory and desirable requirements and the scale of benefits expected from them.

Gung-ho attitudes to managing risks. An assumption that short training courses at the

launch of the system will be sufficient to change well-established working practices and to encourage users to adopt the system.

The above are consistent with findings on BPR.

35

3. Post-implementation Evaluation


Implementation

Design



Post-implementation

36

3. Criteria for Post-Implementation Review(Willcocks and Lester 1996, n=50)

83% comparison to the feasibility study 63% cost-effectiveness 53% quality of product 48% systems availability 44% productivity 22% user satisfaction

15% of organizations used the top six criteria, and a further 15% used the top five (excluding user satisfaction).

37

Question:Of projects that are completed, provide the percentagesthat satisfy the following requirements. Percentagesshould add to 100:[ %] Exceed expected benefits[ %] Produce expected benefits[ %] Produce less than expected benefits[ %] Complete failure

3. Post-implementation Reviews

Source: Seddon, Graeser, and Willcocks (1999), n=80

38

Question:Of projects that are completed, provide the percentagesthat satisfy the following requirements. Percentagesshould add to 100:[ 14 %] Exceed expected benefits[ 55 %] Produce expected benefits[ 26 %] Produce less than expected benefits[ 5 %] Complete failure

Source: Seddon, Graeser, and Willcocks (1999), n=80


39

The remainder of this presentation focuses on post-implementation benefits only, i.e., on benefits from use, once the system is operational.


40






4. A model of factors affecting individual perceptions of IS effectiveness (n=119;266)

Information Quality

Ease ofUse

Perceived Usefulness

Perceived Net Benefit

Importance of the task

Knowledge of the system

Support Staff

0.4

0.3

0.2

0.3

0.2

0.2

0.5

0.1

ns

ns

0.2

Comparing scores on Old and New Systems (September, 1996 and January, 1998. All scales: 1…7)

Dataset 1 (n=70) Dataset 2 (n=130)

MeanStandardDeviation Mean

StandardDeviation

Dataset 2 –Dataset 1

System needs replacing 6.28 1.39

Task is important to me 5.79 1.11 6.05 0.94 0.26

Knowledge of system 5.43 0.98 5.35 1.00 -0.08

Ease of use 3.90 1.28 5.46 1.05 1.56 ***

Information quality 4.09 1.07 4.98 0.99 0.89 ***

Support staff helpful 4.31 1.03 4.56 1.00 0.25

System is useful 4.52 1.37 5.35 1.15 0.83 ***

Perceived net benefit 3.96 1.23 5.53 1.03 1.57 ***

Source: Seddon and Staples (1999)

43






44

Why are senior managers interested in ERP effectiveness?

To improve IT investment decision-making in future To learn from reported experience of others to understand capabilities of ERP systems and so tap possible

sources of competitive advantage to identify areas to target for future development

45

Answers to the five key questions for evaluating IT effectiveness

Stakeholder: senior management System: the entire ERP system Purpose of evaluation: previous slide Time frame: costs and benefits to date, plus costs and benefits for the next 2-

3 years (including software and hardware upgrade costs) Effectiveness criteria: Balanced IT Scorecard

46

Balanced IT Scorecard criteria

1. From the corporate financial perspective, e.g., ROI

2. The customer/user perspective (e.g., on-time delivery rate, satisfaction)

3. Business process (e.g., purchase invoices per employee)

4. An innovation/learning perspective (e.g., rate of cost reduction for IT services)

5. From the systems project perspective (e.g., on-time, quality, cost)

6. A technical perspective (e.g., implementation efficiency, capacity utilization, response times)

Source: Graeser and Willcocks (1998)

47

Norris says for post-implementation evaluation of IT effectiveness, create a task force:

Different stakeholder interests must be represented

Task force must have:– Authority vested from the top– Board-level access– Company-wide emphasis– Business management insight.

Source: Norris (1996)

48

Benefits checklist: Analysis of vendor success stories on WWW (Shang, PhD in progress, 1999)

ERPvendor

Casespublished

Casesselected

Case scope

SAP 256 51 19 industries45 countries.

Peoplesoft 90 65 11 industries,10 countries

Oracle 124 68 No industryclassification,6 countries

Total 470 182

49

Content analysis of the 182 WWW cases suggests that, from the perspective of senior management, there are five categories of benefits from ERP systems (Shang 1999):

1. Operational

2. Management

3. Strategic

4. IT infrastructure

5. Organizational

50

1.Operational benefits (Shang 1999)

Cost reduction: labour, training, inventory Cycle-time reduction:

– faster delivery to customer, – faster administrative processes

Increased productivity Improved data quality Improved customer service

51

Better resource management: inventory, maintenance, production scheduling, workforce management

Better decision making: improved market responsiveness, fast response to work changes, fast response to customer needs

Better control: analysis by line of business, product, customer, geographic area; production costs management

2.Managerial benefits (Shang 1999)

52

Support for future business growth Support for business alliances “Lock-in” customers (realtime data sharing,

interactive customer service) build business innovation build cost leadership enhance product differentiation sustain competitiveness

3.Strategic benefits (Shang 1999)

53

Increased business flexibility reduction in IT costs: legacy systems

maintenance, mainframe replacement, year 2000 compliance

increased infrastructure capability: global platform, database integrity

flexibility: adaptable modern technology, extendable, compatible

4.Infrastructure benefits (Shang1999)

54

Support organizational changes (restructuring)

business and employee skills learning greater customer focus for staff empowerment and accountability teamwork better employee morale and satisfaction

5.Organizational benefits (Shang1999)

55

Vendor success stories do not mention the problems of ERP system use such as:

Dependence on vendor upgrades Poor response times Inflexibility/Expensive customization Staff leaving for better salaries others

Downsides of ERP systems

56

Persuade a number of user organizations to use a combination of the IT Scorecard and the ERP Benefits Checklist to assess the effectiveness of their current systems.

Will also ask them to identify downsides (unwanted consequences) of ERP systems.

Hope to identify some common factors that led to greater perceived success.

Expect user knowledge to be a key determinant of success.

Plan (for next year)

57






58

6. Summary and Lessons IS effectiveness measurement is difficult because there is no

simple causal relationship between IT expenditure and benefits. Financial measures such as Boston Beer’s ROI, and changes in

corporate profitability, are clearly invalid in some circumstances. The previous section contains our best recipe for ERP

effectiveness measurement from the senior management perspective:

Taskforce + scorecard + checklist.

59

Finally:

“Establishing value for money depends on the business judgement of the managers involved - it is no more amenable to numerical analysis than any other value judgement.” (Norris 1996)

Source: Norris, G.D. Post-investment appraisal, in Willcocks L. Investing in Information Systems, London: Chapman and Hall, 1996: 193-221.

60

Questions?

Peter Seddon and Bin Hu



[email protected]

http://www.dis.unimelb.edu.au/staff/peter

ERP success presentation in Shanghai, China, 1999

Documents

effectiveness measurement

typeof system

particular system

erp allit applications

expected roi

sap r3 system implementation

better information

boston beer service