Developing RAE1

Seediscussions,stats,andauthorprofilesforthispublicationat:http://www.researchgate.net/publication/256106276

DevelopingaCombinedFrameworkforEvaluatingITProjectsbasedonIT-BSCandCOBIT

ARTICLEinINTERNATIONALJOURNALOFDIGITALCONTENTTECHNOLOGYANDITSAPPLICATIONS·MAY2011

DOWNLOADS

191

VIEWS

67

1AUTHOR:

RezaSamizadeh

AlzahraUniversity

11PUBLICATIONS7CITATIONS

SEEPROFILE

Availablefrom:RezaSamizadeh

Retrievedon:18August2015

http://www.researchgate.net/publication/256106276_Developing_a_Combined_Framework_for_Evaluating_IT_Projects_based_on_IT-BSC_and_COBIT?enrichId=rgreq-7b46a8cf-64e8-479d-afc7-4f7b634dc967&enrichSource=Y292ZXJQYWdlOzI1NjEwNjI3NjtBUzo5NzY4MTAyMDAzMTAwM0AxNDAwMzAwMzc5MTQ0&el=1_x_2

http://www.researchgate.net/publication/256106276_Developing_a_Combined_Framework_for_Evaluating_IT_Projects_based_on_IT-BSC_and_COBIT?enrichId=rgreq-7b46a8cf-64e8-479d-afc7-4f7b634dc967&enrichSource=Y292ZXJQYWdlOzI1NjEwNjI3NjtBUzo5NzY4MTAyMDAzMTAwM0AxNDAwMzAwMzc5MTQ0&el=1_x_3

http://www.researchgate.net/?enrichId=rgreq-7b46a8cf-64e8-479d-afc7-4f7b634dc967&enrichSource=Y292ZXJQYWdlOzI1NjEwNjI3NjtBUzo5NzY4MTAyMDAzMTAwM0AxNDAwMzAwMzc5MTQ0&el=1_x_1

http://www.researchgate.net/profile/Reza_Samizadeh?enrichId=rgreq-7b46a8cf-64e8-479d-afc7-4f7b634dc967&enrichSource=Y292ZXJQYWdlOzI1NjEwNjI3NjtBUzo5NzY4MTAyMDAzMTAwM0AxNDAwMzAwMzc5MTQ0&el=1_x_4


http://www.researchgate.net/institution/Alzahra_University?enrichId=rgreq-7b46a8cf-64e8-479d-afc7-4f7b634dc967&enrichSource=Y292ZXJQYWdlOzI1NjEwNjI3NjtBUzo5NzY4MTAyMDAzMTAwM0AxNDAwMzAwMzc5MTQ0&el=1_x_6


Developing a Combined Framework for Evaluating IT Projects based on IT-BSC and COBIT Nasrin Dastranj Mamaghani, Reza Samizadeh, Fatemeh Saghafi

International Journal of Digital Content Technology and its Applications Vol.5 No.5, May 2011

Developing a Combined Framework for Evaluating IT Projects based on

IT-BSC and COBIT

1 Nasrin Dastranj Mamaghani, 2 Reza Samizadeh, 3 Fatemeh Saghafi 1, Corresponding Author Iran Telecommunication Research Center, [email protected]

2, Department of Science and Technology, Alzahra University, [email protected] 3, Iran Telecommunication Research Center, [email protected]

doi : 10.4156/jdcta.vol5.issue5.2

ABSTRACT IT projects have special features like high risks, intangible outcomes, hidden benefits and high costs

that make the use of traditional evaluation techniques very difficult and the outcomes unreliable, therefore information technology (IT) investments should be carefully controlled and evaluated. Large efforts have been done to provide appropriate methods for evaluating IT projects. In this paper, traditional methods of measuring projects are reviewed and advantages and disadvantaged of them in measuring IT projects are specified. Then new techniques that are specialized in measuring IT projects are introduced and IT-BSC as an appropriate method with available information is selected. As COBIT framework is defined for IT projects, from the combination of COBIT and IT-BSC, a framework for evaluating IT projects is proposed. Besides, the proposed framework is validated by experts’ opinion with questionnaire and appropriate hypothesis test. Finally, an implemented project in IT area is selected as a case study and is evaluated by measures of the proposed framework. The results show that this framework can be very effective in evaluating IT projects.

Keywords: IT-BSC, COBIT, Delphi Technique, IT Project, Project Management

1. Introduction

In today competitive business environment, there is strong need to control costs and returns on investment along with lower risk of capital investment. Understanding the potential impact of IT on strategic power of organizations and increasing payments of IT projects, has made the evaluation and control of capital investments in IT projects remarkably important and critical [1,3]. According to studies, only 18% of organizations use proper methods of calculating benefits of capital investment in IT [1] and at least 22% of capital investment in IT is regarded as waste. Also, 34 to 40 percent of IT projects have no benefit to be obtained [4.5]. Reasons for this failure are complex. Technical factors, and in addition to that, environmental, organizational, managerial and human resources contribute to this failure. In some studies [1, 2, 3] lack of correct evaluation and control of IT projects is the main factor of failure in this projects. In this paper, different methods of evaluating capital investment of IT projects are classified. In Section 2 the traditional assessment methods of projects are investigated and their advantages and disadvantages for evaluating IT project are expressed. Section 3 introduces new ways to evaluate IT projects. They are specific methods of IT projects assessment. Among these methods IT-BSC1 and COBIT2 are selected as they are specified for IT projects and their information can be available. With merging these two frameworks, a framework for evaluating organizational IT projects is presented in Section 4. Finally, a sample project is evaluated using the proposed framework.

2. Traditional methods of project evaluaiont

Many studies investigated the benefits and drawbacks of traditional methods. Main goal of the

organization is to increase profits and wealth of project stakeholders. To calculate profit and loss of a project, operational methods of assessment are needed like traditional methods. Also these methods are known and understandable and are based on accepted principles [9,13,14]. But there are a lot of

1Balanced Scorecard 2 Control Objectives for Information and related Technology

- 10 -



disadvantages too. Disadvantages of using traditional methods in evaluating IT projects can be categorized in two groups: conceptual reasons and operational reasons.

Conceptual reasons: When a company invests in the IT project, five different partners will be involved: Main Organization (project owner), users, project teams, sponsors and shareholders. Each of the partners follows different objectives to meet outputs of the project. If mentioned methods are used, only the goal of project manager that is the implementation of project in time, budget and provided description will satisfy [15,16,17,18]. Also these methods are naturally conservative and suitable for low-risk projects and they are considered to have quick cost returns [2, 7, 10, 19].

Operational reasons: Benefits of IT projects are classified in three categories: tangible benefits that are quantitative, intangible benefits that are qualitative and difficult to measure, hidden benefits that may be considered very important or to be ignored [8]. For IT projects, proportion of advantages of tangible benefits to intangible and hidden benefits are less. Because IT investments usually have supporting state and their benefits are hidden. Traditional methods only measure the tangible benefits. Although the costs are more measurable than benefits, significant amounts of capital cost of investment in IT projects are intangible or hidden [14, 19]. Approximately 40% of the costs spent on IT projects do not be considered in traditional budget estimates. 65% of IT managers are not able to calculate the full cost of the project through traditional methods [19, 22]. There are many risks in IT capital investments, Such as risk assessment, technical risk, project risk, operational risk, internal political risks, and external environmental risks. By using traditional methods, the risks must be reflected in the discount rate [14, 20]. But only 7% of organizations consider discount rate in IT projects [9]. Even when managers are interested to consider discount rate, the level of risk is usually calculated experimentally or based on numbers [7]. Not considering Discount rate in projects lead to high-risk projects and highly considering it result to conservative projects.

3. New methods of evaluating IT projects

As it is shown in Table2, many studies have acknowledged that traditional methods of evaluating IT

projects are not appropriate. Consequently lots of efforts have been done to replace these methods. These efforts take two different approaches. The first approach is based on traditional methods and decides to improve and resolve the major criticisms of traditional methods. The second approach does not accept the traditional methods and use new methods for evaluating IT projects. Many new methods for evaluating IT projects are still in the conceptual phase and are implemented limitedly [14]. These methods introduced here generally and they are categorized base on the solutions they provide.

Strategic approach: the strategic approach was first introduced by Porter. He proposed strategic dimensions to create competitive advantages clearly. Capital investment of IT projects should create competitive advantages for organizations [23, 24]. The main tool for understanding the role of technology in competitive advantage is Porter's value chain that makes alignment between capital investment of IT projects and company goals. There is no explicit attention to financial analysis in this approach [24]. This method provides the guidelines for selecting different investments, but it is not appropriate when the organization wants to choose between two incompatible projects.

Information economics: Parker and Benson developed the information economy framework and published it in 1987 [25, 26]. Information economy benefits from the allocation of point-rating scores process for evaluating the benefits of capital investment and strategic communications with IT. Information economy introduces a range of procedures which must be performed in the specified order: 1) forming a committee of managers, 2) consensus on intangible results, 3) quantifying importance of benefits and risks in relative scale, 4) Estimating the probability scale in zero or 5, 5) multiplying each estimate in weight and probability, 6) calculating total value, selecting the best alternative which take the largest sum [7, 26]. Drawback of this method is that it needs public consensus. Other various methods have been proposed in the context of Information economy, one of which is Oracle CB – 90 that is a software model for IT managers that have developed by Oracle to do step by step process to make decision of capital investment in IT projects. This framework leads to a decision matrix.

Options model: options model not only are used for improving traditional methods but some new evaluation method have been developed based on theses models. Origin of such evaluating methods is in the evaluating of complex financial transactions such as buying and selling stocks and currency

- 11 -



transactions. These models not only pay attention to financial tracking, storage and analysis of processes but also analyses what occurs in the case of using other options. This model changes dynamically its structure and coefficients to improve the chances in the decision [27]. Equation of Cox-Rubenstein model is an example of options model that is used as a base for creating new Feasibility assessment method [7, 27]. Of course there are some doubts on this method that can be referred as follow: 1– quality of outputs not only depend on the model but depends on the inputs. Low quality of data and conflicts will affect outputs, 2 - Suggestions for capital investment in IT projects happens less in comparison with the number of financial transactions, 3 - Most of these methods are complex and difficult to understand and utilize. Before recruitment of these methods, suggestions must be approved for the applicability of these methods in the management of portfolio investment [7]. Multilayered evaluation process: This method uses assessment methods that more or less are hierarchical. Usually a combination of traditional methods and strategic approach is used. it is believed that the evaluation of capital investment in IT Projects will be effective only when the evaluation process is included in the higher levels of business processes [21, 28] and so traditional methods are not completely put away because they can create many benefits [6, 28]. This method usually leads to a process of two or multilayered process. In the first stage, all capital investments that are not related to business goals of organization are eliminated. In the second stage, selection on the remaining projects is done based on calculations obtained form modified traditional methods. For example Earl [6] has broken the second stage to three sub stages: 1- use NPV on the basis of tangible costs and benefits (quantitative), 2- list the intangible costs and benefits (qualitative), 3 - analyze risks and uncertainties. Finally managers must make their decisions based on the information obtained.

IT-BSC approach: IT projects include the production of IT application programs, programs under Internet, work with the data bases and searching the data and so on. As mentioned in previous sections, these projects have specific characteristics that the most important of them are intangibility and inability to calculate the full cost of the project. Therefore, using traditional criteria that evaluate solely based on financial criteria, is insufficient. So BSC for IT projects is specialized as IT-BSC. Use of IT-BSC from its early concept began since 1985 [36]. Gold in 1992 and Vilkoks in 1995 made a conceptual analysis of it that is more developed by Grembergan in 1997 and later by van Bergen and Tymerman in 1998 [37]. Also in 2005, designing performance management system aimed at monitoring and promoting the use of IT through IT-BSC was investigated [38]. Other similar research with the aim of using the BSC in the evaluation of organizational system performance has been done [39]. In that study all aspects that the researchers of IT has added to general BSC model to evaluate the performance, have been investigated and the framework for IT-BSC is presented based on measurements and strategies used in them. Researches have shown that aspects of BSC should be changed as follows in the field of IT [39, 40, 41]. 1- Company contribution: means that how much value the capital investment in IT projects creates to business and organization. 2 - Users and customers: the evaluation of stakeholders from IT facilities which are created in the project. 3- Operational excellence: the promotion of IT products and services which are created by the assessed project, how much depend on processes that develop and deliver software applications and information systems. 4 - Future orientation: how much opportunities from the perspective of human resources and technology have been created to respond to future needs of organization in the evaluated project.

Each of these aspects must be translated to indicators and measurable criteria related to the leading project to evaluate the current state. This evaluation should be periodically repeated and objectives already determined by using information of the best models, should be compared [37]. Other mechanisms of communication between IT and business is development of waterfall BSC [42] In a study in 1999 by Van der. Zee, designing "operational IT-BSC" and "developmental IT-BSC” for the development of waterfall IT-BSC has been proposed. The following figure shows the model [43].

- 12 -



Figure 1. development of waterfall IT-BSC

This model will align business strategies with IT strategies. This model show how IT creates value

through business [44]. Several studies have used BSC to evaluate projects in the field of e-commerce [34], R & D projects [33], IT projects [29] and ICT projects [35].

COBIT Framework: COBIT framework is designed as a tool for managers in the field of IT governance to understand and consider risk management and benefits from IT and related technologies. This model is independent from technical IT infrastructure that is used in each organization. It’s an open standard for IT control that has developed by the "Institute of IT governance". This framework provides substantial approach for implementing leading projects related to IT in a controlled environment [42]. COBIT framework was presented first by ISACA and ITGI in April 1996. Its goal is providing best IT functions for IT staff, auditors and users. Then other versions are presented by development of managerial guidelines. In May 2007, Version 4.1 has been introduced that in relation to previous versions have more simple definition of goals and includes waterfall development of processes and communications between business, IT goals, and IT processes [43]. Many organizations have used COBIT to manage or control IT. This framework is a set of 138 control objectives, a set of guidelines for auditors and collection of implementing tools [42].

IT Sources in the COBIT framework include: • Application programs; include automated systems and procedures for information processing • Information; contains organization’s data such as inputs, processing and output of mechanized

systems that is used by business • Infrastructure; including technology and equipment (such as hardware, operating system, network,

multimedia and supporting environment) used for the empowerment of processing in application programs

• People; include staff required for planning, organizing, acquisition, implementation, presentation, support, monitoring and assessment systems and internal, outsourcing or contract information services.

Department of IT in organizations should have clear vision of their IT processes for achieving their objectives. These processes carried out business application programs by using people skills and technology infrastructure, in order to strengthen the organization’s information. These resources along with IT processes constitute, IT Architecture of organization.

COBIT conceptual framework includes three aspects of “IT processes”, “IT criteria”, and “IT resources”. Among those in the IT processes aspect, 34 processes in the management of IT in four main areas, including “planning and organizing”, “Acquisition & Implementation”, “delivery and support” and “monitoring and evaluation” has been determined. Figure 2 shows the relation of these four areas. In the second aspect or aspects of information criteria, security, quality and control needs are responded by proposing seven criteria that can be defined as what the organization wants of IT in general. These criteria include: effectiveness, efficiency, confidentiality, integrity, availability, compliance and reliability. In the third aspect or aspects of IT resources, five different types of sources for IT including data, software systems, technology, facilities and human resources has been defined [44].

4. Research Methodology

To extract goals and measures in each aspect of proposed framework, goals and metrics in the first

level of COBIT IT processes has been used and they are allocated to different aspects of the framework.

- 13 -



For allocating goals to each aspect, a complete list of goals in the COBIT IT processes (including 27 goals) were given to IT experts and managers to be selected and prioritized separately for each aspect. As in proposed framework, there are 4 aspects, SO 4 questionnaires were designed and experts’ opinions were analyzed in each one. In the designed questionnaires, to determine importance of each objective in each aspect, different questions were asked which could be answered in likert scale with 5 point: strongly disagree (1), disagree (3), neither agree nor disagree (5), agree (7) and strongly agree (9). The method of model evaluation was based on survey-description research method and by appropriate statistical tests, questionnaires were analyzed.

One of the most popular ways of identification of the data statistical distribution is one-sample Kolmogorov-Smirnov test. The Kolmogorov-Smirnov test compares the observed cumulative distribution function for a variable with a specified theoretical distribution, which may be normal, uniform, Poisson, or exponential [31]. According to our test results, the p-value of all questions was less than 0.05, which showed that the distribution of them was not normal. So, a statistical non-parametric test should be used. In this way, the binomial test is used as a non-parametric test in this paper to determine the importance of the objectives from the view of experts. Binomial test for each question is conducted. In this stage 7 first priority objectives in each aspect have been selected as the main goals in each aspect because when organization can plan achieve its goal that have limited and distinguished objectives (5 to 7) to be able to extract the key success factors among its goals [45][49]. Then by using Friedman test, objectives were prioritized. To evaluate reliability of the questionnaires, Cronbach's Alfa method was used. 5. Presenting integrated IT-BSC and COBIT framework to evaluate IT projects

To provide integrated IT-BSC and COBIT framework, 4 aspects including “Company contribution”,

“users and customers”, “operational excellence” and “Future orientation” which were referred in various studies [39, 40, 41] was considered for IT-BSC. To extract goals and measures in each aspect, goals and metrics in the first level of COBIT IT processes has been used and allocated to different aspects of the framework. For allocating goals to each aspect, a complete list of all goals in the COBIT IT processes (including 27 goals) are given to IT experts and managers to be prioritized separately for each aspect. Questionnaires distributed among 40 experts and 36 of them were received. Among respondents, 20 were between 20 and 30 years old, 13 people were between 30 and 40 and 3 people were over 40 years old and the number of persons having PhD were 10, 6 persons were PhD students, 10 of them had M.S. and 10 were B.S.. Questionnaire distributed between people who had expertise in related field, among them 10 had experience of 5 years, 14 had experience of 12 years and 4 hadn’t determined their experience. By using binomial test, 7 first priority objectives in each aspect have been selected as the main goals and others were rejected because when organization can plan to achieve its goal that have limited and distinguished objectives (5 to 7) to be able to extract the key success factors among its goals [45].

Here the results of aspect of “customers and users” questionnaire analysis are shown. The value of Coronbach’s Alfa for this questionnaire was 0.874 which is more than 0.7 which shows high reliability of results of the questionnaire. Binomial test for each question is conducted, thus one of them is presented in the following as an example: H0: “Respond to business requirements align with business strategy” isn’t important as an objective of company contribution.

H1: “Respond to business requirements align with business strategy” is important as an objective of company contribution.

The result of binomial test for the “customers and users” aspect is shown in Table1:

- 14 -



Table 1. Binomial test for testing hypothesis for importance of each objective of “customers and users” aspect

Code in COBIT

objective Percent

of consensus

Approved or non approved

of H1 hypothesis

Asymp. Sig.(p-value)

PO1PO2،Po،PO10،AI1،AI6

،AI7،DS1،DS3،ME1

Respond to business requirements align with business strategy

50 Isn’t Approved .142

Po2،AI4،AI7ensure integrating of application programs with business processes


PO5،DS6 IT share in business profitability 60 Isn’t Approved .021

PO6،AI7،DS5ensure automating of business transactions and reliability of information transfer


AI1،AI2،AI6Ensure cause and effect relation of control and functional requirements with effective automated solutions


PO6،PO9 Clarify the effects business risks in IT objectives and resources


PO5،PO6،DS1،DS2،DS6،

ME1،ME4

Clarify expenses, benefits, strategies, policies and level of IT services 100 Approved .000

PO1،Po4،Po10،ME1،ME4

Respond to employers governance requirements


PO5،DS6،ME1،ME4

Ensure continuous improvement and readiness for change in future


PO2،DS11 Optimize using of information 90 Approved .008 Po2،Po4،PO7،

AI3 Create agility in IT processes


PO3،AI3،DS3،DS7،DS9

Optimize IT infrastructures, resources and capabilities


PO3،AI2،AI3،AI5

Create and maintain standard and integrated application systems


PO6،AI4،AI7،DS7،DS8

Ensure appropriate use and performance of technology and application’s solutions

80 Approved .008

PO6،DS5،DS11،DS12

Ensure confidentiality of critical information90 Approved .004

PO7،AI5 Create and maintain IT skills that are able to respond to strategies

80 Approved .000

PO8،AI4،DS1،DS2،DS7،DS8،DS10،DS13

Ensure end users satisfaction from presented level of services 100 Approved .000

PO9،Ds10،ME2

Protect achieved objectives 70 Isn’t Approved .045

PO8،PO10 Delivery of services in required time and budget and quality standards

100 Approved .000

PO9،DS5،DS9،DS12،ME2

Protect IT assets80 Approved .005

PO8،AI4،AI6،AI7،DS10

Decrease defects of service delivery and solutions

90 Approved .002

PO6،AI6،DS4،DS12

Ensure minimum effect of defects and changes of IT services in business

80 Approved .012

AI5 Create and maintain standard and integrated IT infrastructure

70 Approved .002

AI6،DS5 Maintain information integrity and processing infrastructure

80 Approved .018

PO6،AI7،DS4،DS5،DS12،

Ensure resistance of services and IT infrastructure from failure, delivered attack


- 15 -



DS13،ME2 and disaster

DS3،DS4،DS8،DS13

Ensure service availability100 Approved .000

DS11،ME2،ME3،ME4

Ensure IT compliance with rules and contracts


As it’s mentioned above, 7 first priorities between approved objectives were selected as key objective

in the aspect. Then by using Friedman test the objectives were prioritized. Results were shown for “customers and users” aspect in Table 2. Hypothesis for Friedman test are as follows:

H0: there is no significant difference between objectives H1: there is significant difference between objectives

Table 2. Friedman test for prioritizing objectives of “customers and users” aspect

Objective Score Rank

1 Clarify expenses, benefits, strategies,

policies and level of IT services 13.5 6

2 Optimize using of information 15 4

3 Ensure confidentiality of critical

information 20.7 2

4 Delivery of services in required time and

budget and quality standards 14 5

5 Decrease defects of service delivery and

solutions 19.4 3

6 Ensure service availability 8.5 7

7 Ensure end users satisfaction from

presented level of services 26.9 1

Based on results of Friedman, related objectives of each aspect are determined. Further by using

metrics of first level objectives in COBIT, appropriate measures of objectives in each aspect are determined. Results are shown in table 3.

- 16 -



Table 3. Combined framework of IT-BSC and COBIT

Customers and users Company contribution measures objectives measures objectives

C21- percent of user’s satisfaction from information model and data availability C22- percent of satisfied IT personnel C23- percent of satisfied stakeholder from quality of IT services, suppliers and data integration in new systems C24- percent of users that are satisfied by achieved performance C25- percent of satisfied stakeholder from service delivery in agreed level C26- percent of satisfied users from service delivery in agreed level C27- number of discontents from service contracts C28- percent of user satisfaction increment from services, systems and new technologies C29- degree of user’s improvement in result of better understanding of systems

1- Ensure service availability 2- Clarify expenses, benefits, strategies, policies and level of IT services 3- Delivery of services in required time and budget and quality standards 4- Optimize using of information 5- Decrease defects of service delivery and solutions 6- Ensure confidentiality of critical information 7- Ensure end users satisfaction from presented level of services

C11- degree of approval of business owners from IT strategic and operational programs C12- degree of alignment with governance and business requirements C13- satisfaction level of business from current state of projects and programs C14- percent of IT investment or provision of predetermined business benefits C15- percent of IT value drivers mapped to business value drivers C16- percent of IT expenses that are stated by value drivers of business C17- percent of IT projects that provide business expectation in determined timeline

1- Respond to business requirements align with business strategy 2- ensure integrating of application programs with business processes 3- ensure automating of business transactions and reliability of information transfer 4- Ensure cause and effect relation of control and functional requirements with effective automated solutions 5- Clarify the effects business risks in IT objectives and resources 6- Clarify expenses, benefits, strategies, policies and level of IT services 7- IT share in business profitability

Future orientation Operational excellence measures objectives measures objectives

C41- percent of satisfied business owners from training of application programs and supporting issues C42- number and type of deviations from infrastructure plan C43- level of stakeholder satisfaction from personnel’s IT skills C44- number of supported business processes by old

1- Ensure continuous improvement and readiness for change in future 2- Optimize IT infrastructures, resources and capabilities 3- Ensure appropriate use and performance of technology and application’s solutions 4- Create and maintain IT skills that are able to respond to strategies

C31- percent of IT investment or provision of predetermined business benefits C32- percent of IT value drivers mapped to business value drivers C33- percent of IT expenses that are stated by value drivers of business C34- amount of IT personnel transactions

1- Respond to employers governance requirements 2- Protect IT assets 3- Protect achieved objectives 4- Ensure confidentiality of critical information 5- Ensure resistance of services and IT infrastructure from failure, delivered attack and disaster

- 17 -



infrastructure C45- number of systems that cover security requirements C46- percent of IT compliance with rules, regulations and contracts C47- degree of senior satisfaction from internal and external monitoring reports C48- percent of spend expenses from competitive procurement

5- Create and maintain standard and integrated IT infrastructure 6- Create and maintain standard and integrated application systems 7- Ensure IT compliance with rules and contracts

C35- percent of critical IT objectives that are covered by risk evaluation C36- percent of IT risk evaluation integrated with IT risk evaluation approach C37- number of instances were confidential information was formed C38- number of business errors in result of IT services errors C39- level of understanding from expenses, benefits, strategies, policies and IT service level C310- number of changes in objectives due to efficiency and effectiveness of IT processes

6- Create agility in IT processes 7- Ensure minimum effect of defects and changes of IT services in business

6. Evaluation of a smaple project using the proposed framework

In this section "system of electronic correspondences" in a research organization will be evaluated by

using the measures obtained in the proposed framework. This project is designed and implemented to make the process of sending and receiving e-mails within the organization electronic and accelerates process of responses to the e-mails. This project has established about three years and makes the electronic communications of 450 staff and managers in Iran Telecommunication Research Center. To evaluate the measures, a questionnaire to suit each of the measures in the various aspects was designed, so that the degree of realization of any measure in qualitative form in mentioned project was called into question. The questionnaires included 34 questions and were distributed between two groups of respondents. The first group consisted of five experts including: project supervisor, two consulter, managing director and project manager to answer three aspects of “company contribution”, “operational excellence” and “future orientation” and the second group consisted of 30 users (Personnel) to respond to questions in “users and customers” aspect. Responses about the agreement or disagreement were analyzed using a five point Likert scale. The extent to which the respondents agrees or disagrees on the realization or non-realization of the measures in the project is graded on a scale of 1 to 9, where 1=strongly isn’t fulfilled, 3=isn’t fulfilled, 5= almost fulfilled, 7= fulfilled, 9= strongly fulfilled. An average, standard deviation, mod and normal weight are also calculated for each aspect. The higher the average score, the most likely the measure is close to realization [46][47]. Further, the reliability or internal consistency of questionnaire was assessed by Cronbach’s alpha [49]. Results are shown in Table4.

Analyses were done as follows: The mean scores from 1 to 4 means "weakness in the realization of system measures". Mean score from 4 to 6 (except 6) is assessed as "average". The mean score equal to 6 and above means "strong" and it shows that the project fulfills the criteria to a high level. The results of averages and weights are shown with two radar diagram in figure 2. Figure 2 shows the degree of realization of each measure.

- 18 -



Table 4. Results achieved from the project evaluation in organization

aspect measure AverageStandard deviation

modeNormal weight

Overall weight score

Company contribution

C11 8.2 .98 9 .16

6.59

C12 7.4 .80 9 .18 C13 6.2 .98 7 .13 C14 5.8 .98 7 .13 C15 5.8 .98 7 .14 C16 5.4 .80 7 .11 C17 6.6 .80 7 .15

Users and customers

C21 4.6 1.50 7 .10

5.68

C22 5.4 .80 7 .12 C23 5.8 .98 7 .12 C24 5.8 2.04 9 .12 C25 5.4 .80 7 .10 C26 6.2 1.60 9 .11 C27 5 1.26 7 .10 C28 7 1.26 9 .10 C29 5.8 .98 7 .12

Operational excellence

C31 5.8 .98 7 .10

5.53

C32 5.8 .98 7 .09 C33 5.4 .80 7 .10 C34 7.4 .80 9 .09 C35 5 1.26 7 .09 C36 4.6 1.50 7 .10C37 3.4 .80 5 .07 C38 6.6 .80 7 .11 C39 6.2 .98 7 .13 C310 4.6 1.50 7 .12

Future orientation

C41 6.2 .98 7 .15

5.07

C42 3.8 .98 5 .11 C43 6.2 1.60 9 .11 C44 2.2 1.60 5 .09 C45 4.2 .98 5 .14 C46 5.8 .98 7 .13 C47 6.6 1.50 9 .15 C48 4.2 1.60 7 .12

As it’s shown in the table the “company contribution’ aspects has higher score. This shows that the

mentioned project realize this aspect more than other ones. Realization of “users and customers”, “operational excellence” and “future orientation” aspects are in the next order.

- 19 -



Figure 2. Realization degree of evaluating measures in project (scores below 4 is weak, scores between

4 and 6 is medium, and scores above 6 is strong) 7. Conclusion

Today information is the most important asset and strategic resources in any organization. Therefore,

management of IT is the major basis of organizational planning and the need to ensure the value of IT, related risk management and information control are considered as key element in the governance of any organization. IT project have specific characteristics such as high risk, limited return on investment, having intangible results, high costs and hidden benefits that make the use of traditional methods to measure the benefits of the project very difficult. This article provided a framework for evaluating IT project with combining IT-BSC and COBIT framework. Aspects of proposed framework are chosen from IT-BSC and objectives and related measures extracted from COBIT and validated using binomial test and survey of experts in the field of information and communication technology. Since strategic alignment issue is one of the most important concerns of today’s IT managers, this framework is greatly contributed to alignment of IT and business in the organization. These measures can be used to evaluate all IT projects and assess the value of various projects to each other in an organization. For this purpose it’s necessary to provide weights to four aspects of "company contribution", "users and customers", "operational excellence" and "Future orientation" in accordance with vision and strategies of organization and then the mean weight scores according to weight of aspects are determined. So it will give only one score to each project that makes it comparable with other projects.

8. References

[1] Hochstrasser B., Griths C., “Controlling IT investment: strategy and management”, Chapman &

Hall, London, 1991. [2] Bacon J., “Why companies invest in information technology”, In: Willcocks L, editor. Information

management: the evaluation of information systems investments. Chapman & Hall, London, 1994. [3] Willcocks L., Lester S., “The evaluation and management of information systems investments:

from feasibility to routine operations”. In: Willcocks L, editor. Investing in information systems: evaluation and management. Chapman & Hall, London,1996.

[4] Fitzgerald G., “Evaluating information systems projects: a multidimensional approach”, Journal of Information Technology, Vol.14, pp.17–30, 1998.

[5] Willcocks L., Lester S., “Evaluating the feasibility of information systems investments: recent UK evidence and new approaches”, In: Willcocks L, editor. Information management: the evaluation of information systems investments. Chapman & Hall, London, 1994.

[6] Earl JM., “Management strategies for information technology”, Prentice Hall, New York, 1989. [7] Strassmann PA., “The squandered computer: evaluating the business alignment of information

technologies”, The Information Economics Press, New Canaan, CT, USA, 1997. [8] Hinton CM, Kaye GR., “The hidden Investments in Information Technology: the role of

organizational context and system dependency”, International Journal of Information Management, Vol.18, No.6, pp.413–27, 1996.

- 20 -



[9] Ballantine J., Stray S., “Financial appraisal and the ICT investment decision making process”,

Journal of Information Technology, Vol.13, pp.3–15, 1998. [10] Apostolopoulos TK, Pramataris KC., “Information technology investment evaluation:

telecommunication infrastructure”, International Journal of Information Management, Vol.19, No.4, pp.287– 96, 1997.

[11] Brealey RA, Stewart CM, Marcus AJ, “Fundamentals of corporate finance”, McGraw-Hill, New York, 1995.

[12] Manigart S, Ooghe H., “Investment selection methods”, Management for engineers, vol.11, No.4, pp.1–18, 1994.

[13] Lumby s., “Investment appraisal and related decisions”, Van Nostrand Reinhold, London, 1981. [14] Clemons EK, Weber BW, “Strategic information technology investments: guidelines for decision

making”, Journal of Management Information Systems, Vol. 7, No.2, pp.10–31, 1990. [15] Turner JR, “The handbook of project-based management”. McGraw-Hill, New York, 1993. [16] Ballantine JA, Galliers RD, Stray SJ, “Information systems/technology evaluation practices:

evidence from UK organizations”, Journal of Information Technology, Vol.12, No.1, pp.29–41, 1996.

[17] Wateridge J., “Training for IS/IT project managers: a way forward”, International Journal of Project Management, Vol.15, No.5, pp.283-288, 1997.

[18] Wright JN, “Time and budget: the twin imperatives of a project sponsor”, International Journal of Project Management, Vol.15, No.3, pp.181–186, 1997.

[19] Willcocks L., “Information management: the evaluation of information systems investments”, Chapman & Hall, London, 1994.

[20] Dos Santos B., “Justifying investments in new information technologies”, Journal of Management Information Systems, Vol.7, No.4, pp. 71–90, 1991.

[21] Whiting R., Davies J., Knul M., “Investment appraisal for IT systems”, In: Willcocks L, editor. Investing in information systems: evaluation and management, Chapman & Hall, London, 1996.

[22] Strassmann P., “The business value of computers”, The Information Economics Press, New Canaan, CT, USA, 1990.

[23] Porter M., “Technology and competitive advantage”, Journal of Business Strategy, Vol. 5, No. 3, pp.60-78, 1985.

[24] Shank JK, Govindarajan X., “Strategic cost analysis of technological investments”, Sloan Management Review, Vol.34, No.1, pp.39–51, 1992.

[25] Parker MM, Trainor HE, Benson RJ, “Information strategy and economics: linking information systems strategy to business performance”, Prentice Hall International, 1989.

[26] Wiseman D., “Information economics: a practical approach to valuing information systems”, In: Willcocks L, editor. Information management: the evaluation of information systems investments. Chapman & Hall, London, 1994.

[27] Moad J., “Time for a fresh approach to ROI”, Datamation archive, Vol.41, No. 3, pp. 57-59, 1995.

[28] Willcocks L., “Investing in information systems: evaluation and management”, Chapman & Hall, London, 1996.

[29] Kaplan RS, Norton DP. “The balanced scorecard: measures that drive performance”, Harvard Business Review, pp.71–80, 1992.

[30] Stewart W.E., “Balanced scorecard for projects”, Project Management Journal, Vol.32, No.1, pp.38–53, 2001.

[31] Barber, E., Miley, F., “Monitoring Project Progress: More than a series of feedback loops”, Australasian Evaluation Society International Conference, October/November 2002.

[32] Stewart W.E., “Balanced scorecard for projects”, Project Management Journal, Vol.32, No.1, pp.38–53, 2001.

[33] Eilat H., “R&D project evaluation: An integrated DEA and balanced scorecard approach”, Omega, Vol.36, No.5, pp.895-912, 2006, DOI: 10.1016/j.omega.2006.05.002.

[34] Van Grembergen, W., Saull, R., “Information Technology Governance through the Balanced Scorecard”, Proceedings of the 34th Hawaii International Conference on System Sciences (HICSS), pp 1-12, 2001.

- 21 -



[35] Milis. K, Mercken, R, “The use of the balanced scorecard for evaluation of information and

communication technology projects”, International Journal of Project Management, No. 22, pp. 87-97, 2004.

[36] Dickson G. Wetherbe J., “The Management of Information Systems”, McGraw Hill, New York, 1985.

[37] Grembergan. “The Balanced Scorecard and IT Governance”, Information Systems Control Journal, Vol.2, pp.40-43 ,2000.

[38] Son, Sertac,Weitzel. “Designing a Process Oriented Framework for IT Performance Management Systems”, The Electronic Journal Information Systems Evaluation, Volume 8, No. 3, pp. 219-228, 2005.

[39] Sedera; Gable;Rosemann;. “A Balanced Scorecard Approach to Enterprise Systems Performance Measurement”, Proceedings of the Twelfth Australasian Conference on Information Systems. 2001.

[40] Van Grembergen, W., De Haes, S., “Measuring and Improving IT Governance through the Balanced Scorecard." Information Systems Control Journal, Vol. 2, 2005.

[41] Grembergen. “IT Governance and its Mechanisms”, 35th Hawaii International Conference on System Sciences (HICCS). IEEE, 2002.

[42] Salle, Mathias, “IT Service Management and IT Governance: Review, Comparative Analysis and their Impact on Utility Computing’, HP Labs Technical Report HPL-2004-98.

[43] Zee, Van der. “Alignment is not enough: integrating business and IT Management with the Balanced Scorecard”, Proceedings of the 1st Conference on the IT Balanced Scorecard. antwerp, 1999.

[44] COBIT 4.1, IT governance institute, USA, 2007. [45] Keeney, R. L., “Value-Focused Thinking: A Path to Creative Decision Making” .Cambridge, MA:

Harvard University Press, 1992. [46] Ruikar, K., Anumba, C., Carrillo, P., “VERDICT-an e-readiness assessment application for

construction companies”, Automation in construction, Vol. 15, No.1,pp 98-110, 2006. [47] Mohammad Kazem HAKI, Maia Wentland Forte, “Service-Oriented Business-IT Alignment: a

SOA Governance Model”, AISS: Advances in Information Science and Service Sciences, Vol. 2, No. 2, pp. 51 - 60, 2010.

[48] Johannes K. Chiang, "E-Commerce Research Trend Forecasting: A Study of Bibliometric Methodology", JDCTA: International Journal of Digital Content Technology and its Applications, Vol. 5, No. 1, pp. 101 - 111, 2011.

[49] Zhou Ping, Le Zhong-Jian, "Analyzing Allocation Strategies of Government Information Resources Using System Dynamic Feedback Archetypes", JDCTA: International Journal of Digital Content Technology and its Applications, Vol. 5, No. 1, pp. 296 - 304, 2011.

- 22 -

Developing RAE1

Documents

cobit framework

proposed framework

combined framework

benefits of capital

appropriate methods

combination of cobit

control of capital investments

information technology