Management Decision - hgtc.irhgtc.ir/darabkola/Pakoassel/930523.2/4- A performance measurement... · Management Decision Emerald Article: A performance measurement framework in portfolio

Management DecisionEmerald Article: A performance measurement framework in portfolio management: A constructivist caseRogério Tadeu de Oliveira Lacerda, Leonardo Ensslin, Sandra Rolim Ensslin

Article information:

To cite this document: Rogério Tadeu de Oliveira Lacerda, Leonardo Ensslin, Sandra Rolim Ensslin, (2011),"A performance measurement framework in portfolio management: A constructivist case", Management Decision, Vol. 49 Iss: 4 pp. 648 - 668

Permanent link to this document: http://dx.doi.org/10.1108/00251741111126530

Downloaded on: 21-10-2012

References: This document contains references to 56 other documents

Citations: This document has been cited by 3 other documents

To copy this document: [email protected]

Access to this document was granted through an Emerald subscription provided by UNIVERSITY OF TORONTO

For Authors: If you would like to write for this, or any other Emerald publication, then please use our Emerald for Authors service. Information about how to choose which publication to write for and submission guidelines are available for all. Please visit www.emeraldinsight.com/authors for more information.

About Emerald www.emeraldinsight.comWith over forty years' experience, Emerald Group Publishing is a leading independent publisher of global research with impact in business, society, public policy and education. In total, Emerald publishes over 275 journals and more than 130 book series, as well as an extensive range of online products and services. Emerald is both COUNTER 3 and TRANSFER compliant. The organization is a partner of the Committee on Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archive preservation.

*Related content and download information correct at time of download.

A performance measurementframework in portfolio

managementA constructivist case

Rogerio Tadeu de Oliveira Lacerda, Leonardo Ensslin andSandra Rolim Ensslin

Federal University of Santa Catarina, Florianopolis, Brazil

Abstract

Purpose – The main aim of this paper is to present a framework to create a better understanding ofthe context and aid the portfolio management process. The paper seeks to present a case to illustratethe proposed methodology, identifying and measuring the success parameters for a project in aportfolio.

Design/methodology/approach – The research method is a qualitative and quantitative mixtureand it is presented as a study case. The primary data were obtained using semi-structured interviewswith decision makers. Bibliographic research is used in order to construct the theoretical frameworkand the intervention instrument adopted is the multicriteria decision aiding methodology –constructivist (MCDA-C).

Findings – The methodology allows the criteria that must be taken into account, according to thedecision makers’ values and preferences in the project selection and sorting processes, to be visualized.The framework supports the ordinal and cardinal measurement of the project performance, making itpossible to compare and rank proposals, as well as providing a process to improve project proposals.

Practical implications – This process has helped in negotiations between stakeholders in aportfolio context and, consequently, has helped the chief project officer (CPO) to select and prioritizestrategic projects within the demand for new products.

Originality/value – This paper offers an empirical understanding of the application of performancemeasurement to portfolio management context and identifies complementary tools in thisinterdisciplinary area.

Keywords Performance management, Portfolio investment, Decision making, Technology led strategy

Paper type Research paper

1. IntroductionPortfolio management presents an alternative to aid the strategic decision processes in acontext that involves complexity, by using problem structuring techniques that improvethe understanding of the consequences on the operational focus of the business.

The competitive forces, such as agility and innovation, has caused portfoliomanagers to need to improve knowledge of the consequences of their decisions in thestrategic objectives of the organization (Gann and Salter, 2000).

Nowadays, the decision to build understanding about the decision contexts,explaining the objectives to be achieved and their association with the operationalactivities, has become key for organizations.

The current issue and full text archive of this journal is available at

www.emeraldinsight.com/0025-1747.htm

The authors would like to thank the English Journal Experts (EJE) Team that reviewed andcorrected the English version of this paper.

MD49,4

648

Management DecisionVol. 49 No. 4, 2011pp. 648-668q Emerald Group Publishing Limited0025-1747DOI 10.1108/00251741111126530

On this issue the research question of this article emerges: how to measure a projectcharter in light of the strategic objectives of a company in order to aid decisions onportfolio management?

In order to answer the research question, the main objective in this study is topropose a methodology that allows those dimensions judged by the decision maker asnecessary and sufficient for evaluating projects to be identified, measured andintegrated and thus enable the creation and ordering of proposals in a cycle of aselection of projects in a software development company.

Thus, the specific objectives of the research are to:. present a performance measurement framework and to generate a better

understanding of the portfolio context; and. present an application in order to illustrate the proposed framework for

identifying and evaluating a project in a portfolio.

To achieve the objectives set out above, the methodology selected by authors was theMCDA-C by its characteristics of constructivism, specifically regarding:

. the ability to promote awareness of the actors in the context that we intend toimprove;

. the ability to structure and evaluate the dimensions considered relevant for theseactors, giving more reliable results;

. the ability to spread generated knowledge; and

. the capability to support the decision-making process.

The relevance of the theme is given by:. the current organization’s context with respect to its business objectives, which

are increasingly using project management as a tool for competitive advantage(PriceWaterhouseCoopers, 2004);

. the feasibility to solve problems using the methodology selected for this study(Bana e Costa et al., 1999);

. extending the scientific publications of Englund and Graham (1999), Coombset al. (1998) and Cooper (2000), suggesting the creation of criteria for selectingprojects, but not focusing their studies on how to perform this activity.

Therefore, following the Introduction, this article is arranged in five sections. In thenext section the theoretical framework of the portfolio management is presented; in thethird section the research methodology used is presented; in the fourth section the builtmodel, analysis and discussion is presented; and the fifth section contains the finalconsiderations of the authors followed by the reference list.

2. Theoretical foundations2.1 Success of projectsGiven that its history had been developed in operational research (OR), the discipline ofproject management was coined within a positivist paradigm (Williams, 2005; Pollack,2007). Thus, it is necessary to consider if such a paradigm is appropriate for a

A performancemeasurement

framework

649

discipline that deals with a complex reality (De Meyer et al., 2002; Pich et al., 2002;Sommer and Loch, 2004).

It is important to note that some hard thinking about project management shouldnot be regarded as wrong or needing to be replaced, but one must bear in mind that thisview is only one point-of-view to examine this field of knowledge (Winter andCheckland, 2003).

On the rationalist view, Roy (1994) considers that:

This is a reductionist conception of OR which I would call unproductive. It is largelyresponsible for what has been called the OR crisis. First of all, this conception of OR tends tocut it off from the milieu which nourishes it and legitimates it as something other than abranch of mathematics. Cutting off OR in this way encourages researchers to work inisolation. This results in naive or impoverished references to managerial reality anddecision-making processes. Those responsible for solving concrete problems are thusinevitably disappointed by the gap between their own expectations and the results theyreceive.

Among the recent published studies on the effects of these two paradigms in theoryand practice of project management, we can highlight the text of Williams (2005). Inthis systemic perspective, the project management has an attitude focused onstakeholders rather than pre-established requirements (Tukel and Rom, 2001), aprecedent in the “iron triangle” (Atkinson, 1999) in project management for a morestrategic view (Jugdev and Muller, 2006).

This evidence is relevant, since the end result of the project is assessed differentlyby various stakeholders in the project, and the success criteria should reflect diverseinterests and viewpoints (Lipovetsky et al., 1997; Baccarini, 1999; Shenhar et al., 2002).Neglecting any point-of-view can mean the failure of the project (Dvir et al., 1998),being then a great opportunity for multi-dimensional studies, enabling us to analyzethe mutual interactions of all variables and managerial success metrics (Dvir et al.,1998).

With such characteristics in question, some authors have understood that the softapproach has positive impacts on the management aspect when:

. the technological uncertainty is high (De Meyer et al., 2002) and long-termconsequences are diffuse;

. the project is susceptible to external factors (Winter and Checkland, 2003); and

. the complexity of the scope and context of the project is high (Atkinson et al.,2006).

2.2 Portfolio management and business strategyCorporate strategy is one of the main organizational processes, Andrews (1980) showsthat this process contains two important aspects: formulation and implementation(Andrews, 1980; Mintzberg, 1994).

However, to achieve implementation success an alignment strategy is required(Shenhar, 2004; Minarro-Viseras et al., 2005). The authors of this article understandthat such organizational alignment can be improved with the portfolio managementand performance measurement.

According to Cooper et al. (2000), portfolio management is the formulation, selectionand implementation of projects and the operationalization of organizational strategy.

MD49,4

650

Despite the growing interest and relevance to portfolio management byorganizations, they face some difficulties (Elonen and Artto, 2003), such as decisionson project prioritization (Cooper et al., 2000), being a critical point of the projectselection, which is the focus of this paper.

Bearing this in mind, Englund and Graham (1999) explore a process model forportfolio management based on four stages, operating in the theoretical framework.

2.2.1 Establishing evaluation criteria. As a first step, it is necessary to define whatshould be done by focusing on the strategic organization objectives (Bourne et al., 2000;Harrison and Pelletier, 2000). This step represents a very crucial point that determineswhether the rest of the process will be successful (Englund and Graham, 1999; Cooper,2007).

At this point it is appropriate to increase the understanding of what the dimensionsto be taken into account by decision makers are, especially in contexts that work withincomplete, fuzzy and conflicting information (Liesio et al., 2007).

In the process for formulating the problem, it is necessary to determine thedimensions in which the projects will be evaluated based on the perception and valuesof the decision makers. These performance criteria are scales on which the judgmentsand decisions will be based to evaluate different projects (Chien, 2002).

It is important to note that portfolio management is under political pressure, giventhe personal interests of executives to use the power to impose their individualpreferences (Englund and Graham, 1999; Chien, 2002; Elonen and Artto, 2003; Engwalland Jerbrant, 2003). Thus, a decision-making process is structured and formalizedthrough a decision maker in managing pressures of interest groups, justifying theirdecisions and communicating with other organization elements (Chien, 2002).

The criteria must be sought in the strategic objectives of the people responsible forportfolio management (Keeney, 1992). The socio-political values of the decision makerand the objective properties of the projects set the environment where the criteriashould be sought (Keeney and Raiffa, 1976; Roy, 1993; Landry, 1995; Roy, 1996; Bana eCosta et al., 1999). Care must be taken to ensure that the criteria are built from thedecision maker’s values and not from the differences between the alternatives and/orsought from past similar situations even if they are successful (Kennerley and Neely,2003; Yahaya and Abu-Bakar, 2007; Ensslin et al., 2010).

Although there are several generic proposals for structuring the criteria forselection, the most important step is to identify the criteria that has greater significancefor the organization (Englund and Graham, 1999; Shenhar et al., 2002; Chen, 2008),taking into account the preferences of decision makers (Roy, 1993) and the adoption ofappropriate management techniques to each particular situation (Shenhar, 2001; Lewiset al., 2002).

Since the establishment of criteria and measurement takes into account the valuesand preferences of the manager responsible for the area being evaluated, this step inperformance evaluation is personalized to the manager responsible for the decision.Roy (1996) and Landry (1995) refer to this condition as the limit of objectivity.

After identifying what are the necessary and sufficient properties of the portfoliothat explain the decision maker’s values (Lacerda, 2009), it is necessary to translatethem into a measurable and unambiguous set of scales. At this stage, the decisionmakers are prompted by the facilitator to clarify the direction of preferences for eachspecific goal set in the previous step.


framework

651

First, the ordinal scales built have a limited degree of knowledge since all theinformation they provide is qualitative. If the decision maker wants to improve thequality of the information he has to transform the ordinal scales into cardinal scalesproviding the information of the difference of attractivity between the level of the scales.

2.2.2 Project information collection. Once the criteria has been clarified, the teamscan “shape” the project scope according to these criteria, in order for them to be alignedto the strategy (Salomo et al., 2007).

Thus, the second step defined proposed by Englund and Graham (1999) for selectionof projects is the project information collection. The main way out of this phase is a listof projects with their charters and information necessary to confront the projectproperties with pre-established criteria for the organization.

2.2.3 Evaluation and recommendations. This stage of evaluation is to determine theimpact of projects on performance indicators to understand the consequences (Keeney,1992). A recurring problem in portfolio management is to give priority to shortduration, low-risk projects and consequently with little impact on improvingcompetitiveness (Fricke and Shenhar, 2000).

Thus, by measuring the impact of each element on the portfolio management, thethird stage ends with the establishment of prioritization in projects within the availableinvestment resources.

2.2.4 Monitoring the portfolio management. The fourth step is proposed to allocatethe financial and human resources in prioritized projects, communicate project teamsand continue managing the initiatives. In this stage, monitoring of projects shouldreturn information to executives in order to close the cycle of four stages, as proposedby Englund and Graham (1999).

Noting that there is not a single tool for effective portfolio monitoring, themethodology presented in this article may be used to build models in line with the twoapproaches described by Cooper et al. (2000) for monitoring portfolio management.

2.3 Portfolio management and performance evaluationAs shown, the contribution of a structured process on the constructivist performanceevaluation allows the identification of opportunities for aiding the portfoliomanagement. Thus, we draw the following parallel with the paradigms for decisionaid to be recognized, as illustrated in Table I.

3. Research methodologyThis item is divided into three sections, the first section presents a methodologicalframework, the second section presents the procedures used in the literature reviewand the third section discusses the adopted intervention instrument.

3.1 Methodological frameworkThis research is exploratory, applied and developed in a case study way because thegoal resides in deepening the knowledge related to innovation within a softwaredevelopment company. The data used are primary, since they were obtained throughunstructured interviews with the technology director and manager of the PMO (ProjectManagement Office) of the company in which the research was performed. Theresearch method is qualitative and quantitative. The intervention tool used was theMCDA-C (Ensslin et al., 2010).

MD49,4

652

Dec

isio

nai

din

gp

arad

igm

sP

arad

igm

des

crip

tion

Con

tex

tin

por

tfol

iom

anag

emen

tR

efer

ence

inp

ortf

olio

man

agem

ent

P1¼

Un

iqu

enes

s,id

enti

tyD

ecis

ion

mak

ers’

val

ues

and

pre

fere

nce

sC

rite

ria

for

eval

uat

ing

pro

ject

sm

ust

be

con

tex

tual

ized

and

dev

elop

edsp

ecifi

call

yin

each

case

ofth

ep

roje

cts

per

form

ance

eval

uat

ion

En

glu

nd

and

Gra

ham

,19

99;

Sh

enh

ar,2

001;

Ch

ien

,200

2;E

ng

wal

l,20

03

P2¼

Lim

ited

kn

owle

dg

eD

ecis

ion

mak

ers’

nee

dto

imp

rov

eth

eir

un

der

stan

din

gof

the

dec

isio

nco

nse

qu

ence

s

Ap

pro

ach

esth

atex

pan

dth

eu

nd

erst

and

ing

ofd

ecis

ion

mak

ers

onth

eco

nte

xt

mu

stb

eu

sed

Wil

liam

s,20

05;

Can

ezan

dG

arfi

as,

2006

;L

iesi

oet

al.,

2007

P3¼

Soc

ial

enti

tyT

ofa

vou

rth

est

akeh

old

ers

wit

hin

tere

sts

inth

ed

ecis

ion

tosu

bm

itth

eir

inte

rest

sto

the

dec

isio

n

Rec

ogn

ize

that

the

crit

eria

for

eval

uat

ion

are

infl

uen

ced

by

soci

alp

arti

cip

ants

inth

ep

ortf

olio

man

agem

ent

dec

isio

np

roce

ss

En

glu

nd

and

Gra

ham

,19

99;

Ch

ien

,20

02;

En

gw

all,

2003

;W

illi

ams,

2005

P4¼

Rec

urs

ive

par

tici

pat

ory

lear

nin

gT

he

dy

nam

icre

curs

ive

pro

cess

ofp

arti

cip

ants

lear

nin

gR

ecog

nit

ion

that

the

lear

nin

gp

roce

ssis

cycl

ical

and

that

the

dec

isio

np

roce

ssco

nte

xt

nee

ds

am

ech

anis

mto

inco

rpor

ate

such

kn

owle

dg

e

En

glu

nd

and

Gra

ham

,19

99;

Coo

per

etal.,

2000

;En

gw

all,

2003

;Wil

liam

s,20

05

P5¼

Pri

nci

ple

sof

mea

sure

men

tP

rop

erti

esof

ord

inal

scal

es,i

nte

rval

,an

dra

tio

Am

ong

refe

ren

ces

ofp

ortf

olio

man

agem

ent

fou

nd

,n

one

incr

ease

sth

eu

nd

erst

and

ing

ofth

ep

roce

ssof

mea

suri

ng

the

lev

elof

dif

fere

nce

sre

cog

nit

ion

bet

wee

nor

din

alan

dca

rdin

alsc

ales

.

Rob

erts

,19

79;

Bar

zila

i,20

01

P6¼

Leg

itim

acy

and

val

idat

ion

Tra

nsp

aren

cyof

par

tici

pat

ion

,re

cog

nit

ion

ofu

sefu

lnes

sof

kn

owle

dg

eg

ener

ated

and

the

scie

nti

fic

stat

us

ofth

eco

nst

ruct

ion

ofk

now

led

ge

use

d

Rec

ogn

itio

nb

yth

ed

ecis

ion

mak

er,

kn

owle

dg

eb

uil

tb

yth

ed

ecis

ion

aid

pro

cess

was

use

fult

ou

nd

erst

and

the

con

seq

uen

ces

ofp

roje

cts

inst

rate

gic

obje

ctiv

esas

wel

las

hav

ing

scie

nti

fic

sup

por

tfo

rco

rpor

ate

use

En

glu

nd

and

Gra

ham

,19

99;

Ch

ien

,20

02;

Can

ezan

dG

arfi

as,

2006

;E

nss

lin

etal.,

2010

Table I.Paradigms for decision

aid and portfoliomanagement


framework

653

3.2 Literature reviewThe theoretical framework of this article began by searching for articles in each journalcited by Shenhar and Dvir (2007), using the keyword “project management”, whichcreated a database of 200 articles that, after systematic reading of their abstracts,answered the previously defined criteria:

. view of knowledge is not reduced to the positivist paradigm;

. recognize that the dimensions of project success are socially constructed; and

. create opportunities to expand the understanding of the degree of success inproject and portfolio management.

Finally, after reading the full articles, eight were selected (Coombs et al., 1998; Frickeand Shenhar, 2000; Chien, 2002; Dvir and Lechler, 2004; Shenhar, 2004; Williams, 2005;Tukel et al., 2006; Cooper, 2007) as the core of the bibliography.

3.3 Construction of the multicriteria model – proceduresThe construction of the model of performance evaluation following the MCDA-Cmethodology is divided into eight steps, described in this sub-section (see Figure 1).

(1) Contextualization. The structuring phase aims to explain the context andachieve an understanding of the problem to be discussed. To achieve such anaim, the players involved in the context are identified and the problemstatement is legitimated with them.

(2) Hierarchical structure of value. The facilitator then encourages the decisionmaker to talk about the context and. by interpreting the interviews. The

Figure 1.The MCDA-C decisionaiding phases

MD49,4

654

primary elements of evaluation (PEE) are identified. These elements are theessential factors in the decision maker’s system of values and concerns. Foreach PEE, a concept representing the decision maker’s choice of preferencedirection is constructed, as well as its psychological opposite pole. Then, thedecision maker is encouraged to group the concepts into areas of concern. Withthe concepts of each area of concern, cognitive maps are constructed (Bititciet al., 2001). In the means-ends relationship map, the clusters of concepts areidentified (Eden et al., 1985) and they represent the map in an exhaustive way.Each cluster in the cognitive map has an equivalent point-of-view in thehierarchical structure of value. This association makes it possible to transferknowledge from cognitive map to the hierarchical structure of value (Ensslinet al., 2000).

(3) Construction of descriptors. The hierarchical structure of value represents thestrategic dimension called fundamental points-of-view (FPsV) and theirconnections to the operational activities, called elemental points-of-view (EPsV).Next, it is necessary to use the information in the cognitive maps to buildordinal scales in the hierarchical structure of value, named descriptors, in orderto measure the range of what is measured (Bana e Costa et al., 1999). In order toestablish the basis for comparing the performance between descriptors, thedecision maker must identify the reference levels “neutral” and “good” (Ensslinet al., 2000). The process of qualitative knowledge generation finishes with thedescriptors.

(4) Independence analysis. To continue the process of building knowledge, thequalitative scales of descriptors must be transformed onto cardinal scales andthen integrated. The MCDA-C uses a compensatory model to build the globalevaluation model. This model assumes that the conversion rates used in theintegration are constant. To achieve this condition, the criteria must beindependent (Ensslin et al., 2000).

(5) Construction of values functions and identification of conversion rates. The nextstep in the methodology is the transformation of the descriptors into cardinalscales called value functions. The value functions are all interval scale. Thistransformation requires the decision makers to describe the different levels ofattractiveness for all the levels of the descriptor. The integration is achieved byassociating the conversion rates with the increase in performance whenimproving from the “neutral” reference level to the “good” reference level foreach descriptor.

(6) Identification of impact profile of alternatives. Then it is possible to evaluate theperformance of every alternative and, among them, the status quo. The modelsconstructed by the MCDA-C methodology make possible an explicit evaluationin numerical and/or graphic form, facilitating the identification andunderstanding of the intensity of strong as well as weak points of thealternatives under evaluation.

(7) Sensitivity analysis. In order to provide a broad view of the stability ofalternative performances, the model allows for the development of a sensitivityanalysis of the impact of alternatives in the scales, in the attractivenessdifference in cardinal scales as well as in the conversion rates.


framework

655

(8) Formulation of recommendations. The knowledge generated allows the decisionmakers to visualize for each criterion where the performance of a givenalternative is “good”, “normal” or “weak”. The scale (levels) of the descriptorsallows the identification of actions to improve performance. Combining thisknowledge with the global evaluation obtained in the previous step, it ispossible to generate new alternatives and measure their impact in the context.

4. Multicriteria model of project evaluationStep 1: contextualizationThe research commenced with meetings with the decision makers of the company inorder to contextualize the problem and expose its function. The company wanted tohave a mechanism to identify and evaluate project proposals of new software products.

The interview resulted in the establishment of a problem focus, with definition of:

(1) Label: to evaluate the competitive advantage of a new company product.

(2) Players:. core decision makers: CTO (Chief Technology Officer) with the CPO;. relevant stakeholders: product manager and sales executive;. those directly affected by decisions: users;. those indirectly affected by decisions: customers.

(3) Problem statement: product managers demand new software, but they do nothave a mechanism to evaluate the competitiveness of a product at the beginningof a project.

Step 2: hierarchical structure of valueAfter legitimating the context of the problem with the decision maker, the PEEs weremapped. A total of 60 PEEs were identified, and classified in two areas of concern forthe decision maker:

(1) compatible products with global players; and

(2) life-cycle cost.

As the study continued, the decision makers were questioned in order to guide thePEEs to action and to determine the concepts inherent in each PEE. When this activitywas concluded, the decision maker started to construct two means-ends relationshipmaps, one for each area of concern. With the maps, it was possible to identify thesub-clusters. Next, the clusters were associated with the concerns and transferred tothe hierarchical structure of value, forming the fundamental points-of-view and theelementary points-of-view.

Figure 2 shows the means-ends relationship map constructed for the area of concern“compatible products with international players”, as well as the identification of itsclusters that generate the FPsV. At this point, a hierarchical structure or a tree of valuewas proposed for the decision context.

Step 3: construction of descriptorsAfter the identification of the FPsV, the construction of scales for the multicriteriamodel was started in order to measure the performance of each action on an ordinal

MD49,4

656

Figure 2.Means-ends relationship

map of an area of concernhighlighting the clusters

“strategic alliances”,“pioneering” and “SOA”


framework

657

scale, using the decision maker’s construction. In order to construct the scales, in somecases it was necessary to decompose the FPsV in the elementary points-of-view (EPsV)in order to operationalize the descriptors.

From the hierarchical value structure, the construction of descriptors was started inorder to express the decision maker’s strategic objectives.

Step 4: independence analysisAs described in section 3.3.4, the MCDA-C methodology uses a compensatory model,thus requiring that the criteria measured be preferentially independent. Therequirement is that the difference of attractiveness between levels of a given criteriaare not affected by the level of performance, where the alternative impacts, on othercriteria.

Figure 3 presents the preferential independency analysis for the criteria: “operationssystems” (OS) and “application services” (AS).

The analysis is done for the interval between the two levels of reference: Neutral andGood. Thus, for the test four fictitious alternatives are created, as showed in Figure 4.

The preferential independence can be tested for ordinary and cardinally (Keeney,1992). For the purpose of this research the cardinal test will be presented which is thestronger.

The cardinal preferential independence (CPI) must be tested for both criteria.

Figure 3.Criteria “operationssystems” (OS) and“application services” (AS)with their levels ofreference, to be used forthe preferentialindependency analysis

MD49,4

658

Figure 4.Four fictitious alternatives

created for preferentialindependency analysis

Figure 5.Cardinal preferential

independence (CPI) checkfor the “operations

systems” (OS) in relationto “application services”

(AS)


framework

659

Thus, let the first test be:A-Is “operations systems” (OS), CPI of “application services”(AS).In order to be the difference in attractiveness of level 5 from level 2 {V1 (5, AS) –V1 (2, AS) ¼ D1} in criteria OS must be the same independently of the level of AS. SeeFigure 5.

The same process was done to test the CPI of AS in relation to OS.Following this process, all the criteria of the model were analyzed to check the

independence of preferences.

Step 5: construction of values functions and identification of conversion ratesWhen the structuring of the decision context was concluded, the evaluation phase wasstarted with the transformation of the descriptors into cardinal scales, so as to expressthe attractiveness of the impact levels of the descriptors. With the use of MACBETHsoftware, Figure 6 shows the transformation phases from the ordinal scale to thecardinal, for one of the EPV descriptor “sign services” from FPV-“SOA”.

Once having the value function, and defined the unit, of all the points-of-view, theprocess to integrate them and form the representation of the model of global evaluationaccording to the perception of the decision maker was started. Integration wasachieved by applying the conversion rates to the points-of-view. At this point, theglobal model of the multicriteria support was used to evaluate the success of a softwareproject of the company investigated.

Step 6: identification of impact profile of alternativesThe project under consideration obtained 50 points, and hence was rated ascompetitive, according to the decision maker’s perception. The evaluation in graphicform is presented in Figure 7, where the arrows beside the points-of-view are the scalerepresentation of success in each dimension evaluated.

Step 7: sensitivity analysisFinally, changes to conversion rates and the impact of each criterion were simulated, asa sensitivity analysis of the model. This analysis resulted in an instrument to show tothe decision maker and project agents, where the best opportunities for improvement inperformance are, thus, improving the competitiveness of new products developed.

Figure 6.Transformation process ofa descriptor in a specificcriterion (cardinal scale),using the MACBETHsoftware

MD49,4

660

Figure 7.Hierarchical value

structure and the graphicof consequences of the

project


framework

661

Step 8: formulation of recommendationsThe generated knowledge has helped to measure, on a cardinal scale, the contributionthat improvement actions may make to the strategic objectives as can be observed inthe measurement of 50 points for the initial version of the project evaluated. With thisknowledge, the CPO started a new activity to sort the improvement requests andimprove the project’s performance, linking with the FPV “Speed in developing newfeatures”, that would improve the global performance by 13 points (see Figure 8). Thiswould involve the average rate of errors dropping from four to two (EPV–“ProjectManagement”), the project deploying ten features requested by stakeholders per monthinstead of eight (EPV-“Productivity”), and the project reusing 20 percent of the lines ofcode instead of the original 15 percent (EPV-“Components Reuse”). This illustratesStep 8 of the MCDA-C, recommendations.

5. Conclusions and recommendations for further researchThe present article used recent views of knowledge to propose solutions to someinherent problems in portfolio management as observed in theoretical foundations (seeTable II).

In addition, the case study findings about the parallels between portfoliomanagement theory (see Table I) and decision aiding paradigms are summarized inTable III.

Figure 8.The representation of theimpact of improvementaction a’ in the globalperformance against thecurrent project a

MD49,4

662

With the above point in mind, the research question guiding this work is revisited: howto measure a project charter in light of the strategic objectives of a company in order toaid decisions on portfolio management?

To answer this question, the present work has presented a methodology that, byusing the instruments described here, helps the creation process to evaluate and sortthe project proposals and makes it possible to increase the alignment with the strategicplanning of the organization.

Section Concerns in portfolio management Contribution of the MCDA-C

2.1 Success of projects The success of a project isevaluated by different criteria andvaries according to the views ofstakeholders

The MCDA-C recognizes thevalues and preferences ofdecision-makers andoperationalizes them in anobjective instrument of ordinaland cardinal measurement

2.2 Portfolio managementand business strategy

Portfolio management is based onthe formulation, selection andimplementation of projects to meetthe strategic objectives oforganizations

The MCDA-C is presented as amethodology to explain what isimportant in a given context(formulation), sort the projects bytheir contribution to strategy(selection) and monitor the projectthroughout its implementation

2.2.1 Establishing evaluationcriteria

This is a crucial step in theprocess of portfolio management,because it is based on the selectioncriteria that projects will bedecided on and the strategy of anorganization is achieved fromtheir execution

The MCDA-C enables theidentification, organization,measurement and integration ofcriteria judged necessary andsufficient by the decision maker,in order to measure thecontribution of the portfolio in thebusiness strategy

2.2.2 Project informationcollection

In this step, the team can createactions aligned with the strategicobjectives of a portfolio

The models created by MCDA-Ccan be used by management andtechnical team to identify thefactors that most contribute to thebusiness strategy and identifyactions to improve theirperformances

2.2.3 Evaluation andrecommendations

Each project is evaluated based oncriteria of the model and theprojects with more contributionwill be funded

The MCDA-C provides a modelthat allows local and globalcardinal evaluation, facilitatingthe sorting of the projects withgreater strategic impact

2.2.4 Monitoring the portfoliomanagement

The projects must be continuouslymeasured by its strategiccontribution, and not only in itsapproval

The models constructed by theMCDA-C may be used in line withthe two approaches described byCooper et al. (2000) to monitor theportfolio management

Table II.Contribution of MCDA-Cin portfolio management


framework

663

As a result, the decision makers may apply the model to guide the search for actions inorder to select and prioritize projects put forward by managers. Before the model, thedecisions were ad hoc, resulting in many negotiations choosing which project to beginwith in order.

A limitation of the present research is its tightly centered focus on the technicalconcerns of product development, which is the decision maker’s main concern. So, themodel could be expanded by the unpacking of the core problem in order to observeother company areas, such as services, sales and marketing.

The MCDA-C methodology has demonstrated its usefulness in the process ofsupporting portfolio management, and it is recommended that it be applied in othercontexts, so as to test its generality in similar situations, such as measuring acontribution of a strategic initiative in operations management, requirementsmanagement and risk management.

However, as models generated in each situation are specific to the context, the modelwill need to be adapted, even in contexts that are very similar.

Decision aiding paradigms/portfoliomanagement theory Case study notes

P1 ¼ Uniqueness, identity The model was built with and for the decision-maker at agiven moment and, thus, can be considered unique

P2 ¼ Limited knowledge The decision maker realized at the start that the context ofsorting and selecting projects in a portfolio was complexand ill-structured. This is evidenced by the lack of clearcriteria for the selection and sort process before thedevelopment of the model

P3 ¼ Social entity The managers, to take their decisions, together with theirpreferences and values, may take into account the views ofother stakeholders, such as superiors, peers andsubordinates

P4 ¼ Recursive participatory learning The decision maker’s participation was required in order toexpand his understanding of the context and to assure thathis values and preferences was incorporate in the model

P5 ¼ Principles of measurement The scales used to measure the performance of theproperties being assessed were, in a first stage, ordinal(descriptor). In a second step, in order to integrate the scales,they were transformed into cardinal scales. All inaccordance with measurement’s principles theory

P6 ¼ Legitimacy and validation In order to ensure that the model reflects decision makers’values and preferences he was requested, during all theprocesses, to inform if the model reflected his main concerns.This secures the legitimacy of the model. In order to ensurethat all that is being done is in accordance with scientifictheories, the process must be approved by a scientificsociety in the subject. This stage is known as the validationof the process

Table III.The study case findingsabout the parallelbetween the portfoliomanagement theory anddecision aidingparadigms

MD49,4

664

References

Andrews, K.R. (1980), The Concept of Corporate Strategy, rev. ed., Irwin, Homewood, IL.

Atkinson, R. (1999), “Project management: cost, time and quality, two best guesses and aphenomenon, its time to accept other success criteria”, International Journal of ProjectManagement, Vol. 17 No. 6, pp. 337-42.

Atkinson, R., Crawford, L. and Ward, S. (2006), “Fundamental uncertainties in projects and thescope of project management”, International Journal of Project Management, Vol. 24 No. 8,pp. 687-98.

Baccarini, D. (1999), “The logical framework method for defining project success”, ProjectManagement Journal, Vol. 30 No. 4, p. 25.

Bana e Costa, B.E., Ensslin, L., Correa, E.C. and Vansnick, J.-C. (1999), “Decision support systemsin action: integrated application in a multicriteria decision aid process”, European Journalof Operational Research, Vol. 113 No. 2, pp. 315-35.

Barzilai, J. (2001), “On the foundations of measurement”, Proceedings of the 2001 IEEEInternational Conference on Systems, Man, and Cybernetics, Vol. 1 The Hague, 10-13October.

Bititci, U.S., Suwignjo, P. and Carrie, A.S. (2001), “Strategy management through quantitativemodelling of performance measurement systems”, International Journal of ProductionEconomics, Vol. 69 No. 1, pp. 15-22.

Bourne, M., Mills, J., Wilcox, M., Neely, A. and Platts, K. (2000), “Designing, implementing andupdating performance measurement systems”, International Journal of Operations &Production Management, Vol. 20 No. 7, pp. 754-71.

Canez, L. and Garfias, M. (2006), “Portfolio management at the Mexican petroleum institute”,Research Technology Management, Vol. 49 No. 4, p. 46.

Chen, C.C. (2008), “An objective-oriented and product-line-based manufacturing performancemeasurement”, International Journal of Production Economics, Vol. 112 No. 1, pp. 380-90.

Chien, C.-F. (2002), “A portfolio-evaluation framework for selecting R&D projects”, R&DManagement, Vol. 32 No. 4, pp. 359-68.

Coombs, R., Mcmeekin, A. and Pybus, R. (1998), “Toward the development of benchmarkingtools for R&D project management”, R&D Management, Vol. 28 No. 3, pp. 175-86.

Cooper, R.G. (2007), “Managing technology development projects”, Engineering ManagementReview, Vol. 35 No. 1, pp. 66-7.

Cooper, R.G., Edgett, S.J. and Kleinschmidt, E.J. (2000), “New problems, new solutions: makingportfolio management more effective”, Research Technology Management, Vol. 43 No. 2,pp. 18-33.

De Meyer, A., Loch, C.H. and Pich, M.T. (2002), “From variation to chaos”, MIT SloanManagement Review, Winter.

Dvir, D. and Lechler, T. (2004), “Plans are nothing, changing plans is everything: the impact ofchanges on project success”, Research Policy, Vol. 33 No. 1, pp. 1-15.

Dvir, D., Lipovetsky, S., Shenhar, A. and Tishler, A. (1998), “In search of project classification: anon-universal approach to project success factors”, Research Policy, Vol. 27 No. 9,pp. 915-35.

Eden, C., Jones, S. and Simms, D. (1985), “Messing about in problems”, R&D Management, Vol. 15No. 3, pp. 254-5.


framework

665

Elonen, S. and Artto, K.A. (2003), “Problems in managing internal development projects inmulti-project environments”, International Journal of Project Management, Vol. 21 No. 6,pp. 395-402.

Englund, R.L. and Graham, R.J. (1999), “From experience: linking projects to strategy”, Journal ofProduct Innovation Management, Vol. 16 No. 1, pp. 52-64.

Engwall, M. (2003), “No project is an island: linking projects to history and context”, ResearchPolicy, Vol. 32 No. 5, pp. 789-808.

Engwall, M. and Jerbrant, A. (2003), “The resource allocation syndrome: the prime challenge ofmulti-project management?”, International Journal of Project Management, Vol. 21 No. 6.

Ensslin, L., Dutra, A. and Ensslin, S.R. (2000), “MCDA: a constructivist approach to themanagement of human resources at a governmental agency”, International Transactionsin Operational Research, Vol. 7 No. 1, pp. 79-100.

Ensslin, L., Giffhorn, E., Ensslin, S.R., Petri, S.M. and Vianna, W.B. (2010), “Avaliacao doDesempenho de Empresas Terceirizadas com o Uso da Metodologia Multicriterio de Apoioa Decisao- Construtivista”, Revista Pesquisa Operacional, Vol. 30 No. 1, pp. 125-52.

Fricke, S.E. and Shenhar, A.J. (2000), “Managing multiple engineering projects in amanufacturing support environment”, IEEE Transactions on Engineering Management,Vol. 47 No. 2, pp. 258-68.

Gann, D.M. and Salter, A.J. (2000), “Innovation in project-based, service-enhanced firms: theconstruction of complex products and systems”, Research Policy, Vol. 29 Nos 7-8,pp. 955-72.

Harrison, E.F. and Pelletier, M.A. (2000), “Levels of strategic decision success”, ManagementDecision, Vol. 38 No. 2, pp. 107-18.

Jugdev, K. and Muller, R. (2006), “A retrospective look at our evolving understanding of projectsuccess”, Project Management Journal, Vol. 36 No. 4, pp. 19-31.

Keeney, R.L. (1992), Value-Focused Thinking: A Path to Creative Decision Making, HarvardUniversity Press, Cambridge, MA.

Keeney, R.L. and Raiffa, H. (1976), Decisions with Multiple Objectives: Preferences and ValueTradeoffs, Wesley, New York, NY, pp. 224-41.

Kennerley, M. and Neely, A. (2003), “Measuring performance in a changing businessenvironment”, International Journal of Operations & Production Management, Vol. 23No. 2, pp. 213-29.

Lacerda, R.T.O. (2009), “O sucesso em gerenciamento de projetos: a estruturacao de um modelode avaliacao a partir de uma visao construtivista”, Dissertacao de mestrado emEngenharia de Producao da Universidade Federal de Santa Catarina, Florianopolis.

Landry, M. (1995), “A note on the concept of ‘problem’”, Organization Studies, Vol. 16 No. 2,p. 315.

Lewis, M.W., Welsh, M.A., Dehler, G.E. and Green, S.G. (2002), “Product development tensions:exploring contrasting styles of project management”, Academy of Management Journal,Vol. 45 No. 3, pp. 546-64.

Liesio, J., Mild, P. and Salo, A. (2007), “Preference programming for robust portfolio modelingand project selection”, European Journal of Operational Research, Vol. 181 No. 3,pp. 1488-505.

Lipovetsky, S., Tishler, A., Dvir, D. and Shenhar, A. (1997), “The relative importance of projectsuccess dimensions”, R&D Management, Vol. 27 No. 2, pp. 97-106.

MD49,4

666

Minarro-Viseras, E., Baines, T. and Sweeney, M. (2005), “Key success factors when implementingstrategic manufacturing initiatives”, International Journal of Operations & ProductionManagement, Vol. 25 No. 2, p. 151.

Mintzberg, H. (1994), The Rise and Fall of Strategic Planning: Reconceiving Roles for Planning,Plans, Planners, Free Press, New York, NY.

Pich, M.T., Loch, C.H. and Meyer, A.D. (2002), “On uncertainty, ambiguity, and complexity inproject management”, Management Science, Vol. 48 No. 8, p. 1008.

Pollack, J. (2007), “The changing paradigms of project management”, International Journal ofProject Management, Vol. 25 No. 3, pp. 266-74.

PriceWaterhouseCoopers (2004), “Boosting business performance through programme andproject management”, Connected Thinking, PriceWaterhouseCoopers, Leeds.

Roberts, F.S. (1979), Measurement Theory, Addison-Wesley, Reading, MA.

Roy, B. (1993), “Decision science or decision-aid science?”, European Journal of OperationalResearch, Vol. 66 No. 2, pp. 184-203.

Roy, B. (1994), “On operational research and decision aid”, European Journal of OperationalResearch, Vol. 73 No. 1, pp. 23-6.

Roy, B. (1996), Multicriteria Methodology for Decision Aiding, Kluwer Academic, Dordrecht.

Salomo, S., Weise, J. and Gemunden, H.G. (2007), “NPD planning activities and innovationperformance: the mediating role of process management and the moderating effect ofproduct innovativeness”, Journal of Product Innovation Management, Vol. 24 No. 4,pp. 285-302.

Shenhar, A.J. (2001), “One size does not fit all projects: exploring classical contingency domains”,Management Science, Vol. 47 No. 3, pp. 394-414.

Shenhar, A.J. (2004), “Strategic project leadership: toward a strategic approach to projectmanagement”, R&D Management, Vol. 34 No. 5, pp. 569-78.

Shenhar, A.J. and Dvir, D. (2007), “Project management research-the challenge and opportunity”,Project Management Journal, Vol. 38 No. 2, p. 93.

Shenhar, A.J., Tishler, A., Dvir, D., Lipovetsky, S. and Lechler, T. (2002), “Refining the search forproject success factors: a multivariate, typological approach”, R&D Management, Vol. 32No. 2, pp. 111-26.

Sommer, S.C. and Loch, C.H. (2004), “Selectionism and learning in projects with complexity andunforeseeable uncertainty”, Management Science, Vol. 50 No. 10, p. 1334.

Tukel, O.I. and Rom, W.O. (2001), “An empirical investigation of project evaluation criteria”,International Journal of Operations & Production Management, Vol. 21 No. 3, p. 400.

Tukel, O.I., Rom, W.O. and Eksioglu, S.D. (2006), “An investigation of buffer sizing techniques incritical chain scheduling”, European Journal of Operational Research, Vol. 172 No. 2,pp. 401-16.

Williams, T. (2005), “Assessing and moving on from the dominant project managementdiscourse in the light of project overruns”, IEEE Transactions on EngineeringManagement, Vol. 52 No. 4, pp. 497-508.

Winter, M. and Checkland, P. (2003), “Soft systems – a fresh perspective for projectmanagement”, Proceedings of the Institution of Civil Engineers: Civil Engineering, Vol. 156No. 4, pp. 187-92.

Yahaya, S.Y. and Abu-Bakar, N. (2007), “New product development management issues anddecision-making approaches”, Management Decision, Vol. 45 No. 7, pp. 1123-42.


framework

667

About the authorsRogerio Tadeu de Oliveira Lacerda is currently a Doctoral Student in Production Engineering.He has a Master’s degree in Production Engineering from the Federal University of SantaCatarina (2009). He graduated in Business Administration and has a post-graduate qualificationin Information Engineering. Lacerda has spent many years helping large companies in ProjectManagement, Business Process Management and IT Management. His doctoral research focuseson continuous improvement, portfolio management and performance evaluation. He is a ProjectManagement professional and has contributed to the OPM3 standard of the Project ManagementInstitute. Rogerio Tadeu de Oliveira Lacerda is the corresponding author and can be contactedat: [email protected]

Leonardo Ensslin has a post-doctoral position in Multicriteria Decision Aiding at LancasterUniversity (2000) and has a PhD in Industrial Systems from the University of SouthernCalifornia (1974). Leonardo Ensslin is Professor and Coordinator of Organizational Intelligence inthe Department of Systems Engineering and Production at the Federal University of SantaCatarina (UFSC) on undergraduate and post-graduate courses. His degree in mechanicalengineering was awarded by UFRGS, and he has a Master’s in Production Engineering fromUFSC. He is a Consultant and Lecturer in analysis and performance evaluation, organizationalimprovement systems, and innovation and decision aiding processes.

Sandra Rolim Ensslin has a degree in Accounting Science, gained from the CatholicUniversity of Pelotas, a Master’s in Production Engineering from the Federal University of SantaCatarina (UFSC) and Doctor in Production Engineering by UFSC (2003). She is ProgramCoordinator of Graduate Studies in Accounting and Assistant Professor at the UFSC. She hasexperience in Accounting and Production Engineering, working in the following topics:methodology of multicriteria decision aiding constructivist, organizational performanceevaluation, intellectual capital, intangible assets and accounting research.

MD49,4

668

To purchase reprints of this article please e-mail: [email protected] visit our web site for further details: www.emeraldinsight.com/reprints

Management Decision - hgtc.irhgtc.ir/darabkola/Pakoassel/930523.2/4- A performance measurement... · Management Decision Emerald Article: A performance measurement framework in portfolio

Documents