Cloud services evaluation framework

Cloud Services Evaluation FrameworkMiguel Reixa

ISCTE-IUL

Lisboa, Portugal

[email protected]

Carlos Costa

Adetti-IUL/ ISCTE-IULLisboa, Portugal

[email protected]

Manuela Aparicio

Adetti-IUL/ ISCTE-IULLisboa, Portugal

[email protected]

ABSTRACTThis paper intends to propose an easy and quick methodology to compare and choose the cloud service that best fit the small size organization needs. Approach strategies and methodologies were studied and chosen the most relevant to meet objective. Cloud computing characteristics, definitions, organization types, classifications and important taxonomy were chosen in the literature to give basic understanding it. In this work is presented one practical application to illustrate how to use it. Using this methodology, small size organizations can optimise their resources and at the same time be confidante to select the best cloud service option.

Categories and Subject DescriptorsD. Software, D.4 OPERATING SYSTEMS, D.4.8 Performance; D. Software, D.4 OPERATING SYSTEMS, D.4.7 Organization and Design, Subjects: Distributed systems

General TermsManagement, Measurement, Performance, Reliability, Standardization, Legal Aspects, Verification.

KeywordsCloud Computing; Cloud Adoption; Decision Support Systems; Cloud ROI; Cloud Taxonomy; Software-as-a-Service; Multi-attribute Decision Making.

1. INTRODUCTIONSome organization managers hear cloud computing market saying that changing their IT systems to cloud computing is a good opportunity and strategy to get less costly IT services ([2], [9], [30]), others may only heard this words in their daily business about one competitor or client suggestion but have no clue what that really mean. Nevertheless, when they decide to move to cloud computing they must be aware of what impacts, risks and benefits those decisions may have in their organization.

Following step is to decide which service is best to that organization.

This paper intends to present one proposal of a simple methodology to choose the cloud computing service that best fit to one organization.

It will start with one brief perspective and definition, types and characteristics of cloud computing. This will give a base to understand the general concept and organization of cloud computing and will open the way to presented the proposal and with one application example we will show how to use it.

In the end of this paper is shown the conclusion, brief comments about the feedbacks and the next steps of this work.

2. UNDERSTANDING WHAT IS CLOUD COMPUTING

There are already several authors describing the organization and defining the types, classifications, characteristics, important taxonomy and definitions for cloud computing ([1], [3], [4] [6], [12], [13], [15], [16], [17], [21], [25], [42], [43], [44], [45], [52]). In this section will be referred the definitions, types, classifications and characteristics that are most commonly referred in literature and easier to understand.

2.1 Cloud computing perspectives and definitions

Cloud computing can be seen from two sides, the supplier side and the user side.

In the point of view of suppliers side, cloud computing is one way to rent there free computer resources. With that strategy they can optimize and make profitable there resources.

In the user or consumer side, he sees the cloud computing as a mean to get almost infinite, immediate and reliable computer resources at very low cost.

There are many definitions to cloud computing and the one most cited is from National Institute of Standards and Technology ([10], [33]):

“Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model promotes availability.”

Other explanation of cloud computing can be that cloud computing is a platform or infrastructure that enables execution of code (services, applications etc.), with reliability according to pre-

61

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.OSDOC'12, June 11, 2012, Lisbon, Portugal. Copyright 2012 ACM 978-1-4503-1284-4/12/0006..$10.00.

defined quality parameters that is automatically ensured and where the resources are put to use according to each moment requirements observing overarching requirement definitions including not only both up and downward scalability of resources and data, but also load-balancing of data throughput [50].

2.2 Types and classification of cloudsThere are many cloud computing classifications in the literature. The most common and widely accepted classifications are ([5], [46]):

By deployment, cloud computing can be classified as private, public, mixed or hybrid, community, federated and special purpose. Private are the ones where only the owners of that cloud computing can use it. Public are the ones where the owner allows that it can be accessed by everyone outside of his organization that requests him to do it. The mixed one is the mix of both, which means that only part of that cloud can be accessed to some internal or external users. Community clouds are the ones that are based on public or private ones yet they are used as one sub-cloud for local users that have the same requirements or needs. Federated clouds are owned and shared by more than one owner that use it to each one needs. The special purpose clouds are clouds that are extended with additional capabilities like distributed documentation management or collaboration tools.

By services it can be classified as ([20], [22], [47]):

Infrastructure as a Service (IaaS), is a large set of low-level computing resources that are split through virtualization and can be dynamically managed (assigned and resized) according the user needs.

Platform as a Service (PaaS), provide one additional level of abstraction, the user will have one software level that will be used to run, develop and manage the service according his needs.

Software as a Service (SaaS), provide final software applications that can be directly used like the ones that are locally installed.

2.3 Cloud characteristicsThe European Commission “Expert Group Report” [50] makes a summary of most common cloud computer characteristics divided in three groups:

- Non-functional aspects – “represent qualities or properties of a system, rather than specific technological requirements”.

- Economic considerations – “are one of the key reasons to introduce cloud systems in a business environment in the first instance”.

- Technological aspects – come from the opposed of non-functional and economical aspects and “typically imply a specific realization”.

Table 1. Cloud computing non-functional characteristics

Cloud characteristic Description

Elasticity The capability of the underlying infrastructure to adapt to changing. E.g.: The capability to adapt to the

Cloud characteristic Description

amount and size of data supported by an application, number of concurrent users, among others.

Reliability The capability to ensure constant operation of the system without disruption. E.g.: The capability to ensure no loss of data, no code reset during execution, etc.

Quality of Service The capability where specific services and/or resources requirements have to be met.

E.g.: Basic QoS metrics must be guaranteed at least, like: response time, throughput, reliability, etc.

Agility and adaptability The capability of automatically and on-real-time react and adapt the changing environments and conditions to resources or there management. E.g.: The capability of automatically and on-real-time manage the amount of different requests, size of resources, types of resources, different quality or routes, etc.

Availability The ability to introduce redundancy for services and data so failures can be masked transparently. E.g.: The ability to have fault tolerance particularly achieved through replication of data/services and distributing them across different resources to achieve load-balancing.

Table 2. Cloud computing economic characteristics

Cloud characteristic Description

Cost reduction Reduce cost for infrastructure maintenance and acquisition. E.g.: Quickly adapt to changing consumer behaviour with reduced cost for infrastructure, maintenance and acquisition.

Pay per use The capability to build up cost according to the actual consumption of resources. E.g.: Payment by unit of traffic, unit of data stored in one amount of time or amount of data processed in one

62

Cloud characteristic Description

unit of time.

Improved time to market The capability to provide infrastructures, potentially dedicated to specific use cases that take over essential capabilities to support easy provisioning and thus reduce users time to market services. E.g.: Providing easy and case adapted solutions to implement user services.

Return of investment (ROI)

Outsourcing resources versus increasing the local infrastructure and employing (private) cloud technologies need to be outweighed and critical cut-off points identified in order to choose the most profitable one if exists. E.g.: Make ROI studies.

Turning CAPEX (capital expenditure) into OPEX (operational expenditure)

CAPEX is required to build up a local infrastructure, but with outsourcing computational resources to cloud systems on demand and scalable, a company will actually spend OPEX for provisioning of its capabilities, as it will acquire and use the resources according to operational need. E.g.: The actual cost benefit is not always clear, is required to study which one is more beneficial, if it exists.

“Going Green” Capacity to adapt resources to requirements with adjustment of energy consumption and carbon footprint to what is exactly required. E.g.: Significantly reduction of energy consumption and carbon footprint due to better controlled and automatically adjusted up and downscaling of resources.

Table 3. Cloud computing technological characteristics

Cloud characteristic Description

Virtualization Technological abstraction layer that hide complex technical systems from the user and gives additional flexibility. It allows: Ease of use: hiding infrastructure, making easy to develop new

Cloud characteristic Description

applications and reducing system controlling actions. Infrastructure independence: higher interoperability by making the code platform independent. Flexibility and adaptability: with virtualization the underlying infrastructure can change easier according to different conditions and requirements. Location independence: Access to services is independent from user or the resource location.

Multi-tenancy Property in which the same resource, in unknown location, may be assign to several users, possibly at same time and available in multiple isolated instances. E.g.: Databases which are concurrently altered but maintained in isolated tenants.

Security, privacy and compliance

Security is a process or set of steps that helps keep data from being seen or accessed by unauthorised person or system. Many times privacy is seen as one aspect of security. Compliance is the act of being in conformance or in accordance to laws or rules.

Data Management Property that assures that all processed data is flexibly distributed across multiple resources. It assures consistency, awareness of data location,, latencies, work load, horizontal and vertical scalability and other aspects. E.g.: As data may constantly vary system must automatically adjust the required space in the resources to needs and assure that it is consistent and isolated.

APIs and / or Programming Enhancements

Are tools offered to exploit the cloud features. E.g.: Environment features that are provided in a fashion that allows the user to leave automatic management to the system.

Metering Any kind of measurement that permits to offer elastic prices, charging and billing to elastic

63

Cloud characteristic Description

resources or services.

Tools Programs or add-ons that can be used to support development and adaptation to usage.

This is not one complete and extensive set of characteristics but show the most common referred ones found in the literature.

3. METHOD DEVELOPMENTSeveral methodologies were investigated ([7], [8], [24], [26], [28], [34], [36], [37], [38], [39], [40], [48], [53], [54]) to select the ones that potentially give us reliable, quick and easy evaluation process.

3.1 Evaluation approachThe activities of MCDM (Multi-Criteria Decision Making) should include ([18], [51]):

- Select and agree on one set of attributes that the chosen product should meet.

- Make one evaluation resulting in one value representing the importance of that attribute to achieve the final requirements and objectives.

- Make another evaluation resulting in one value representing how much that product attribute fit in the user requirements and objectives.

- Rank the products based on how well each product attributes fit the user needs.

These four steps added with the selection of products candidates summarise our approach.

3.2 Attributes to choose cloud servicesMADMAC (Multiple Attribute Decision Methodology for Adoption of Clouds) [49] and SFA (Sales Force Automation) (SFA) factors [19] analysis already suggests a set of attributes that should be considered to choose cloud services. To that suggestion we added some others to make a more complete set that can be considered the start set of attributes. To this list, can be added other attributes that users consider necessary for their particular case. This step should be carefully examined to see if the new attributes are not yet partial or totally covered by the existing ones. Overlapping attributes lead to evaluations that can be time consuming and give errors in the end rating results. In table 4 is suggested the attributes to use in the evaluation. This is based in MADMAC, SFA factors and complemented with items from the previous tables like “Turning CAPEX into OPEX”, “Going Green” and “Quality of Service”.

Table 4. Suggested comparison attributes

1. Suitability 1.1. Functionality 1.2. Legacy Compatibility.1.3. Network Quality

1.3.1. Latency 1.3.2. Bandwidth

2. Economic Value 2.1. Capital Cost 2.2. Operating Cost 2.3. ROI2.4. ROEI2.5. Turning CAPEX into OPEX2.6. “Going Green”

3. Control 3.1. Integrability 3.2. Manageability

3.2.1. Monitoring Ease 3.2.2. Autonomy3.2.3. Adaptability (API)3.2.4. Service Help Desk

4. Usability (Consumability) 4.1. Application Launch Time 4.2. Graphics Agility 4.3. Simplicity

5. Reliability 5.1. Elasticity 5.2. High Availability 5.3. Disaster Recovery 5.4. Compliance 5.5. Trust5.6. Quality of Service

6. Security 6.1. Confidentiality 6.2. Integrity 6.3. Availability 6.4. Auditability 6.5. Multi-tenant Trust

The inclusion of “Turning CAPEX into OPEX” and “Quality of Service” is easy to understand, the first is related with one evaluation of one economic advantage that is not evaluated in the other attributes; the second, is measuring the guaranties given in contract by service suppliers that the service offered will met minimum requirements. The inclusion of “Going Green” attribute result from the “carbon footprint” and environmental awareness pressure in society. Clients from organisations and governments are taking in attention the initiatives in this area, so having one process that take in consideration this issue is one good and important strategic decision and one additional decision factor that can make the difference in case of very similar services to choose.

To avoid overlap with the “Operating Cost” (regarding energy consumption) and “Compliance” (regarding environmental legislation) attributes it should be clarified. In this case, the attribute intent to evaluate the processes, strategies, efforts and

64

commitments from cloud service suppliers to provide services that assure the smaller environmental impact.

3.3 Attributes evaluationThe evaluation is done by experts that know very well all cloud services solutions in the market and by stakeholders.

Stakeholders [35] are persons knowing the organisation needs and goals that are directly or indirectly leading or involved in this process and that are inside operational and strategic goals, but do not need to be inside the organisation.

The experts should decide, according their experience and the result of stakeholders goals [55], the most appropriated set of products to evaluate. This is only one previous selection that should only focus on getting all services that may achieve the major stakeholder goals. E.g.: If is required one Office service it should be only selected services with those functionality, even if it have only one part of the required functionality it should be included. It is nonsense and one waste of time and resources to include other services that do not have the minimum required Office functional characteristics. The number of experts needed [23] is based in the importance of accuracy and independence of results versus the costs and resources required to make the evaluation.

All involved persons must have experienced all product proposals. Most of cloud services let users to experience it freely prior to adopt it. Ideally, some research can be also done to provide some additional information; this work can be done by specialists taking in mind to get the information that are most independent and scientifically based.

To each involved person should be done one inquiry matrix in order to evaluate all the attributes.

The evaluations are done according Likert scale ([14], [27], [32]) using five decision options [31]:

1- Not important.

2 - Little importance.

3 - Average importance.

4 - Big importance.

5 - Extremely important.

The experts must evaluate the products in order to answer the question: What importance this attribute have, in respect to the service in evaluation, to meet technical implementation that accomplish stakeholder goals?

In the other hand, stakeholder must evaluate the importance and the relative weight of each attribute. They must answer the question: What importance this attribute have to meet my operational and strategic goals?

The first evaluation is seen from the implementation technical side and the second from the operational and strategic side.

3.4 Product rankingThe evaluations are joined in two: one show the average results of experts evaluation and the other the average results from stakeholders evaluation.

To get the final service ranking is done one final matrix were the attributes final evaluation result for one service j is given by:

A j=∑ S i⋅E ij

A j - Final score for service j

E ij - Experts average evaluation to attribute i of service j

S i - Stakeholders average evaluation to attribute i

After comparing all A j values the biggest one corresponds to service that best fit in the organisation goals.

4. FRAMEWORK PRACTICAL APPLICATION

To validate and get feedback about this approach was done one practical case that is here presented.

4.1 Organisation descriptionThis practical case was done in one small multimedia company with 6 people. There business goes from the preparation of movies to be ready to production in DVD writing plants by making all the intros and extras, films format conversion, making web sites, and computer aided design to make marketing spots and virtual environments to be used in events.

They are using proprietary Office software which has associated costs in licences and hardware, to permit good collaboration and interaction between users and between all software modules. Note that the Office software is used to support the business but is not used as production software to this business.

Their main goal in going to cloud computing is to reduce licence costs while improving collaboration with their suppliers and customers without scalability and compatibility problems, promoting at same time the mobility.

4.2 Involved peopleIn this practical application were involved 12 specialists and one stakeholder. All the specialists work in the IT area.

The stakeholder is the company owner. He is also responsible for all the IT area inside the company.

4.3 Choosing the cloud servicesBased on that company requests were selected by the experts tree Office cloud services: Zoho Docs [29], Google Docs [11] and Microsoft Office 365 [41].

4.4 Services evaluation and rankingThe evaluation results to this case are shown in table 5. Only the lower attribute level is considered because the lower ones are components from the higher ones.Table 5. Office services practical evaluation and ranking

65

Attribute S i

1 » Zoho Docs

2 » Google Docs

3 » Microsoft Office 365

E i1 E i2 E i3

1.1. 4 2,8 4,3 3,5

1.2. 4 3,3 4,5 3,8

1.3.1. 4 3,2 4,4 3,6

1.3.2. 3 4,0 4,3 4,1

2.1. 2 3,3 3,1 3,2

2.2. 4 3,3 3,1 2,8

2.3. 3 4,0 3,8 3,3

2.4. 3 4,0 3,8 3,7

2.5. 2 3,5 3,4 3,3

2.6. 3 3,6 3,6 3,4

3.1. 4 3,3 4,3 3,8

3.2.1. 4 2,8 4,2 3,8

3.2.2. 4 3,1 4,4 3,8

3.2.3. 4 3,0 3,7 3,7

3.2.4 3 3,0 4,1 3,8

4.1. 4 3,0 4,0 3,7

4.2. 5 3,0 4,4 4,0

4.3. 5 3,2 4,8 4,0

5.1. 4 3,0 3,8 3,4

5.2. 5 3,3 4,6 4,1

5.3. 5 3,0 4,1 3,9

5.4. 5 3,0 3,6 3,6

5.5. 4 2,8 4,7 3,8

5.6. 5 2,9 4,3 3,9

6.1. 5 3,2 4,4 3,9

6.2. 5 3,1 4,4 4,1

6.3. 5 3,3 4,4 3,1

6.4. 3 2,6 3,6 3,0

6.5. 4 2,7 3,5 3,4

Attribute S i

1 » Zoho Docs

2 » Google Docs

3 » Microsoft Office 365

E i1 E i2 E i3

A j -- 10614,5 13524 12132,5

S i - Stakeholders average evaluation to attribute i ; E i j - Experts average evaluation to attribute i of service j ; A j - Final score for service j

In this practical application Google Docs have the higher evaluation score. It should be this service to be implemented.

5. CONCLUSIONS AND FUTURE WORK

At the actual point of this work we already tested that the application of this framework is feasible and easy to use. The results need to be verified and validated. The validation, verification and corrections will be done after the implementation of proposed service in the company.

The main goal of getting the evaluation process quicker and easier was achieved implementing two main changes in usual literature framework proposals:

- The use one already defined set of attributes that can be adjusted to particular needs but avoid a lot of work.

- The use of Likert scale with 5 evaluation degrees that simplify and speed up people evaluation process.

Feedback from all involved persons was positive stating that the process was quick, easy and have one additional positive effect: the knowledge acquired about the services characteristics will permit better use and optimisations.

The main difficulties found in this application process stated in the feedback are:

- the need that all involved people, speciality the stakeholders, understand at least the basic characteristics and implications to their organisation in order to make one informed and conscientious evaluation.

- the need of good knowledge about all the services.

- the correct clarification of all attributes.

Additional improvements may be implemented in the future after making the validation, verification and receiving critics and suggestions to this work.

6. REFERENCES[1] Anderson, J. M. (2003). Why we need a new definition of

information security. Computers & Security, 22(4), 308–313. doi:10.1016/S0167-4048(03)00407-3

[2] Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz, R., Konwinski, A., Lee, G., et al. (2010). A view of cloud

66

computing. Commun. ACM, 53(4), 50–58. doi:10.1145/1721654.1721672

[3] Behrend, T. S., Wiebe, E. N., London, J. E., & Johnson, E. C. (2010). Cloud computing adoption and usage in community colleges. Behaviour & Information Technology, 30(2), 231–240. doi:10.1080/0144929X.2010.489118

[4] Brindley, C. (2011). Authentication and Trust: Turning the Cloud inside out. Em N. Pohlmann, H. Reimer, & W. Schneider (Eds), ISSE 2010 Securing Electronic Business Processes (pp 72–79). Wiesbaden: Vieweg+Teubner. Available at http://www.springerlink.com/content/x633345273p13236/

[5] Calheiros, R. N., Ranjan, R., Beloglazov, A., De Rose, C. A. F., & Buyya, R. (2011). CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software: Practice and Experience, 41(1), 23–50. doi:10.1002/spe.995

[6] Chang, W. Y., Abu-Amara, H., & Sanford, J. F. (2011). Building and Configuring Enterprise Cloud Services3,2. Transforming Enterprise Cloud Services (pp 273–308). Dordrecht: Springer Netherlands. Available at http://www.springerlink.com/content/t7p4x10578225802/

[7] Chung, L., & Cooper, K. (2002). A knowledge-based COTS-aware requirements engineering approach, 175–182. doi:10.1145/568760.568791

[8] Chung, L., & Cooper, K. (2004). Matching, ranking, and selecting components: a COTS-aware requirements engineering and software architecting approach. International Workshop on Models and Processes for the Evaluation of COTS Components MPEC 2004 W7S Workshop 26th International Conference on Software Engineering, 41–44. doi:10.1049/ic:20040432

[9] Das, R. K., Patnaik, S., & Misro, A. K. (2011). Adoption of Cloud Computing in e-Governance. Em N. Meghanathan, B. K. Kaushik, & D. Nagamalai (Eds), Advanced Computing (Vol. 133, pp 161–172). Berlin, Heidelberg: Springer Berlin Heidelberg. Available at http://www.springerlink.com/content/n5783250w212v435/

[10] Dillon, T., Wu, C., & Chang, E. (2010). Cloud Computing: Issues and Challenges. 2010 24th IEEE International Conference on Advanced Information Networking and Applications (AINA) (pp 27–33). Presented in 2010 24th IEEE International Conference on Advanced Information Networking and Applications (AINA), IEEE. doi:10.1109/AINA.2010.187

[11] Docs – Google Apps. Available at http://www.google.com/apps/intl/en/business/docs.html#utm_campaign=en&utm_source=docs_en_US&utm_medium=et&utm_term=login_promo, last visited March 2012.

[12] Dudin, E., & Smetanin, Y. (2011). A review of cloud computing. Scientific and Technical Information Processing, 38(4), 280–284. doi:10.3103/S0147688211040083

[13] Durkee, D. (2010). Why cloud computing will never be free. Commun. ACM, 53(5), 62–69. doi:10.1145/1735223.1735242

[14] Edwards, A. L., & Kenney, K. C. (1946). A comparison of the Thurstone and Likert techniques of attitude scale construction. Journal of Applied Psychology, 30(1), 72–83. doi:10.1037/h0062418

[15] Ferrer, A. J., Hernández, F., Zsigri, C., Tordsson, J., & Elmroth, E. OPTIMIS: A holistic approach to cloud service provisioning. ScienceDirect - Future Generation Computer Systems : OPTIMIS: A holistic approach to cloud service provisioning. Available at http://www.sciencedirect.com/science/article/pii/S0167739X1100104X, last visited September 2011.

[16] Fouquet, M., Niedermayer, H., & Carle, G. (2009). Cloud computing for the masses. Proceedings of the 1st ACM workshop on User-provided networking: challenges and opportunities, U-NET ’09 (pp 31–36). New York, NY, USA: ACM. doi:10.1145/1659029.1659038

[17] Furht, B. (2010). Handbook of Cloud Computing. (A. Escalante, Ed) (1st Edition.). Available at http://www.springer.com/computer/communication+networks/book/978-1-4419-6523-3

[18] Godse, M., & Mulik, S. (2009). An Approach for Selecting Software-as-a-Service (SaaS) Product. IEEE International Conference on Cloud Computing, 2009. CLOUD ’09 (pp 155–158). Presented in IEEE International Conference on Cloud Computing, 2009. CLOUD ’09, IEEE. doi:10.1109/CLOUD.2009.74

[19] Godse, Manish, & Mulik, S. (2009). An Approach for Selecting Software-as-a-Service (SaaS) Product (p 4). Presented in 2009 IEEE International Conference on Cloud Computing, IEEE. Available at http://ieeexplore.ieee.org/ielx5/5283544/5283545/05284286.pdf?tp=&arnumber=5284286&isnumber=5283545

[20] Goyal, P. (2010). Enterprise Usability of Cloud Computing Environments: Issues and Challenges. 2010 19th IEEE International Workshop on Enabling Technologies: Infrastructures for Collaborative Enterprises (WETICE) (pp 54–59). Presented in 2010 19th IEEE International Workshop on Enabling Technologies: Infrastructures for Collaborative Enterprises (WETICE), IEEE. doi:10.1109/WETICE.2010.15

[21] Grobauer, B., Walloschek, T., & Stocker, E. (2011). Understanding Cloud Computing Vulnerabilities. IEEE Security & Privacy, 9(2), 50–57. doi:10.1109/MSP.2010.115

[22] Hoefer, C. N., & Karagiannis, G. (2010). Taxonomy of cloud computing services. 2010 IEEE GLOBECOM Workshops (GC Wkshps) (pp 1345–1350). Presented in 2010 IEEE GLOBECOM Workshops (GC Wkshps), IEEE. doi:10.1109/GLOCOMW.2010.5700157

[23] Hwang, W., & Salvendy, G. (2010). Number of people required for usability evaluation: the 10+2 rule. Commun. ACM, 53(5), 130–133. doi:10.1145/1735223.1735255

[24] Jadhav, A. S., & Sonar, R. M. (2008, 30). Evaluating and selecting software packages: A review. ELSEVIER.

[25] Khajeh Hosseini, A., Greenwood, D., Smith, J. W., &‐ Sommerville, I. (2012). The Cloud Adoption Toolkit: supporting cloud adoption decisions in the enterprise.

67

Software: Practice and Experience, 42(4), 447–465. doi:10.1002/spe.1072

[26] Lewis, G. A., & Morris, E. J. (2004). From System Requirements to COTS Evaluation Criteria. Evaluation, 159–168.

[27] Likert Scaling. Available at http://www.socialresearchmethods.net/kb/scallik.php, last visited March 2012.

[28] Lopes, R., & Carlos, C. (2011). A COTS Selection Approach with a Decision Support System (p 25). Presented in CAPSI’2011, ISEG - Instituto Superior de Economia e Gestão.

[29] Make Your Business Do More Online. Available at http://www.zoho.com/, last visited March 2012.

[30] Marston, S., Li, Z., Bandyopadhyay, S., Zhang, J., & Ghalsasi, A. (2011). Cloud computing — The business perspective. Decision Support Systems, 51(1), 176–189. doi:10.1016/j.dss.2010.12.006

[31] Matell, M. S., & Jacoby, J. (1972). Is there an optimal number of alternatives for Likert-scale items? Effects of testing time and scale properties. Journal of Applied Psychology, 56(6), 506–509. doi:10.1037/h0033601

[32] Maurer, T. J., & Pierce, H. R. (1998). A comparison of Likert scale and traditional measures of self-efficacy. Journal of Applied Psychology, 83(2), 324–329. doi:10.1037/0021-9010.83.2.324

[33] Mell, P., & Grance, T. (2011). The NIST Definition of Cloud Computing - Recommendations of the National Institute of Standards and Technology. NIST. Available at http://rszt.pmmk.pte.hu/uploads/8f/23/8f23a309550830fa62395163ecec6fd3/nist_SP800-145.pdf

[34] Misra, S. C., & Mondal, A. (2011). Identification of a company’s suitability for the adoption of cloud computing and modelling its corresponding Return on Investment. Mathematical and Computer Modelling, 53(3–4), 504–521. doi:10.1016/j.mcm.2010.03.037

[35] Mitchell, R. K., Agle, B. R., & Wood, D. J. (1997). Toward a theory of stakeholder identification and salience: Defining the principle of who and what really counts. Academy of management review, 853–886.

[36] Mohamed, A., Ruhe, G., & Eberlein, A. (2007a). MiHOS: an approach to support handling the mismatches between system requirements and COTS products. Requirements Engineering, 12(3), 127–143. doi:10.1007/s00766-007-0041-5

[37] Mohamed, A., Ruhe, G., & Eberlein, A. (2007b). Decision Support for Handling Mismatches between COTS Products and System Requirements. 2007 Sixth International IEEE Conference on CommercialofftheShelf COTSBased Software Systems ICCBSS07, 63–72. doi:10.1109/ICCBSS.2007.13

[38] Mohamed, A., Ruhe, G., & Eberlein, A. (2007c). COTS Selection: Past, Present, and Future (p 10). Presented in 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS’07), IEEE. Available at

http://home.exetel.com.au/jbrady/HTMLobj-183/COTS_Selection.pdf

[39] Mohamed, A. S. A. (2007). Decision support for selecting cots software products based on comprehensive mismatch handling. University of Calgary, Calgary, Alta., Canada, Canada.

[40] Neubauer, T., & Stummer, C. (2007). Interactive Decision Support for Multiobjective COTS Selection. 40th Annual Hawaii International Conference on System Sciences, 2007. HICSS 2007 (p 283b–283b). Presented in 40th Annual Hawaii International Conference on System Sciences, 2007. HICSS 2007, IEEE. doi:10.1109/HICSS.2007.283

[41] Office Online Services - Hosted in the Cloud - Microsoft Office 365. Available at http://www.microsoft.com/en-us/office365/online-software.aspx#fbid=forwkiNhyzI, last visited March 2012.

[42] Otsuka, H., & Lutfiyya, H. (2011). Using strategy trees in change management in clouds. Proceedings of the 7th International Conference on Network and Services Management, CNSM ’11 (pp 133–142). Laxenburg, Austria, Austria: International Federation for Information Processing. Available at http://dl.acm.org/citation.cfm?id=2147671.2147693

[43] Peng, J., Zhang, X., Lei, Z., Zhang, B., Zhang, W., & Li, Q. (2009). Comparison of Several Cloud Computing Platforms. 2009 Second International Symposium on Information Science and Engineering (ISISE) (pp 23–27). Presented in 2009 Second International Symposium on Information Science and Engineering (ISISE), IEEE. doi:10.1109/ISISE.2009.94

[44] Prodan, R., & Ostermann, S. (2009). A survey and taxonomy of infrastructure as a service and web hosting cloud providers. 2009 10th IEEE/ACM International Conference on Grid Computing (pp 17–25). Presented in 2009 10th IEEE/ACM International Conference on Grid Computing, IEEE. doi:10.1109/GRID.2009.5353074

[45] Rastogi, D. A. (2010). A Model based approach to Implement Cloud Computing in E-Governance. International Journal of Computer Applications, 9(7), 15–18. doi:10.5120/1399-1888

[46] Rimal, B. P., & Eunmi Choi. (2009). A Conceptual Approach for Taxonomical Spectrum of Cloud Computing. Proceedings of the 4th International Conference on Ubiquitous Information Technologies & Applications, 2009. ICUT ’09 (pp 1–6). Presented in Proceedings of the 4th International Conference on Ubiquitous Information Technologies & Applications, 2009. ICUT ’09, IEEE. doi:10.1109/ICUT.2009.5405687

[47] Rimal, B. P., Eunmi Choi, & Lumb, I. (2009). A Taxonomy and Survey of Cloud Computing Systems. Fifth International Joint Conference on INC, IMS and IDC, 2009. NCM ’09 (pp 44–51). Presented in Fifth International Joint Conference on INC, IMS and IDC, 2009. NCM ’09, IEEE. doi:10.1109/NCM.2009.218

[48] Ruhe, G. (2003). Web, Web-Services, and Database Systems. Lecture Notes in Computer Science (Vol. 2593, pp 34–45).

68

Springer Berlin / Heidelberg. Available at http://www.springerlink.com/content/a8mgdd9up5bnh5ua/abstract/

[49] Saripalli, P., & Pingali, G. (2011). MADMAC: Multiple Attribute Decision Methodology for Adoption of Clouds. 2011 IEEE International Conference on Cloud Computing (CLOUD) (pp 316–323). Presented in 2011 IEEE International Conference on Cloud Computing (CLOUD), IEEE. doi:10.1109/CLOUD.2011.61

[50] Schubert, L. (2010). The Future Of Cloud Computing, Opportunities for European Cloud Computing Beyond 2010. Em K. Jeffery & B. Neidecker-Lutz (Eds), . Available at http://cordis.europa.eu/fp7/ict/ssai/docs/cloud-report-final.pdf

[51] Triantaphyllou, Shu, B., Sanchez, S. N., & Ray, T. (1998). Multi-Criteria Decision Making : An Operations Research Approach. Electronics, 15, 175–186.

[52] Vaquero, L. M., Rodero-Merino, L., Caceres, J., & Lindner, M. (2008). A break in the clouds: towards a cloud definition. SIGCOMM Comput. Commun. Rev., 39(1), 50–55. doi:10.1145/1496091.1496100

[53] Wanyama, T., & Far, B. H. (2005). Towards providing decision support for COTS selection. Canadian Conference on Electrical and Computer Engineering, 2005 (pp 908–911). Presented in Canadian Conference on Electrical and Computer Engineering, 2005, IEEE. doi:10.1109/CCECE.2005.1557124

[54] Wile, D., Balzer, R., Goldman, N., Tallis, M., Egyed, A., & Hollebeek, T. (2010). Adapting COTS products, 1–9. doi:10.1109/ICSM.2010.5609658

[55] Zardari, S., & Bahsoon, R. (2011). Cloud adoption: a goal-oriented requirements engineering approach. Proceedings of the 2nd International Workshop on Software Engineering for Cloud Computing, SECLOUD ’11 (pp 29–35). New York, NY, USA: ACM. Doi:10.1145/1985500.1985506

69

70