Cloud Evaluation: Benchmarking and Monitoring

CHAPTER 1

CLOUD EVALUATION: BENCHMARKINGAND MONITORING

Massimo Ficco, Massimiliano Rak, Salvatore Venticinque,Luca Tasquier and Giuseppe Aversano

Dipartimento di Ingegneria Industriale e dell’InformazioneSeconda Universita’ degli studi di Napoli, via Roma 29, Aversa, 81031, Italy

Abstract. The Cloud paradigm has largely been adopted in several different con-texts and applied to an incredibly large set of technologies. It is an opportunity forIT users to reduce costs and increase efficiency providing an alternative way of usingIT services. It represents both a technology for using computing infrastructures in amore efficient way and a business model for selling computing resources. It gives tosmall and medium enterprises the possibility of using services and technologies thatwere prerogative of large ones, by paying only for the resources needed and avoid-ing upfront investment. The possibility of dynamically acquire and use resources andservices on the base of a pay-per-use model, implies incredible flexibility in terms ofmanagement, which is otherwise often hard to address. On the other hand, becauseof this flexibility new problems emerge, which involve the evaluation and monitoringof the quality of the acquired resources and services. This chapter presents a surveyon the solutions and technologies proposed by research and open community to facesuch issues.The final publication is available at link: https://books.google.it/books?hl=en &lr=&id=gXZSCAAAQBAJ&oi=fnd&pg=PA175&dq=Cloud+Evaluation:+Benchmarking+and+Monitoring &ots=wAXJZ4JL0O&sig=bknfuPM7srLMEfD-3 hvNtmV5e8#v=onepage

Please enter \offprintinfo{(Title, Edition)}{(Author)}at the beginning of your document.

1

2 CLOUD EVALUATION: BENCHMARKING AND MONITORING

&q=Cloud%20Evaluation%3A%20Benchmarking %20and%20Monitoring&f=falseKeywords. Cloud Computing, Benchmarking, Monitoring, mOSAIC, Cloud Agency.

1.1 Introduction

Cloud Computing is an emerging paradigm that allows customers to easily obtainservices according to a pay-per-use business model. A side effect of such a model isthat customers need to clearly evaluate the performance of acquired Cloud service,in order to estimate whether they are paying the right price for the resources actu-ally used. Such an evaluation can be based only on well-designed benchmarks. Abenchmark is a program that generates a well-known workload on the system undertest, and enables the expert to measure a set of predefined performance indexes. Ingeneral, benchmarking activities aim at performing static performance indexes eval-uations, that fit the flexibility and dynamicity of the Cloud paradigm. Moreover,monitoring performances of services is mandatory because of chargeback, controland provisioning. In the Cloud context the monitoring of service levels becomescritical because of the conflicts of interest that might occur between provider andcustomer in case of outage. Being the Cloud conceived as a market utility, it isnecessary to ensure that vendor and consumer meet the conditions of the achievedagreement, which can have financial or legal consequences. A first issue is what tobe monitored. It depends on the terms of services included in the SLA, but customerscould also be interested in a number of metrics that depends on the kind of applica-tion they will run in the Cloud, such as throughput, reliability, load balancing andelasticity. A second problem to be addressed concerns how and by whom monitor-ing should be performed. Monitoring and tools are available for real and virtualizedresources, but different scenario have to be considered. In private Cloud all is avail-able to users, whereas in a public Cloud, customers can only access to their virtualresources, meanwhile the provider manages hardware and hypervisor.

In this Chapter, the roles of benchmarking and monitoring of service performancein Cloud Computing are discussed. Moreover, a survey on the proposed solutions andtechnologies to face such issues are presented.

1.2 Benchmarking

The main goal of benchmarking is to offer an offline evaluation of Cloud services thathelps in configuring and adapting application before they comes into use and startconsuming cloud resources (virtual machines, memory, storage) and being charged.Benchamrking aims at offering solutions for a clear comparison of different servicesin the same executing conditions.

Due to such considerations benchmarking tools aim at clearly defining and repro-ducing execution conditions for the target system (service, application, resource ) tobe evaluated. The key to success for a benchmarking procedure is the identificationof the goals of the performance analysis. This section aims at stating the different

BENCHMARKING 3

goals that the benchmarking procedure may assume. In order to identify such goals itis important to make some key assumptions: the target of our benchmarks, the actorsinvolved in the benchmarking process.

We have identified a few main benchmarking scenarios that represent possibleconditions under which a benchmarking procedure should take place. We describeeach of these scenarios through three elements: Target User, Main Goal, Target Re-source. The Target User is the user interested in performing the benchmarking. Hecould be a Developer, which aims at creating a new servcies to be offered usingresources acquired on the cloud, or a Final User, i.e., the one that just invokes theservices and use them. The Target Resource is the subject of the benchmarking pro-cedure, i.e., what the Benchmark aims at stressing (the system under test).

The main scenarios we focus on are summarized in Table 1.1. Each scenariois enriched with a brief description of the goal to be achieved by the performanceanalysis.

1.2.1 Benchamrking state of art

This section aims at offering a clear and global view of the state of the art aboutbenchmarking in cloud environment. Its goal is to outline the new challenges incloud environment, the currently available solution and the new proposals. The rolesof benchmarking and of monitoring of services are sometimes misunderstood. Inthe following we will refer to benchmarking activities when we aim at performingstatic performance evaluations of indexes. A benchmark is a program that generatesa well-known workload on the system under test (shortly, sut) and enables the expertto measure a set of predefined performance indexes. At the state of the art, fewsolutions exist that tackle the problem of benchmarking in cloud environments, evenif a lot of different approaches exist in different contexts. This section will firstlypresent benchmark results and products in many different contexts (benchmark inHPC systems, comparison between IaaS providers, existing frameworks, etc.), Then,a brief critical analysis of the solution presented is performed, outlining the openresearch problems.

1.2.1.1 HPC Benchmarking Even if recently cloud providers have started to of-fer HPC-oriented services [3], HPC users started to be interested to cloud paradigmfrom its birth (as shown by the incredible amount of discussions comparing GRIDand Cloud). In HPC systems, performance benchmarking is a well known prac-tice. Many stable benchmarking suites of codes exist, among which NAS ParallelBenchmarks (NPB) [28], LinPack [38] (just to mention the most famous ones). Suchbenchmarks were adopted to compare HPC systems, like the ones in the Top500 list.An interesting survey of HPC-related problems can be found in [26].

The first results in the context of cloud benchmarking can be found in the tentativeof adapting such benchmarks to cloud based clusters, in order to understand the realapplicability of solutions based on virtual machines in this context. Probably themost relevant paper in this field is the one by Walker [73] that tries to outline the


Table 1.1 Summary of Benchmarking Scenarios

Target User Goal Target Resource Goal Description

Developer CompareProviders

Cloud resourcesand/or services

The goal is to enable a devel-oper to choose among ser-vices offered by differentproviders

Developer Evaluate newlydeveloped appli-cations

ApplicationComponents

The goal is to enable a devel-oper to choose among differ-ent components and evalu-ate how each component be-haves

Developer Compare appli-cations configu-rations

Applications The goal is enable the De-veloper to compare two dif-ferent applications in orderto configure them at best

Final User Cost Evaluation Cloud Services The goal is to compare dif-ferent similar services of-fered by Cloud Providers inorder to predict costs of theirusage.

Developer Predictions Applications,components andresources

The goal is to help devel-oper to build up benchmarkswhose goal is to predict thebehavior of a target applica-tion

Final User Compare Appli-cations

Applications The goal is to enable the fi-nal user to know the perfor-mance offered by the targetapplications for comparisonwith other offerings (SLA-related problem)

real performance of cloud based clusters. The paper results are now outdated, butthe approach followed is interesting.

Papers as [41, 51] propose the systematic benchmarking of clusters of virtual ma-chines, obtained from IaaS cloud providers, and offer some interesting considerationabout the performance perceived by users.

BENCHMARKING 5

1.2.1.2 Benchmark Standard Solutions It is a matter of fact that, at the state ofthe art, there exists no standard for cloud benchmarking. Here, we focus on two ofthe most important benchmark consortiums that are addressing cloud-related bench-marking: SPEC [22] and TPC [24].

The Standard Performance Evaluation Corp (SPEC) has developed through theyears some solutions that can be adopted in the context of cloud environments, likeSPECvirt and SPA SOA. Moreover, the SPECweb benchmark, which focuses on webservers/application server benchmarking, is also of high interest.

Moreover, from March 2011, has formed a new group to research and recommendapplication workloads for benchmarking cloud computing performance. The groupfunctions under SPECs Open Systems Group (OSG) and is working in cooperationwith other SPEC committees and subcommittees to define cloud benchmark method-ologies, determine and recommend application workloads, identify cloud metrics forexisting SPEC benchmarks, and develop new cloud benchmarks. Current partici-pants in OSGCloud include AMD, Dell, IBM, Ideas International, Intel, KarlshuheInstitute of Technology, Oracle, Red Hat and VMware. Long-time SPEC benchmarkdeveloper Yun Chao is a supporting contributor. The group collaborates with theSPEC Research Cloud group, which is working on gaining a broader understandingof cloud behavior and performance issues. The group targets three types of users forthe workloads and benchmarks it will create: Hardware and software vendors pro-viding products that enable cloud services. Cloud providers entities that offer cloudservices, such as IaaS, PaaS or SaaS. Business customers who would use bench-mark results to help them select cloud providers. SPEC publishes several hundreddifferent performance results each quarter spanning a variety of system performancedisciplines. The goal of SPEC is to ensure that the marketplace has a fair and use-ful set of metrics to differentiate candidate systems. The basic SPEC methodologyis to provide the benchmarker with a standardized suite of source code based uponexisting applications that has already been ported to a wide variety of platforms byits membership. The benchmarker then takes this source code, compiles it for thesystem in question and then can tune the system for the best results.

The TPC [24] is a non-profit corporation founded to define transaction processingand database benchmarks and to disseminate objective, verifiable TPC performancedata to the industry. In the context of clouds, which are born to offer transactional ser-vices, TPC benchmarks are of great interest. TPC can be considered as application-level benchmarks in cloud environments and they are a basis for the evaluation ofthe actual performance offered by standard transactional software on the top of IaaS-delivered machines.

1.2.1.3 Benchmarking Services The adoption of standard benchmarks, as de-scribed above, is interesting to compare the offerings from different cloud providers,mainly at IaaS level. The main drawback of the standard benchmarks is that theyare not designed to cope with the elasticity of cloud offerings. They just provide anevaluation of static systems, and are not able to manage systems that dynamicallymay change over time, with highly variable performance. A possible solution is tobenchmark the systems after any elastic mutation, for example by adopting the ubiq-


uitous as-a-service approach. This is the solution adopted in [64], where the use of abenchmarking service is proposed, which starts up the benchmark execution on anynewly-delivered machine. CloudHarmony [7] is probably the most interesting solu-tion of this kind. It offers a very large collection of customizable benchmarks thatis continuously executed on remote resources. The benchmarks provided fall into 3categories: Performance, network and Availability. Performance, that includes CPU,disk IO, memory, and other performance factors, measured using about 100 syn-thetic and real-world benchmarks like Unixbench, IOzone, SPECjvm2008 and manyothers. Organizations should consider these characteristics when selecting a cloudservice in order to establish reasonable expectations and growth planning. Networkperformance measurements include both network performance within the cloud aswell as out of the cloud. Typical indexes are latency and throughput, measured usingdifferent techniques: For cloud-to-cloud performance CloudHarmony utilizes a net-work of about 50 servers located throughout the world to periodically measure andrecord latency and throughput measurements to other clouds. For cloud-to-consumerperformance they host a cloud speedtest which measures latency and throughput us-ing a browser-based application. Service availability or uptime, is a critical cloudperformance characteristic. To measure uptime, CloudHarmony has setup serviceswith most Public Cloud vendors and have monitors in place to track any outages ofthose services.

While CloudHarmony is built on the top of standard benchmarks, CloudSleuth [9]can build up a cloud application benchmark, which generates a standard workloadon target services in order to perform comparisons.

1.2.1.4 CloudOriented Benchmarks Currently only few cloud-specific propos-als for benchmarking exist in the literature. The contribution can be summarized bymentioning the following Cloudstone [69] and CloudCmp [55] projects. These areacademic projects, which aim at offering flexible frameworks able to build up customworkloads to be run in cloud environments. Cloudstone is an academic open sourceproject from the UC at Berkeley. It provides a framework for testing realistic per-formance. The project does not publish as of yet comparative results across clouds,but provides users with the framework that allows them to get their measurements.The reference application is a social Web 2.0-style application. The Cloudstonestack can be divided into three subcategories: web application, database, and loadgenerator. When running benchmarks, the load generator generates load against theweb application, which in turn makes use of the database. The Web 2.0 applica-tion used in Cloudstone is Olio, an open source social-event application with bothPHP and Ruby implementations. Cloudstone uses the Ruby implementation. Ng-inx and HAProxy are used to load balance between the many Rail servers (i.e., thinservers). Cloudstone supports the use of either MySQL or PostgreSQL. Replicationwhen using MySQL is supported using the built-in master-slave replication features.Replication when using PostgreSQL is supported using PGPool-II, a type of loadbalancer for PostgreSQL database servers. CloudCmp instead focuses on typicalCloud-based IaaS services (storage and virtual machines) for which a set of customkernel benchmark were proposed that aims at comparing key features of the sys-

BENCHMARKING 7

tems. The academic project, also supported by some private companies, was appliedto compare few common cloud providers. The main limit of this kind of benchmarksis that they do not take into account the variability of results of the same provider indifferent conditions. Moreover, the results, even if offer a general idea of providercloud behavior, can hardly be used by a service provider to make a real choice, tak-ing into account how its application behaves. CloudCmp [55] can guide customersin selecting the best-performing provider for their applications. While many PublicCloud providers offer pay-as-you-go computing, their varying approaches to infras-tructure, virtualization, and software services lead to a problem of plenty. To helpcustomers pick a cloud that fits their needs, we develop CloudCmp, a systematiccomparator of the performance and cost of cloud providers. CloudCmp measuresthe elastic computing, persistent storage, and networking services offered by a cloudalong metrics that directly reflect their impact on the performance of customer appli-cations. CloudCmp strives to ensure fairness, representativeness, and compliance ofthese measurements while limiting measurement cost.

CloudBench [56] has been developed to conduct scalable, controllable, and re-peatable experiments in virtualized cloud infrastructures. CloudBench, integrates anefficient discrete-event simulation with the cloud infrastructure under test in a closedfeedback control loop. The technique supports the decomposition of complex re-source usage patterns and provides a mechanism for statistically multiplexing appli-cation requests of various characteristics to generate realistic and emergent behavior.It also exploits parallelism at multiple levels to improve simulation efficiency, whilemaintaining temporal and causal relationships with proper synchronization.

C-MART [70] is a benchmark designed to emulate a modern web application run-ning in a cloud computing environment. It is designed using the cloud computingparadigm of elastic scalability at every application tier and utilizes modern web-based technologies such as HTML5, AJAX, jQuery, and SQLite. C-MART consistsof a web application, client emulator, deployment server, and scaling API. The de-ployment server automatically deploys and configures the test environment in ordersof magnitude less time than current benchmarks. The scaling API allows users todefine and provision their own customized datacenter. The client emulator gener-ates the web workload for the application by emulating complex and various clientbehaviors, including decisions based on page content and prior history.

Cloud Spectator [5] has worked with various IaaS cloud computing providers tohelp optimize market messaging, pricing, and increase revenue by enlarging marketshare. CloudSpecs Benchmarking System, developed by Cloud Spectator, is an au-tomated benchmarking application with over 30 tests for CPU, RAM, storage, andinternal network as well as overall virtual server and application performance tests.CloudSpecs features an optimal blend of tests for a thorough understanding of per-formance without an overwhelming amount of data.

The CliQr CloudCenter platform [23] provides the necessary visibility and flex-ible control to optimize the price/performance benefits of migrating an applicationto the cloud. An application’s price/performance results can vary greatly from cloudto cloud. CliQr’s CloudCenter Manager benchmarking environment enables appli-cations to be benchmarkedunchanged and simultaneouslyacross any cloud provider


and/or configuration, size, or instance type. As a critical part of the deployment pro-cess, the CloudCenter benchmarking environment automatically selects the clouddeployment options based on the application requirements (e.g., memory require-ments, database support, etc.). The end user simply clicks on the eligible clouds tobe benchmarked. CliQr measures the price, performance, and the price-performanceindex for each cloud environment selected. The results are graphically presented,enabling the user to make an informed decision about deploying the application onthe cloud.

In recent years many online graph database service providers have started theiroperations in Public Clouds due to the growing demand for graph data storage andanalysis. While this trend continues to rise there is little support available for bench-marking graph database systems in cloud environments. XGDBench [34] is a graphdatabase benchmarking platform for cloud computing systems. XGDBench has beendesigned to operate in current cloud service infrastructures as well as on futureexascale clouds. We extend Yahoo! Cloud Serving Benchmark to the domain ofgraph database benchmarking. The benchmark platform is written in X10 which isa PGAS language intended for programming future exascale systems. UtilizzandoXGDBench si sono valutate le prestazioni di four famous graph data stores Allegro-Graph, Fuseki, Neo4j, an OrientDB using XGDBench on Tsubame 2.0 HPC cloudenvironment.

CloudGauge [39] is a dynamic and experimental benchmark for virtualized andcloud environments, and the key requirements and characteristics of virtual systems’performance metrics, evaluation and workload’s characterization which can be con-sidered a step further to implement a production-ready virtual systems and cloudbenchmarking suite and performance models.

OFBench [61] is a new research benchmark for enterprise applications. OFBenchis a load generator for the demo e-commerce component of the ApacheOFBiz en-terprise resource planning (ERP) framework. ERP applications are increasingly im-portant in the cloud market due to the growing popularity of Software-as-a-Service(SaaS) solutions, hence OFBench is useful to explore the resource consumption pat-terns arising in these applications. On client-side, OF Bench features randomizedcustomer surfing patterns, workload generation based on automation of real webbrowsers, reproducible burstiness in inter-request and inter-session think times, non-stationary workload generation. On server-side, it features JMX performance moni-toring, business logic instrumentation, and synchronization with client-side monitor-ing via TCP sockets.

Acumen Cloud [1] is a web-based schedule benchmarking utility that gives youthe power to compare your Fuse Schedule Index diagnostics results to other similarprojects in terms of size and/or nature. The Fuse Schedule Index is made up ofnine different schedule quality checks and their proven thresholds. Each of thesekey indicators is individually calculated and weighted before being compiled intothe Index. Standard thresholds have been established to help you determine if yourproject makes the cut or not. Acumen Cloud determine not only your overall FuseSchedule Index, but also see where your score ranks compared to other projectsof similar size and nature. Use this benchmarking result to track improvement of

BENCHMARKING 9

quality over time and gain insight as to whether your project is being developed tosufficient quality and detail. And since its web-based, the benchmarking data youare being ranked against is constantly updated in real time for a more comprehensivecomparison of your project.

1.2.2 Benchmarking Big Data services

Management of Big data is one of the key services in the cloud paradigm, and a greatnumber of new systems were developed for cloud data serving. Examples of systemsproposed for cloud serving use include BigTable, PNUTS, Cassandra, HBase, Azure,CouchDB, SimpleDB, Voldemort, and many others. Further, they are being appliedto a diverse range of applications that differ considerably from traditional (e.g., TPC-C like) serving workloads. The number of emerging cloud serving systems and thewide range of proposed applications, coupled with a lack of apples-to-apples perfor-mance comparisons, makes it difficult to understand the tradeoffs between systemsand the workloads for which they are suited. Yahoo! Cloud Serving Benchmark(YCSB) framework [33], aims at facilitating performance comparisons of the newgeneration of cloud data serving systems. We define a core set of benchmarks andreport results for four widely used systems: Cassandra, HBase, Yahoo!’s PNUTS,and a simple sharded MySQL implementation. The key feature of the YCSB frame-work/tool is that it is extensible: it supports easy definition of new workloads, inaddition to making it easy to benchmark new systems.

Moreover data sizes are breaking the petabyte barrier, this imposes new problemsand research possibilities for the database community. Usually, data of this sizeis stored in large clusters or clouds. Although clouds have become very popularin recent years, there is only little work on benchmarking cloud applications. AData Generator for Cloud-Scale benchmarking [63] presents a data generator forcloud sized applications. Its architecture makes the data generator easy to extendand to configure. A key feature is the high degree of parallelism that allows linearscaling for arbitrary numbers of nodes. We show how distributions, relationships anddependencies in data can be computed in parallel with linear speed up.

1.2.2.1 A Critical Analysis The above presented solutions clearly state the mainproblem of benchmarking in cloud environments. These are: how to represent witha single (or a few) indexes the dynamicity of a cloud environment? How usefulis to perform few static measurement? As a matter of fact, at the state of the art,the main direction for performance evaluation of cloud offerings is oriented to theuse of monitoring. Benchmarking solutions, like the ones proposed commerciallyby CloudHarmony or CloudSleuth, try to adapt benchmarking solutions to typicalmonitoring approaches. Moreover, currently it is impossible to identify a singleworkload representing the incredibly large variety of cloud services offered. Thedirection adopted is to reuse benchmarks targeted at specific services, and to try todeliver them in a more flexible way, as in [53], which uses TPC-W to evaluate thecloud services delivered. It should be noted that benchmark results are not reliable incloud environments. The same request of resource may lead to completely different


behavior, as shown in [35, 36]. This means that benchmarking should be appliedon the single specific resource, more than on generic resources offered by a givencloud provider. This approach, on the other side, has a high impact on the costs.It is questionable whether it is acceptable to pay for a resource usage which is justtargeted to execute a benchmark?

In conclusion, benchmarking tools should be more flexible than the those com-monly adopted in other environments. The adoption of a single performance index isnot acceptable and workload definition should be customizable by the benchmarkeruser, according to its specific needs.

1.3 Benchmarking with mOSAIC

The Benchmark generation step aims at producing a set of benchmarking kernels thatstress the components of the mOSAIC application, collecting performance indexes.The mOSAIC environment provides a built-in framework for the development ofbenchmark applications, which has been described in [66, 27]. We rely on the mO-SAIC benchmarking facilities to offer a fixed tool that performs a set of tests that areof general purpose, benchmarking the basic mOSAIC components, namely QueueServers, KV Stores and Filter Cloudlets (i.e., cloudlets that receive messages, elab-orate them and send out or store the result). Extending the benchmarking suit tomore complex components, e.g., cloudlets that do not operate as a filter, is part ofour future work.

The proposed suite performs the tests under simple repetitive workloads, thatstresses the target component, in order to identify their behaviour in well-knownworking condition. This kind of benchmarks can be used to predict more complexapplications, as proposed in [65]. For each class of components, we build up adedicated benchmark application, which stresses the component and measures itsperformance indexes. The benchmarking applications can be executed on the actualtarget resources, leased from a cloud provider, and stress the components while theyare executed, varying their load (e.g., number of concurrent requests, number andlength of concurrent SET/GET operations). The results of benchmarking applicationexecution are then collected in a simple csv file.

In order to illustrate the approach, we describe briefly the three main benchamrkapplications: the Queue benchmark, the KV benchamrk and the Filter benchmark.The main goal of the former is to measure the waiting time of each message in a cloudqueue. We assume that, for each message, such waiting time depends on the totalnumber of messages that are already waiting for being processed when the messageis put in the queue. As a consequence, it is possible to stress the Queue compo-nent through a set of different experiments, each of which submits a fixed number ofconcurrent messages, and evaluates the average response time. The KV Store bench-mark application stresses the KV store component (like an S3 bucket or a dedicatedKV store system), by submitting a fixed number of concurrent requests to the storingcomponent. The stressing cloudlets perform a sequence of SET/GET operations onthe system. The application evaluates the average number of completed operations

MONITORING 11

per second. The last application stresses a cloudlet, the mosaic abstraction used tobuild cloud components, which act in reaction to the receipt of a message. When theinput has been processed, the component sends out a new message. In this case, wemeasure the processing time (i.e., the time spent from the moment in which the re-quest is sent on the queue and the time when the result is received by the consumer).

This applications are collected together in a package, composed of two moduesthe benchmark-bundle.sh and the mosaic bundle. The last one, mOSAIC bun-dle [15] is a standalone application that can transform a cloud virtual machine ina mOSAIC Platform, i.e., it installs on the VM the mOSAIC components and runsthem. The benchmark bundle, instead, is a client application that, once the mosaicbundle is started over the remote machine, automatically deploys and runs all thebenchamrk applciations, collecting the data in a single csv file.

The benchmark bundle enables the developers to automatize the benchamrkingprocess and the collection of the data on the target resources in order to tune theirapplications.

1.4 Monitoring

1.4.1 Monitoring Problem Scenarios

As specified in Cloud Computing Use Cases whitepaper [49] “all cloud servicesmust be metered and monitored for cost control, chargebacks and provisioning”.In this direction, several usage patterns related with the runtime maintenance usecases, as described in [37], have been developed. They cover different use cases,like ‘monitoring of sensor data and sensor diagnosis’, ‘monitoring the status of anintelligent maintenance system’, ‘monitoring simulations in a model exploration sys-tem’, ‘resource monitoring at infrastructure level [48, 62] and for security purposes[45, 58, 46, 44, 43]. On provider side monitoring is necessary for effective utilizationof Computing resources, to check service levels, which have been agreed with thecustomers, to lead negotiation strategies, such as overbooking, in order to maximizethe total incoming. At this level the monitoring addresses the full range of resources,both hardware and virtual. Different requirements are based on consumers’ needs,like monitoring their own resources for checking that service level agreements arecontinuously respected and for detecting under-utilization or overloading conditionsof their resources in order to renegotiate the agreement if it is necessary. Monitoringand tools are available for physical and virtualized resources, but we have to dealwith different scenarios. In private Cloud all is available to the user, but in a pub-lic Cloud customers can only access the virtual resources, meanwhile the providermanages hardware and hypervision. This means that monitoring information is ac-cessible only to providers for administration purposes, and eventually shared to thecustomers. However, Cloud customers cannot check the compliance of the ServiceLevel Agreement (SLA) trusting the monitoring service of their own same provider.In fact the Cloud provider has a conflicting interest ensuring the guarantees on ser-vice levels it provides.


A monitoring solution controlled by the user is necessary to advance the stateof art in the current Cloud context. In fact, GoGrid, as an example, obliges itscustomers to trust its own monitoring service for checking the agreed service lev-els. From GoGrid’s SLA: individual servers will deliver 100% uptime as monitoredwithin the GoGrid network by GoGrid monitoring systems (Server Uptime Guaran-tee). Only failures due to known GoGrid problems in the hardware and hypervisorlayers delivering individual servers constitute Failures and so only those Failuresare covered by this Server Uptime Guarantee and this SLA. In order to address themonitoring in the current Cloud scenario, it is necessary to face with the semanticgap between SLA offered by different Cloud Vendors. The main monitored param-eters for compute resources are cumulative availability, outage length, server reboottime, which are used to check the service level objectives. Usually, there are no ser-vice levels for compute performance; customers are simply guaranteed to have thecompute resource for which they paid, with technical enforcement at the hypervisorlevel. However, performance information at hypervisor level are always available tothe vendors and sometimes to the customers. Service levels can be monitored andprovided for internal network between nodes, within the providers data center, and,if the provider has multiple data centers, among those data centers. Values of keyperformance indicators are usually related to geographical locations. About stor-age, monitoring covers availability and performance. However, sometimes a storageperformance profile will actually be an internal network profile, covering latency be-tween compute and storage. Finally, security is not offered as a service level, butis granted by the availability of firewall and configurable policies. Many providersoffer a 100% availability for their virtual servers and for the internal network, anddeclare to pay a penalty, not exceeding the maximum monthly credit allowed to theuser, when a lower value has been measured by their own monitoring service. Thedefinition of downtime is different for each provider. For example the availabilitycriteria of Azure is more related to the downtime of its virtual machine, but Ama-zon refers manly to the availability of the external network. Rackspace offer 100%availability of network and servers. GoGrid is another Cloud provider that offers100% availability for its internal and external networks. Besides it guarantees 100%uptime of its servers and also network performance, in terms of packet loss, latencyand jitter. Also in this case, the user must trust the provider’s monitoring services tocheck failures. Performance guarantees change from internal to external networks,and they also change in different geographical regions. Azure and GoGrid provideperformance guarantees for their Storage services, but Amazon only grants persis-tence. It means that Amazon grants that after its data centers fail (only because ofhardware or hypervisor), configuration of network, computing infrastructure, anddata will be restored.

1.4.2 Monitoring Problem Analysis

Problem we have to address concerns how and by whom monitoring should be per-formed. Monitoring and tools are available for real and virtualized resources, but wehave to deal with different scenarios. In private Cloud all is available to the user,

MONITORING 13

but in a public Cloud customers can only access the virtual resources, meanwhile theprovider manages hardware and hypervisor. Monitoring is required at each Clouddeployment level, including the infrastructure level (IaaS), the platform level (PaaS)and the application-level (SaaS).

Service Monitored Parameters and Performance Indexes

Compute

Cumulative Availability (connectivity uptime). E.g. 99.5%, 99%

Role instance uptime. E.g. 99.9%, 99%

Outage length. E.g. Monthly time

Server reboot time. E.g. 15 minutes

Network

Availability. E.g. 100%

Packet loss. E.g. 0.1% per month

Latency within the Cloud network or within the same VLAN. E.g. 1 ms

Mean jitter, maximum jitter. E.g. 0.5 ms within any 15 minute period

Bandwidth. E.g. Mb/s, Gb/s

Storage

IOPS. E.g. I/O per second.

Latency with the internal compute resource. E.g. <1 ms

Bandwidth with the internal compute resource. E.g. Gb/s

Processing time (it does not include transferring).E.g. Blob copy <60 s, time to process data <2 s x amount of MB,

list tables returns or continues within 10 s

Maximum restore time. E.g. Number of minutes

Uptime. E.g. If <99.9% refund 10% credits,

If <99% refund 25% credits

Table 1.2 Different monitored resources at infrastructure (IaaS) level

1.4.2.1 Monitoring at IaaS level IaaS delivers a computing infrastructure overthe Internet and is enabled to split, assign and dynamically resize these resources tobuild custom infrastructures, just as demanded by customers.

Cloud IaaS SLAs are similar to SLAs for network services, hosting and data cen-ter outsourcing. The main issues at IaaS concerns the mapping of high level applica-tion requirements on monitored parameters which measure low level service levels.

For IAAS there are many standardization efforts like OCCI (Open Cloud Com-puting Interface) [18], SOCCI (Service-Oriented Cloud-Computing Infrastructure)by The Open Group [20], the ISO Study Group on Cloud Computing (SGCC) [12],and there is a de-facto industry alignment on IaaS service level objectives, warranties,guarantees, performance metrics, etc.

Table 1.2 captures the most important monitored resources at infrastructure (IaaS)level and associated monitored parameters, together with their performance indexes.


Compute, network and storage are the main abstractions of services at this level, ascaptured in Table 1.2. Different monitored parameters for compute, like cumulativeavailability, outage length, server reboot time, are used to check the service levelobjectives. Usually there are no service level agreements established for computeperformance: customers are simply guaranteed to have the compute for which theypaid, with technical enforcement at the hypervisor level. However performance in-formation at hypervisor level are always available to the vendors and sometimes tothe customers.

As providers usually offer a network SLA, some support for monitoring infor-mation of the cloud provider’s data center connectivity to the Internet is provided.This information can be monitored and provided also for internal communicationbetween different resources (devices) within provider’s data center or, if there aremultiple data centers, between those data centers. Values of key performance indi-cators are usually related to geographical locations.

1.4.2.2 Monitoring at PaaS level With platform as a service (PaaS) an additionalabstraction level is introduced: rather than supplying a virtualized hardware infras-tructure, a software platform is provided where consumer services can be executed.Sizing of the hardware resources demanded by the execution of consumer servicesis made transparently by a specialized PaaS provider.

PaaS providers offer a wide array of services that covering development, deploy-ment and execution of applications, with the advantage that the developer can lever-age a fully functional development and/or production environment. Additional ser-vices, like load balancing, domain name services, access control, content deliverynetworks, databases, and communication mechanisms, could be provided and mon-itored at PaaS level, as exemplified in Table 1.3. Hovever, at this level performanceindexes, which are not necessary to check service level agreements, are not available.

1.4.2.3 Monitoring at SaaS level With software as a service (SaaS), several cloudsystems could be grouped in order to deliver a complete aggregated service, underthe control of a Cloud governance implementation. Since resources at this level areusually complete applications, there will be no difficulties to map low level moni-toring information to application level. The objective of monitoring will be to un-derstand how the application works. Relevant parameters, like monthly cumulativeapplication downtime, maximum error response to a valid query during two ore moreconsecutive time defined intervals, errors rate of content delivery, average downloadtime less that a specified value, are captured in Table 1.4 (the ‘File Server’ service isprovided as a reference example). However, it is really difficult to provide a compre-hensive and representative list of performance parameters.

1.4.3 Monitoring state of the art

The increasing use of Cloud Computing to manage applications, services and re-sources puts even more emphasis on the necessity of monitoring the QoS parametersin order to have the desired performances [31]. Furthermore the resources control

MONITORING 15

PAAS Service Monitored Parameters

SQL DatabaseConnection time E.g. < 30s

Number of read/write/query failures after connec-tion

Load BalancingLatency from VIP to server E.g. <05 ms

Throughput E.g. 500 Mb/s, 5000 transactions/sec-ond

Concurrent transactions E.g. 10.000

CDN No. of errorless transactions E.g. 50KB-1M httpsample,

(Content Delivery Network) objects are performed in a month,

by a commercial test application,

of a third party.

Access Control Service downtime (total) E.g. Sum of intervals dur-ing which all,

customer’s attempts to establish,

a connection to Access Control,

service fail.

Table 1.3 Different monitored resources at platform (PaaS) level

is very important when a customer wants to check the compliance of the provideroffered services with the ones that he/she has signed into the SLA.

A reference SLA model, which is composed of SLA objectives (SLOs) is offeredin [68]. Here, a number of service levels and performance metrics for each resourceresults in multiple SLOs for every service. Similarly, in [30] there is an attempt to de-fine various QoS metrics. A different approach is offered in [52], where SLA-basedresource negotiations are combined with virtualized resources in terms of on-demandservice provision. An approach for policy based automatic SLA negotiation for WebServices has been studied in [75] and [74] for Internet-based Virtual Computing En-vironments. In the paper [42], a QoS-based approach to SLA definition is presented,also handling dynamic reconfiguration. A different implementation, following the‘as-a-service’ paradigm, is offered QoS monitoring component with QoSMONaaS,built on the top of the SRT-15 project [21].

Infrastructure-level resource monitoring [32] aims at the measurement and re-porting of system parameters related to real or virtual infrastructure services offeredto the user (e.g., CPU, RAM, or data storage parameters). General monitoring toolsavailable today include Nagios [29], an open source network monitoring system usedto monitor services and hosts, and MonALISA [54], a Jini-based monitoring frame-work. These approaches can be considered as metering tools that do not provide a


SaaS Service Objective

Generic

Service Availability E.g. < 0.01% per month

Response time E.g. seconds

Service Outage Resolution Time, Failover Window On Disaster Re-covery

E.g. The time necessary to recover the service after

an outage, or after the outage of all the datacenter

RFO E.g. Reason for Outage

Customer Service Response Time, Customer Service AvailabilityE.g. Time needed for reacting to a request by the

customer service, and time of the day,

days of week during which the service is available.

Reclaiming Customer Data E.g. Possibility to get back the data after

the end of agreement, and eventual support for migration.

Maintenance Notification, Proactive Service Outage NotificationE.g. Time within which an outage (minutes), or

scheduled or special maintenance are notified

File ServerConnection uptime E.g. 99.9%

Server error E.g. Server error after 2 or more consecutive 90s inter-vals

Avg. download time E.g. Fails to download a 1-byte document in 0.3sec.

Table 1.4 Different monitored resources at software (SaaS) level, and ’File Server’ sampleservice objectives

lot in the direction of advanced monitoring functionality. Furthermore, traditionalmonitoring technologies for single machines or Clusters are restricted to locality andhomogeneity of monitored objects and, therefore, cannot be applied in the Cloud inan appropriate manner [40]. In the field of Grid computing, the Globus Toolkit Mon-itoring and Discovery System (MDS4) [67] provides mechanisms for monitoring anddiscovering resources and services. MDS lacks metering tools of each own and re-lies on external information providers. GridICE [25] follows a centralized approachthus requiring multiple Nagios servers in order to facilitate a large distributed envi-ronment. Many available tools need to run on the physical machine and use facilitiesof the hypervisor. In a Private Cloud they can be used for management purpose, butin a Public Cloud they are controlled by the Cloud vendor. Cloud vendors share asubset of information to the customers, monitoring services and supporting differentperformance measurements, depending on the monitoring technology.

CLOUD MONITORING IN MOSAIC’S CLOUD AGENCY 17

Monitoring solutions in Clouds are usually tailored to specific vendors. For exampleAmazon CloudWatch [2] provides the functionality to receive information about spe-cific metrics, such as the number of read bytes, from EBS services. Windows Azure[60] provides monitoring capabilities through the Azure Diagnostic Manager. Anopen source software targeting Clouds is the Lattice Monitoring Framework [32],which follows the notion of producers and consumers of information in trying tominimize network utilization for monitoring information.Other third party technologies, which provide Cloud monitoring facilities, are Cloud-kick [8], Nimsoft Monitor[17], Monitis [14], RevealCloud [19]. All of them areproprietary solutions and do not aim at defining a standard for monitoring interoper-ability. Some technologies for monitoring network and host like S-Flow have beenextended to support the transport of monitoring information of virtualized resources.For example host-sflow [10] exports physical and virtual server performance met-rics using the sFlow protocol. Ganglia and other collectors of performance measuresare already compliant with its specification. Some of the most promoted Cloud-monitoring solutions include the Nimsoft Monitor, Monitis, or the Tap in CloudManagement Service [4].

About PaaS monitoring we could have two different solutions for monitoring sup-port. The first one needs to provide an API that allow users for software componentsfor develop ad-hoc monitoring service inside their own application, or to bind appli-cations to monitoring services from third party. In this case, as an example Cloud-Bees [6] uses a set of shared services that the externally-facing services depend on.For example, an application might use the CloudBees MySQL database service and aPartner Service like New Relic monitoring. In the second case the PaaS vendor pro-vides information about memory, badwidth, I/O utilization of deployed applications.An example in this case is Jelastic [13].

VMware vFabric Hyperic [11], e.g., is specialized in web-application monitor-ing, and is capable of monitoring applications that utilize any of the supported webtechnologies. A more general approach is to utilize the JMX Java framework [16],that is employed by most Java application containers, and is capable of providinginformation on the status of the application running in the container. This, however,requires that the application is written in Java, and that it allows to publish informa-tion through the JMX subsytem.

1.5 Cloud monitoring in mOSAIC’s Cloud Agency

As resource usage and the service workload can change very frequently, a continu-ous and updated answer must be provided to the user by efficiently monitoring theenvironment (e.g., a change in the answer must be detected as soon as possible whileminimizing the communication cost). Furthermore, the Cloud elasticity enables tochange dynamically over time the system configuration, and so the distributed mon-itoring must adapt itself quickly to the new requirements. In [50] authors claims thatan approach based on software agents is a natural way to tackle the monitoring tasksin the aforementioned distributed environments. Agents move and distribute them-


selves to perform their monitoring tasks. In [57] an optimal control of mobile mon-itoring agents in artificial-immune-system-based (AIS-based) monitoring networkshas been studied. That is why we propose an agents based monitoring service.

Cloud Agency (CA) [71] [72] is a multi agent system (MAS) that accesses, onbehalf of the user, the utility market of Cloud computing to maintain always thebest resources configuration that satisfies the application requirements. This systemis being designed and developed within the research activities of the FP7 mOSAICproject [59]. Cloud Agency uses agent technologies to offer its services by an OCCIcompliant interface. They are not limited to infrastructure management, but extendedwith provisioning and monitoring facilities.

Obtaining a common interface is the focus of the OCCI group that defined a modelfor Cloud management at IaaS level, by offering definitions for entities, API andprotocol. The specification of Cloud API is a Resource Oriented Architecture (ROA)that uses REpresentational State Transfer (REST) protocol.

CA Monitoring System offers an agent-based infrastructure whose goal is to gen-erate warnings when the target application and/or the associated resources are inconditions which may lead to a violation of one of the requirements defined in theSLA [47]. Cloud Agency allows the user to monitor the infrastructure in terms ofresource utilization and for the definition of some strategy of autonomic reconfigu-ration. Reconfiguration can use management facilities by stopping, starting, movinginstances, but it could ask for provisioning of additional resources.

The CA-Client GUI allows the configuration of the monitoring infrastructure onthe acquired resources. It is possible to select a set of metrics, each one representinga Meter Agent, which is able to perform the chosen measurements. Once the userhas chosen the metric, he/she can start the measures by using the same monitoringconsole. The CA-Client sends a request to the Cloud Agency by using its exposedRESTful interface. This request is forwarded to the Archiver Agent (which is also re-sponsible for the collection and the management of monitoring information), whichasks for deploying a Cloud Agency instance on the selected resource to the relatedCloud provider embodied by a specific Vendor Agent. Vendor Agent launches aspecific script that allows the starting of a new Cloud Agency instance and the con-nection of it to the already existing one. As described in Figure 1.1, after finishingthe deployment operations the Vendor Agent sends an acknowledge to the ArchiverAgent that creates the user selected Meter Agent. This one loads the required li-braries to perform the measures, retrieves from the Archiver Agent the location onwhich the measurements have to be executed and migrates on the resource carringon its data. After the migration, the Meter Agent installs the measurement librarieson the resource and starts its own work by sending a notification to the ArchiverAgent that forwards it to the CA-Client in an asynchronous way. At this point, theMeter Agent periodically performs measures on the resource and sends them to theArchiver.

CONCLUSIONS 19

Figure 1.1 Starting monitoring on a resource

1.6 Conclusions

In Cloud paradigm resources are dynamically acquired and charged on the base of apay-per-use model. The Cloud management system needs to automatically scale andallocate new resources to match the SLAs. Therefore, in such a dynamic provision-ing scenario, the monitoring and evaluation of service performance play an impor-tant role, giving the possibility to maintain always the best resources allocation thatsatisfies the user requirements. Benchmarking and monitoring of provided servicesallow to infer an accurate assessment of system performance, workload distribution,overloads, bottleneck, underused resource, and malicious resource exhaustion. Thischapter has outlined the additional problems that such issues raise in Cloud, as wellas presented an overview of state of art about the solutions and the technologiesproposed to face them.

REFERENCES

[1] Acumen Cloud. http://www.projectacumen.com.

[2] Amazon CloudWatch.

[3] Amazon HPC service. http://aws.amazon.com/hpc-applications/.

[4] Cloud Management Service. http://www.tapinsystems.com/cloud-services/.

[5] Cloud Spectator. http://www.cloudspectator.com/about.

[6] CloudBees. http://www.cloudbees.com/resources/white-papers.cb.

[7] Cloudharmony.

[8] Cloudkick. https://www.cloudkick.com/.

[9] CloudSleuth.

[10] host-sflow. http://host-sflow.sourceforge.net/.

[11] Hyperic. http://www.vmware.com/products/vfabric-hyperic/.

[12] ISO Study Group on Cloud Computing. http://www.iso.org/iso/jtc1 sc38 home.

[13] Jelastic. http://jelastic.com/.

[14] Monitis. http://portal.monitis.com/index.php/products/cloud-monitoring.

[15] mOSAIC bundle. http://developers.mosaic-cloud.eu/confluence/display/MOSAIC/How+to+test+mOS#HowtotestmOS-mOS-bundleinstaller.

[16] MX Java framework. http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html.

Please enter \offprintinfo{(Title, Edition)}{(Author)}at the beginning of your document.

21

22 REFERENCES

[17] Nimsoft Monitor. http://www.nimsoft.com/solutions/nimsoft-monitor/cloud/public-cloud.

[18] Open Cloud Computing Interface (OCCI). http://occi-wg.org/.

[19] RevealCloud. http://copperegg.com/.

[20] Service-Oriented Cloud-Computing Infrastructure (SOCCI).https://collaboration.opengroup.org/projects/soa-soi/.

[21] SRT-15 Project. http://srt-15.unine.ch/.

[22] Standard Performance Evaluation Corp (SPEC). http://www.spec.org/osgcloud/press/cloudannouncement20120613.html.

[23] The CliQr CloudCenter platform. http://www.cliqr.com/products/benchmark.php.

[24] TPC: Transaction Processing Performance Council.

[25] Sergio Andreozzi, Natascia De Bortoli, Sergio Fantinel, Antonia Ghiselli, Gian LucaRubini, Gennaro Tortone, and Maria Cristina Vistoli. GridICE: a monitoring service forGrid systems. Future Generation Computer Systems, 21(4):559–571, 2005.

[26] Rocco Aversa, BENIAMINO Di Martino, MASSIMILIANO Rak, Salvatore Ven-ticinque, and Umberto Villano. Performance prediction for HPC on clouds. CloudComputing: Principles and Paradigms, R. Buyya, J. Broberg, and AM Goscinski, Ed.,John Wiley &Sons, pages 437–454, 2011.

[27] G. Aversano, M. Rak, and U. Villano. The mosaic benchmarking framework: Devel-opment and execution of custom cloud benchmarks. Scalable Computing, 14(1):33–45,2013. cited By (since 1996)0.

[28] D.H. Bailey, E. Barszcz, L. Dagum, and H.D. Simon. NAS parallel benchmark results.Parallel Distributed Technology: Systems Applications, IEEE, 1(1):43–51, 1993.

[29] Wolfgang Barth. Nagios: System and Network Monitoring. No Starch Press, 2008.

[30] Bu-Qing Cao, Bing Li, and Qi-Ming Xia. A Service-Oriented Qos-Assured and Multi-Agent Cloud Computing Architecture. In Proceedings of the 1st International Confer-ence on Cloud Computing, CloudCom ’09, pages 644–649, Berlin, Heidelberg, 2009.Springer-Verlag.

[31] Stuart Clayman et al. Monitoring service clouds in the future internet. Towards theFuture Internet, pages 115–126, 2010.

[32] Stuart Clayman, Alex Galis, Clovis Chapman, Giovanni Toffetti, Luis Rodero-Merino,Luis M. Vaquero, Kenneth Nagin, and Benny Rochwerger. Monitoring Service Clouds inthe Future Internet. In Georgios Tselentis, Alex Galis, Anastasius Gavras, Srdjan Krco,Volkmar Lotz, Elena Paslaru Bontas Simperl, Burkhard Stiller, and Theodore Zahariadis,editors, Future Internet Assembly, pages 115–126. IOS Press, 2010.

[33] Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears.Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM sym-posium on Cloud computing, SoCC ’10, pages 143–154, New York, NY, USA, 2010.ACM.

[34] Miyuru Dayarathna and Toyotaro Suzumura. XGDBench: A benchmarking platform forgraph stores in exascale clouds. In CloudCom, pages 363–370. IEEE, 2012.

[35] Jiang Dejun, Guillaume Pierre, and Chi-Hung Chi. EC2 performance analysis for re-source provisioning of service-oriented applications. In Service-Oriented Computing.ICSOC/ServiceWave 2009 Workshops, pages 197–207. Springer, 2010.

REFERENCES 23

[36] Jiang Dejun, Guillaume Pierre, and Chi-Hung Chi. Resource provisioning of web appli-cations in heterogeneous clouds. In Proceedings of the 2nd USENIX conference on Webapplication development, pages 5–5. USENIX Association, 2011.

[37] DMTF. Architecture for Managing Clouds, June 2010.http://dmtf.org/sites/default/files/standards/documents/DSP-IS0102 1.0.0.pdf.

[38] Jack J. Dongarra, Piotr Luszczek, and Antoine Petitet. The LINPACK Benchmark:past, present and future. Concurrency and Computation: Practice and Experience,15(9):803–820, 2003.

[39] M.A. El-Refaey and M.A. Rizkaa. CloudGauge: A Dynamic Cloud and VirtualizationBenchmarking Suite. In Enabling Technologies: Infrastructures for Collaborative En-terprises (WETICE), 2010 19th IEEE International Workshop on, pages 66–75, 2010.

[40] Vincent C. Emeakaroha, Ivona Brandic, Michael Maurer, and Schahram Dustdar. Lowlevel Metrics to High level SLAs - LoM2HiS framework: Bridging the gap betweenmonitored metrics and SLA parameters in cloud environments. In Waleed W. Smari andJohn P. McIntire, editors, HPCS, pages 48–54. IEEE, 2010.

[41] Constantinos Evangelinos and C Hill. Cloud Computing for parallel Scientific HPCApplications: Feasibility of running Coupled Atmosphere-Ocean Climate Models onAmazon’s EC2. ratio, 2(2.40):2–34, 2008.

[42] Stefano Ferretti, Vittorio Ghini, Fabio Panzieri, Michele Pellegrini, and Elisa Turrini.QoS. Cloud Computing, IEEE International Conference on, 0:321–328, 2010.

[43] M. Ficco and M. Rak. Intrusion tolerance as a service: A SLA-based solution. Pro-ceedings of the 2nd International Conference on Cloud Computing and Services Science(CLOSER), pages 375–384, 2012.

[44] M. Ficco and M. Rak. Intrusion tolerance in cloud applications: The mOSAIC approach.Proceedings of the 6th International Conference on Complex, Intelligent, and SoftwareIntensive Systems (CISIS), pages 170–176, 2012.

[45] M. Ficco, M. Rak, and B. Di Martino. An intrusion detection framework for supportingSLA assessment in Cloud Computing. Proceedings of the 2012 4th International Con-ference on Computational Aspects of Social Networks (CASoN), pages 244–249, 2012.

[46] M. Ficco, S. Venticinque, and B. Di Martino. mOSAIC-based intrusion detection frame-work for cloud computing. Lecture Notes in Artificial Intelligence and Lecture Notes inBioinformatics, 7566 LNCS:628–644, 2012.

[47] M. Ficco, S. Venticinque, and B. Di Martino. An advanced intrusion detection frame-work for cloud computing. Computer Systems Science and Engineering, 28(6):401–411,2013.

[48] Teodor-Florin Fortis, Gorka Esnal, Imanol Padillo, Tamas Mahr, and Gabor Ferschl.Cloud Patterns for mOSAIC-Enabled Scientific Applications. In Euro-Par 2011 Work-shops, Part I, volume 7155 of Lecture Notes in Computer Science, pages 83–92. SpringerBerlin/Heidelberg, 2012.

[49] Cloud Computing Use Cases Group. Cloud Computing Use Cases White Paper. July2010.

[50] Sergio Ilarri, Eduardo Mena, and Arantza Illarramendi. Using cooperative mobile agentsto monitor distributed and dynamic environments. Inf. Sci., 178(9):2105–2127, 2008.

24 REFERENCES

[51] Alexandru Iosup, Simon Ostermann, M Nezih Yigitbasi, Radu Prodan, ThomasFahringer, and Dick HJ Epema. Performance analysis of cloud computing services formany-tasks scientific computing. Parallel and Distributed Systems, IEEE Transactionson, 22(6):931–945, 2011.

[52] Attila Kertesz, Gabor Kecskemeti, and Ivona Brandic. An SLA-based resource virtu-alization approach for on-demand service provision. In Proceedings of the 3rd inter-national workshop on Virtualization technologies in distributed computing, VTDC ’09,pages 27–34, New York, NY, USA, 2009. ACM.

[53] Donald Kossmann, Tim Kraska, and Simon Loesing. An evaluation of alternative archi-tectures for transaction processing in the cloud. In Proceedings of the 2010 ACM SIG-MOD International Conference on Management of data, pages 579–590. ACM, 2010.

[54] I. Legrand, H. Newman, R. Voicu, C. Cirstoiu, C. Grigoras, C. Dobre, A. Muraru,A. Costan, M. Dediu, and C. Stratan. MonALISA: An agent based, dynamic servicesystem to monitor, control and optimize distributed systems. Computer Physics Com-munications, 180(12):2472–2498, 2009.

[55] Ang Li, Xiaowei Yang, Srikanth Kandula, and Ming Zhang. CloudCmp: comparingpublic cloud providers. In Proceedings of the 10th ACM SIGCOMM conference onInternet measurement, IMC ’10, pages 1–14, New York, NY, USA, 2010. ACM.

[56] Qi Liu, Marcio A. Silva, Michael R. Hines, and Dilma Da Silva. Hardware-in-the-loop simulation for automated benchmarking of cloud infrastructures. In Proceedingsof the Winter Simulation Conference, WSC ’12, pages 226:1–226:12. Winter SimulationConference, 2012.

[57] Wenjia Liu and Bo Chen. Optimal control of mobile monitoring agents in immune-inspired wireless monitoring networks. Journal of Network and Computer Applications,34(6):1818–1826, 2011.

[58] Ficco M. Security event correlation approach for cloud computing. Journal of HighPerformance Computing and Networking, 7(3), 2013.

[59] Beniamino Di Martino, Dana Petcu, Roberto Cossu, Pedro Goncalves, Tamas Mahr, andMiguel Loichate. Building a Mosaic of Clouds. Lecture Notes in Computer Science,pages 529–536, 2011.

[60] Cade Metz. VMware open sources Microsoft Azure killer,April 2011. http://www.theregister.co.uk/2011/04/12/vmware open-sources platform as service code/.

[61] Jean Moschetta and Giuliano Casale. OFBench: An Enterprise Application Benchmarkfor Cloud Resource Management Studies. In SYNASC, pages 393–399. IEEE ComputerSociety, 2012.

[62] D. Petcu. Identifying Cloud computing usage patterns. In Cluster Computing Workshopsand Posters (CLUSTER WORKSHOPS), 2010 IEEE International Conference on, pages1–8, sept. 2010.

[63] Tilmann Rabl, Michael Frank, HatemMousselly Sergieh, and Harald Kosch. A DataGenerator for Cloud-Scale Benchmarking. In Raghunath Nambiar and Meikel Poess,editors, Performance Evaluation, Measurement and Characterization of Complex Sys-tems, volume 6417 of Lecture Notes in Computer Science, pages 41–56. Springer BerlinHeidelberg, 2011.

REFERENCES 25

[64] M. Rak, A. Cuomo, and U. Villano. A Service for Virtual Cluster Performance Evalu-ation. In Enabling Technologies: Infrastructures for Collaborative Enterprises (WET-ICE), 2010 19th IEEE International Workshop on, pages 249–251, 2010.

[65] M. Rak, A. Cuomo, and U. Villano. Cost/performance evaluation for cloud applicationsusing simulation. pages 152–157, 2013. cited By (since 1996)0.

[66] Massimiliano Rak and Giuseppe Aversano. Benchmarks in the cloud: The mosaic bench-marking framework. In Symbolic and Numeric Algorithms for Scientific Computing(SYNASC), 2012 14th International Symposium on, pages 415–422. IEEE, 2012.

[67] Jennifer M Schopf, Laura Pearlman, Neill Miller, Carl Kesselman, Ian Foster, MikeD’Arcy, and Ann Chervenak. Monitoring the grid with the Globus Toolkit MDS4. Jour-nal of Physics: Conference Series, 46(1):521, 2006.

[68] Kwang Sim. Towards Complex Negotiation for Cloud Economy. In Paolo Bellavista,Ruay-Shiung Chang, Han-Chieh Chao, Shin-Feng Lin, and Peter Sloot, editors, Ad-vances in Grid and Pervasive Computing, volume 6104 of Lecture Notes in ComputerScience, pages 395–406. Springer Berlin / Heidelberg, 2010.

[69] Will Sobel, Shanti Subramanyam, Akara Sucharitakul, Jimmy Nguyen, Hubert Wong,Arthur Klepchukov, Sheetal Patil, Armando Fox, and David Patterson. Cloudstone:Multi-platform, multi-language benchmark and measurement tools for web 2.0. In Proc.of CCA, 2008.

[70] Andrew Turner, Andrew Fox, John Payne, and Hyong S. Kim. C-MART: Benchmarkingthe Cloud. IEEE Transactions on Parallel and Distributed Systems, 24(6):1256–1266,2013.

[71] Salvatore Venticinque. Agent Based Services for Negotiation, Monitoring and Reconfig-uration of Cloud Resources. European Research Activities in Cloud Computing, pages178–202, 2012.

[72] Salvatore Venticinque, Luca Tasquier, and Beniamino Di Martino. Agents based cloudcomputing interface for resource provisioning and management. In Complex, Intelligentand Software Intensive Systems (CISIS), 2012 Sixth International Conference on, pages249–256. IEEE, 2012.

[73] Edward Walker. Benchmarking amazon EC2 for high-performance scientific computing.LOGIN, 33(5):18–23, 2008.

[74] Zan Xiao and Donggang Cao. A Policy-Based Framework for Automated SLA Nego-tiation for Internet-Based Virtual Computing Environment. In Proceedings of the 2010IEEE 16th International Conference on Parallel and Distributed Systems, ICPADS ’10,pages 694–699, Washington, DC, USA, 2010. IEEE Computer Society.

[75] F.H. Zulkernine and P. Martin. An Adaptive and Intelligent SLA Negotiation System forWeb Services. Services Computing, IEEE Transactions on, 4(1):31–43, jan. 2011.

Cloud Evaluation: Benchmarking and Monitoring

Documents