11Cloud-Spectator-Report_final.pdf

Cloud IaaS Performance & Price-Performance Comparing Linux Compute Performance of 1&1, Amazon AWS, Aruba Cloud,

CloudSigma, and Microsoft Azure

Prepared for 1&1 on Behalf of SolidFire

Commercial Report

Published on 6/2015

Cloud Spectator Performance Report

1

Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

Contents

1. Introduction 2

Why Performance Matters 2

2. Executive Summary 4

Findings 4

3. Methodology 6

Process 6

Tests Used 7

Provider Data Center/Region Locations 8

VM Configurations and Pricing 8

Understanding Performance Results 10

Understanding The CloudSpecs Score (Price-Performance) 10

Key Considerations 11

4. Detailed Performance Findings 12

Processor & Memory Bandwidth 12

Disk IOPS: Sequential and Random Operations 14

Internal Network 17

5. Detailed Price-Performance Findings 19

Processor & Memory Bandwidth 19

Disk IOPS: Sequential and Random Operations 21

Internal Network 24

6. Conclusion 26

7. About 26


2


Introduction

In an effort to simulate an end-user experience regarding performance of virtual machines across various cloud providers, Cloud Spectator

ran its iterative benchmark suite for 72 hours on each of the following providers: 1&1, AWS, Aruba Cloud, Microsoft Azure, and CloudSigma. SolidFire sponsored this study on behalf of its client, 1&1.

In most cases, much of the work was straightforward regarding provisioning and the setup process. Occasionally, vendor-side issues

occurred during the provisioning process, and the Cloud Spectator team contacted the corresponding vendor’s support team in order to resolve

issues such as VM provisioning errors. Three primary VMs of each size were tested on all providers for 24 hours each (72 hours total). This was done sequentially; once one VM had run the test suite for 24 hours, that VM was terminated and a new VM was created.

This study not only examined the performance of each vendor, but also tracked performance variability for each of the three 24-hour

periods. The methodology allowed Cloud Spectator to capture performance variability both over time on the same VM as well as across different VMs on multiple physical hosts. Some providers, such as 1&1, show strong processor and memory bandwidth performance stability for all of its VMs

throughout the course of the study. Other providers, such as AWS, exhibited controlled periods of burst followed by throttled performance on network

storage depending on the size of the storage volume. Others, such as CloudSigma, exhibited unstable performance across all resources throughout

the study, possibly due to server-side issues at the time of the study, which contributed to provisioning problems as well. Taking performance and stability a step further, price-performance analyses are conducted to help readers understand the value ratio

between the cost of the VM and the performance output. While the performance output is limited to the data points collected in the study, by

comparing the price-performance ratio, readers can gain better insight into the overall user experience seen on these providers.

Why Performance Matters

Performance and pricing are both key considerations in the public cloud industry, together having a substantial impact on a company’s annual

operating costs. Cloud users may need fewer resources on better performing services, which can lower costs. Since many users only consider price and not price-performance, these users may be paying more because they require additional resources to achieve a desired level of performance.

While some providers try to differentiate their offerings by cutting prices, others try to differentiate by focusing on improved performance and user

experience.

Differences in performance outputs of VMs across IaaS providers

can greatly impact quality of service as well as annual operating

costs. The graph on the right illustrates an example of the average processor performance from a sample of six Cloud

Service Providers (CSPs) as studied by Cloud Spectator. CSP 1

has a processor performance three times as high as CSP 6

(names removed), which gives CSP 1 a notable advantage in many processor-intensive workloads. CSPs 2-5 exhibit a closer

resemblance in processor performance, but do not offer nearly as

much processing power as CSP 1 does.


3


The table below lists the 3 hardware components studied in this project, and each purpose as a function in the server.

CPU & MEMORY PERFORMANCE

STORAGE PERFORMANCE

NETWORK PERFORMANCE

The performance of all applications is highly

dependent on the CPU. The CPU is responsible for

the processing and orchestration of all applications.

The relationship between CPU performance and RAM

is also observed by examining RAM bandwidth. While

memory performance is not considered one of the key

bottlenecks in performance for many applications, a

subset of applications—particularly HPC and in-

memory databases—is highly dependent on large

sustained memory bandwidth.

Because most applications and all data reside

on the disk, having fast disk performance is a

key consideration for best application

performance in many cases.

In a cloud environment, network performance

is a critical piece. Scalability, in many cases,

is dependent on the availability of additional

VMs that must maintain a strong network

backbone.


4


Executive Summary

On behalf of 1&1, SolidFire commissioned Cloud Spectator to gauge the performance of VMs on five different cloud providers’ European data

centers: 1&1, Amazon AWS, Aruba Cloud, CloudSigma, and Microsoft Azure. Both performance and price-performance were examined to evaluate the value of each provider’s VMs. The purpose of the study was to understand, from an end-user perspective, the disparity of performance and value

(defined as price-performance) among cloud providers with similarly sized VMs. Overall, 1&1 exceled in performance and price-performance for all

component resources of the VMs tested. Its high performance rankings, combined with hourly pricing, introduces powerful, scalable cloud

infrastructure at low cost to its users.

Findings

vCPU & Memory Performance Findings

For this study, Cloud Spectator evaluated vCPU and memory

bandwidth performance by benchmarking the VMs using Geekench

3, a suite of benchmark tests that simulate tasks such as cryptographic encoding and image processing. Testing occurred over

the course of a 72-hour testing period. Pricing was examined in

conjunction with the performance tests.

vCPU & Memory Performance Key Findings:

• 1&1’s VMs achieved the highest performance across processor and memory performance in the study.

• 1&1’s VMs achieved the highest CloudSpecs ScoreTM in the test

group, indicating the strongest price-performance value for

processor and memory bandwidth.

• CloudSigma’s processor performance varied the most. Its

coefficients of variation (CV), which is a percentage expressing the

relationship between the average and standard deviation (useful for determining variability in performance), ranges up to 43.7%.

• The virtual processors on 1&1, AWS, and Azure exhibited the most

stability throughout the study, resulting in less than 3% coefficients of variation.


5


Storage Performance Findings

Storage was evaluated using the FIO disk benchmark, which tests

the sequential read/write and random read/write operations of storage. In this study an 8KB block size was used. Testing occurred

over a 72-hour test period. Persistent storage (offered as “block

storage” or “redundant storage”) was used in all storage tests. Pricing

was examined in conjunction with the performance tests.

Storage Performance Key Findings:

• 1&1 VMs offered the highest disk performance among all included in this study.

• Although AWS offers SSD technology on its block storage offering,

EBS, the performance of that offering is more dependent on the size of the block storage volume provisioned.

• Despite being one of the lower-tier performers in disk IOPS, Azure

displayed the most stable disk performance throughout the study.

• Out of all providers examined in the study, only AWS appeared to

provide a period of burst performance for its block storage. On the

2vCPU VMs, which have 100GB of block storage, AWS volumes

displayed a burst behavior. After the period of burst, sequential read/write operations and random read/write operations dropped to

become 10% and 20%, respectively, of the initial IOPS achieved

during burst. This burst behavior was not seen on VMs with 400GB and 800GB of block storage, due to the larger number of IOPS.

• 1&1 displayed the best price-performance value for disk IOPS. Its

high-performance SAN disk offering is designed with SolidFire technology.

Internal Network Performance Findings

Internal network performance was measured as the throughput between VMs within the same data center of the cloud provider

(measured using iperf and ping respectively) over the course of a 72-

hour test period. Pricing was examined in conjunction with the performance tests.

Internal Network Performance Key Findings:

• CloudSigma VMs achieved the highest internal network

throughput, although the high throughput is unstable and

fluctuates between less than 500Mbits to over 10Gbits.

• CloudSigma displayed the best price-performance value for

internal network. Despite its large fluctuation in internal network

throughput, its median score range significantly outperformed all other providers.

• 1&1 achieved the second-highest internal network CloudSpecs

ScoreTM.

• Other than CloudSigma, only certain Azure VMs exceeded 1 GB/s

throughput.


6


Methodology

Cloud Spectator strives to create a transparent and detailed methodology to allow readers to understand the testing process and recreate any

studies. If any information is unclear or if you have any questions, please email the team at [email protected] or call +1 (617) 300-0711.

Process

1. Three iterations of 24-hour test cycles were run for each VM on each provider for a total of 72 hours of testing per VM size. After each 24-

hour block, VMs were terminated before beginning another cycle of tests on newly provisioned machines. 2. Each VM was provisioned with a Linux Ubuntu 14.04 OS by default, available from all providers. For AWS, the HVM image was used.

3. Before each 24-hour test period, and after provisioning the VMs, system updates and upgrades were conducted via apt-get.

4. The following dependencies were installed for testing: a. Git. Git was used to clone the test repository on the VM.

b. MySQL. For automation purposes, mysql-server was installed to automate data uploads.

c. Pip. Used to download the appropriate libraries for Python in order to run the testing. SQL Alchemy was downloaded to interact

with MySQL and upload data. d. Libmysqlclient-dev. MySQL database’s development files, which are necessary for the SQL Alchemy and MySQL interaction.

5. Each test cycled through in the following sequence: Geekbench 3 (process & memory), fio sequential operations, fio random operations,

Iperf internal network throughput (for more information on testing, see Tests Used). a. For fio testing (to measure disk IOPS), sequential operations ran first. Files from the sequential tests were deleted, and fio

recreated files before running random operations. Once random operations completed, the files were also deleted. Thus, before

each disk IOPS test, the files associated with the tests were deleted and recreated.

6. Internal network testing was conducted in one of the following manners: a. On AWS and Azure, where VMs demonstrated varying internal network throughput depending on size and/or instance type, a

clone of that VM was created in the same region/availability zone. The cloned server listened for a TCP connection via Iperf.

E.g., two c4.large instances were created in Amazon AWS’s EU West 1 region to test throughput. The cloned VM was terminated alongside the tested VMs at the conclusion of each 24-hour test cycle.

b. On 1&1, Aruba Cloud, and CloudSigma, where VMs did not demonstrate varying internal network throughput depending on size

and/or instance type, a screen session was created on each VM to listen for a TCP connection. Each category of VMs, which

contains 2 VMs, connected with each other to perform throughput testing; i.e., 1&1’s 2 vCPU 4GB RAM and 2 vCPU 8GB RAM virtual machines conducted network throughput tests across one another.

7. A total of approximately 1.4 million data points were collected throughout the period of the study.

8. At the end of each test iteration, results were uploaded into Cloud Spectator’s database through use of SQL Alchemy (Python—see 4c in Process).


7


Tests Used

Processor & Memory Bandwidth: Geekbench 3

Geekbench 3, a licensable product created by Primate Labs, is a cross-platform processor benchmark that can measure single-core and multi-core performance by simulating real-world workloads. The Geekbench 3 test suite is comprised of 27 individual tasks/workloads: 13 integer workloads, 10

floating point workloads, and 4 memory-bandwidth tasks. While processor and memory bandwidth are both performance factors that contribute to the

final score provided by Geekbench 3, the test suite weighs processing performance much more heavily than memory bandwidth. Also, memory

bandwidth is not necessarily affected by the amount of memory available for the VM, so VMs with larger amounts of memory may not exhibit larger bandwidth. For more information on Geekbench 3 and to see its individual workloads, please see http://www.primatelabs.com/geekbench/.

Geekbench 3 Tasks (Figure 3.1)

TEST TOOL TASK Separate CPU tests that are all aggregated into a final

score.

• Subtests include: Integer Math, Floating Point Math

DESCRIPTION

Integer Geekbench 3 AES, Twofish, SHA1, SHA2, BZip2 Compression, BZip2 Decompression, JPEG Compression, JPEG Decompression,

PNG Compression, PNG Decompression, Sobel, Lua, Dijkstra

Integer and Floating Point tasks together represent vCPU performance. The performance of all applications

is highly dependent on the vCPU since the vCPU is responsible for the processing and orchestration of all

applications. Floating Point Geekbench 3 Black Scholes, Mandelbrot, Sharpen Filter, Blur Filter,

SGEMM, DGEMM, SFFT, DFFT, N-Body, Ray Trace

Memory Geekbench 3 STREAM Copy, STREAM Scale, STREAM Add, STREAM

Triad

While memory performance is not considered one of

the key bottlenecks in performance for many common applications, a subset of applications—particularly HPC

and in-memory databases—is highly dependent on

large sustained memory bandwidth.

Sequential and Random Disk IOPS: fio

Fio is an open source I/O generator that spawns a number of threads and processes to conduct a particular type of I/O action specified. For the purpose of this study, fio was used to measure disk IOPS by tracking direct I/O to the VM’s network storage. 5 x 200mb files were created for

sequential operations testing, and 5 x 200 mb files were created for random operations testing. All operations were 50% read and 50% write. Each

test iteration used an 8kb block size. Each test iteration lasted 60 seconds.

Internal Network Throughput: Iperf

Iperf is an open source tool used to measure TCP or UDP network bandwidth performance. In this study, Cloud Spectator used Iperf to measure the network throughput between VMs residing in the same region/availability zone. A screen session was created for Iperf as a server machine on each

appropriate VM (see Process 6). Each Iperf test iteration lasted 60 seconds, and data was transferred in one direction, from the test VM to the Iperf

server VM.


8


VM Configurations & Pricing

VM configurations were matched to standardize by virtual processors. 2 vCPUs, 4 vCPUs, and 8 vCPUs machines from each provider were used in

this study that most closely matched in size. While some providers may offer the option for local storage, none was used and all disk testing was

conducted on persistent SAN storage. The storage columns in Figures 3.3 A through C are reflective of the SAN storage provisioned for each VM. Azure uses Blob Storage, which automatically provides the user with as-needed storage. Thus, only the space on the volume that is needed

(depending on the OS and installed applications) is given to the user. With Blob Storage, users cannot deploy volumes with a pre-defined amount of

storage, although the user can specify how large he or she expects the blob to grow. For more information, see this article: https://msdn.microsoft.com/en-us/library/azure/ee691964.aspx.

2 vCPU Virtual Machines (Figure 3.3 A)

Provider Instance vCPU RAM (GB) Storage (GB) Monthly (€)

1&1 4GB 2 4 100 €29.99

1&1 8GB 2 8 100 €79.20

AWS C4.large 2 3.75 100 (EBS Optimized) € 95.04

AWS M3.large 2 7.5 100 (EBS Optimized) € 108.72

Aruba Cloud 4GB 2 4 100 € 64.80

Aruba Cloud 8GB 2 8 100 € 79.20

Azure A2 2 3.5 Blob Storage € 68.10

Azure D2 2 7 Blob Storage € 104.60

CloudSigma 4GB 2 4 100 € 49.33

CloudSigma 8GB 2 8 100 € 73.73


9


4 vCPU Virtual Machines (Figure 3.3 B)


1&1 8GB 4 8 400 €93.60

1&1 15GB 4 15 400 €144.00

AWS C4.xlarge 4 7.5 400 (EBS Optimized) € 208.08

AWS M3.xlarge 4 15 400 (EBS Optimized) € 236.16

Aruba Cloud 8GB 4 8 400 € 158.40

Aruba Cloud 15GB 4 15 400 € 183.60

Azure A3 4 7 Blob Storage € 143.65


CloudSigma 8GB 4 8 400 € 135.27

CloudSigma 15GB 4 15 400 € 177.97

8 vCPU Virtual Machines (Figure 3.3 C)


1&1 15GB 8 15 800 (2 x 400) € 172.80

1&1 30GB 8 30 800 (2 x 400) € 280.80

AWS C4.2xlarge 8 15 800 (EBS Optimized) € 416.88

AWS M3.2xlarge 8 30 800 (EBS Optimized) € 473.76

Aruba Cloud 15GB 8 15 800 € 313.20

Aruba Cloud 30GB 8 30 800 € 367.20

Azure A4 8 14 Blob Storage € 287.24


CloudSigma 15GB 8 15 800 € 276.63

CloudSigma 30GB 8 30 800 € 368.15 Conversion Rates: £1.00 = €1.40 $1.00 = €0.89


10


Understanding Performance Results

The virtual machines’ performance information was depicted using the minimum, 5th percentile, median, 95th percentile, and maximum scores

retrieved from all data points collected for each of the tests mentioned above during the testing period. The information was integrated into percentile

graphs and value tables designed to visualize performance variation captured while testing over time. An example of a performance percentile graph is displayed below:

Understanding The CloudSpecs Score (Price-Performance)

Cloud Spectator uses the CloudSpecs ScoreTM as an indication of price-performance value for each resource of the VM, separated by 1) processor & memory bandwidth, 2) disk IOPS, and 3) internal network throughput. By definition, the CloudSpecs ScoreTM provides information on how much

performance the user receives for each unit of cost. The CloudSpecs ScoreTM is an indexed, comparable score ranging from 0-100 indicative of

value based on a combination of cost and performance. The calculation of the CloudSpecs ScoreTM is:

price-performance_value = [VM performance score] / [VM cost]

best_VM_value = max{price-performance_values}

CloudSpecs ScoreTM = 100*price-performance_value / best_VM_value

In this report, Cloud Spectator uses the aggregated performance scores as

the [provider performance score] to calculate each machine’s CloudSpecs

ScoreTM.

The graph on the left is an example of how Cloud Spectator’s price-

performance analysis is visualized. The closer the score is to 100, the

higher price-performance value it indicates. The score 100 represents the

best-value VM among all in the comparison. The value is scaled; e.g., the

100

44 41 41 25

0

20

40

60

80

100

CSP1 CSP2 CSP3 CSP4 CSP5

Clou

dSpe

cs S

core

TM

Cloud Service Provider (CSP)

Legend

Maximum: highest score achieved on this VM over the duration of the testing.

95TH Percentile (High-Score Category): 95% of all scores on this VM achieved this score or lower.

Median (Median-Score Category): The number separating the higher half of the scores of that VM from the lower half. If the median is closer to the

95th percentile, then more high scores were observed than low scores; vice versa.

5TH Percentile (Low-Score Category): 5% of all scores on this provider achieved this score or lower.

Minimum: lowest score achieved on this VM over the duration of the testing.

0

500

1000

1500

2000

2500

3000

3500

4000

AWS Azure Google Rackspace Softlayer

Mpixe

ls/se

c

Sample Performance Graph


11


VM from Cloud Service Provider 1 (CSP1) with a score of 100 gives 4x the value of the VM from CSP5 with a score of 25.

The CloudSpecs ScoresTM of any VM can change depending on the participants in the comparison. For example, if the highest score in a comparison changes, the price-performance value represented by score 100 will change accordingly, and so will the other CloudSpecs ScoreTM

values.

If you have questions regarding Cloud Spectator’s price-performance calculation, please contact us at [email protected].

Key Considerations

• Pricing used in this study (for price-performance comparisons) is up to date effective June 10, 2015. Pricing may change for the specified

VMs after the release of this report.

• Testing was conducted on specific VM types for each provider. Different VM configurations may yield different comparative results between the providers. AWS and Azure offered fixed VM configurations, while 1&1, Aruba Cloud, and CloudSigma offered independently

customizable resource configurations.

• Users may experience different performance across different physical hosts. Factors such as user contention or malfunctions of the physical hardware can cause suboptimal performance. Cloud Spectator terminated and created new VMs for each test iteration to increase

the likelihood of testing on different physical hosts.

• VMs selected were the base offerings across providers; greater performance may be obtained on certain providers by paying for additional features/services.


12


Detailed Performance Findings

Processor and Memory Bandwidth

Below are the results of processor and memory bandwidth testing on all providers. Because memory bandwidth is not affected by the amount of provisioned RAM, the VMs with larger amounts of RAM do not necessarily have higher Indexed Scores. The tables on the right specify the scores

achieved by each provider’s VMs. The lowest score in each category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in

each category is highlighted and bolded in green in the corresponding tables.

0 500

1000 1500 2000 2500 3000 3500 4000 4500 5000

4GB 8GB c4.large m3.large 4GB 8GB A2 D2 4GB 8GB

1&1 AWS ArubaCloud Azure CloudSigma

Inde

xed

Scor

e

Processor & Memory PERFORMANCE: 2vCPUs Figure 4.1 A

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

10000

8GB 15GB c4.xlarge m3.xlarge 8GB 15GB A3 D3 8GB 15GB


Inde

xed

Scor

e

Processor & Memory PERFORMANCE: 4vCPUs Figure 4.1 B

0 2000 4000 6000 8000

10000 12000 14000 16000 18000

15GB 30GB c4.2xlarge m3.2xlarge 15GB 30GB A4 D4 15GB 30GB


Inde

xed

Scor

e

Processor & Memory PERFORMANCE: 8vCPUs Figure 4.1 C

Provider VM Min 5TH Median 95TH Max

2 vCP

Us

1&1 4GB 4575 4619 4711 4731 4746 8GB 4059 4663 4708 4732 4748

AWS c4.large 3901 3934 3952 3973 4002 m3.large 3125 3139 3155 3173 3192

Aruba Cloud

4GB 3731 4017 4085 4165 4203 8GB 2973 3883 4084 4167 4197

Azure A2 1781 2132 2166 2185 2304 D2 2176 3557 3621 3693 3746

CloudSigma 4GB 2457 2599 2896 2965 2986 8GB 1150 1374 2826 2981 3000


4 vCP

Us

1&1 8GB 7993 8493 8764 9088 9132 15GB 8304 8688 8852 9101 9144

AWS c4.xlarge 6932 7672 7736 7775 7808 m3.xlarge 5699 6105 6261 6286 6308

Aruba Cloud

8GB 6872 7459 7693 7842 7868 15GB 5660 7388 7663 7840 7881

Azure A3 3648 4096 4118 4136 4161 D3 4978 6650 6677 6700 6716

CloudSigma 8GB 1149 1454 3698 5782 5888 15GB 1192 1474 4538 4927 5387


8 vCP

Us

1&1 15GB 14218 15648 16138 16281 16349 30GB 14858 15768 16097 16767 16910

AWS c4.2xlarge 12718 13257 13705 14056 14140 m3.2xlarge 10580 11228 11519 11588 11659

Aruba Cloud

15GB 10739 11848 13775 14119 14337 30GB 10765 11692 13874 14206 14400

Azure A4 5541 7418 7839 7917 7963 D4 8687 12646 12739 12793 12866

CloudSigma 15GB 4019 5076 7281 7848 8056 30GB 1336 1550 5900 6803 7640


13


Observations: Processor & Memory Bandwidth Performance

The scores in Figures 4.1 A – C are indexed through a combination of floating point and integer performance on processors and memory bandwidth

on RAM. CloudSigma’s performance varied the most across processors. Its coefficient of variation (CV), which is a percentage expressing the

relationship between the average and standard deviation (useful for determining variability in performance), ranges up to 43.7% (on the 8 vCPU, 30GB RAM VM). By contrast, the CVs on 1&1’s tested VMs ranged up to 2.6% (on the 4 vCPU, 15GB RAM VM).

Additional Observations

• 1&1’s VMs achieved the highest performance across processor and memory performance in the study.

• The Indexed Score, which weighs heavily on processor performance, is similar between VMs of the same vCPU count for 1&1, Aruba

Cloud, and CloudSigma. All three providers offer independently customizable VMs, which are backed with the same hardware. AWS and Azure exhibit differences in performance results. AWS’s C4 Family and M3 Family are provisioned with different processors; Azure’s A

Series and D Series are provisioned with different processors as well.

• Although AWS and Azure VMs’ processor and memory performance scored lower than 1&1 VMs, both providers also achieved similar levels of performance stability in the 72-hour test period, with CVs lower than 3% on all VMs. Aruba Cloud VMs’ CVs ranged up to 9.3%.

• CloudSigma VMs’ large variation in performance, seen in the processor and memory performance results, are present in disk IOPS and

network performance as well.

• CloudSigma and Azure VMs continue to display the lowest processor and memory bandwidth performance in each category (see tables in Figure 4.1 A through C).


14


SAN Disk IOPS: Sequential Operations Below are the results of disk IOPS testing on all providers, specifically for sequential operations (50% read, 50% write). Disk IOPS was tested with

direct I/O, so results are not reflective of cached performance, which may sustain higher IOPS on each provider. The tables on the right specify the

scores achieved by each provider’s VMs. The lowest score in each category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables.

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

10000



IOPS

Sequential Read/Write- 2vCPU (100GB) Figure 4.2 A

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

10000



IOPS

Sequential Read/Write- 4vCPU (400GB) Figure 4.2 B

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

10000



IOPS

Sequential Read/Write - 8vCPU (800GB) Figure 4.2 C


2 vCP

Us

1&1 4GB 4477 6138 7143 7947 8341 8GB 4622 6082 6996 7972 8358


Aruba Cloud

4GB 379 713 1493 2494 4060 8GB 267 597 1372 4414 6204

Azure A2 586 1056 1380 1384 1399 D2 480 1220 1382 1412 1436

CloudSigma 4GB 214 349 753 2170 4538 8GB 208 328 749 2011 3621


4 vCP

Us

1&1 8GB 3052 5437 7675 9133 9437 15GB 2845 6570 7933 8804 9138


Aruba Cloud

8GB 241 618 1186 2483 4496 15GB 278 655 1200 2228 3892

Azure A3 580 1069 1380 1384 1421 D3 427 1166 1381 1385 1404

CloudSigma 8GB 219 311 603 1768 4191 15GB 177 306 633 2155 3909


8 vCP

Us

1&1 15GB 3198 4821 6755 8703 9280 30GB 3265 5709 7650 9214 9577


Aruba Cloud

15GB 342 617 1221 2352 4345 30GB 367 599 1300 3000 5843

Azure A4 396 620 663 687 712 D4 531 1195 1385 1414 1435

CloudSigma 15GB 202 316 646 1852 4724 30GB 195 306 596 2074 4012


15


SAN Disk IOPS: Random Operations Below are the results of disk IOPS testing on all providers, specifically for random operations (50% read, 50% write). Disk IOPS was tested with

direct I/O, so results are not reflective of cached performance, which may sustain higher IOPS on each provider. The tables on the right specify the

scores achieved by each provider’s VMs. The lowest score in each category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables.

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

10000



IOPS

Random Read/Write - 2vCPU (100GB) Figure 4.3 A

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

10000



IOPS

Random Read/Write - 4vCPU (400GB) Figure 4.3 B

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

10000



IOPS

Random Read/Write - 8vCPU (800GB) Figure 4.3 C


2 vCP

Us

1&1 4GB 4710 6002 6620 7346 8234 8GB 4772 5943 6621 7314 7695


Aruba Cloud

4GB 165 291 612 952 1181 8GB 115 307 552 1064 1709

Azure A2 163 1021 1383 1430 1438 D2 149 1127 1386 1428 1436

CloudSigma 4GB 243 325 661 1741 3224 8GB 210 351 652 1435 2997


4 vCP

Us

1&1 8GB 2830 5110 7272 8605 8896 15GB 3868 6232 7505 8359 8667


Aruba Cloud

8GB 117 296 532 983 1167 15GB 150 305 537 951 1173

Azure A3 167 963 1376 1424 1436 D3 113 1116 1376 1380 1436

CloudSigma 8GB 232 308 561 1674 3049 15GB 115 302 558 1557 2778


8 vCP

Us

1&1 15GB 2814 4557 6463 8042 8911 30GB 3087 5490 7364 8751 9109


Aruba Cloud

15GB 0 295 534 992 1629 30GB 155 296 518 919 1175

Azure A4 164 341 369 382 392 D4 83 1085 1385 1401 1432

CloudSigma 15GB 114 314 651 2156 3897 30GB 168 319 611 2151 3474


16


Observations: Sequential and Random Disk IOPS Performance

Providers offer unique approaches to SAN disk, from hardware components (SSD vs. traditional magnetic drives) to performance throttling. Even the

similarities can be quite different; for example, while 1&1 and CloudSigma both offer SSD-backed storage volumes, the performance difference

between the two providers is noticeable (see Figure 4.2-4.3), with 1&1 producing more than 6,500 IOPS, while CloudSigma produces less than 700 in the median range.


• Providers with SSD offerings exhibit little performance difference between random and sequential IOPS, which is expected for SSDs.

These providers include 1&1, AWS and CloudSigma. One exception is on the small VMs for AWS, which exhibited different IOPS

performance results due to throttling.

• Although 1&1 and CloudSigma both offer SSDs, 1&1’s volumes achieved 4.4 – 4.7x more IOPS than CloudSigma’s SSDs when examining

95TH percentile figures.

• 1&1, Aruba Cloud, and CloudSigma express large performance variability across the 24 hours of testing. Despite the variability, the low points of 1&1’s disk IOPS in both sequential and random operations still surpass other providers for the majority of tests. Aruba Cloud and

CloudSigma’s variability results in some of the lowest performance observed during the study.

• Although AWS offers SSD technology on its block storage offering, EBS, the performance of that offering is more dependent on the size of

the block storage volume provisioned. Although the actual IOPS performance varies between sequential and random operations, the pattern of performance remains similar based on the size of the provisioned storage (in this test scenario, 100GB, 400GB, and 800GB

sizes were provisioned on 2 vCPU, 4 vCPU, and 8 vCPU machines, respectively). For its General Purpose SSD volumes, AWS offers 3

IOPS per GB with burst up to 3000 IOPS. Figures 4.2 A and 4.3 A illustrate AWS’s burst for a 100GB volume. While burst for the 100GB volume never exceeded 3065 IOPS (see Max in the corresponding tables), the non-burst range, expressed by the median, stays at 299

IOPS for sequential operations, but is higher for random operations. The minimum guaranteed IOPS, 300 (100GB * 3 IOPS per GB), is

sustained for the most part, although minimum values showed lower IOPS, with dips to 227 IOPS. At 800 GB, AWS sustains a fairly stable

rate of approximately 2750 – 3000 IOPS. Burst credits are assigned to volumes, and as long as the volume still has credits, it can burst to 3000 IOPS. In the period of the study, for 4 vCPU and 8 vCPU machines on AWS, the 400GB and 800GB volumes sustained enough

credits for a continuous 24-hours of high performance; interestingly, the c4.xlarge VMs dropped in performance for IOPS on sequential

operations in all three 24-hour iterations, which suggests that credits for IOPS are being used up on the c4.xlarge VMs much faster than on others.

• Azure and Aruba Cloud do not offer SSD technology-backed storage. Therefore, both providers offer higher IOPS performance on

sequential operations when compared with random operations, which is common behavior for magnetic drives. Figures 4.2 and 4.3 illustrate this behavior for both providers. In one of the sequences for the 8 vCPU with 15GB RAM VM, Aruba Cloud did not successfully

complete a cycle of random read/write operations, and therefore achieved a minimum of 0 in the total 72-hour study.

• 1&1 achieved much higher IOPS than all other providers tested, with maximums exceeding 9500 IOPS for sequential operations. Even in

the median, 1&1’s SolidFire-backed SSD volumes obtained a minimum of 6463 IOPS, which is 2.15x more IOPS than can be achieved per 800GB volume on AWS, unless a user decides to purchase provisioned IOPS at an additional charge.

• For random IOPS, Azure and Aruba Cloud displayed the lowest results in most scenarios. Azure and Aruba Cloud, as mentioned before,

are the only providers in this study that use magnetic storage for their storage volumes.


17


Internal Network Throughput Below are the results of internal network testing on all providers. Internal network was examined by running TCP connections between two servers

within the same data center/region/zone. The tables on the right specify the scores achieved by each provider’s VMs. The lowest score in each

category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables.

0 500

1000 1500 2000 2500 3000 3500 4000 4500 5000



Thro

ughp

ut (M

bit/s

)

Internal Network Throughput PERFORMANCE: 2 vCPUs Figure 4.4 A

0 2000 4000 6000 8000

10000 12000 14000 16000 18000



Thro

ughp

ut (M

bit/s

)

Internal Network Throughput PERFORMANCE: 4 vCPUs Figure 4.4 B

0

1000

2000

3000

4000

5000

6000



Thro

ughp

ut (M

bit/s

)

Internal Network Throughput PERFORMANCE: 8 vCPUs Figure 4.4 C


2 vCP

Us

1&1 4GB 700 929 955 955 956 8GB 774 928 955 955 956


Aruba Cloud

4GB 792 834 875 895 895 8GB 794 838 894 895 895

Azure A2 193 302 708 734 746 D2 640 863 880 895 909

CloudSigma 4GB 914 1466 2760 4359 4832 8GB 968 1979 2900 3660 4361


4 vCP

Us

1&1 8GB 619 885 956 956 956 15GB 661 896 956 956 956


Aruba Cloud

8GB 779 826 857 882 895 15GB 297 743 853 881 895

Azure A3 570 863 888 910 930 D3 1451 1566 1624 1683 1735

CloudSigma 8GB 1123 1942 6526 13265 16152 15GB 1423 2517 6970 11396 12821


8 vCP

Us

1&1 15GB 696 841 953 956 956 30GB 709 891 956 956 956


Aruba Cloud

15GB 724 801 852 895 895 30GB 658 820 861 893 895

Azure A4 123 890 910 925 936 D4 1432 1529 1590 1649 1684

CloudSigma 15GB 1344 2001 3208 4657 5983 30GB 1005 2191 3493 5105 5792


18


Observations: Internal Network Throughput Performance

With regards to internal network throughput, three out of five providers exhibited fairly stable performance of less than 10% CV. Azure’s A2 and A4

VMs expressed higher variability at 33.3% and 12.1%, respectively. CloudSigma’s variability ranged much higher, up to 45.9% on the 4vCPU 8GB

RAM VM. When examining the median value of the same VM size, CloudSigma achieves much higher throughput (up to almost 7 Gbits) than the other VMs. Comparing with the same VM’s minimum value, the almost 7 Gbits drops to 1.5 Gbits. Despite the sharp decrease in performance,

CloudSigma’s unstable performance still achieved higher throughput, for the most part, than all other providers.


• AWS’s VMs deliver a specific network throughput depending on the size and family of VM. As the size of the VMs increase, the network

throughput increases as well. The difference in network performance between the C4 Family and M3 Family are comparable. By contrast, Azure’s A Series provides less throughput than its D Series.

• Azure’s VMs, which scale depending on size and Series, exceeds 1GB/s network throughput for D3 and D4s (see Figures 4.4B and C). All

other providers, with the exception of CloudSigma, never exceed 1GB/s, although 1&1 VMs and AWS’s larger VMs come close.

• 1&1 provides a continuous and fairly steady throughput of slightly less than 1GB/s (956 GB/s max) regardless of VM size. Similarly, Aruba Cloud provides a lower, continuous throughput at a little less than 900 MB/s, regardless of VM size as well.

• While AWS displayed some of the lowest internal network throughput numbers for 2 vCPU and 4 vCPU VMs, Aruba Cloud, which does not

scale internal network throughput with increased VM size, displayed the lowest throughput numbers for 8 vCPU VMs.


19


Detailed Price-Performance Findings

Processor & Memory

Figures 5.1 A through F illustrate the price-performance results for processor and memory bandwidth performance across all tested providers. Rows

are organized by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.

100

40

26 20

37

0 10 20 30 40 50 60 70 80 90

100

4GB

c4.la

rge

4GB

A2

4GB


Clo

udSp

ecs

Scor

e

Processor & Memory Price-Performance: 2 vCPUs Figure 5.1 A

100

87

49 58

64

0 10 20 30 40 50 60 70 80 90

100

8GB

m3.

larg

e

8GB

D2

8GB


Clo

udSp

ecs

Scor

e

Processor and Memory Price-Performance: 2 vCPUs Figure 5.1 B

100

52

40 31 29

0 10 20 30 40 50 60 70 80 90

100

8GB

c4.x

larg

e

8GB

A3

8GB


Clo

udSp

ecs

Scor

e

Processor & Memory Price-Performance: 4 vCPUs Figure 5.1 C

100

68

43 50

41

0 10 20 30 40 50 60 70 80 90

100

15G

B

m3.

xlar

ge

15G

B

D3

15G

B


Clo

udSp

ecs

Scor

e

Processor and Memory Price-Performance: 4 vCPUs Figure 5.1 D

100

47

35 29 28

0 10 20 30 40 50 60 70 80 90

100

15G

B

c4.2

xlar

ge

15G

B

A4

15G

B


Clo

udSp

ecs

Scor

e

Processor & Memory Price-Performance: 8 vCPUs Figure 5.1 E

100

66

42 51

28

0 10 20 30 40 50 60 70 80 90

100

30G

B

m3.

2xla

rge

30G

B

D4

30G

B


Clo

udSp

ecs

Scor

e

Processor and Memory Price-Performance: 8 vCPUs Figure 5.1 F


20


Observations: Processor & Memory Price-Performance

In all cases illustrated from Figure 5.1 A through F, 1&1’s VMs achieved the highest CloudSpecs ScoreTM in the test group, indicating the strongest

price-performance value for processor and memory bandwidth. AWS’s m3.large VM comes closest to matching in value with a 1&1 VM counterpart,

at a CloudSpecs ScoreTM of 87 (see Figure 5.1 B). Additional Observations

• For VMs with 2 vCPUs and approximately 8GB RAM (see Figure 5.1 B), all values are most evenly matched, compared with other VM sizes.

• As VM sizes scale up in processors and RAM, CloudSigma’s CloudSpecs ScoreTM drops, and its rank falls in relativity to other tested

providers’ VMs, due largely to performance (see Figure 4.1).

• Relatively, Azure’s D Series, which has an average CloudSpecs ScoreTM of 53 across all VM sizes, provides more relative price-

performance value than its A Series, which has an average CloudSpecs ScoreTM of 27. Because charts cannot be relatively compared,

though, the difference between the CloudSpecs ScoresTM does not equate to an almost 2x value on Azure’s D Series.


21


SAN Disk IOPS: Sequential Operations Figures 5.2 A through F illustrate the price-performance results for sequential disk IOPS performance across all tested providers. Rows are

organized by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.

100

2 7 8 6

0

10

20

30

40

50

60

70

80

90

100

4GB

c4.large

4GB A2

4GB


Clou

dSpe

cs Score

Sequen9al Read/Write: 2 vCPUs (100GB) Figure 5.2 A

100

4 14 15 11

0 10 20 30 40 50 60 70 80 90 100

8GB

m3.large

8GB

D2

8GB


Clou

dSpe

cs Score

Sequen9al Read/Write: 2 vCPUs (100GB) Figure 5.2 B

100

23

7 12

5

0 10 20 30 40 50 60 70 80 90 100

8GB

c4.xlarge

8GB A3

8GB


Clou

dSpe

cs Score

Sequen9al Read/Write: 4 vCPUs (400GB) Figure 5.2 C

100

30

9 12 6

0 10 20 30 40 50 60 70 80 90 100

15GB

m3.xlarge

15GB

D3

15GB


Clou

dSpe

cs Score

Sequen9al Read/Write: 4 vCPUs (400GB) Figure 5.2 D

100

25

8 6 6

0 10 20 30 40 50 60 70 80 90

100

15GB

c4.2xlarge

15GB

A4

15GB


Clou

dSpe

cs Score

Sequen9al Read/Write: 8 vCPUs (800GB) Figure 5.2 E

100

30

10 12 6

0 10 20 30 40 50 60 70 80 90 100

30GB

m3.2xlarge

30GB

D4

30GB


Clou

dSpe

cs Score

Sequen9al Read/Write: 8 vCPUs (800GB) Figure 5.2 F


22


SAN Disk IOPS: Random Operations Figures 5.3 A through F illustrate the price-performance results for random disk IOPS performance across all tested providers. Rows are organized

by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.

100

5 3 9 6

0

10

20

30

40

50

60

70

80

90

100

4GB

c4.large

4GB A2

4GB


Clou

dSpe

cs Score

Random Read/Write: 2 vCPUs (100GB) Figure 5.3 A

100

12 6

16 11

0 10 20 30 40 50 60 70 80 90 100

8GB

m3.large

8GB

D2

8GB


Clou

dSpe

cs Score

Random Read/Write: 2 vCPUs (100GB) Figure 5.3 B

100

24

3 12

5

0 10 20 30 40 50 60 70 80 90 100

8GB

c4.xlarge

8GB A3

8GB


Clou

dSpe

cs Score

Random Read/Write: 4 vCPUs (400GB) Figure 5.3 C

100

31

4 12

6

0 10 20 30 40 50 60 70 80 90 100

15GB

m3.xlarge

15GB

D3

15GB


Clou

dSpe

cs Score

Random Read/Write: 4 vCPUs (400GB) Figure 5.3 D

100

25

3 4 6

0 10 20 30 40 50 60 70 80 90

100

15GB

c4.2xlarge

15GB

A4

15GB


Clou

dSpe

cs Score

Random Read/Write: 8 vCPUs (800GB) Figure 5.3 E

100

31

4 12

6

0 10 20 30 40 50 60 70 80 90 100

30GB

m3.2xlarge

30GB

D4

30GB


Clou

dSpe

cs Score

Random Read/Write: 8 vCPUs (800GB) Figure 5.3 F


23


Observations: Sequential and Random Disk IOPS Price-Performance

For sequential and random operations across all VMs, 1&1 offers the highest value due to a combination of high-performance SSDs and low overall

cost of the virtual machine (see Figures 5.2 and 5.3). While CloudSigma also offers SSDs, the disk IOPS achieved on CloudSigma VMs display

much lower performance (see Figures 4.2 and 4.3). AWS, which also offers SSD-based storage (General Purpose) offering was used in this study, throttles performance at a maximum of 3,000 IOPS unless users purchase additional provisioned IOPS; therefore, performance on AWS’s SSDs did

not exceed that of 1&1 either, which had a median range of 6,500-7,500 IOPS.


• 1&1 achieved the highest CloudSpecs ScoresTM as well as most IOPS for each VM tested.

• Aruba Cloud offered the lowest value for random IOPS on SAN disk on all tested VMs, due to its magnetic drives and cost of the VMs.

• AWS’s c4.large and m4.large (see Figure 5.2 A and B) offered lower price-performance value due to throttled IOPS after burst limits were

exceeded for the 100GB storage volume.

• Although CloudSigma used SSD technology for its storage volume, the price-performance value of those storage volumes tested are similar to Aruba Cloud and Azure volumes, which use magnetic disks.


24


Internal Network Throughput Figures 5.4 A through F illustrate the price-performance results for internal network throughput performance across all tested providers. Rows are

organized by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.

Observations: Internal Network Throughput Price-Performance

CloudSigma VMs deliver the highest available throughput out of all providers tested. Despite having higher prices than 1&1, the high network throughput performance resulted in a better CloudSpecs ScoreTM for CloudSigma. Similar with Processor & Memory price-performance,

57

13 16 19

100

0

10

20

30

40

50

60

70

80

90

100

4GB

c4.large

4GB A2

4GB


Clou

dSpe

cs Score

Internal Network Price-‐Performance: 2 vCPUs Figure 5.4 A

31

20 21 21

100

0 10 20 30 40 50 60 70 80 90 100

8GB

m3.large

8GB

D2

8GB


Clou

dSpe

cs Score

Internal Network Price-‐Performance: 2 vCPUs Figure 5.4 B

21

9 9 13

100

0 10 20 30 40 50 60 70 80 90 100

8GB

c4.xlarge

8GB A3

8GB


Clou

dSpe

cs Score

Internal Network Price-‐Performance: 4 vCPUs Figure 5.4 C

17 13 9

19

100

0 10 20 30 40 50 60 70 80 90 100

15GB

m3.xlarge

15GB

D3

15GB


Clou

dSpe

cs Score

Internal Network Price-‐Performance: 4 vCPUs Figure 5.4 D

48

25 18

27

100

0 10 20 30 40 50 60 70 80 90

100

15GB

c4.2xlarge

15GB

A4

15GB


Clou

dSpe

cs Score

Internal Network Price-‐Performance: 8 vCPUs Figure 5.4 E

36 26

19

39

100

0 10 20 30 40 50 60 70 80 90 100

30GB

m3.2xlarge

30GB

D4

30GB


Clou

dSpe

cs Score

Internal Network Price-‐Performance: 8 vCPUs Figure 5.4 F


25


Azure also achieved a better CloudSpecs ScoreTM for its D Series in regards to internal network throughput. The D series offered approximately 1.7x

more network throughput on VMs tested in the study.

Although AWS’s virtual machines increased throughput based on size and family, that increase did not surpass the price-performance ratio of 1&1 VMs’ throughput and cost; therefore, 1&1 achieved a higher CloudSpecs ScoreTM on network throughput than AWS.

Aruba Cloud, which offered higher throughput than AWS on smaller VMs (see Figure 4.4 A), displayed higher internal network throughput

price-performance than AWS until VM sizes scale up. As machine sizes increased, AWS’s throughput scaled while Aruba Cloud’s did not, which is

the reason AWS has higher CloudSpecs ScoresTM for the larger VMs. For these VMs, (see Figure 5.4 B and C), Aruba Cloud fell short of AWS, as well as all other competitors in the study.


• Azure’s D3 and D4 VMs exceeded a throughput of 1 GB/s (see Figures 4.4 B and C), giving it better price-performance value in internal

network than all other providers for those VM sizes with the exception of CloudSigma (see Figures 5.4 D and F).

• In all other VM sets, 1&1 achieves the highest CloudSpecs ScoreTM with the exception of CloudSigma for internal network.

• Aruba Cloud, which does not scale internal network throughput with the size of the VM (see Figure 4.4), drops in value ranking as VMs are

scaled up in size; other providers, such as AWS and Azure, which scale internal network throughput with the corresponding VM size and

family/series, surpass Aruba Cloud in value for internal network as the VMs are scaled up.


26


Conclusion

Selecting the right provider and virtual machines requires a thorough and accurate performance comparison. The study conducted for this

report offers a general understanding of performance and price-performance strengths and weaknesses across each included vendor’s VMs, and

should be considered a source of information to help guide readers in their own testing and analyses. The processor, memory bandwidth, network storage, and internal network were all examined and results clearly show that no single provider can offer everything to fit everyone’s needs.

Therefore, in order to accurately select the right provider for a business or an application, performance and price-performance analysis is crucial.

Results from this study show that, 1&1’s VMs displayed strong performance and price-performance. 1&1’s VMs demonstrated high performance and stability for processing and memory bandwidth both on the same VM, and also across VMs of the same size. Its network-attached

storage produced the largest amount of IOPS seen in the study as well, regardless of VM size and without the need for purchasing additional

volumes, sizing up, or provisioning IOPS. These high-performance results, combined with the low cost of the VMs, reflect on the value that 1&1’s

VMs can deliver to potential users, as seen with the price-performance results. 1&1’s performance and price-performance offers an excellent alternative for high-performance environments such as distributed file systems and data analytics processing.

Performance in the industry cannot be assumed to be equal or even similar, as illustrated in this report. When it comes to processor and

memory bandwidth performance, tiered providers such as AWS and Azure may offer varying performance depending on the family/series of the VM, despite having equivalent amounts of vCPUs and similar amounts of memory. For disk IOPS, although both 1&1 and CloudSigma advertise SSD

volumes, 1&1’s SSDs achieved 4.4 – 4.7x more IOPS than their CloudSigma counterparts. Internal network performance on CloudSigma, though,

exceeds all providers examined in the study.

While this study was conducted in the manner of understanding a typical end user experience, it should not be assumed to be accurate for all use cases. Stress testing was conducted to better understand fluctuation and theoretically sustained performance, and should be seen as a

general indication of provider performance. For more detailed analysis on any specific use case, please contact Cloud Spectator at

[email protected] or by phone at +1 (617) 300 0711.

About

About Cloud Spectator

Cloud Spectator is a cloud analyst agency focused on cloud Infrastructure-as-a-Service (IaaS) performance. The company actively monitors several

of the largest IaaS providers in the world, comparing VM performance (i.e., CPU, RAM, disk, internal network, and workloads) and pricing to achieve transparency in the cloud market. The company helps cloud providers understand their market position and helps business make intelligent

decisions in selecting cloud providers and lowering total cost of ownership. The firm was founded in early 2011 and is located in Boston, MA.

For questions about this report, to request a custom report, or if you have general inquiries about our products and services, please contact Cloud

Spectator at +1 (617) 300-0711 or [email protected].

For press/media related inquiries, please contact: Ken Balazs

VP Sales & Marketing

[email protected]

11Cloud-Spectator-Report_final.pdf

Documents

performance variability

performance output

throttled performance

unstable performance

performance matters

priceperformance analyses

detailed performance

cloud spectator team