Transcript
UC Berkeley
Evaluating Amazon’s EC2 As A Research Platform
Michael Armbrust and Gunho Lee
1
2
RAD Lab Overview
High
level
spec
Low
level
spec
Drivers Drivers Drivers
New apps, equipment, global policies (eg SLA)
Com- piler
Offered load, resource
utilization, etc.
Instru
menta
tion B
ackpla
ne
Training data
Ruby on Rails environment
VM monitor
local OS functions
trace collection
web svc APIs
Web 2.0 apps
Berkeley DB
local OS functions
trace collection
SCADS
Au
tom
atic
Work
loa
d
En
gin
e
Director
performance & cost
models
Log Mining
Policy-aware switching
Overview of EC2
Platform Units Memory Disk
Small - $0.10 / hour 32-bit 1 1.7GB 160GB
Large - $0.40 / hour 64-bit 4 7.5GB 850GB – 2 spindles
X Large - $0.80 / hour 64-bit 8 15GB 1690GB – 4 spindles
High CPU Med - $.20 64-bit 5 1.7GB 350 GB
High CPU Large-$.80 64-bit 20 7GB 1690 GB
One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.
•! Elastic virtual machine capacity •! Programmatically controlled through web service API •!~5 minutes to allocate new machines •! Charges for machine time and data transfer in/out of datacenter
Appeal
•! Automatically dynamically scale your computing resources as demand changes
•! For researchers:
–!Run experiments for $100 per 1000 machine hours!
–!AMIs provide simple containment of experimental setup and allow for easy verification
Rapid Changes
•! Changing machines and processor types
•! New features
–! Availability Zones
–! Elastic IP Addresses
–! Persistent Storage
Reverse Engineering
•! How much hardware do they have?
–!Not as much as we expected
–!Seems to be changing
•! What have we seen in our tests?
–!402 VMs
–!379 Physical Machines
–!Overlap of as many as 7 VMs per machine
Goal
•! Characterize the performance and variance of different aspects of EC2
–!Clock Accuracy
–!CPU
–!Memory Bandwidth
–!Disk
–!Network Throughput / Latency
•! Provide recommendations to researchers hoping to use EC2
Benchmark Architecture
•! Suite of benchmarks controlled from the R Cluster –! SSH to machines –! Installed required binaries –! Collect and record results to MySQL
Usage: util/runTests.rb [options] -r, --reservation RESERVATION_ID Include the specified reservation in tests
-d, --disk SIZE Execute a disktest of specified GBs -c, --cpu_info Caputure CPU Info
-t, --trace_path FILE Run traceroute and capture to file (for graphing) -n, --network N Perform a network throughput test between random pairs of
nodes N times
-s, --skewserver SERVER Install and start the skew monitor in the background reporting into SERVER
-m, --memorytest Perform the stream memory benchmark -x, --detectcap Attempt to detect Xen CPU Capping
Clock Accuracy
•! Time difference between EC2 & local server
•! Frequency (1~3 times / hour) and magnitude (~200ms) of clock skews seen by many machines over some period
CPU Performance
•! EC2 uses Xen CPU caps to provide fairly consistent performance across different processor type
•! Programs may experience scheduling artifacts with small or latency sensitive computation
CPU Performance
Memory Bandwidth
•! Used Stream Memory Benchmark
•! Low variation between machines of the same type
–! Std Dev < 55 MB/s
•! Very high between different processor types
Disks – Warm Up Effect
•! When first using /mnt (ephemeral storage) there are significant allocation performance artifacts
•! Oddly, these seem to correlate with the processor type
Disks – Warm Up Effect
Disks – Long Term Performance
•! Overtime aggregate disk performance is more consistent •! Over 90% of writes occur at 40mb/s or greater •! Average 54.88mb/s, Standard Deviation 9.05
Individual Performance
•! There is a much greater discrepancy between individual machines
•! Most likely due to collocation with other disk intensive customers
Original Topology
•! Originally we saw three distinct sections of their network
New Topology
Network Quirks and Latency
•! 96% RTT < 1ms
•! Occasional route changes lead to weird paths and increased latency
•! Current EC2 bug limits the number of concurrent connections that can be established cluster wide
–!Expected fix in a few weeks
Network Performance
Long Term Network
Long Term Network
Large – CPU
•! No CPU cap
–!Slower CPU, but full speed
Large – Disk
•! Similar performance
Large – Network
•! 2x performance at 4x cost
Small vs. Large
•! Small instance shows statistically better performance/$ –! Due to lack of Xen I/O caps coupled with low overall
cluster utilization
•! Large instance has higher network bandwidth –! Explicitly specified –! Not as high as it costs
•! No CPU caps for large instance –! Seems like not sharing physical core with other
instances –! No future guarantee
Evaluation for Research
•! Typical experimental environment for (distributed) system research
•! Single machine (simulation)
•! Private cluster (many nodes)
•! Emulab (network topology emulation)
•! PlanetLab (geographically distributed)
–! Many experiments on private cluster and Emulab can be done better on EC2
Size Cost Usability Control
Private Cluster Small Expensive Easy Full
Emulab / PlanetLab Medium Free Hard Partial
EC2 Large Cheap Easy Little
Evaluation for Research
•! EC2 is useful when you do…
•! Experiment on “large” cluster
-! distributed storage/computation/service…
•! Computation with “power” of large cluster
-! computational biology, vision, ML…
•! Research on virtualized environment just like EC2, of course
-! cloud computing…
Recommendations
•! Design for I/O variability
–!Do disk writes in bulk in the background
–!Be topology aware
–!Dynamically redistribute work
•! Be aware of the difference between small & large instances
–!Overall, better performance/$ for small
–!Some advantages for large
Q&A
•! Thanks!
top related