Empirical Evaluation of Latency-Sensitive Application ...lass.cs.umass.edu/slides/2010/mmsys10.pdf · Game Server Latency 21! Server crippled without bandwidth controls (tc off)!

Department of Computer Science

Empirical Evaluation ofLatency-Sensitive Application

Performance in the Cloud

Sean Barker and Prashant ShenoyUniversity of Massachusetts Amherst

University of Massachusetts Amherst - Department of Computer Science

Cloud Computing

! Cloud platforms built with data centers: large-scale, concentrated servers clusters• Machines rented out to

companies or individuals• Hosting for arbitrary applications• May supplement local resources

! Cheap enough to rent machines by the hour

2

Type CPUs Memory Disk Cost/hr

Small 1 1.7 GB 160 GB $0.085

Large 4 7.5 GB 850 GB $0.34

XL 8 15 GB 1690 GB $0.68

Current prices on Amazon Elastic Compute Cloud (EC2)


Multimedia Cloud Computing Scenarios

! Clouds designed primarily for web & e-commerce apps, but may also be used for multimedia

! Rent game server for an evening• No firewall or bandwidth issues, only a few dollars

! Rent high-CPU machines for HD video transcoding• Home PC may take several hours to transcode one video,

cloud can transcode many in a fraction of this time

! Rent servers for webcast of live event• Large, inexpensive temporary bandwidth allocation

3

! Data center servers are typically well-equipped• Providers share individual

machines machines among multiple users

! Example: one user runs game server, another runs high-performance database on same machine

! Multimedia has unique performance requirements• Low latency games, low jitter & high bandwidth streaming

! Are cloud platforms designed for conventional web applications suitable for multimedia?


Resource Sharing in the Cloud

4

8 GB RAM

Core 1

Core 2

Core 3

Core 4

1000 GB Disk

1000 GB Disk

4 GB RAM

Core 1

Core 2

Core 3

Core 4

1000 GB Disk

1000 GB Disk

4 GB RAM


Outline

! Motivation

! Virtualized clouds

! Amazon EC2 study

! Laboratory cloud study

! Real world multimedia case studies

! Related work & conclusions

5


Virtualized Clouds

! Cloud platforms are virtualized data centers! Virtualization facilitates machine distribution

among multiple users with virtual machines (VMs)

6

VM

Hardware

VM VM

Game Server

Web Server

Media Server

Customer A

Users

Customer C

Customer B

! Each VM is assigned slice of physical resources! VM access to hardware managed by hypervisor• Enforces limits and isolates VMs from each other

! Are these resource sharing mechanisms suitable for the timeliness constraints of multimedia?

VM VM VM

AppA

App C

Users

App B

Hardware

Hypervisor


Virtual Machine Isolation

8

resourcestarvation

Hypervisor

VM VM VM

App A

Users

Hardware

App B App C


Outline

! Motivation

! Virtualized clouds

! Amazon EC2 study

! Laboratory cloud study

! Real world multimedia case studies

! Related work & conclusions

9


EC2 Study – Overview

! Amazon Elastic Compute Cloud (EC2)• Popular virtualized cloud platform

! Unknown applications coexisting on machine• No control over VM placement

! Goal: evaluate performance with unknown background server load

! Methodology: measured CPU, disk, and network consistency over period of days

10


EC2 CPU Performance

0

200

400

600

800

1000

1200

1400CPU time (ms)

Time (5 minute intervals)

EC2Local

11

• Volatility on EC2 vs stability on dedicated server

2.5x average outliers:

1.5-2x avg

no competing VMs: no outliers


EC2 Disk Performance

0

10000

20000

30000

40000

50000

60000

70000

80000

90000Long write time (ms)


EC2Local

12

• Similarly: inconsistent EC2 disk performance

widely fluctuatingdisk performance


EC2 Network Latency (LAN)

0

50

100

150

200

250First three hops latency (ms)


13

• Latency variations in EC2 LAN


EC2 Study – Summary

! Performance variations observed on EC2• Not observed on local server running a single VM

! Can only speculate on causes without access to the hypervisor

! Need to experiment on a controlled platform similar to Amazon’s

14


Laboratory Cloud Study – Overview

! Local cloud running the Xen hypervisor• Same virtualization technology used by EC2• Advantage: local cloud gives us control of interference

! Built-in mechanisms for sharing hardware between VMs• CPU credit scheduler• Round-robin disk servicing• Linux-level tool tc for network sharing

! How well do these tools isolate background work?

! Methodology: evaluated performance impact of competing VM

15


CPU Performance with Background Load

0

50

100

150

200CPU time (ms)

Time (5 second intervals)

16

• Default 1 to 1 sharing with variable background load

No background work: VM gets 100% CPU

Max background work: VM gets 50% CPU


Disk Performance with Background Load

0

20

40

60

80

100

1 2 3 4 8

Performance Impact (%)

Disk Thread Pairs on Collocated VM

Fair ShareSmall Read

Small WriteRead Throughput

Write Throughput

17

• Degraded by half over ‘fair’, but stable with increasing load

‘unfair’ impact


Laboratory Cloud Study – Summary

! Significant interference possible from background VMs

! Xen configuration can guarantee share of CPU• Default settings allow fluctuation in shared CPU

! Disk sharing less fair and harder to control• Consistent with observed EC2 behavior

! Network sharing effects evaluated in case studies on laboratory cloud (next)

18


Case Study 1 – Doom 3 Game Server

! Multiplayer Doom 3 game server

! Introduced controlled interference as before

! Measured map load times and server latency

! Network sharing configuration via tc:• Idle: No bandwidth usage by resource-hog VM• Off (default): No rate-limiting, network free-for-all• Shared: 50% (min) to 100% (max) of bandwidth per VM• Dedicated: 50% (max) of bandwidth per VM

19


Game Server Map Load

0

1000

2000

3000

4000

5000

Idle Disk CPU Disk + CPU

Average Server Load Time (ms)

Collocated VM Activity

20

• Interference produces up to 50% degradation


Game Server Latency

21

! Server crippled without bandwidth controls (tc off)

! Dedicated vs shared bandwidth:• Dedicated: lower latency, higher jitter• Sharing: higher latency, lower jitter

Configuration Avg. Latency (ms)

Std. Deviation (jitter) Timeouts

No interference 8.1 10.2 0%

tc off (free-for-all) N/A N/A 100%

tc, sharing b/w 33.9 16.9 2%

tc, dedicated b/w 23.6 29.6 7%


Case Study 2 – Darwin Streaming Server

! Streaming video to multiple clients

! Introduced controlled interference as before

! Measured sustained streaming bandwidth and stream jitter (latency variation)

! Varied tc settings and number of clients• Max video stream rate of 1 Mbps per client

22


Streaming Server Bandwidth

0

200

400

600

800

1000

idle (fair) off shared dedicated

average bitrate per stream (kbps)

tc sharing type

4 streams8 streams

23

• both tc configurations recovered bandwidth

decreased stream quality


Streaming Server Jitter

0

2

4

6

8

10

12

14

16

idle (fair) off shared dedicated

average stream jitter (ms)

tc sharing type

4 streams8 streams

24

• Jitter improved by shared, but worsened by dedicated


Real World Case Studies – Summary

! Real applications show substantial impacts from background interference

! Network is particularly vulnerable without administrative controls

! Proper configuration is important• CPU and network isolation tools fairly well-developed• Disk isolation needs better mechanisms

25


Related Work

! Fair-share schedulers and quality-of-service• Nieh and Lam (SOSP ‘97) for multimedia• Sundaram et al. (ACM MM ‘00) for QoS-aware OS

! Virtualization and hypervisors• Xen, VMware ESX Server

! Improving performance isolation• Gupta et al. (Middleware ‘06) for Xen mechanisms

! We focus on evaluation of existing mechanisms with specific attention to multimedia

26


Conclusions

! Clouds exhibit performance variations• Applications with timeliness requirements are

particularly sensitive

! Appropriate hypervisor configuration can help• In some cases, prevents resource starvation• Some resource sharing mechanisms need improvement

! Future work: evaluation of non-Xen platforms

! Questions?

27

Empirical Evaluation of Latency-Sensitive Application ...lass.cs.umass.edu/slides/2010/mmsys10.pdf · Game Server Latency 21! Server crippled without bandwidth controls (tc off)!

Documents