Maximizing EC2 and Elastic Block Store Disk Performance

© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Maximizing EC2 and Elastic Block

Store Disk PerformanceVinay Kumar, Sr. Product Manager, EBS

What We’ll Cover

Maximizing EC2 and Elastic Block Store

• Four key pieces

• Configuration Options

• Tips and Best Practices

For most builders AWS is get in and go!

What is Amazon EBS?

• Network block storage

• Designed for five nines of availability

• Attach to EC2 within the same AZ

• Point-in-time snapshots to S3

• Provision and pay just for what you need

EBS Volume Types

• General Purpose (SSD)

• Provisioned IOPS (SSD)

• Magnetic

When performance matters, use SSD-backed volumes

Amazon EBS General Purpose(SSD)• New default volume type for EBS

• Every volume can burst up to 3,000 IOPS

• Larger volumes can burst for longer periods

• 3 IOPS per GB baseline performance

• Maximum of 3,000 on a 1TB volume

• Throughput bursts up to 128MB/s per volume

• 99% performance consistency

• Latency: Single digit milliseconds

Sizing General Purpose (SSD)• For generic boot, developer, test/dev and web apps

– Just provision GB’s required for your applications

• For database apps

– Calculate your required IOPS needed in steady state

– Provision the GB = (Steady state IOPS) ÷ 3

• For example, customer requiring1500 steady state IOPS, provision 1500

÷ 3 = 500 GB

– IO bursts will support • Database load or table scan operations

• Spike in IO workload

7

Amazon EBS PIOPS (SSD)• Provision up to 4,000 IOPS per volume

• Supports IOPS to GB ratio of 30

• Designed to offers 99.9% consistency

• Throughput: up to 128MB/s per volume

• Latency: Single digit millisecond

• Best for IO intensive databases that require highest consistency

NEW

Amazon EBS Magnetic

• IOPS: ~100 IOPS steady-state, with best-effort bursts

• Throughput: variable by workload, best effort to 10s of MB/s.

• Latency: Varies, reads typically ~20-40 ms, writes typically ~2-10ms

• Best for cold workloads

EBS Volume Types (Cheat Sheet)General Purpose (SSD) Provisioned IOPS (SSD) Magnetic

Recommend Use CasesBoot Volumes Small to Med DBs Dev and test

I/O intensive Large DBs

Cold Storage

Storage Media SSD-backed SSD-backed Magnetic-backed

Volume Size 1GB- 1TB 1GB- 1TB 1GB- 1TB

Max IOPS per volume3 IOPS/GB Burst up to 3000

IOPS4,000 IOPS ~100 IOPS

Read and Write Peak Throughput

128 MB/s 128 MB/s ~50-90 MBps

Max I/O per node 48,000 48,000 ~ 1000-5000

Peak Throughput Node 800 MB/s 800 MB/s ~800MB/s

Latency (random read) 1-2ms 1-2 ms 20-40 ms

API Name gp2 io1 standard

Price* $.10/GB-Month$.125/GB-Month

$.065/provisioned IOPS$.05/GB-Month $.05/ 1M

I/O

Always use General Purpose (SSD) for boot volumes

Performance Optimization is Measured by

1. IOPS:

2. Latency:

3. Throughput:

Four Key Pieces

(1) Amazon EC2

(2) I/O

(4) Amazon

EBS

(3) Network

link

Tools available for tuning

1. EC2 Instance:

2. EBS Optimized:

3. Workload:

4. Queue Depth:

5. RAID:

6. Pre-warm:

1. EC2 Instance

Amazon

EBS

Amazon

EC2

Select EC2 instance that has the right Network, RAM and CPU

resources for your applications.

2. EBS-Optimized

• Offers “SAN like” experience

• All new instance family supports EBS-

optimized flag

• EBS-optimized now supports up-to 2Gb/s– Drive 16,000 16K IOPS or 256MB/s

• EC2 8XL instances support 10Gb/s network

• Max IOPS per node supported is ~100,000

IOPS @ 4K IO

* Max 2X for 8kB IO & * Max 4X for 4kB IO

b

Instance Type Dedicated EBS (Mbps)

Max 16K IOPS*

Max MB/s

c1.xlarge 1,000 8,000 128c3.xlarge 500 4,000 64c3.2xlarge 1,000 8,000 128c3.4xlarge 2,000 16,000 256g2.2xlarge 1,000 8,000 128i2.xlarge 500 4,000 64i2.2xlarge 1,000 8,000 128i2.4xlarge 2,000 16,000 256m1.large 500 4,000 64m1.xlarge 1,000 8,000 128m2.2xlarge 500 4,000 64m2.4xlarge 1,000 8,000 128m3.xlarge 500 4,000 64m3.2xlarge 1,000 8,000 128r3.xlarge 500 4,000 64r3.2xlarge 1,000 8,000 128r3.4xlarge 2,000 16,000 256r3.8xlarge NA 48,000 800c3.8xlarge NA 48,000 800i2.8xlarge NA 48,000 800

cc2.8xlarge NA 48,000 800

Use EBS-Optimized instances for consistent EBS

performance

3. Workload• I/O size

– 4 KB to 64 MB

• I/O pattern

– Sequential and random

EBS SSD-backed volumes measure IO size up to 256KB

EBS SSD-backed delivers same performance for read and write

• I/O type

– Read and write

• I/O concurrency

– Number of concurrent IO

NEW

IOPS and Throughput limits on EBS

4,000 IOPS

PIOPS volume

4,000 IOPS

128MB/s throughput

You can achieve 4,000 IOPS

when driving smaller IO

You can achieve up to 128MB/s

when driving larger IO

IOPS and Throughput limits on EBS

4,000 IOPS

PIOPS volume

4,000 IOPS

128MB/s throughput

4,000 X 4KB = 16MB/s

4,000 X 32KB =128MB/s

8,000 x 8KB = 64MB/s

4,000 x 8KB = 32MB/s

2,000 x 128KB=256MB/s

1,000 x 128KB =128MB/s

Block (IO) size determines whether your application is IOPS bound or

throughput bound

4. Queue Depth

An I/O

Amazon

EBSAfter it’s gone, it’s gone

Amazon

EC2

Queue depth is the pending IO in flight.

IO Latency

• Elapsed time between IO submission and it’s completion

time

• Performance requirements may be driven by IOPS or

latency or both

• There is an interdependency among IOPS, queue depth,

and latency

Random Read Latency

0.075

35.1

0

5

10

15

20

25

30

35

40

1 4 8 12 16 20 24 28 32

La

ten

cy T

P9

0 (

ms

)

Queue Depth

Random READ latency across various QDs

Latency (TP90)

Read latency linearly increases with increase in queue depth

Random Read Latency

0.075

35.1

2.09

1,865

4,152

3,851

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

5

10

15

20

25

30

35

1 4 8 12 16 20 24 28 32

La

ten

cy

TP

90

( m

s)

Queue Depth

16 KB random READ IOPS, Latency across various QDs

Latency (TP90) Avg Read IOPS

IOP

S

Queue depth of one has the lowest latency, but has the lowest IOPS

Random Read Latency

0.075

35.1

2.09

1,865

4,152

3,851

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

5

10

15

20

25

30

35

1 4 8 12 16 20 24 28 32

La

ten

cy

TP

90

( m

s)

Queue Depth

16 KB random READ IOPS, Latency across various QDs

Latency (TP90) Avg Read IOPS

IOP

S

Queue depth between 4-8 has the optimal IOPS and latency performance

Random Read Latency

0.075

35.1

2.09

1,865

4,152

3,851

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

5

10

15

20

25

30

35

1 4 8 12 16 20 24 28 32

La

ten

cy

TP

90

( m

s)

Queue Depth

16 KB random READ IOPS, Latency across various QDs

Latency (TP90) Avg Read IOPS

IOP

S

Higher queue depths have negative impact on IOPS and latency

Random Write Latency

0.08

7.71

845

4,152

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

1

2

3

4

5

6

7

8

9

10

1 4 8 12 16 20 24 28 32

La

ten

cy

TP

90

( m

s)

Queue Depth

16 KB random WRITE IOPS, Latency across various QDs

Latency (TP90) AvgIOPS

IOP

S

Write latency, queue depth and IOPS interaction is similar to

that of read latency

Optimal queue depth to achieve lower latency and highest IOPS is

between 4-8; 1 QD per 500 IOPS

EBS-Optimized offers consistent latency experience

5. RAID

Amazon

EBS

Amazon

EC2

Amazon

EBS

Amazon

EC2

Maximum performance per instance

• How should you think about taking snapshots on a striped volume?

– Quiesce file systems and take snapshot

– Unmount file system and take snapshot

– Use OS-specific tools

12×400 GB PIOPS, pre-warmed, RAID 0 LVM, Stripe size 64 KB, attached to CR1 instance

Use stripe size of 128KB or 256KB

6. Pre-warming• First write penalty for new volumes

• Lazy loading for volumes created from snapshots

• Typically 5%, extreme worst case of 50% performance reduction in IOPS and

latency when volumes are used without pre-warming – Performance is as provisioned when all the chunks are accessed

• Recommendation if testing or you have spare setup time:– For new volumes

• Linux: DD write

• Windows: NTFS Full format

– Takes roughly an hour to pre-warm 1 TB 4 KB PIOPS/General Purpose (SSD) volumes

– Be warned, can take up to a day for a 1 TB standard Amazon EBS volume

Use large block size to speed up your pre-warming.

Ex: sudo dd if=/dev/xvdf of=/dev/xvdf conv=notrunc bs=1M

Performance / Stability Tips

• Ext4 or XFS (understand journal impact!)

• EBS encryption has no visible IOPS

performance impact for many EC2 instances

Workload/

Software

Typical

block size

Random

/Seq?

Typical

correct

EBS type

Max EBS @

500 Mb/s

Max EBS @

1 Gb/s

Max EBS @ 10Gb/s

instances

Oracle DB Configurable:2 KB –

16 KB

Default 8 KB

random PIOPS/GP2 RAID 0, 2 × 4000

PIOPS:

~7800 IOPS

RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 24 × 4000 PIOPS:

~96,000 IOPS

Microsoft

SQL Server

8 KB w/ 64 KB

extents

random PIOPS/GP2 RAID 0, 2 × 4000

PIOPS:

~7800 IOPS

RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 20 × 4000 PIOPS:

~80,000 IOPS

MySQL 16 KB random PIOPS/GP2 1 x 4000 PIOPS:

~4000 IOPS

RAID 0, 2 × 4000

PIOPS:

~7,800 IOPS

RAID 0, 12 × 4000 PIOPS:

~48,000 IOPS

PostgreSQL 8 KB random PIOPS/GP2 RAID 0, 2 × 4000

PIOPS:

~7,800 IOPS

RAID 0, 4 × 4000

PIOPS:

~15,600 IOPS

RAID 0, 24 × 4000 PIOPS:

~96,000 IOPS

MongoDB 4 KB serialized PIOPS/GP2 RAID 0, 4 ×4000 PIOPS:

~15,600 IOPS

RAID 0, 8 × 4000

PIOPS:

~31,000 IOPS

RAID 0, 24 × 4000

PIOPS:

~96,000 IOPS

Cassandra 4 KB random PIOPS/GP2 RAID 0, 4 ×4000 PIOPS:

~15,600 IOPS

RAID 0, 8 × 4000

PIOPS:

~31,000 IOPS

RAID 0, 24 × 4000

PIOPS:

~96,000 IOPS

GlusterFS 128 KB sequential PIOPS/GP

21 × 4000

PIOPS:

~500 IOPS

RAID 0, 2 × 4000

PIOPS:

~1000 IOPS

RAID 0, 12 × 4000

PIOPS:

~6,000 IOPS

(768MB/s)

Cheat Sheet Sample! Storage Workloads on AWS

EBS-optimized

Four Key Pieces: Balanced Oh, YEAH!!

Amazon

EC2

A “boatload” of I/O

Right sized EBS

Tools available for tuning

1. EC2 Instance:

2. EBS Optimized:

3. Workload:

4. Queue Depth:

5. RAID:

6. Pre-warm:

© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Maximizing EC2 and Elastic Block

Store Disk PerformanceVinay Kumar, Sr. Product Manager, EBS

Thank you!