This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Maximizing EC2 and Elastic Block Store Disk Performance
Todd Varland, Solutions Architect, AWS
What We’ll Cover- Maximizing EC2 and Elastic Block Store
Performance – Best Practices- As Measured by…- Configuration Options- Deployment Patterns- Tips and Best Practices
So.. what storage workloads?• Relational DB
• NoSQL DB
• Data warehousing
• Files
• Media
For most builders AWS is get in and go!
Other AWS choices we won’t cover
• Amazon RDS (managed RDBMS)
• Amazon DynamoDB (managed NoSQL)
• Amazon ElastiCache (in-memory caching service)
• Amazon Redshift (managed data warehouse)
• Amazon S3 (object store service)
• Amazon Glacier (archive storage service)
A “Normal” Hard Drive
EBS =
What is Amazon EBS?
• Very flexible service with lots of choice – Used with Amazon EC2 instances– Attach/detach/copy/delete volumes– Point-in-time snapshots of volumes -> Amazon S3– Automatically replicated within its Availability Zone to protect
from component failure– Paying a low price for only what you provision
Key Pieces
AmazonEC2
An I/O
AmazonEBS
Network link
Measured by
1. Throughput: Read/Write rate to storage (MB/s)
2. Latency: Delay between request and completion (ms)
3. Cost: How much to run this ($)
Tools available for tuning
1. EC2 Instance: Network transfer rate (Mbps)
2. EBS Optimized: EC2 instance option (On/Off)
3. PIOPS: Provisioned I/O operations per second (#)
4. Queue Depth: The number of outstanding I/Os (#)
cc2.8xlarge 32 NA 800MB/s 50,000 m2.xlarge 2 No 64MB/s 4000
m2.2xlarge 4 Yes 64MB/s 4000m2.4xlarge 8 Yes 128MB/s 8000 cr1.8xlarge 32 NA 800MB/s 50,000 hi1.4xlarge 16 NA 800MB/s 50,000 cg1.4xlarge 16 NA 800MB/s 50,000
Max 8k = 2x
Max 4k = 4x*
Max 2k = 8x*
*Maximum IOPS is also limited to ~100,000 per 32 vCpu, irrespective of block size/throughput.
EBS-Optimized• EBS-optimized offers a “SAN-like” experience
• Network interference results:
No impact on IOPS or Amazon EBS throughput
Row Labels AvgBW AvgIOPs
m3.2xlarge (EBS-optimized)
no network loadrandom
read 57,542 3,596
write 61,713 3,857
rw (70/30) 66,997 4,186
sequential
read 61,708 3,856
write 61,651 3,853
rw (70/30) 66,996 4,187
with network load-test1random
read 59,835 3,739
write 63,407 3,962
rw (70/30) 68,859 4,303
sequential
read 61,736 3,858
write 63,360 3,959
rw (70/30) 68,859 4,302
Network interference tests
No Difference
InThroughput
EC2 Instance
An I/O
AmazonEBS
• BandwidthJust because Amazon EC2 sends morework doesn’t mean there’s enough bandwidth to handle it!
AmazonEC2
EC2 Instance
An I/O
• Bandwidth Without more bandwidth, more Amazon EBS volumes or higher PIOPS won’t help!
AmazonEC2
❶ Select a new type of Provisioned IOPS volume
❸ Specify the number of I/O operations per second your application needs, up to 4000 IOPS per volume. The volume will deliver the specified I/O operations per second.
• If replicas on SSD instance types, disable integrity features such as fsync and full_page_writes on those hosts to improve performance
Performance – Extra-large Production Scale
Sta
ble
Testing Random 4 KB Reads
EBS PIOPS+ SSD
Performance / Stability Tips• Ext4 or XFS (understand journal impact!)
• nobarrier, noatime, noexec, nodiratime
• Raise file descriptor limits
• Set read-aheads low
• AWS business-level support – Trusted Advisor
• Amazon CloudWatch metrics in general
• SNAPSHOT SNAPSHOT SNAPSHOT
Pre-warming Amazon EBS volumes
• Typically 5%, extreme worst case of 50% performance reduction in IOPS and latency when volumes are used without pre-warming
– Performance is as provisioned when all the chunks are accessed
• Recommendation if testing or you have spare setup time:– Write to every 4 MB block before using new volumes
• Linux: DD • Windows: NTFS Full format
– Takes roughly an hour to pre-warm 1 TB 4 KB PIOPS volume– Be warned, can take up to a day for a 1 TB standard Amazon EBS volume
Architecting for Performance: Latency
• Performance requirements may be driven by IOPS or latency or both
• Recommendation is to start with queue depth of 4 and tune based on IOPS and latency requirement – Some customers may need lowest possible latency; this can be achieved at
queue depth of 1 or 2
• Very high queue depths ( >24) may decrease IOPS count as well as increase latency
Write Latency• Database applications care
about latency as much as IOPS delivered
• There is an interdependency among IOPS, queue depth, and latency
• Current guidance is queue depth of 1 for every 200 IOPS, but if latency-bound and write-heavy, 1:500 – 1:1000 is better.