AWS re:Invent 2016: Optimizing Network Performance for Amazon EC2 Instances (CMP315)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Brian Barrett, Principal Software Engineer, AWS

Nick Matthews, Partner Solutions Architect, AWS

November 2016

CMP315

Optimizing Network Performance

for Amazon EC2 Instances

What to Expect from the Session

• High Performance networking applications

• Enhanced networking

• Deep dive

• Performance

• What next?

• Elastic Network Adapter

• What is ENA?

• Driver details

• Customer benefits

Networking on AWS

AWS proprietary networking

• Highest performance in largest EC2 instance sizes

• Full bi-section bandwidth in placement groups, with no

network oversubscription

Enhanced Networking

• Over 1 MPPS performance, reduced instance-to-instance

latencies

• Supporting Intel SR-IOV for consistent network

performance in a virtualized environment

• Amazon Elastic Network Adaptor – new for 2016

Traditional EC2 Networking

Instance

Virtualization LayerX

en P

V

Xen P

V

eth0 eth1

Traditional EC2 Networking

Instance

Virtualization LayerX

en P

V

Xen P

V

eth0 eth1

Enhanced Networking (Intel 82599)

Instance

Virtualization Layer

PCI VFeth0

eth1 PCI VF

Intel

82599

Enhanced Networking (Intel 82599)

Instance

Virtualization Layer

PCI VFeth0

eth1 PCI VF

Intel

82599

Elastic Network Adapter (ENA)

• New PCI network device developed for EC2

• Supported on X1, P2, and m4.16xlarge instance types

• Ability to scale across variety of bandwidths

• 10 and 20 Gbps instance types today

• Device interface supports up to 400 Gbps networking

• Increased number of request queues per device

• Improved debugging features

Enhanced Networking Latency Improvements

0

50

100

150

200

250

300

tp50 tp99

cc2.8xlarge c3.8xlarge c4.8xlarge m4.16xlarge

TCP Bandwidth

0

5000

10000

15000

20000

25000

64 128 256 512 1024 2048 4096 8192

Mb

ps

TCP Packet Size

cc2.8xlarge c3.8xlarge m4 m4.16xlarge

TCP Bandwidth

0

5000

10000

15000

20000

25000

64 128 256 512 1024 2048 4096 8192

Mb

ps

TCP Packet Size

cc2.8xlarge c3.8xlarge m4 m4.16xlarge m4.16xlarge Multistream

ENA Driver Support

• Supported in Amazon Linux out of the box

• Linux and Windows drivers available

• Linux: https://github.com/amzn/amzn-drivers

• Windows Server 2008 R2, 2012, 2012 R2 support

• Part of Linux 4.9

• DPDK support as of 16.04, recommend 16.11

• FreeBSD driver coming soon

High Performance Network Application

Communications

• Global communications company providing real time

communications to 1M developers

• Routing calls between AWS regions is done internally

and requires low latency

How does Twilio benefit?

• HVM instances that allows them to have access to

physical network cards getting 10 Gbps

• Enhanced networking allows them to achieve high PPS

that helps them in getting less variance in packet

processing times

• Quicker call routing and happy customers

• Distributed application with thousands of TCP connections between

instances

• Migrated Clash of Clans from EC2 Classic to VPC

• 100% on Enhanced Networking instances

Internet

Front-end

Proxy Layer

Application

Service Layer

Storage Layer

Supercell

How did Supercell benefit?

• In EC2 Classic, hundreds of servers each with thousands

of TCP connections took 30 minutes to register, discover

services, and establish TCP connections

• With Enhanced Networking on VPC, it takes less than 1

minute

• TCP reliability

• Monitoring

• Database connections

• Message queue depths dramatically smaller

Getting players back in the game

Network maintenance and

update downtimes are 20

minutes instead 1 hour

Cisco Cloud Services Router

• Same router experience on AWS or on-premises

• Advanced features for VPN, routing, and security

AWS Marketplace offering

Cisco Transit VPC

• Connect multiple VPCs

• Centralize connectivity

• Routers terminate VPN and

pass all traffic

• Fully automated using tags

See a demo at booth #200

https://aws.amazon.com/answers/transit-vpc/

Cisco benefits from Enhanced Networking

CEF Forwarding: 100%+ increase to 4.6 Gbps

IPsec forwarding: 100%+ increase to 4.2 Gbps

Enhanced Networking Requirements

• Supported instance types:

• ixgbe: C3, C4, D2, I2, M4 (except m4.16xlarge), R3

• ena: m4.16xlarge, P2, X1

• AMI requires either ixgbevf or ena driver

• ixgbevf version 2.14.2 or later

• Any ENA release

• AMI tag to enable Enhanced Networking

Instance Enablement

% aws ec2 describe-instance-attribute \

--instance-id i-0ec179afafe026710 \

--attribute sriovNetSupport

{

"InstanceId": "i-01286b75ff50326ea",

"SriovNetSupport": {}

}

Not yet!

Instance Enablement

% aws ec2 describe-instance-attribute \

--instance-id i-07312ca8e93d69514 \

--attribute sriovNetSupport

{

"InstanceId": "i-07312ca8e93d69514",

"SriovNetSupport": {

"Value": "simple”

}

}

82599

Support!

Instance Enablement

% aws ec2 describe-instances

--instance-id i-07a94b1806d6cd309 \

--query "Reservations[].Instances[].EnaSupport”

[

true

] ENA

Support!

AMI Check

• 82599 Enhanced Networking:% aws ec2 describe-image-attribute --image-id ami_id \

--attribute sriovNetSupport

• ENA Enhanced Networking:% aws ec2 describe-image-attribute --image-id ami_id \

--attribute enaSupport

Instance Enablement

1. Start an instance without Enhanced Networking

support

2. Build the ENA kernel driver

3. Build a new AMI or modify the instance# aws ec2 modify-instance-attribute \

--instance-id instance_id --ena-support

Or

# aws ec2 register-image --ena-support ...

4. Restart / relaunch and enjoy!

More info: https://docs.aws.amazon.com

Yes![ec2-user ~]$ ethtool -i eth0

driver: vif

version:

firmware-version:

bus-info: vif-0

supports-statistics: yes

supports-test: no

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: no

Is this thing on (ixgbe)?

No [ec2-user ~]$ ethtool -i eth0

driver: ixgbevf

version: 2.14.2

firmware-version: N/A

bus-info: 0000:00:03.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: yes

supports-priv-flags: no

Yes![ec2-user ~]$ ethtool -i eth0

driver: vif

version:

firmware-version:

bus-info: vif-0

supports-statistics: yes

supports-test: no

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: no

Is this thing on (ena)?

No [ec2-user ~]$ ethtool -i eth0

driver: ena

version: 0.6.6

firmware-version:

bus-info: 0000:00:03.0

supports-statistics: yes

supports-test: no

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: no

Thank you!

Remember to complete

your evaluations!