What is AWS Direct Connect…
• Dedicated, private pipes into AWS
• Create private (VPC) or public interfaces to AWS
• Cheaper data-out rates than Internet (data-in still free)
• Consistent network performance compared to Internet
• Multiple AWS accounts can share a connection
Why use AWS Direct Connect?
$0.000
$0.050
$0.100
$0.150
First 10TB Next 40TB
Next 100TB Next
350TB Direct Connect
Internet
Public Subnet
Availability Zone A
Private Subnet
Public Subnet
Availability Zone B
Private Subnet
Instance A 10.1.1.11 /24
Instance C 10.1.3.33 /24
Instance B 10.1.2.22 /24
Instance D 10.1.4.44 /24
VPC CIDR: 10.1.0.0 /16
Virtual Private
Gateway (VGW)
Internet Gateway
(IGW)
Only 1 IGW and 1 VGW per VPC
VPN connection Customer
data center Customer
data center
AWS Direct Connect
Route Table
Destination Target
10.1.0.0/16 local
Internal CIDR VGW
Direct Connect – Single Link, Single CGW
VPC 1 10.1.0.0/16 AWS Direct
Connect Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet Service Provider
Network ` `
VGW AWS
Network DX POP Location
Cross Connect
Customer Gateway Router
Circuit to Customers
Network
Customers Network
Backbone
Circuit to Customers
Site
Customer Provider
Edge Router Customers
Local Network
Demarcation
Direct Connect – Single Link, Single CGW
VPC 1 10.1.0.0/16 AWS Direct
Connect Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet Service Provider
Network ` `
From - To
CGW to VGW VGW to CGW
Route
172.16.0.0/16 10.1.0.0/16
Metric
- -
eBGP From - To
Customer to CGW CGW to Customer
Route
172.16.0.0/16 10.1.0.0/16
Routing – Probably eBGP
Layer 2 VLAN Connectivity
BGP is a requirement for Direct Connect: http://aws.amazon.com/directconnect/faqs/
VLAN Y
VLAN X
VIFs
virtual private cloud 1
virtual private cloud 2
virtual private cloud N
…
public endpoints
Region Direct Connect Location
private VIF 1 public virtual interface (VIF)
private VIF 2 VLAN Z
VLAN N
AWS DX Router
Customer Router
Each interface can be associated with a different
AWS Account. (Hosted Virtual Interfaces)
Direct Connect – Single Link, Single CGW
VPC 1 10.1.0.0/16 AWS Direct
Connect Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet Service Provider
Network ` `
With IPSEC Failover
Direct Connect – Single Link, Single CGW
VPC 1 10.1.0.0/16 AWS Direct
Connect Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet Service Provider
Network ` `
IPSEC over The Internet
From - To
CGW to VGW VGW to CGW
Route
172.16.0.0/16 10.1.0.0/16
Metric
- -
eBGP
With IPSEC Failover
Direct Connect – Dual Links, Single CGW
VPC 1 10.1.0.0/16 AWS Direct
Connect Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet Service Provider
Network ` `
Direct Connect – Dual Links, Single CGW
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet Service Provider
Network ` `
Direct Connect – Dual Links, Single CGW
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet Service Provider
Network ` `
From - To
CGW to VGW VGW to CGW
Route
172.16.0.0/16 10.1.0.0/16
Metric LP 150
eBGP
From - To
CGW to VGW VGW to CGW
Route
172.16.0.0/17 10.1.0.0/16
Metric LP 90
eBGP
- You can split your route advertisements to the VGW - Instead of using AS Path Prepend
CGW to VGW 172.16.128.0/17
Direct Connect – Dual Links, Dual CGW
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet Service Provider
Network ` `
Direct Connect – Dual Links, Dual CGW
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet
`
Service Provider Network
`
Direct Connect – Dual Links, Dual CGW
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet
` `
`
Service Provider Network
Direct Connect – Dual Links, Dual CGW
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet
`
`
Service Provider Network
`
Direct Connect – Dual Links, Dual CGW
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet
`
`
Service Provider Network
`
BGP AS - Y
BGP AS - X iBGP between Routers iBGP between Routers
Direct Connect – Dual Links, Dual CGW
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet
`
`
Service Provider Network
`
- So far so good? - What’s wrong with this topology?
- SPoF!
Direct Connect – Dual Links, Dual CGW
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location
Customer Edge Router
172.16.0.0/16
Customers DC Customers Subnet
`
`
Service Provider Network
`
Direct Connect – Dual Locations, Dual Links
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location - 1
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet
`
`
Service Provider Network
`
AWS Direct Connect
Customer Gateway
Colocation
`
DX Location - 2
VPC 1 10.1.0.0/16
AWS Direct Connect
Customer Gateway
Colocation
DX Location - 1
Customer Edge Router
172.16.0.0/16
Customers DC Customers
Subnet
`
`
`
AWS Direct Connect
Customer Gateway
Colocation
`
DX Location - 2
DirectConnect – Dual Locations, Dual Links, Dual Routers
Service Provider Network
Multi Account DX
VPC 1 10.1.0.0/16
Customer Gateway
VLAN320 Private VI
AWS Direct Connect
Ethernet Trunk
SVI/Sub 320
IP 169.x.x.2 IP 169.x.x.1
BGP AS 65xxx BGP AS 17493
VLAN 320
Colocation
Multi-Account Direct Connect
Customer Gateway
VLAN320
Ethernet Trunk
SVI/Sub 320
IP 169.x.x.2
BGP AS 65xxx
VPC 1 10.1.0.0/16
Private VI
IP 169.x.x.1
BGP AS 17493
VLAN 320
Colocation
VPC 2 10.2.0.0/16 IP 169.y.y.1
BGP AS 17493
VLAN 330
VLAN330
SVI/Sub 330
IP 169.y.y.2
BGP AS 65xxx
Private VI
AWS Direct Connect
AWS Account 1
Multi-Account Direct Connect
Customer Gateway
VLAN320
Ethernet Trunk
SVI/Sub 320
IP 169.x.x.2
BGP AS 65xxx
VPC 1 10.1.0.0/16
Private VI
IP 169.x.x.1
BGP AS 17493
VLAN 320
Colocation
VPC 2 10.2.0.0/16 IP 169.y.y.1
BGP AS 17493
VLAN 330
VLAN330
SVI/Sub 330
IP 169.y.y.2
BGP AS 65xxx
Private VI
AWS Direct Connect
AWS Account 1
AWS Account 2
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
Avai
labi
lity
Zone
B
Choose your VPC address range • Your own private, isolated
section of the AWS cloud • Every VPC has a private IP
address space • That maximum CIDR block
you can allocate is /16 • For example 10.0.0.0/16 –
this allows 256*256 = 65,536 IP addresses
Select IP addressing strategy • You can’t change the VPC
address space once it’s created
• Think about overlaps with other VPCs or existing corporate networks
• Don’t waste address space, but don’t’ constrain your growth either
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24 10.0.5.0/24 10.0.4.0/24
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
10.0.5.0/24 10.0.4.0/24
EC2 App
Log
EC2 Web
Bastion
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
10.0.5.0/24 10.0.4.0/24
EC2 App
“Web servers can connect to app servers on port 8080”
Log
EC2 Web
Bastion
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
10.0.5.0/24 10.0.4.0/24
EC2 App
“Web servers can connect to app servers on port 8080”
“Allow outbound
connections to the log server”
Log
EC2 Web
Bastion
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
10.0.5.0/24
Bastion
10.0.4.0/24
EC2 App
“Web servers can connect to app servers on port 8080”
“Allow outbound
connections to the log server”
“Allow SSH and ICMP from
instances in the Bastion security
group”
Log
EC2 Web
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
10.0.5.0/24 10.0.4.0/24
EC2 App
Log
EC2 Web
Security groups • Operate at the instance level • Supports ALLOW rules only • Are stateful • Max 50 rules per security group • Max 5 groups per instance
Bastion
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
Router 10.0.5.0/24 10.0.4.0/24
EC2 App
Log
EC2 Web
Bastion
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
Router 10.0.5.0/24 10.0.4.0/24
EC2 App
Log
EC2 Web
“Deny all traffic between the web server subnet and the database server subnet”
Bastion
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
Router 10.0.5.0/24 10.0.4.0/24
EC2 App
Log
EC2 Web
Bastion
NACLs are optional • Applied at subnet level • Stateless and permit all by default • ALLOW and DENY • Applies to all instances in the subnet • Use as guard rails (port 21, 135,…)
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
Router 10.0.5.0/24 10.0.4.0/24
EC2 App
Log
EC2 Web EC
2 Web
Elastic Load Balancer
Bastion
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
10.0.3.0/24
EC2
Router 10.0.5.0/24 10.0.4.0/24
EC2 App
Log
EC2 Web EC
2 Web
Elastic Load Balancer
Bastion
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
EC2
10.0.3.0/24
EC2
Router 10.0.5.0/24 10.0.4.0/24
EC2 App
Log
EC2 Web EC
2 Web EC
2 EC2 Web
Elastic load balancers • Instances can automatically be
added and removed from the balancing pool using rules
• You can add instances into security groups at launch time
Elastic Load Balancer
Auto scalin
g
Bastion
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
EC2
EC2
EC2 App
EC2 Web EC
2 Web EC
2 EC2 Web
Internet Gateway
VPC Router
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
EC2
EC2
EC2 App
EC2 Web EC
2 Web EC
2 EC2 Web
Internet Gateway
VPC Router
Internet routing • Add route tables to subnets to
control Internet traffic flows – these become Public subnets
• Internet Gateway routing allows you to allocate a static Elastic IP address or use AWS-managed public IP addresses to your instance
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
EC2
EC2
EC2 App
EC2 Web EC
2 Web EC
2 EC2 Web
Internet Gateway
VPC Router
Internet routing • Use a NAT instance to
provide Internet connectivity for private subnets - required to access AWS update repositories
• This will also allow back-end servers to route to AWS APIs – for example storing logs on S3, or using Dynamo, SQS, SNS and SWS
NAT
VPC A - 10.0.0.0/16
Avai
labi
lity
Zone
A
10.0.1.0/24
10.0.2.0/24
EC2
10.0.3.0/24
EC2
NAT
10.0.4.0/24
EC2 App
EC2 Web EC
2 Web EC
2 EC2 Web VPC
Router
Amazon S3
DynamoDB
Amazon SNS
Amazon SQS
Internet Gateway
NAT
To NACL or not to NACL? Pros
Another layer of defense Can speed up deals
• Fits legacy IT models • Network/FW Engineer’s friend
Can help with networking compliance • Separate groups for SGs/NALCs
Explicit deny rules Apply to an entire subnet
Cons Adds complexity Can slow down adoption
• Fits legacy IT processes • DevOps Enemy
Potentially not necessary for compliance
• Third-party proactive controls • SG audits (programmable infra)
Stateless FW rules Apply only to subnets/CIDR addresses
Routing Instances Love Them
NAT instances VPN tunnels (between VPCs) Data loss prevention
Intrusion detection
Hate Them Single point of failure Extra costs (EC2, third-party licenses) More for customer to manage
Potential network bottleneck
AWS region
Public-facing web app
Internal company app
What’s next?
VPN connection
Customer data center
Multiple VPCs
Public-facing web app
Internal company app #2
HA pair VPN endpoints Customer data center
Internal company app #3
Internal company app #4
Internal company app #1
Internal company Dev
Internal company QA
AWS region
Backup AD, DNS Monitoring Logging
Multiple VPCs over IPSEC VPN
Public-facing web app
Internal company app #2
HA pair VPN endpoints Customer data center
Internal company app #3
Internal company app #4
Internal company app #1
Internal company Dev
Internal company QA
AWS region
Backup AD, DNS Monitoring Logging
Multiple VPCs over AWS Direct Connect
Direct Connect Facility
Customer Data Center
Physical Connection
Logical Connections
VLANs
Logical Connections
VLANs
• Security groups and NACLs still apply
AWS region
Public-facing web app
Internal company app #1
HA pair VPN endpoints
company data center
Internal company app #2
Internal company app #3
Internal company app #4
Services VPC
Internal company Dev
Internal company QA
AD, DNS
Monitoring Logging
• Security groups still bound to single VPC
Multiple VPCs over VPC Peering
10.1.0.0/16
10.0.0.0/16
• VPCs within same region Peer
Request
Peer Accept
• Same or different accounts
• IP space cannot overlap
• Only 1 between any 2 VPCs
VPC peering configuration
10.0.0.0/16 10.0.0.0/16
PCX-1 PCX-2
Subnet 1
10.1.1.0/24 Subnet 2
10.1.2.0/24
10.1.0.0/16 Route Table Subnet 1
Destination Target
10.1.0.0/16 local
10.0.0.0/16 PCX-1
Route Table Subnet 2
Destination Target
10.1.0.0/16 local
10.0.0.0/16 PCX-2
A
B C
10.1.0.0/16
10.4.0.0/16 10.0.0.0/16
10.3.0.0/16
172.16.0.0/16 192.168.0.0/16
10.2.0.0/16
172.17.0.0/16
C A
10.1.0.0/16
10.4.0.0/16 10.0.0.0/16
10.3.0.0/16
172.16.0.0/16 192.168.0.0/16
10.2.0.0/16
172.17.0.0/16
company data center
10.10.0.0/16
10.1.0.0/16
10.4.0.0/16 10.0.0.0/16
10.3.0.0/16
172.16.0.0/16 192.168.0.0/16
10.2.0.0/16
172.17.0.0/16
company data center
10.10.0.0/16
10.4.0.0/16 10.0.0.0/16
172.16.0.0/16 192.168.0.0/16 172.17.0.0/16
10.1.0.0/16 10.2.0.0/16 10.3.0.0/16
Peer review
• Shared infrastructure services moved to VPC
• 1 to 1 peering = app isolation
• Security groups and NACLs still apply
AWS region
Public-facing web app
Internal company app #1
HA pair VPN endpoints
company data center
Internal company app #2
Internal company app #3
Internal company app #4
Services VPC
Internal company Dev
Internal company QA
AD, DNS
Monitoring Logging • Security groups still bound to
single VPC
Multiple accounts
CloudTrail – Log & monitor these!
• API actions with potential impact – Internet Gateway – Routes and Route Tables – Network ACLs – EC2 instances (run/create/launch/terminate) – Security Groups – CloudTrail (stop/delete/update) – Put[Group/Role/User]Policy – ModifyAccount – ModifyBilling, ModifyPaymentMethods – "Type":"Root" – Create[User/Role/Group] – CreateAccessKey
Continuous Change Recording Changing Resources
AWS Config History
Stream
Snapshot (ex. 2014-11-05) AWS Config
segregate duties
With AWS IAM you get to control who can do what in your AWS environment and from where Fine-‐grained control of your AWS cloud with two-‐factor authen;ca;on Integrated with your exis;ng corporate directory using SAML 2.0 and single sign-‐on
AWS account owner
Network management
Security management
Server management
Storage management