Simplifying the network stack with Romana Pani Networks OpenStack Meetup, Auckland, May 2016
Simplifying the network stack with Romana
Pani NetworksOpenStack Meetup, Auckland, May 2016
romana.io Simplifying the network stack with Romana @romanaproject
Agenda
● “Cloud native”, why does it matter?● A better network for cloud native architectures● Demos
romana.io Simplifying the network stack with Romana @romanaproject
About us
● Team background:– Data center networks
– Low-level traffic management
● Created L2 overlay network startup– Bought by Cisco
● OpenStack networking● There's got to be a better way
– Time is right
What is 'cloud native'?
romana.io Simplifying the network stack with Romana @romanaproject
The past: Enterprise networking
● Full control● Applications need L2 and L3
– May need hard-wired IP addresses
– Broadcasts
● Servers are pets, not cattle: “Careful!”– VM migration
● Complex!
romana.io Simplifying the network stack with Romana @romanaproject
Cloud native applications
● Automate all the things!– Infrastructure as code
– Cattle, not pets: “Meh... just kill it.”
– Workloads come and go quickly
– Build for resiliance
● IP is all you need– No hardcoded IP addresses, discovery
– No special network requirements
– Basic IP connectivity
The problem
romana.io Simplifying the network stack with Romana @romanaproject
We have a mismatch
● Building cloud native applications…● … on top of enterprise networking
– SDN controllers use overlay L2 domains
– VLAN, VXLAN, OVS, etc.
● Complexity and brittleness– Lose benefits of simplicity
– Lose performance (encap, blinded hardware)
– Difficult to maintain and trouble shoot
romana.io Simplifying the network stack with Romana @romanaproject
The price you pay: Complexity
VXLAN Decap
VXLAN Decap
VXLAN Encap
VXLAN Encap
2 Top of Rack Round Trips
East/West Traffic
Per Instance Security
romana.io Simplifying the network stack with Romana @romanaproject
The price you pay: Performance
Router
Endpoint A Endpoint B
Router
L2 overlay A
L2 overlay B
VRouter
romana.io Simplifying the network stack with Romana @romanaproject
Why do we do this to ourselves?
● We don't need any L2 features● Except maybe traffic segmentation
– Multi tenancy
– Tiers and policies
The solution
romana.io Simplifying the network stack with Romana @romanaproject
Networking the way it was intended
● Use native L3 capabilities● No overlays● De-emphasize IP address ranges● Still provide segmentation, multi tenancy● Simple, clear and scalable network setup
romana.io Simplifying the network stack with Romana @romanaproject
Truly cloud native networking
● Project Romana● Open source● Apache 2.0 license● Mostly written in Go● Kubernetes and OpenStack
romana.io Simplifying the network stack with Romana @romanaproject
Truly cloud native networking
● Use only IP routing– No overlays
– All workload addresses are 'real'
– Simplicity!
● Use smart addressing– Encode tenant or segment in IP address
– Assign “virtual” addresses with host prefixes
– Massive (!) collapse of route table
● Routes are static– No route updates, no broadcasts for new endpoint
romana.io Simplifying the network stack with Romana @romanaproject
Romana Architecture
● On each host: Agent– Configures routes– Connects endpoint interfaces– Sets policy implementations
● Controller: Cooperating microservices– Each service with RESTful interface– Specialized for different tasks
● Environment: Different integration points– APIs, drivers for various parts of OpenStack or
Kubernetes
romana.io Simplifying the network stack with Romana @romanaproject
Romana Architecture
Host A Host B Host C
Agent Agent Agent
Tenant
Topology
IPAM
Root
Environment (OpenStack or Kubernetes)
Policy
Beautifully simple networking
romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:192.168.8.11
Host B
eth0:192.168.8.22
Host C
eth0:192.168.8.33
romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:192.168.8.11
romana-gw:10.0.0.1/16
Host B
eth0:192.168.8.22
romana-gw:10.1.0.1/16
Host C
eth0:192.168.8.33
romana-gw:10.2.0.1/16
romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:192.168.8.11
romana-gw:10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:192.168.8.22
romana-gw:10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:192.168.8.33
romana-gw:10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:192.168.8.11
romana-gw:10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:192.168.8.22
romana-gw:10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:192.168.8.33
romana-gw:10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:192.168.8.11
romana-gw:10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:192.168.8.22
romana-gw:10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:192.168.8.33
romana-gw:10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:192.168.8.11
romana-gw:10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Routes:10.1/16 → 192.168.8.2210.2/16 → 192.168.8.33
Host B
eth0:192.168.8.22
romana-gw:10.1.0.1/16
10.1.3.52
10.1.9.2
Routes:10.0/16 → 192.168.8.1110.2/16 → 192.168.8.33
Host C
eth0:192.168.8.33
romana-gw:10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
Routes:10.0/16 → 192.168.8.1110.1/16 → 192.168.8.22
romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
Rack A Rack B
romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
Host A2 Routes
0.0.0.0 192.168.1.200→10.68/14 192.168.1.1→10.76/14 192.168.1.3→10.80/14 192.168.1.4→
romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
ToR A Routes
10.128/10 192.168.2.200→10.68/14 192.168.1.1→10.72/14 192.168.1.2→10.76/14 192.168.1.3→10.80/14 192.168.1.4→
Host A2 Routes
0.0.0.0 192.168.1.200→10.68/14 192.168.1.1→10.76/14 192.168.1.3→10.80/14 192.168.1.4→
romana.io Simplifying the network stack with Romana @romanaproject
Larger network: Full L3
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
ToR A Routes
10.128/10 192.168.2.200→10.68/14 192.168.1.1→10.72/14 192.168.1.2→10.76/14 192.168.1.3→10.80/14 192.168.1.4→
Host Routes
0.0.0.0 192.168.1.200→
Scalable distributed firewalland
traffic policies
romana.io Simplifying the network stack with Romana @romanaproject
Romana: Traffic segmentation
● Tenant traffic separated:– Tenants don't get whole CIDR prefix or L2 domain
– But fully isolated from other tenants' traffic
● Tenants can define segments:– Like tiers, provide isolation and policies
● Use segment and tenant bits in IP addresses:– Apply policies (iptables) based on that
– Segments can stretch across hosts
romana.io Simplifying the network stack with Romana @romanaproject
Semantic and topological addressing
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bitsThe network prefix. In this example, we are using the 10/8
address space.
6
Host ID Segment IDWe currently
store tenant ID in upper bits of segment ID.
4 67
Endpoint ID
Widths are configurable, don't have to use byte boundaries.
romana.io Simplifying the network stack with Romana @romanaproject
Semantic and topological addressing
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bitsThe network prefix. In this example, we are using the 10/8
address space.
6
Host ID Segment IDWe currently
store tenant ID in upper bits of segment ID.
4 67
Endpoint ID
Widths are configurable, don't have to use byte boundaries.
Encode thetenant ID
romana.io Simplifying the network stack with Romana @romanaproject
Host BHost A
Allowing traffic within tenant
10.0.0.5 10.1.0.12
iptables:check src/dst addrs“tenant/segment bits
must match”
Src: 10.0.0.5Dst: 10.1.0.12
Same tenant/segment bits
romana.io Simplifying the network stack with Romana @romanaproject
Host BHost A
Isolating tenant traffic: Default
10.0.0.5 10.1.128.9
iptables:check src/dst addrs“tenant/segment bits
must match”
Src: 10.0.0.5Dst: 10.1.128.9
Different tenant/segment bits
Differenttenant
romana.io Simplifying the network stack with Romana @romanaproject
Host BHost A
Apply network policy between segments (full isolation as default)
10.0.0.5 10.1.1.9
iptables:Does policy chain
exist?Otherwise: DROP
Src: 10.0.0.5Dst: 10.1.1.9
Same tenant, different segment
policy-chain:From segment 0?Protocol TCP?To port 80?
Demo 1:
Kubernetes + Romana clusteron top of Catalyst OpenStack cloud
romana.io Simplifying the network stack with Romana @romanaproject
Baking layered cakes
● Kubernetes on OpenStack? Why?– On demand clusters
– Full tenant isolation
● Not all workloads fit into containers– Seamless connection between pods and VMs
● Really nice with fully routed networking– No double encapsulation
– Logical, efficient packet forwarding
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Jump host withpublic IP address
romana.io Simplifying the network stack with Romana @romanaproject
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Install OpenStackcommand line tools
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
$ neutron port-update \ e925b70e-031e-4ef7-a27c-583b4b775290 \ --allowed-address-pairs type=dict list=true \ mac_address=fa:16:3e:e1:df:59,ip_address=10.0.0.0/8
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
$ git clone https://github.com/romana/romana$ cd romana/romana-install$ ./romana-setup -p static -i my-inventory -s kubernetes install
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Romanainstaller
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Kubernetes + Romana
Romana clusteraddress range:
10/8
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Kubernetes + Romana
Podswith containers.
Pods have RomanaIP addresses.
romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - What you will see
● Creation of pods● Network configuration● Application of network policies
Demo 2:
Mixing containers with legacy workloads
romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - Overview
bar-1 bar-2foo
Kubernetes + Romana
romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - Overview
bar-1 bar-2foo
Kubernetes + Romana
vm-workload
Legacy applicationin VM
romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - Overview
bar-1 bar-2foo
Kubernetes + Romana
vm-workload
Direct connection:- No gateway- No encap/decap- No NAT
romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - What you will see
● Creation of pods● Contact pod from VM● See the packet route
Demo 3:
Romana + Kubernetes clusteron top of Romana + OpenStack cluster
romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
HW1 HW2 HW3 HW4
romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
HW1 HW2 HW3 HW4
$ ./romana-setup -p static -i hw-inventory -s devstack install
romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
HW1 HW2 HW3 HW4
OpenStack + Romana
Romana cluster 1address range:
10/8
romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
OpenStack VMs
VMs haveIP addresses
ofRomana cluster 1
romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
$ ./romana-setup -p static -i vm-inventory -s kubernetes install
romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3
Kubernetes + Romana
VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
Romana cluster 2address range:
172.16/12
romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3
Kubernetes + Romana
VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
Podswith containers.
Pods haveIP addresses
ofRomana cluster 2
romana.io Simplifying the network stack with Romana @romanaproject
OpenStack + Romana
Kubernetes + Romana
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
romana.io Simplifying the network stack with Romana @romanaproject
OpenStack + Romana
Kubernetes + Romana
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
Remember this one?
2 Top of Rack Round Trips
East/West Traffic
Per Instance Security
Without pure L3 networklayered clusters
would be even morecomplex.
romana.io Simplifying the network stack with Romana @romanaproject
OpenStack + Romana
Kubernetes + Romana
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
But with Romana, networkingeven in layered clusters becomes
really easy...
romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - What you will see
● Creation of pods● Pods and VMs with fully routable addresses● Ease of use showcase: Trouble shooting
romana.io Simplifying the network stack with Romana @romanaproject
Conclusion
● Cloud native architectures simplify things● Need cloud native networking to enjoy benefits● Romana:
– Cloud native without compromises
– Native network performance
– Mostly static config: Solid network
– Very easy to work with and understand
● Easy to try:– Simple installers for Kubernetes and OpenStack
romana.io Simplifying the network stack with Romana @romanaproject
Thank you!
● Romana Links– http://romana.io - Project home
– http://romana.io/blog - Blog
– https://github.com/romana/romana - Sources
● Contact– @romanaproject - Twitter
– [email protected] - Email
– https://romana.slack.com/ - Slack channel