Network Node is Not Needed Anymore - Completed Distributed Virtual Router / Fujitsu part

Copyright 2015 FUJITSU LIMITED1

Network Node is Not Needed Anymore- Completed Distributed Virtual Router

OpenStack Summit October 2015 TokyoTuesday, October 27 - 2:50pm – 3:30pm

Takanori MiyagishiFujitsu LimitedE-mail address: [email protected] name: miygishi_t

mailto:[email protected]


Agenda

VisionIssuesSolutionEfforts in LibertyFuture Plan


Vision


Vision Deployment of enterprise systems on OpenStack

To achieve this from the network perspective, we need more scalability and availability. Current Neutron has the following issues. Performance bottle neck Single point of failure (SPOF)


Issues


Legacy Network Architecture (Legacy Router)

Neutron centralizes networking services to Network Node Most of traffics need to go through Network Node

Compute Node Network Node

ML2 pluginOVS-agent

DHCP-agent

Controller Node

Neutron Server

L3-agent

ML2 pluginOVS-agentOVS-agent

metadata-agent

: Management network: External network: Data network


Legacy Network Architecture (Legacy Router)


ML2 pluginOVS-agent

DHCP-agent

Controller Node

Neutron Server

L3-agent

ML2 pluginOVS-agentOVS-agent

metadata-agent

: Management network: External network: Data network

Neutron centralizes networking services to Network Node Most of traffics need to go through Network Node

What happens due to this Architecture?


Issue #1: Network Node is Bottle neck There are 6 Network Traffics

East-West Traffic• Between VMs

- Same Subnet 　 -- (1)

- Different Subnets 　 -- (2)• VM accessing to DHCP server 　 -- (3)• VM getting metadata from vRouter -- (4)

North-South Traffic• Between VM and out of OpenStack

- Using Floating IP 　 -- (5)

- Using SNAT 　 -- (6)

Compute node #1 Network NodeCompute node #2

br-tun

br-int

br-tun

br-int

br-ex

VM VM VM

: External Network: Data Network

DHCP

br-tun

br-int

vRouter


Issue #1: Network Node is Bottle neck There are 6 Network Traffics

East-West Traffic• Between VMs





- Using SNAT 　 -- (6) Not need to go through Network Node in theboth cases:- VMs are on the same host- VMs are on different hosts


br-tun br-tun

br-int

br-ex

VM VM VM


DHCP

br-tun

br-int

vRouter

br-int


Issue #1: Network Node is Bottle neck


br-ex

VM VM VM


DHCP

br-tun

br-int

vRouter

br-tun

br-int

br-tun

br-int

There are 6 Network Traffics East-West Traffic• Between VMs





- Using SNAT 　 -- (6) Not need to go through Network Node in theboth cases:- VMs are on the same host- VMs are on different hosts




br-ex

VM VM VM


DHCP

vRouter

br-tun

br-int

br-tun

br-int

br-tun

br-int






- Using SNAT 　 -- (6) Need to go through Network Node in theboth cases:- VMs are on the same host- VMs are on different hosts



Need to go through Network Node in theboth cases:- VMs are on the same host- VMs are on different hosts


br-ex

VM VM VM


DHCP

vRouter

br-tun

br-int

br-tun

br-int

br-tun

br-int










br-ex

VM VM VM


DHCP

vRouter

br-tun

br-int

br-tun

br-int

br-tun

br-int










br-tun

br-int

br-tun

br-int

br-ex

VM VM VM


DHCP

br-tun

br-int

vRouter










br-tun

br-int

br-tun

br-int

br-ex

VM VM VM


DHCP

br-tun

br-int

vRouter








Issue #1: Network Node is Bottle neck Five out of six traffic patterns need to go through Network Node


br-tun

br-int

br-tun

br-int

br-ex

VM VM VM DHCP

br-tun

br-int

vRouter

Need through Network Node?

East-West

(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes(3) VM accessing to DHCP Yes(4) VM getting metadata Yes

North-South

(5) Using Floating IP Yes(6) Using SNAT Yes

(1)

(1)

(2),(3),(4),(5),(6)

Bottle neck


Issue #2: Network Node is SPOF If Network Node fails, that affects many traffics


br-ex

VM VM VM DHCP

br-tun

br-int

vRouter

Affected Trafficsby Network Node Failure

East-West


North-South


br-tun

br-int

br-tun

br-int


Solution


DVR Architecture L3-agent and Metadata-agent can be distributed on each Compute

Node But a part of L3-agent (SNAT) and DHCP-agent still need to

run on Network Node


ML2 pluginOVS-agent

DHCP-agent

Controller Node

Neutron Server

L3-agent(SNAT)

ML2 pluginOVS-agentOVS-agentL3-agent

metadata-agent Metadata-agent


DVR Architecture


ML2 pluginOVS-agent

DHCP-agent

Controller Node

Neutron Server

L3-agent(SNAT)


metadata-agent Metadata-agent

L3-agent and Metadata-agent can be distributed on each Compute Node But a part of L3-agent (SNAT) and DHCP-agent still need to

run on Network NodeWhat are benefits of the distribution of agents?- Will explain, comparing with Legacy Router

21

Traffics with Legacy Router again five of six types of traffics needs through Network Node

Copyright 2015 FUJITSU LIMITED


East-West

(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes(3) VM accessing to DHCP Yes(4) Gets metadata Yes

North-South



br-tun

br-int

br-tun

br-int

br-ex

VM VM VM DHCP

br-tun

br-int

vRouter

(1)

(1)

(2),(3),(4),(5),(6)

22

Changes in the Architecture by DVR These components are distributed to each compute node

vRouter, Floating IP, and bridge (interface) to external network



br-tun

br-int

br-tun

br-int

br-ex

VM VMDHCP

br-tun

br-int

br-exbr-ex

fip fip


East-West


North-South


23

Network flow of DVR



br-ex

VM VMDHCP

br-tun

br-int

br-exbr-ex

fip fip

br-tun

br-int

br-tun

br-int

East-West traffic - Between VMs(same subnet) Not need to go through Network Node in the both cases:

VMs are on the same host VMs are on different hosts


East-West


North-South


24

Network flow of DVR



br-ex

VM VMDHCP

br-tun

br-int

br-exbr-ex

fip fip

br-tun br-tun

br-intbr-int

East-West traffic - Between VMs(different subnet) No longer need to go through Network Node VM access to vRouter that exists on own Compute Node


East-West

(1) Between VMs (same subnet) No(2) Between VMs (different subnet) Yes -> NO

(3) VM accessing to DHCP Yes(4) VM getting metadata Yes

North-South


25

Network flow of DVR



br-ex

VM VMDHCP

br-tun

br-int

br-tun

br-int

br-tun

br-int

br-exbr-ex

fip fip

East-West traffic – Access to DHCP Still need to go through Network Node


East-West


(3) VM accessing to DHCP Yes(4) VM getting metadata Yes

North-South


26

Network flow of DVR



br-ex

VM VMDHCP

br-tun br-tun

br-int

br-tun

br-int

br-exbr-ex

fip fipbr-int

East-West traffic – Gets metadata No longer need to go through Network Node VM access to vRouter that exists on own Compute Node


East-West


(3) VM accessing to DHCP Yes(4) VM getting metadata Yes -> NO

North-South


27

Network flow of DVR



br-ex

VM VMDHCP

br-tun br-tun

br-int

br-tun

br-int

br-ex

fipbr-int

br-ex

fip

North-South traffic – Using Floating IP No longer need to go through Netowork Node VM access to vRouter and translate to floating IP that exists on own

Compute node


East-West



North-South

(5) Using Floating IP Yes -> NO(6) Using SNAT Yes

28

Network flow of DVR



VM VMDHCP

br-tun

br-int

br-exbr-ex

fip fip

br-exbr-tun

br-int

br-tun

br-int

North-South traffic – Using SNAT Still need to go through Network Node


East-West



North-South


29

Network flow of DVR 3 types of traffic are no longer need to go through

Network Node due to DVR



br-ex

VM VMDHCP

br-ex

fip

br-tun

br-int

(4)

(5)

(2)

br-tun

br-int

br-ex

fip

br-tun

br-int


East-West



North-South


30

Network flow of DVR 2 types of traffics still need to go through Network Node

Meaning there are still two issues (Bottleneck and SPOF)



VM VMDHCP

br-tun

br-int

br-exbr-ex

fip fip

(6)

(3)

br-exbr-tun

br-int

br-tun

br-int


East-West



North-South



Completely Distributed Virtual Router

To eliminate bottle neck and SPOF completely Distributed SNAT and DHCP are required


ML2 pluginOVS-agent

DHCP-agent

Controller Node

Neutron Server

L3-agent(SNAT)


metadata-agentDHCP-agent


Efforts in Liberty


Distributed SNAT Distribute remaining l3-agent(SNAT) on Network Node to each Compute

Node


ML2 pluginOVS-agent

DHCP-agent

Controller Node

Neutron Server

L3-agent(SNAT)


metadata-agent


Conventional behavior of SNAT Create SNAT on Network Node for each vRouter

If a tenant has multiple vRouters, then Neutron creates multiple SNATs The amount of public IP consumption

# of vRouter that sets a gateway - 3 public IPs are needed in the following configuration

Compute Node #1 Network NodeCompute Node #2VM VM VM

SNATvRouterA1 vRouterA1'vRouterA2 vRouterA2'vRouterB1 vRouterB1'SNAT

SNATVM VM VM VM


Possible Idea for Distributed SNAT Five ideas that I devised w/ some hints in the past discussions

#1 One SNAT for one Router #2 One SNAT for one Compute Node #3 One SNAT for one Compute Node w/o Security Concern #4 Double NAT #5 BGP(Border Gateway Protocol) Understood why Distributed SNAT has not been implemented yet

It’s not easy at all to devise the best design…

Will explain these ideas from the next page For each idea, will show an actual amount of public IP consumption in an example system

configuration Compute Node#1 and #2 have two VMs of tenant#1 and a VM of tenant#2 respectively


#1 One SNAT for one Router Distribute SNAT simply … Follow the conventional behavior

The number of SNAT increases because each vRouter requires one SNAT

Compute Node #1 Compute Node #2VM VM

SNATSNAT SNAT

VMVM VM

SNATSNAT SNAT

VM

.1 .3 .5 .2 .4 .6


#1 One SNAT for one Router The amount of public IP consumption

(# of Compute Nodes) * (# of vRouters per Compute Node) … Too many!! 6 public IPs are needed in the example system configuration


SNATSNAT SNAT

VMVM VM

SNATSNAT SNAT

VM

.1 .3 .5 .2 .4 .6


#2 One SNAT for one Compute Node One SNAT shared by vRouters in Compute Node

In other words, sharing SNAT between different tenantsNeeds an exclusion of address scope between different tenants


SNAT

VMVM VM

SNAT

VM

.1 .2


#2 One SNAT for one Compute Node The amount of public IP consumption

# of Compute nodes … Sufficiently small 2 Public IPs are needed in the example system configuration

Security concern exists If an admin makes a wrong setting on SNAT, VMs in the different tenants can

communicate.


SNAT

VMVM VM

SNAT

VM

.1 .2


#3 One SNAT for one Compute Node w/o Security Concern SNAT shared by vRouters of the same tenant in Compute Node

Compute Node #1 Compute Node #2VMVM VM

SNAT

VM

.1 SNAT .3 SNAT .4

VM VM

SNAT .2


#3 One SNAT for one Compute Node w/o Security Concern The amount of public IP consumption

(# of Compute nodes) * (# of Tenants) If each tenant only has one vRouter, the consumption will be the same

as idea #1 because (# of Tenants) becomes (# of vRouter) 4 public IPs are needed in the example system configuration

Compute Node #1 Compute Node #2VMVM VM

SNAT

VM

.1 SNAT .3 SNAT .4

VM VM

SNAT .2


#4 Double NAT Double NAT on SNAT and upstream physical router

SNAT: translates private IP to a dedicated local IP (like ISP shared IP) Upstream physical router: translates the local IP to a public IP

Need some changes to Floating IP mechanism

- Floating IP needs to be able to allocate the local IPs to VMs


SNATSNAT SNAT

VMVM VM

SNATSNAT SNAT

VM

.1 .3 .5 .2 .4 .6

Upstream Router

NAT table


#4 Double NAT Dynamic setting to upstream physical router is required

OpenStack doesn’t have such a feature yet The amount of public IP consumption

(# of vRouter) 3 public IPs are needed


SNATSNAT SNAT

VMVM VM

SNATSNAT SNAT

VM

.1 .3 .5 .2 .4 .6

Upstream Router

NAT table(Need dynamic setting)


#5 BGP(Border Gateway Protocol) vRouters in the same group share one public IP

Upstream physical router as a gateway BGP speaker uses BGP to advertise the router which SNAT

to connect in the same group Neutron doesn’t have a feature like BGP Speaker yet

Compute Node #1 BGP SpeakerCompute Node #2

Upstream Router

Peering

VM

SNAT.6

VM VM

SNAT .2 SNAT .4

VM

SNAT.5

VM VM

SNAT .1 SNAT .3


#5 BGP(Border Gateway Protocol) High cost

Development cost: BGP support CAPEX: Linux server for BGP speaker Special Requirement: upstream physical router that supports BGP


Upstream Router

Peering

VM

SNAT.6

VM VM

SNAT .2 SNAT .4

VM

SNAT.5

VM VM

SNAT .1 SNAT .3


#5 BGP(Border Gateway Protocol) The amount of public IP consumption

# of vRouter … same as the current DVR where SNAT is not distributed “BGP support” is under development now

SPEC(merged): https://review.openstack.org/#/c/90833/


Upstream Router

Peering

VM

SNAT.6

VM VM

SNAT .2 SNAT .4

VM

SNAT.5

VM VM

SNAT .1 SNAT .3

https://review.openstack.org/#/c/90833/

https://review.openstack.org/#/c/90833/


#5 BGP(Border Gateway Protocol) Need more enhancements for “BGP support” to realize this idea because

BGP Speaker will be another SPOF Possible solution is to make vRouter behave as BGP Speaker


Upstream Router

Peering

VM

SNAT.6

VM VM

SNAT .2 SNAT .4

VM

SNAT.5

VM VM

SNAT .1 SNAT .3


Summary of the Ideas Ideas in the order of the amount of public IP consumption

Name Consumption Cost for Infrastructure

#1 One SNAT for one Router (# of Compute nodes) * (# of Routers)

#3 One SNAT for one Compute Node w/o Security Concern

(# of Compute nodes) * (# of Tenants)

#2 One SNAT for one Compute Node # of Compute Nodes

#4 Double NAT # of vRouters Physical router required

#5 BGP # of vRouters A server for BGP Speaker and physical router required


Summarized in table Trade-off between IP consumption and cost for infrastructure

Name Consumption Cost for Infrastructure

#1 One SNAT for one Router (# of Compute nodes) * (# of Routers)

#3 One SNAT for one Compute Node w/o Security Concern

(# of Compute nodes) * (# of Tenants)

#2 One SNAT for one Compute Node # of Compute Nodes

#4 Double NAT # of vRouters Physical router required

#5 BGP # of vRouters A server for BGP Speaker and physical router required

Many

A fewHeavy

Light


Future Plan of Distributed SNAT


Future Plan

I’ll propose idea #3 again at the Design Summit, “One SNAT for one Compute Node w/o Security Concern” It doesn’t require any specific hardware Still many public IPs can be consumed in a particular environment If you cannot get much advantages in your environment, you can use the

centralized SNAT (Network Node)

Let’s discuss at the Design SummitWill discuss at Neutron’s Contributors meetup on Friday!

Network Node is Not Needed Anymore - Completed Distributed Virtual Router / Fujitsu part

Software