Kubernetes SDN performance and architecture

Copyright © 2016 Mirantis, Inc. All rights reserved

www.mirantis.com

Kubernetes SDN Performance and Architecture

Jakub PavlikMarek Celoud


Presentation Agenda

1. Overlay vs Non-Overlay2. Calico3. OpenContrail4. Connection/comparison5. Q&A


About us

Marek [email protected]@MCeloud

Jakub Pavlí[email protected]@JakubPav


Networking in Kubernetes

● Networking in containers used to be an issue● Kubernetes solved the biggest problems of port mapping● Different approaches for different use cases

● Overlay vs. Non-overlay● Multitenancy and security● Performance and scaling

● Multiple plugins similar like OpenStack Neutron


Network solutions in Kubernetes

SDNs:● Calico● OpenContrail● Romana● Weave● Contiv● OpenVSwitch● ...


Overlay vs. Non-overlay

Common Overlay concerns:● Loose benefit of simplicity● Loose performance● Difficult to maintain and

troubleshoot

Overlay benefits:● Multitenancy, Security,

Micro-segmentation● L2, L3, EVPN, L3VPN

capability● Analytics

From performance perspective not using an overlay, it is still necessary to use an internal bridge to demux the container virtual-ethernet interface pairs.

“The key aspect to consider is operational complexity!” Pedro Marques


Test environment

● Run various functional and performance tests● Calico bare metal● OpenContrail bare metal● OpenContrail running on Kubernetes with Calico● OpenContrail and Kubernetes next together● Calico in OpenStack with OpenContrail● OpenContrail Kubernetes in OpenStack with OpenContrail

● 100 nodes with 32GB RAM with 8 CPUs and 2x 10Gb links


Calico


Calico Overview

● CNI network plugin● BIRD routing daemon● Etcd● Confd● Felix● Pure L3


Calico


Calico

CalicoPros:

● No overhead● Reduce Complexity● Using standard

protocols

Cons:

● Underlay depended● No L2


Calico with k8s

● Using CNI● Calico 0.22.0 version with kubernetes 1.4● Kubernetes Policy for security


Production consideration for Calico

● Use separate etcd cluster for Calico● Use at least etcd v3● Disable BGP full mesh peering● Do not run Calico in k8s manifests, but as separated

systemd/docker


OpenContrail


OpenContrail Overview

● Overlay SDN● Control, config, analytics, database, agent● Multiple encapsulations (MPLSoverGRE/UDP, VXLAN)● Uses (usually) physical gateways


OpenContrail overview


OpenContrail overview

OpenContrailPros:

● Underlay agnostic● Advanced networking

features● Uses physical

gateways

Cons:

● Overhead● Complex


OpenContrail with s8s

● Network manager which provides bridge between Contrail and k8s

● Using ECMP instead of kube-proxy (iptables) balancing● Networks created based on labels in manifests● Security and Multi-tenancy done by policy● Contrail 3.0.3 supports Kubernetes 1.4


Production consideration for OpenContrail

● Separate Cassandra cluster for analytics● Use physical routers as gateways


Comparison


Performance


Why not both?


Multi-cloud examples

● Connection Baremetal, VMs, container

● Run k8s on top of OpenStack with same Contrail (VM sub-interfaces)


Kubernetes production findings

● build own binaries (Mirantis Downstream) instead of reusing existing docker containers with unknown origin

● use single or high available cluster setup● run ETCD control services in systemd not only in

manifests and docker● cleanup from mixing bash, salt, and unrelated features for

production● manage native SSL cert by Salt or external cert entity● pull images from private docker registry with

authentication


Calico vs OpenContrail comparison


MCP


Q&A

Thank you for your time


Backup Slides

Kubernetes SDN performance and architecture

Technology

Kubernetes SDN performance and architecture