Networking for Kubernetes A Tale from the Trenches Cloud Engineering, eBay Sreekanth Pothanis
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Networking for KubernetesA Tale from the Trenches
Cloud Engineering, eBaySreekanth Pothanis
Networking is inherently hard!
Complexities of running on openstack
Scale
Multitenancy
Interoperability with Legacy
Private Network model with Openstack SDN
Dedicated kube router provisioned in neutron
Private Networks
Subnet per node
L3 Routed Model
NIPAP as IPAM
Subnet per node
Fully routable pods
Network 2.0
Abstract out network boundaries from nodes to arbitrary network scopes
IP blocks are allocated to these network scopes
Scopes can represent a host or a higher level aggregation
Supports legacy and other complex network zoning
Network 2.0
node
pod
pod Network Scope
IPAM
node
pod
pod
Allocation Pools
Network Scope
Allocation Pool
1 uuid1
2 uuid2
IPAM controllerCluster admin creates network scopes + allocation pools
Kubernetes Nodes are associated with Scopes
IPAM Controller assigns IP based on scope of the node selected by Kube scheduler
Pods are annotated with IPs
Tessnet plugin configures the pods with annotated IP
Kube Scheduler
IPAM controller
TessNetPlugin
Pod: myPod
Host: A
Pod:myPod
Node A
notMyPod
myPod 10.10.11.4
Tessnet Pluginkubelet
Network Scope1
Allocation Pools10.10.12.0/2210.11.1.0/24
Node: A
Node: B
Node: C Network Scope2
API Server
Host: AIP: 10.10.1.4
Pod: myPod
Host: AIP: 10.10.1.4
Pod: myPod
“network_scope”: “netscope1”
Networking 2.0 -- host
OVS
ARP Proxy
Service to POD
Kube’s default implementation creates LBs on Nodes
Load balance on pods directly
Neutron LBaaS Pool
Neutron LBaaS VIP
POD POD POD POD
eBay Ingress
Application Topology
POOL
Application VIP
VIP
GTM Load Balanced Pool
POOL
VIP
POOL
VIP
Region 1 Region 2 Region 3
Global Name (omg.g.ebay.com)
MONITOR MONITOR MONITOR
Application VIPApplication VIP
Ingress controller
Ingress: myIngressStatus: VIP-1 IP GTM name
Ingress controller
API Server
LBMS
DNS
GTM
Ingress: myIngress
Ingress: myIngressStatus: VIP-1 IP
Ingress: myIngressStatus: VIP-1 IP GTM name
DNS
apiVersion: v1kind: Servicemetadata: annotations: network.tess.io/kube2udns: "nginx.spothanis.svc.32.tess.io.\t3600\tIN\tA\t10.149.4.27"
Kube2DNS controller
DNS
APIServer
Future work
Network Policy Enforcement
Globally federated Ingress -- SLB based
Related Documents
