Using ansible to core os & kubernetes clusters

Using Ansible to CoreOs & Kubernetes ClustersAnsible Meetup Berlin 28.4.2016

Agenda

● Kubernetes?● Coreos?● Components

○ Etcd○ kubernetes master○ Kubernetes nodes

● Doing it all with ansible● Demo

Motivation

● GIFEE - Google Infrastructure for Everyone Else● Useful if you can't do Public Cloud for example for legal issues● Platform for microservices

CoreOs

What is CoreOs

● Modern Minimal Linux Os● designed for security, consistency, and reliability● No packages, additional applications come via containers● Configured via cloud config● https://coreos.com/os/docs/latest/cloud-config.html● No python !

Etcd

● /etc distributed● Consistent reliable distributed key-value store● Used to keep configuration in a cluster● Backend for kubernetes state● Leader election● Uneven number of nodes to be able to form a majority even in case of

network partitions

Etcd small cluster

● Etcd runs on every node● Etcd competes for resources with the actual

workloads of the cluster● Might be a problem in production when the

cluster gets high load

Etcd Production deployment

Kubernetes

Kubernetes

● Container Orchestration Platform Initiated by Google● Automatic bin packing● Service discovery and load balancing● Automated rollouts and rollbacks● Secret and configuration management● Self-healing

Kubernetes Architecture overview

Deploying all this with ...

The inventory - groups: etcd & coreos

[etcd-node]# nodes to run etcd and be part of the etcd cluster ( uneven # )

[etcd-proxy]# nodes to run etcd in proxy mode and proxy requests to the cluster

[coreos:children]Etcd-nodeetcd-proxy

Inventory groups: kubernetes

[kubernetes-master]# api server and master elected components (scheduler & kube-controller)

[kubernetes-node]# kubernetes worker node - runs kubelet and kube-proxy

[kubernetes:children]Kubernetes-masterkubernetes-node

[loadbalancer]# run a nginx-reverse-proxy (kubernetes daemonset)

[ceph-mon]# run monitor for ceph on kubernetes (daemonset) (should be 3 nodes)

[ceph-osd]# run osd for ceph on kubernetes (daemonset)

Inventory additional groups

Inventory: Global Configuration Variables

Kube_master_dns_name # dns netry for kube master nodes if you have dnsKube_master_ip # the master node kubectl should talk to if you have no dns kube_cluster_name=privatkubernetes_version=1.2.3k8s_service_ip_range=10.100.0.0/16k8s_service_ip=10.100.0.1k8s_dns_service_ip=10.100.0.10k8s_dns_domain=cluster.local

Bare metal Provider

● Designed to use every baremetal provider that provides an api to boot servers into a rescue system and access via ssh

● Sample implementations for:○ Hetzner○ Kimsufi ( ovh )

● Minimal hardware requirements ○ One network interface is enough

Exposing Services

https://github.com/cornelius-keller/kubernetes-reverseproxy

● Nginx reverse proxy ● Runs as a daemonset on dedicated nodes● Wirtes an nginx config from specially annotated services

How it works

● Generate all certificates needed for kubernetes and etcd● Boot nodes into a rescue system● Install coreos ● Reboot the system● Install python (pypi) to be able to work with ansible● Template the cloudconfig ● Reboot again

Takes about 8-10 min for a three node cluster

Where to find

https://github.com/cornelius-keller/ansible-coroeos-kubernetes

Thanks!