How to Secure Your Kubernetes Cluster by Stanislav Kolenkin, Senior DevOps.
Introduction
2
Kubernetes, or k8s (k, 8 characters, s...get it?), or “kube” is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.
Why containers and Kubernetes?
3
Kubernetes makes containers easier to manager• Scheduling• Lifecycle management• Scaling• Load balancing
Kubernetes has security features built in• Logging and monitoring• Debugging and introspection• Identity and authorization
Why containers and Kubernetes?
4
Running containers allows you to adopt a fundamentally different security model.
How is securing a container different than a VM?
5
Surface of attack Minimalist host OS and limits the surface of an attack.Resource isolation Host resources are separated using namespaces and
cgroups.Permissions Access controls are for app privileges and shared
resources.Lifetime Containers have a shorter average lifetime.
Container security
6
Privilege escalation Unpatched vulnerability DDoS
kubernetes API compromise
Zero day exploit in common library
Node compromise and exploit
Container security
7
Application security Are my application secure?Infrastructure security Software supply chain Container runtime security
Is my infrastructure secure for developing containers?
Is my container image secure build and deploy?
Is my container secure to run?
Platform security Is my (cloud provider’s) infrastructure secure?
Kubernetes
8
Chenxi Wang, Jane Bond Project
Dr. Chenxi Wang is the founder of the Jane Bond Project, a cybersecurity strategy consulting firm. A well known strategist, speaker, and writer in the cybersecurity industry. Dr. Wang also serves as Vice Chair on the Board of Directors for the OWASP foundation
Kubernetes is so new thatlots of practitioners don’tknow what securitycontrols come with it.
So one of the first things to do isstudy up on what controls are thereand use them to strengthen yoursecurity posture
– Chenxi Wang, Jane Bond Project
Managed Kubernetes Services
15
GKE AKS EKSNetwork Policy Calico Self-deployed
(kube-router) Self-deployed
(Calico)RBAC Yes Yes YesMgmt via (vendor) CLI Complete Complete MinimalUpgrades Auto/On-demand On-demand UnclearMulti AZ Yes Partial Yes
Protecting Cluster Metadata
Yes
Kubernetes: Enforce bastion model via firewall rules in GCE
19
- name: kubernetes-allow-ssh-to-controller-api type: compute.v1.firewall properties: allowed: - IPProtocol: tcp ports: ["6443"] sourceTags: [ "ssh-bastion" ] sourceRanges: ["10.10.0.0/16"] targetTags: [ "k8s-controller" ] network: global/network/default
Kubernetes Engine: Authorized networks
20
Authorized networks are a way of specifying a restricted range of IP addresses that are permitted to access your container cluster's Kubernetes master endpoint.
GKE uses both Transport Layer Security (TLS) and authentication to provide secure access to your container cluster's Kubernetes master endpoint from the public internet. This provides you the flexibility to administer your cluster from anywhere; however, you might want to further restrict access to a set of IP addresses that you control. You can set this restriction by specifying an authorized network.
List up to 20 authorized networks to connect to the master over HTTPs.
RBAC
22
In Kubernetes, Identity and User management are not integrated in the platform and should be managed by external IAM platforms like Keycloak, Active Directory, Google’s IAM, etc. However, authentication and authorization are handled by Kubernetes.
The RBAC model in Kubernetes is based on three elements:● Roles: definition of the permissions for each Kubernetes resource type● Subjects: users (human or machine users) or groups of users● RoleBindings: definition of what Subjects have which Roles
Network Policy
25
Network Policies is a new Kubernetes feature to configure how groups of pods are allowed to communicate with each other and other network endpoints. In other words, it creates firewalls between pods running on a Kubernetes cluster. This guide is meant to explain the unwritten parts of Kubernetes Network Policies.
By default, Kubernetes does not restrict traffic between pods running inside the cluster. This means any pod can connect to any other pod as there are no firewalls controlling the intra-cluster traffic.
Network Policy
26
Network Policies give you a way to declaratively configure which pods are allowed to connect to each other. These policies can be detailed: you can specify which namespaces are allowed to communicate, or more specifically you can choose which port numbers to enforce each policy on.
Kubernetes Engine: Set a Network Policy
28
Create cluster with a network policy using the flag: --enable-network-policyUse a Network Policy as normalExample: restricting traffic to a pod
Introduction
29
We can configure the following Network policy scenarios:
• DENY all traffic to an application• LIMIT traffic to an application• ALLOW all traffic to an application• DENY all non-whitelisted traffic to a namespace• DENY all traffic from other namespaces (LIMIT access to the current namespace)• ALLOW traffic to an application from all namespaces• ALLOW all traffic from a namespace• ALLOW traffic from some pods in another namespace
Introduction
30
We can configure the following Network policy scenarios:
• ALLOW traffic from external clients• ALLOW traffic only to a port of an application• ALLOW traffic from apps using multiple selectors• DENY egress traffic from an application• DENY all non-whitelisted traffic from a namespace• LIMIT egress traffic to the cluster (DENY external egress traffic)
Introduction
31
We could configure soon the following Network policy scenarios:
• LIMIT egress traffic from an application to some pods• ALLOW traffic only to Pods in a namespace
Why disable the dashboard? Tesla hack
34
• Hackers accessed the Kubernetes console, which was not password protected
• Console contained privileged cloud account credentials
• Used credentials to access resources and mine cryptocurrency
https://redlock.io/blog/cryptojacking-tesla
Remove the dashboard’s admin access
35
1. Don’t deploy it2. Reduce its permissions3. Reduce its permissions, and add authentication
Disable Kubernetes dashboard
36
The Kubernetes dashboard does not have admin access in GKE 1.7+ clusters, and is disabled by default in GKE 1.10+ clusters.
Use Google Console to perform the same tasks you would use the Kubernetes dashboards for.
Kubernetes: already disabled
39
Service account disabled by default (as RBAC is enabled) for Kubernetes 1.8+ clusters.
apiVersion: v1kind: ServiceAccountmetadata: name: build-robot automountServiceAccountToken: false
…
Kubernetes: already disabled
40
The default service account has no privileges if ABAC is disabled.Attribute-based access control (ABAC) defines an access control paradigm
whereby access rights are granted to users through the use of policies which combine attributes together.
RBAC is enabled by default (and ABAC is disabled) for GKE 1.8+ clusters.
Linux security constructs
42
AppArmor is a Linux kernel security module that supplements the standard Linux user and group based permissions to confine programs to a limited set of resources. AppArmor can be configured for any application to reduce its potential attack surface and provide greater in-depth defense. It is configured through profiles tuned to whitelist the access needed by a specific program or container, such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either enforcing mode, which blocks access to disallowed resources, or complain mode, which only reports violations.
Linux security constructs
43
Seccomp
Secure Computing Mode (seccomp) is a kernel feature that allows you to filter system calls to the kernel from a container. The combination of restricted and allowed calls are arranged in profiles, and you can pass different profiles to different containers. Seccomp provides more fine-grained control than capabilities, giving an attacker a limited number of syscalls from the container.
The default seccomp profile for docker is a JSON file and can be viewed here: https://github.com/docker/docker/blob/master/profiles/seccomp/default.json. It blocks 44 system calls out of more than 300 available. Making the list stricter would be a trade-off with application compatibility.
Linux security constructs
44
AppArmor• Linux Security Module that lets you restrict your program’s actions• Controls and audits various process actions, e.g., file functions like read, write,
execute
seccomp• Filters a process’ syscalls to limit what syscalls the process allows• Puts applications in `secure` state with whitelist of allowed syscalls• Docker seccomp default denies ~150 uncommon or potentially unsafe
syscalls
PodSecurityPolicy (PSP)
46
To use, enable the admission controller for Pod Security Policykube-apiserver --enable-admission-plugins=PodSecurityPolicy
The ‘restricted’ policy provides a good starting point.
PodSecurityPolicy (PSP)
47
apiVersion: policy/v1beta1kind: PodSecurityPolicymetadata: name: restrictedspec: privileged: false # Don't allow privileged pods! allowPrivilegeEscalation: false # Don’t allow privilegedescalation: runAsUser: # Require the container to run without rootprivileges: rule: ‘MustRunAsNonRoot’ supplementalGroups: # Forbid adding to the root group rule: ‘MustRunAs’ ranges:
- min: 1 max: 65535
Hardening your cluster
49
• Enable Protecting Cluster Metadata• Using clair or other solutions for scanning images for known vulnerabilities • Verify binaries that are deployed• Consider sandboxing - gVisor• Limit the identity used by pods - OSS Proposal• Use a service mesh for authentication/encryption - Istio
Protecting Cluster Metadata
51
Google Kubernetes Engine uses instance metadata to configure node VMs, but some of this metadata can be used to attack a cluster, and so should be concealed from workloads running on the cluster that don't need access to it.
This page explains how to use GKE's metadata concealment feature to protect only potentially sensitive system metadata from user workloads running on your cluster.
Protecting Cluster Metadata
52
Instance metadata is data about your instance that you can use to configure or manage the running instance. Instance metadata is divided into categories. AWS: http://169.254.169.254, GCP: http://metadata.google.internal
GKE: Protecting cloud metadata
53
Beginning with Kubernetes version 1.9.3, you can enable metadata concealment to prevent user Pods from accessing certain VM metadata for your cluster's nodes, such as Kubelet credentials and VM instance information. Specifically, metadata concealment protects access to kube-env (which contains Kubelet credentials) and the VM's instance identity token.
Metadata concealment firewalls traffic from user Pods (Pods not running on HostNetwork) to the cluster metadata server, only allowing safe queries. The firewall prevents user Pods from using Kubelet credentials for privilege escalation attacks, or from using VM identity for instance escalation attacks.
GKE: Protecting cloud metadata
54
Limitations● Metadata concealment only protects access to kube-env and the node's
instance identity token● Metadata concealment does not restrict access to the node's service account● Metadata concealment does not restrict access to other related instance
metadata
To create a cluster with metadata concealment enabled, you need add the following parameter:
--workload-metadata-from-node=SECURE
Protecting cloud metadata
55
• Metadata-proxyhttps://github.com/GoogleCloudPlatform/k8s-metadata-proxy
• Configure NetworkPolicy to prevent access metadata API• kube2iam
https://github.com/jtblin/kube2iam
Secrets and Configmaps
57
Objects of type secret are intended to hold sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image. Secrets use Base64 for encrypt and decrypt.
ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable.
Dynamic secrets on Kubernetes pods using Vault
60
Summary:
• Vault provides a secure solution for sharing and distributing secrets• Some challenges that require customization for your needs• Stakes are extremely high and need to prepare accordingly • Kubernetes and Vault are excellent tools and can work together well, but
proper integration is a non-trivial task.
Mesh network
62
A mesh network is a local network topology in which the infrastructure nodes connect directly, dynamically and non-hierarchically to as many other nodes as possible and cooperate with one another to efficiently route data from/to clients.
Istio
63
Istio gives:• Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic.• Fine-grained control of traffic behavior with rich routing rules, retries,
failovers, and fault injection.• A pluggable policy layer and configuration API supporting access controls,
rate limits and quotas.• Automatic metrics, logs, and traces for all traffic within a cluster, including
cluster ingress and egress.• Secure service-to-service authentication with strong identity assertions
between services in a cluster.
Istio
65
GKE ( for k8s version 1.7.6 +) uses k8s network policies to manage which pods and services can communicate . This is relatively straightforward to configure. Istio also has network policies but they’re not the k8s policies you know and love so what’s the difference and why both? This post explains it well so I won’t repeat the reasoning here but this table summarises the differences and why you need both.
KubeATF
68
The Kubernetes Ansible Testing Framework is a tool meant to spin up, test, save results, and spin down Kubernetes clusters using the various installation tools/kits/templates readily available in a human and CI/CD friendly way.
Test Cases:1. Kubernetes Installation Tool Testing - If you are a maintainer of a
Kubernetes installation tool, this provides a mechanism to create a repeatable process for creating test clusters to improve consistency and reduce time spent during development.
2. Kubernetes Release Testing - If you are running the e2e (end to end) testing suite on various releases/configurations, this tool may help you.
KubeATF
69
3. Kubernetes Application Testing - If you are deploying an application or a helm chart inside Kubernetes and you are concerned with ensuring compatibility across clouds and versions of Kubernetes, this tool may help you.
4. Kubernetes Security Testing - If you are assessing multiple installation tools for their security posture using things like CIS Benchmarking checks, this tool can help automate the testing process.
Kubeaudit
70
Kubeaudit is a command line tool to audit Kubernetes clusters for various different security concerns: run the container as a non-root user, use a read only root filesystem, drop scary capabilities, don't add new ones, don't run privileged.
Kubeaudit
71
Kubeaudit has multiple checks:● Audit all● Audit security context○ Audit readOnlyRootFilesystem○ Audit runAsNonRoot○ Audit allowPrivilegeEscalation○ Audit privileged○ Audit capabilities
● Audit image● Audit Service Accounts● Audit network policies● Audit resources
K8Guard
72
K8Guard is an auditing system for Kubernetes clusters. It monitors different entities on your cluster for possible violations. K8Guard notifies the violators and then takes action on them. It also provides metrics and dashboards about violations in the cluster through Prometheus.
NeuVector
73
The tools are in the form of scripts which implement tests for the 100+ recommendations in the Kubernetes 1.6 CIS Benchmark for security. They are available here at https://github.com/neuvector/kubernetes-cis-benchmark. There are different security recommendations for the Kubernetes master node and for the worker node, so there are two separate scripts.In general the recommendations focus on auditing key security areas of
NeuVector
74
Kubernetes, including:● Use of privileged containers● API server authentication and authorization● Kubelet authentication● etcd security● Data security, e.g. files, secrets● Certificate management● Security for pods.
TwistLock
75
Twistlock is the industry’s most complete, automated and scalable container cybersecurity platform. From precise, full-lifecycle vulnerability and compliance management to application-tailored runtime defense and cloud native firewalls, Twistlock secures your containers and modern applications against the next generation of threats across the entire application lifecycle.
Why GKE security?
76
• Google Kubernetes Engine provides best practices with security by default.There will always be app-level hardening and tuning to do.
• Google Kubernetes Engine patches you to the latest version, automatically.This keeps you up to date with security patches and with new features.
• Google Kubernetes Engine provides best of Google Cloud securityWith integrations with IAM, Audit Logging, VPC, and more.
Conclusion
77
• Many possibilities to make your cluster more secure• Each with it’s benefits and drawbacks• Generally - Security means (hard) work• Depends on your use Case what to use• Play around with and get a feeling (in a playground)• Monitoring • Logging