Veer's Container Security

@OpenShift

RHOpenShift

Container Security

Presenter: Veer MuchandiTitle: Principal Architect - Container SolutionsSocial Handle: @VeerMuchandiBlogs: https://blog.openshift.com/author/veermuchandi/

What Are Containers?

● Sandboxed application processes on a shared Linux OS kernel

● Simpler, lighter, and denser than virtual machines

● Portable across different environments

● Package my application and all of its dependencies

● Deploy to any environment in seconds and enable CI/CD

● Easily access and share containerized components

INFRASTRUCTURE APPLICATIONS

It Depends on Who You Ask

2

Understanding Containers

Virtualization vs Containerization

Container Host SecurityLinux Technologies

- Namespaces for Isolation- Seccomp- Read Only Mounts- SELinux - MCS- RHEL Atomic

Base Image build your applications - RHEL or RHEL-Atomic

Linux Containers Architecture

Kernel provides process isolation by creating separate namespaces for containers

PID namespaces allow processes in different containers to have the same PIDNetwork namespaces allows container to use separate virtual network stack, loopback device and process space

Mount namespaces isolate the set of file system mount pointsUTS namespaces isolate system identifiers – nodename and domainnameIPC namespaces isolate certain interprocess communication (IPC) resourcesUser allow you to specify a range of host UIDs dedicated to the container. A process can have full root privileges for operations inside the container.

Cgroups ensure that a single container cannot exhaust a large amount of system resources

Cgroups allocate CPU time, system memory, network bandwidth, or combinations of these among user-defined groups of tasks

9

SELinux and Multi Category Security

● Mandatory Access Control● SElinux is a LABELING system● Every Process has a Label● Every file, Directory, System object has a Label● Policy rules control access between labeled processes and labeled

objects● The Kernel enforces the rules

10

SELINUX - MAC - MCS - Process

system_u:system_r:container_runtime_t:s0

SElinux Policy module for the container

The OOTB SElinux policy container.te defines what you can execute and access with the label container_runtime_t

[root@osemaster ~]# ps -efZ | grep docker-containerd-shim-currentsystem_u:system_r:container_runtime_t:s0 root 3035 1479 0 Feb15 ? 00:00:01 /usr/bin/docker-containerd-shim-current 4d254785cbc6ee7aae8facc48555251e2385f65d89553b319b6324b1501e4b16 /var/run/docker/libcontainerd/4d254785cbc6ee7aae8facc48555251e2385f65d89553b319b6324b1501e4b16 /usr/libexec/docker/docker-runc-current

11

SELINUX - MAC - MCS - Files

container_var_lib_t / svirt_sandbox_file_t

SElinux Policy module for the container

[root@osemaster ~]# ls -lZ /var/lib/docker/containers/97de4217a04b6532e312cfb3e4638529aeb7dfa281a2cc067e092fcee82e6737/-rw-r-----. root root system_u:object_r:container_var_lib_t:s0 97de4217a04b6532e312cfb3e4638529aeb7dfa281a2cc067e092fcee82e6737-json.log-rw-rw-rw-. root root system_u:object_r:container_var_lib_t:s0 config.v2.json-rw-rw-rw-. root root system_u:object_r:container_var_lib_t:s0 hostconfig.json-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0 hostname-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0:c0,c1 hosts-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0 resolv.conf-rw-r--r--. root root system_u:object_r:container_var_lib_t:s0 resolv.conf.hashdrwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c0,c1 secretsdrwx------. root root system_u:object_r:container_var_lib_t:s0 shm

12

SECCOMP - DROPPING PRIVILEGES FROM CONTAINERS

CAP_SETPCAPCAP_SYS_MODULECAP_SYS_RAWIOCAP_SYS_PACCTCAP_SYS_NICECAP_SYS_RESOURCECAP_SYS_TIMECAP_SYS_TTY_CONFIGCAP_AUDIT_WRITECAP_AUDIT_CONTROLCAP_MAC_OVERRIDECAP_MAC_ADMINCAP_SYSLOGCAP_NET_ADMINCAP_SYS_ADMIN

Modify process capabilitiesInsert/Remove kernel modulesModify Kernel MemoryConfigure process accountingModify Priority of processesOverride Resource LimitsModify the system clockConfigure tty devicesWrite the audit logConfigure Audit SubsystemIgnore Kernel MAC PolicyConfigure MAC ConfigurationModify Kernel printk behaviourConfigure the network:

- Setting the hostname/domainname- mount(),unmount()- nfsservctl- ….

13

SECCOMP - REMOVE PRIVILEGES FROM CONTAINERS

A root user inside a container running in OpenShift has none of the previous capabilities available!

14

Read Only Mounts

/sys/proc/sys/proc/sysrg-trigger/proc/irq/proc/bus

Container Build Time Security- Content in the containers, Trusted Containers- Responsibility of maintaining Container Images- Container Registry - RBAC, Public and Private Registries- Scanning for Vulnerabilities- Integrating Container Scanning and SAST into CI/CD- Future Vulnerabilities- Build Automation to rebuild during fixes

● Red Hat Container Registry

● Policies to control who can deploy which containers

● Certification Catalog

● Trusted content with security updates

HOST OS

CONTAINER

OS

RUNTIME

APP

HOST OS

CONTAINER

OS

RUNTIME

APP

16

Image governance and private registries

● What security meta-data is available for your images?

● Are the images in the registry updated regularly?

● Are there access controls on the registry? How strong are they?

Trusting Container Content

Red Hat provides Trusted Container Images (registry.access.redhat.com)Languages: PHP, Python, Ruby, Perl, Node.js, Java, .Net Core

Enterprise Grade JBoss Middleware: EWS, EAP, BPM, BRMS, RH SSO, Data Grid, DataVirt, 3Scale

Databases: MySQL, Mongo, PostgreSQL, Maria

CICD: Jenkins

Partners Images: Container Certification by Red Hat.

Publish fixes to the content in this registry and notify you.

18

Red Hat Registry: Container Health Index

https://access.redhat.com/articles/2803031

Private RegistriesWhat if public registry (like DockerHub) is down?

Enterprise security policies may not allow your Container Images to be pushed outside.

Caching images in Private Registry

- Atomic Registry with RBAC- JFrog Artifactory- Docker Trusted Registry

20

RESTRICT WHERE YOUR CONTAINERS COME FROM

- name: allow-images-from-internal-registry onResources: - resource: pods - resource: builds matchIntegratedRegistry: false - name: allow-images-from-dockerhub onResources: - resource: pods - resource: builds matchRegistries: - docker.io

Responsibilities in the new paradigmOperations

- Maintain Trusted OS Base Images (RHEL, RHEL-Atomic)- Ensure these are good with Linux Kernel

System Engineers/Architects

- Maintain Middleware Images- Control build process - S2I, CICD- Reference Architectures with Trusted middleware

Development teams

- Write code that layers on approved images- Ensure code is clean, opensource software vullnerabilities are handled

Container ScanningFrequency

Scan containers as soon as they are created

Scan containers that get into registry

Ongoing basis- Identify any new vulnerabilities

“Deny execution of containers that are vulnerable”

23

Scanning containers when created

https://www.youtube.com/watch?v=65BnTLcDAJI

Run the scan at this point

Scanning ToolsAtomic Scan - configurable with different scanners

OpenScap Scanner - also built into CloudForms

BlackDuck

JFrog XRay

25

Identify Vulnerabilites on an ongoing basis

26

Vulnerable? CloudForms takes Action!

27

Prevent Image From Running

28

Build Automation - when there are updated images

Container Platform Security- Authentication- Authorization- Running Privileged Containers- Network Isolation- Securing communication between hosts- Storage Security- Deploying workloads to specific hosts of your choice

API Authentication• OAuth Access Token

– Obtained from OAuth server using endpoints

– Sent as

• X.509 Client Certs

– HTTPS connection

– API Server verifies against a trusted cert authority bundle

– API Server creates and distributes certs to Controllers to authenticate themselves

• Request with invalid token or cert gets a 401

• No cert or token gets system:anonymous user and system: unauthenticated virtual group

<master>/oauth/authorize <master>/oauth/token

Authorization: Bearer access_token=…

Authentication

OAuth Client

OAuth Server

Master

Client Types:openshift-web-consoleopenshift-browser-clientopenshift-challenging-client

Identity Provider

1. Authentication Request

2. Determine Identity

3. Access token

Configured Identity Providers:- LDAP- GitHub- GitLab- Google- OpenID Connect- HTPasswd- Keystone

Authentication Requests:<master>/oauth/authorizeWWW-authenticate<master>/oauth/token/request

AuthorizationRoleBinding associates Users/Groups with Roles

- Cluster RoleBindings- Project RoleBindings

33

Container Deployment Permissions (SCC)

Network Isolation with Network Policy Objects

Add Network Policies To Allow Specific Incoming Traffic

35

Secured Communications between Hosts

Secures cluster communications with IPsec

● Encryption between all Master and Node hosts (L3)

● Uses OpenShift CA and existing certificates

● Simple setup via policy defn○ Groups (e.g. subnets)○ Individual hosts

Master

P1

Nodes

P2

172.16.0.0/16

36

Secure storage by using

● SELinux access controls

● Secure mounts

● Supplemental group IDs for shared storage

Securing Storage attached to Containers

37

Isolate Workloads by labeling Nodes

Node 1east

Node 2east

Node 1west

Node 2west

Master / Scheduler

$ oadm new-project myproject \ --node-selector='type=user-node,region=east'

pod pod

38

Securing federated clusters across data centers or environments

● Authentication and authorization

● API endpoints

● Secrets

● Namespaces

FEDERATED CLUSTERS (Roadmap)ROLES & ACCESS MANAGEMENT

Source: Building Globally Distributed Services using Kubernetes Cluster Federation. October 14, 2016

39

FEDERATED CLUSTERS (Roadmap)ROLES & ACCESS MANAGEMENT

Source: Building Globally Distributed Services using Kubernetes Cluster Federation. October 14, 2016

API

Repl Ctrl

Ubernetes

state

API

Repl Ctrl

Kubernetes Cluster

state

API

Repl Ctrl

Kubernetes Cluster

state

Application Security - Running as Containers- API Management- Red Hat Single Sign-on- HTTP(s)- Securing sensitivedata as Secrets- Calling services running outside a Container Platform

41

Container platform & application APIs

● Authentication and authorization

● LDAP integration

● End-point access controls

● Rate limiting

API Management

No SSL

My Apphttp://myapp.mydomain.com

Router

My AppMy App

Edge Termination

My App

https://myapp.mydomain.com Router

My AppMy App

Passthrough Termination

My App

https://myapp.mydomain.com Router

My AppMy App

Reencrypt Termination

My App

https://myapp.mydomain.com Router

My AppMy App

Secrets

Sensitive Info: Passwords, Client Config files, dockercfg etc

used by application containers provided as secrets

Never come to rest on Nodes

Stored in ETCD. Encrypted starting OCP 3.6.1

kubernetesMasterConfig:

apiServerArguments:

experimental-encryption-provider-config:

- /path/to/encryption.config

47

Calling External Services using Egress Router

The OpenShift egress router runs a service that redirects egress pod traffic to one or more specified remote servers, using a pre-defined source IP address that can be whitelisted on the remote server.

NODEIP1

EGRESSROUTER

PODIP1

EGRESS SERVICE

INTERNAL-IP:8080

EXTERNAL SERVICE

Whitelist: IP1

POD

POD

POD

...- name: EGRESS_DESTINATION value: | 80 tcp 1.2.3.4 8080 tcp 5.6.7.8 80 8443 tcp 9.10.11.12 443 13.14.15.16...

Signing

GPG Keyringpolicy.json

Signed OpenShift Node

OpenShift Registry

image-auditor rolegpg2 --gen-keyatomic push --sign-byoc adm verify-image-signature

Cluster Admin

Unsecure Container

Ansible for Key Propagation

OpenShift NodeTenant

Select from ContentimageStreams

templatesimages

if/then

sandboxProject

productionProject

nodeLabelSelector

nodeLabelSelector

1

2

3

1

Tenant Usage StepsCluster Admin Setup Steps

23

Upcoming!!

Thank you!!