{code} and containers

{code} and containers

Kendrick ColemanDeveloper Advocate@kendrickcoleman

WHY IS OSS IMPORTANT?

Intangible Benefits of Open Source

Not reinventing the wheel

Customization with benefits

Motivated workforce Attracting top talent

Standardized practices

Business acceleration

Cleaner software

Cheaper

Customer goodwill

Community support

Innovation

Flexibility

Freedom

Integration

True Benefits of Open Source

Not reinventing the wheel

Customization with benefits

Motivated workforce Attracting top talent

Standardized practices

Business acceleration

Cleaner software

Cheaper

Customer goodwill

Community support

Innovation

Flexibility

Freedom

Integration

FIFI and Integration

• Freedom – to run software for any purpose and to redistribute it

• Innovation – the ability to leverage a collaborative community of developers

• Flexibility – to deploy the software in a manner that best meets the organization’s requirements

• Integration – Ability to easily integrate open source software with existing infrastructure

It’s not about cost, it’s about integration

“{code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure. Enabling Dell EMC to be relevant in DevOps and open source communities through contribution, engagement and technical solution leadership.”

codedellemc.com

Open source at Dell EMC

– Contribute to meaningful OSS projects– Create new thought leading OSS applications– Drive awareness of OSS opportunities with

product teams– Participate in relevant community

engagement projects– Acting in the interest of building a community

“{code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure. They enable Dell EMC to be relevant in DevOps and open source communities through contribution, engagement and technical solution leadership.”

codedellemc.com

Digital Business is Driving Software Innovation

Microservices

APIs

Open source

Containers

Cloud Native

Frameworks

AnalyticsInsights Drive New Functionality,

Which Drives New Data

Applications/IoTTransforms Business

DataGenerated By

New Applications

Open source

Containers

Digital Business is Driving Software Innovation

Microservices

APIs

Cloud Native

Frameworks

AnalyticsInsights Drive New Functionality,

Which Drives New Data

Applications/IoTTransforms Business

DataGenerated By

New Applications

Keys to Freedom and Flexibility

• Embrace DevOps and open source

• Remove unnecessary complexity

• Run all applications from containers

• Operate everything as software, including storage

• Run applications as services

• Use platform that orchestrates and consumes all infrastructure natively for applications

Applications are Changing

Loosely Coupled Services

Many Servers

~2000 Today

Monolithic

Big Servers

Slow Changing

Rapidly updated

Operating 3rd Platform Infrastructure

• Maturity improves as you move upward

• Skipping levels causes limitations

• Composable layers require integration and automation

• Requires developer oriented approaches to operating infrastructure

}• Applications get abstracted from infrastructure

• Identity and state no longer encapsulated with infrastructure

• Resiliency moves into application

}

Containers are Key

• Lightweight

• Becoming defacto application packaging standard

• Package of software binaries and dependencies

• Easily portable across environments (on-prem and cloud)

• Allows ecosystem to develop around its standard

• Docker, Mesos, and Kubernetes are currently the most popular container technologies

• A persistent container stores data in an external volume

• Additionally an opportunity to run old applications in new ways

Code andDependencies

Container

Optimizing and Enabling Rapid Innovation

Virtual machines

Server Public Cloud

Disaster Recovery

Developer Laptop

Server Cluster

Data Center

Generic Persistence

Web Front EndBackground Workers

SQL Database

NoSQLScale-Out Database

Queue API Endpoint

Development Test & QA Production Scale Out

Deployment and Compute

RackHD

github.com/rackhd

github.com/codedellemc/docker-

machine-rackhd

Challenges of deploying container platforms

Visibility & Ease of Use

Elasticity

Data Storage & Protection

Platform Sprawl & Siloed Organizations

Financial Viability & Existing Investments

Many Applications, Different Needs

BusinessChallenges

InfrastructureChallenges

THERE IS NO SUCH THING AS A

”STATELESS” ARCHITECTURE

IT’S JUST SOMEONE ELSE’S PROBLEM

- Jonas Boner, CEO of Lightbend

hub.docker.com

7 of the top 12 require persistence

Do you have any of these in your data center?

• Databases– Postgres, MongoDB, MySQL, MariaDB, Redis, Cassandra

• Search, Analytics, Messaging– ElasticSearch, LogStash, Kafka, RabbitMQ

• Content Management– Wordpress, Joomla, Drupal, SugarCRM

• Service Discovery– Consul, Zookeeper, etcd

• Continuous Integration and Delivery– Jenkins, GitLab, SonarQube, Selenium, Nexus

• Custom Applications– That Java app your company built

Stateful andpersistent applications

What's the problem?

• When I run a persistent application in a container, where does my data get stored?– The container holds the data

directory and structure of the entire application

– Optionally use local volumes

• Stateless applications work well – nginx, httpd, kibana, haproxy,

memcached, solr, celery

$ docker run -v redisData:/data redis

redisData

/etc /var/bin /opt/data

What's the problem?

• Lose a container– Lose the data

• Lose a server– Lose the data

• Local data storage– Failed hard drives or failed RAID– Can not scale beyond the

physical limit of the server


Introducing REX-Ray

REX-Ray

The leading container storage orchestration engine enabling persistence for cloud native workloads

rexray.codedellemc.com

Cloud Native Interoperabilityrexray.codedellemc.com

DOCKER APACHE MESOS KUBERNETES AND OPENSHIFT

CLOUD FOUNDRY

Use the stand alone Docker Engine to run a stateful application or

combine it with Docker Swarm Mode to turn your application into a robust

service.

Use any framework that orchestrates containers

such as Marathon or Aurora to provide

persistent storage for stateful services.

Run stateful applications in pods and stateful sets through FlexREX, and

benefit from a broader set of storage platforms and

CLI management capabilities.

Introduce a storage broker to move beyond

just 12-factor applications to support

persistence requirements in your

PaaS.

REX-Ray Featuresrexray.codedellemc.com

PERSISTENT STORAGE ORCHESTRATION FOR

CONTAINERS

ENTERPRISE READY OPEN SOURCE TRUSTED INTEROPERABILITY

• Run any application using multiple storage platforms.

• Resume state and save data beyond the lifecycle of a container.

• Containers aren’t just for stateless applications anymore.

• High-availability features for container restarts across host

• CLI intuitiveness• Contributions from the

cloud native community and the leading storage vendor in the world.

• A completely open and community driven project

• Constantly innovating and providing new integration points.

• Community contributed drivers, features, and additional functionality

• Built using an interface library called libStorage.

• Includes reusable orchestration and lifecycle operations that satisfy the integration of cloud native and storage platforms today and tomorrow.

REX-Ray Featuresrexray.codedellemc.com

• A single interface to all of your storage platforms.

• A common orchestration and volume lifecycle operations no matter what type of platform you choose.

• Includes support for storage platforms types that cover block, file, and object.

• Run any application that has any type of storage requirement

• Installed as a single binary or deployed as a container.

• Can be configured to include one or multiple storage platforms from a single stateless service.

• Multiple architectural choices allow flexibility for deployments.

• Configure as standalone to serve in a decentralized architecture.

• Leverage the client/agent and controller for a centralized architecture

MULTIPLE STORAGE PLATFORM SUPPORT

STORAGE AGNOSTIC EFFORTLESS DEPLOYMENT

STANDALONE OR CENTRALIZED

Storage Platform Integrationrexray.codedellemc.com

• Elastic Block Storage (EBS)

• Elastic File Storage (EFS)

• Simple Storage Service (S3)

• Use thin-provisioned EBS volumes as an alternative to reduce costs in AWS.

• ScaleIO (block)• Isilon (NFS)• ECS (object)

• Persistent Disks (PD)

• Cloud Storage Buckets (CSB)

• Digital Ocean Block Storage attaches persistent disks your droplets.

• Blob Storage Unmanaged Disks (block)

• S3 compatible object storage

• Local Disk

• RADOS Block Devices (RBD)

• RADOS Gateway (RGW)

• Mount a S3 bucket as a directory using Fuse

Solving the problem

• REX-Ray is installed and configured on all hosts in the cluster as a stateless service

• REX-Ray acts as storage plug-in and container engines re-direct storage operations to REX-Ray– Create/Mount/Unmount/Delete/

Snapshot

$ docker run --volume-driver=rexray -v redisData:/data redis

/redisData


Solving the problem

• Lose the container or lose the server– Data persists and remains in tact

on the remote storage platform

/etc /var/bin /opt

/redisData

Solving the problem

• Attach the volume to a new container on a different host– Equivalent of a hard reset.

Application starts and resumes from the last write to disk

– Container schedulers can perform automated failover

• Scalability– Application data can scale to the

maximum supported by the storage platform

/etc /var/bin /opt/data /redisData

Development to Production Lifecycle

Deploying containers using REX-Ray stays consistent from local development, testing in the cloud, to production in on-premise datacenters

ScaleIOIsilon

all supported storage platforms are not shown*

Who’s Using REX-Ray?

…and many more

Docker Integration

• Use the stand alone Docker Engine to run a stateful application or combine it with Docker Swarm Mode to turn your application into a robust service.

• REX-Ray development started when the Docker Volume Driver interface was introduced in 1.7 Experimental

• One of 6 available drivers during DockerCon 2014 debut of the Docker Volume Driver

• {code} team has submitted 6 patches and over 100 lines of code to the Docker Engine for Storage

• Provides full volume lifecycle capability

Docker Integration

• Deploy as a host based service or as a Docker PluginAvailable in the Docker Store

$ docker service create \--name redis \--replicas 1 \--mount type=volume,src=redisData, dst=/data,volume-driver=rexray \redis

redisData


Storage Platform

Persistent Volume

Docker Swarm Integration

Mesos Integration

• Docker Integration• Use Docker as the underlying container technology• Specify REX-Ray as the Volume Driver for any persistent storage use case

REX-Rayrexray.codedellemc.com

Mesos Integration

• Docker Volume Driver Isolator Module• No remote storage capability using Mesos before September 2015• {code} was first to bring an abstracted persistent storage model• Freedom to use the Mesos Containerizer with ANY docker volume driver

(that means you don’t need Docker)• Merged upstream into Mesos 1.0 to provide storage platform support

for any vendor who has written a docker volume driver, including REX-Ray

• Every company that boasts storage integration with Mesos uses the {code} contributed module.

DC/OS Integration

• DC/OS is a distributed operating system based on the Apache Mesos distributed systems kernel.

• DC/OS 1.7 Release ships with REX-Ray embedded• DC/OS 1.8 Release ships with DVDI embedded for Mesos 1.0• Featured in the documentation - https://dcos.io/docs/1.8/usage/storage/external-storage/

Mesos Container Framework Integration

• Marathon from Mesosphere

• Apache Aurora (originally used by Twitter)

• ElasticSeach

ScaleIO Framework for Mesos

• A Software-based Storage Framework using ScaleIO to turn commidy hardware and direct attached storage (DAS) into a globally accessible and scalable storage platform that can be deployed anywhere

• Framework installs and configures ScaleIO on all Mesos Agent (compute) nodes. As more Agents are added to the cluster, ScaleIO is installed and adds storage resources to the cluster.

• Deploy and Configure ScaleIO without knowing operational details

• Centralized Monitoring of the storage platform

• Persistent storage native to the container scheduling platform using REX-Ray

• https://github.com/codedellemc/scaleio-framework

https://github.com/codedellemc/scaleio-framework



{code} + Mesos

Mesos Module DVDI is officially merged into Mesos 1.0 and allows

any Docker Volume Driver to be used with Mesos Frameworks

REX-Ray is installed on every host and performs storage

orchestration

MARATHONScaleIO-Framework

installs ScaleIO software on all Mesos Agents

SUPPORTED STORAGE PLATFORMS WITH REX-RAY

Kubernetes Integration

• Kubernetes offers two approaches for storage integration.

• An “in-tree” volume plugin for a platform. The storage interface code is directly embedded into Kubernetes. The downside is that plugin velocity (the speed at which a plugin can be added, enhanced, or patched) is gated by the Kubernetes release cycle.

• The second approach is to leverage the FlexVolume volume plugin. This plugin uses externally installed and managed software to handle basic Attach/Detach/Mount/Unmount storage operations.


• REX-Ray provides an adapter script called FlexRex which integrates with the FlexVolume plug-in to interact with the backing storage system

• Allows pods to consume data stored on volumes that are orchestrated by REX-Ray. Using Kubernetes' FlexVolume plug-in, REX-Ray can provide uniform access to storage operations such as attach, mount, detach, and unmount for any configured storage provider.

• Run stateful applications in pods and stateful sets through FlexREX, and benefit from CLI management capabilities.

• Use any REX-Ray supported storage platform• Dynamic Provisioning and storage classes are not available as a

limitation of FlexVolume and requires volume create/delete to be managed outside of Kubernetes

$ rexray flexrex install


• ScaleIO is part of the core Kubernetes code and a first class native storage provider

• ScaleIO can take full advantage of the Kubernetes volume lifecycle features including dynamic provisioning and storage classes

• ScaleIO driver is embedded in the standard distribution of Kubernetes

• Contributed code from the {code} by Dell EMC team passes “Google” standard of quality

• Opens a new opportunity for those running Kubernetes in on-premise data centers. It allows utilization of your commodity x86 server hardware for very high performance and highly available storage for running stateful apps in containers.

OpenShift Integration

• Kubernetes is used under the covers of OpenShift.

• Get all the benefits of FlexRex and native use of ScaleIO without any additional configuration

• Support persistence requirements for applications

Cloud Foundry Integration

• Introduce a storage broker to move beyond just 12-factor applications to support persistence requirements in your PaaS.

REX-Rayrexray.codedellemc.com

Dell EMC ScaleIO

Software that creates an enterprise class virtual SAN on top of commodity server hardware

OS, Hypervisor, and Media Agnostic

• Utilizes commodity hardware• “Pay as you grow”— linear predictable costs• No dedicated storage components (FC network, HBAs)• Add servers of any type or configuration to the pool• Retire servers on your own schedule• Eliminate migration• Scale performance linearly

Scale Out Storage for Scale Out Apps

#CodeOpen

Demo

#CodeOpen

Demo

Take our projects for a spin at the {code} Labs

http://github.com/codedellemc/labs

Dell - Internal Use - Confidential54 of Y

codedellemc.comcommunity.codedellemc.com@codeDellEMCblog.codedellemc.com

“{code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure.”

rexray.codedellemc.com

github.com/codedellemc/labs

{code} and containers

Technology