San Francisco HashiCorp User Group at GitHub

HASHICORP

Deploying and Discovering at Scale with Consul & Nomad

HASHICORP

Jon Benson @jm_benson

HASHICORP

Agenda

• Nomad Overview

• Nomad Architecture

HASHICORP

Agenda

• Consul Overview

• Consul Architecture

• Prepared Queries

HASHICORP

Agenda

• Demo

• Questions!

HASHICORP

Advantages of a Scheduler

Higher Resource Utilization

Decouple Work from Resources

Better Quality of Service

HASHICORP

Advantages of a Scheduler

Bin Packing

Over-Subscription

Job Queueing

Higher Resource Utilization

Decouple Work from Resources

Better Quality of Service

HASHICORP

Advantages of a Scheduler

Abstraction

API Contracts

Standardization

Higher Resource Utilization

Decouple Work from Resources

Better Quality of Service

HASHICORP

Advantages of a Scheduler

Priorities

Resource Isolation

Pre-emption

Higher Resource Utilization

Decouple Work from Resources

Better Quality of Service

HASHICORP

NomadHASHICORP

Cluster Scheduler

Easily Deploy Applications

Job Specification

HASHICORP

example.nomad

# Define our simple redis jobjob "redis" {

# Run only in us-east-1 datacenters = ["us-east-1"]

# Define the single redis task using Docker task "redis" { driver = "docker"

config { image = "redis:latest" }

resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 dynamic_ports = ["redis"] } } }}

HASHICORP

Job Specification

Declares what to run

HASHICORP

Job Specification

Nomad determines where and manages how to run

HASHICORP

Job Specification

Abstract work from resources

NomadHASHICORP

Higher Resource Utilization

Decouple Work from Resources

Better Quality of Service

NomadHASHICORP

Multi-Datacenter

Multi-Region

Flexible Workloads

Job Priorities

Bin Packing

Large Scale

Operationally Simple

HASHICORP

Thousands of regions

Tens of thousands of clients per region

Thousands of jobs per region

HASHICORP

Built on Experience

gossip consensus

HASHICORP

• Cluster Management

• Gossip Based (P2P)

• Membership

• Failure Detection

• Event System

HASHICORP

• Gossip Protocol

• Large Scale

• Production Hardened

• Operationally Simple

HASHICORP

• Service configuration and discovery

• Monitoring at scale

• High-availability

HASHICORP

• Service configuration and discovery

• Monitoring at scale

• High-availability

HASHICORP

• Service configuration and discovery

• Monitoring at scale

• High-availability

HASHICORP

• Service configuration and discovery

• Monitoring at scale

• High-availability

HASHICORP

HASHICORP

• Multi-Datacenter

• Raft Consensus

• Large Scale

• Production Hardened

• Coordination (Locking)

• Central Servers + Distributed Clients

• Network Tomography

• Prepared Queries

HASHICORP

Prepared Queries

• Multiple instances of a given service exist in multiple

datacenters

• Clients can talk to any of them, and always prefer the

instances with lowest latency

• Policies can change, desire to not have the clients know

the details of how to locate a healthy service

HASHICORP

Prepared Queries

• New query namespace, similar to services

• Register queries to answer for parts of this namespace

• Clients use APIs, or “.query.consul” DNS lookups to run

queries

HASHICORP

gossip consensus

Mature Libraries

Design Patterns

No Scheduling Logic

HASHICORP

Built on Research

gossip consensus

NomadHASHICORP

Inspired by Google Omega

Optimistic Concurrency

Internal State and Coordination

Service and Batch workloads

Pluggable Architecture

HASHICORP

Single Region Architecture

SERVER SERVER SERVER

CLIENT CLIENT CLIENTDC1 DC2 DC3

FOLLOWER LEADER FOLLOWER

REPLICATIONFORWARDING

REPLICATIONFORWARDING

RPC RPC RPC

HASHICORP

Multi Region Architecture

SERVER SERVER SERVERFOLLOWER LEADER FOLLOWER

REPLICATIONFORWARDING

REPLICATION

REGION B GOSSIP

REPLICATION REPLICATIONFORWARDING

REGION FORWARDING

REGION A

SERVERFOLLOWER

SERVER SERVERLEADER FOLLOWER

NomadHASHICORP

Region is Isolation Domain

1-N Datacenters Per Region

Flexibility to do 1:1 (Consul)

Scheduling Boundary

HASHICORP

Data Model

HASHICORP

Evaluations ~= State Change Event

HASHICORP

Create / Update / Delete JobNode Up / Node Down

Allocation Failed

HASHICORP

External Event

Evalua?on Crea?on

Evalua?on Queuing

Evalua?on Processing

Op?mis?c Coordina?on

State Updates

HASHICORP

Server Architecture

Omega Class Scheduler

Pluggable Logic

Internal Coordination and State

Multi-Region / Multi-Datacenter

HASHICORP

Client Architecture

Broad OS Support

Host Fingerprinting

Pluggable Drivers

HASHICORP

Fingerprinting

Operating System

Hardware

Applications

Environment

Type Examples

Kernel, OS, Versions

CPU, Memory, Disk

Java, Docker, Consul

AWS, GCE

HASHICORP

Fingerprinting

Constrain Placement and Bin Pack

HASHICORP

Fingerprinting

“Task Requires Linux, Docker, and PCI-Compliant Hardware” expressed as Constraints

HASHICORP

Fingerprinting

“Task needs 512MB RAM and 1 Core” expressed as Resource Ask

HASHICORP

Drivers

Execute Tasks Provide Resource Isolation

HASHICORP

Containerized

Virtualized

Standalone

Docker

Qemu / KVM

Java Jar

Static Binaries

Rocket

HASHICORP

Containerized

Virtualized

Standalone

Docker

Rocket

Windows Server Containers

Qemu / KVM

Hyper-V

Xen

Java Jar

Static Binaries

C#

NomadHASHICORP

Workload Flexibility:

Schedulers

Fingerprints

Drivers

Job Specification

NomadHASHICORP

Operational Simplicity:

Single Binary

No Dependencies

Highly Available

NomadHASHICORP

Cluster Scheduler

Easily Deploy Applications

Job Specification

NomadHASHICORP

Higher Resource Utilization

Decouple Work from Resources

Better Quality of Service

NomadHASHICORP

• Million Container Challenge

• hashicorp.com/c1m.html

• github.com/hashicorp/c1m

• Nomad 0.4

• Volume support across drivers

• Advanced networking

HASHICORP

Thanks!We’ll do a quick demo then answer questions…

Jon Benson @jm_benson