Pacemaker: OpenStack's Pid 1

David Vossel <[email protected]>

PACEMAKER

OpenStack SummitMay 21, 2015David Vossel <[email protected]>

OpenStack's PID 1


Story Time


And how Pacemaker Saved the DayThe Future of HA


There once was a database



DB


Not like the other databases


DB


DB



A distributed self replicating database




A distributed self replicating database

Active/Active Replicated Database

DB DB DB DB



DB DB DB DB

Everyone: HOORAY!



DB DB DB DB

Everyone: Load Balance?

Clients

????????????????????


HAProxy enters the scene.



DB DB DB DB

proxy

HA Proxy: No Problem

Clients



DB DB DB DB

proxy


Clients

Everyone:Hooray!



DB DB DB DB

proxy

Clients

* Everyone: ummmmm


Everyone: What if a node dies?


DB DB DB DB

proxy

Clients



DB DB DB DB

Clients

proxy


Everyone: What if a node dies?



DB DB DB DB

Clients

proxy

* Everyone: ummmmm



DB DB DB DB

Clients

proxy

Everyone: But... What if proxy dies?



DB DB DB DB

Clients

proxy

HA Proxy:...

Everyone: But... What if proxy dies?



DB DB DB DB

Clients

proxy

Everyone:What?????

HA Proxy:...



DB DB DB DB

Clients

proxy

Everyone:Uhhh???!!!!??

HA Proxy:...

????????????????????



DB DB DB DB

Clients

Everyone:Hello????

HA Proxy:...

????????????????????


Clients

Everyone:Anyone?

????????????????????


Everyone:I knew this cloud thing wouldn't work...


KeepaliveD: Wait! Guys... I've got an idea!!!

Everyone:I knew this cloud thing wouldn't work...


KeepaliveD enters the scene.


keepaliveD


DB DB DB DB

proxy proxy proxy proxy

VIP VIP VIP VIP

Clients

Everyone:Whoa, Proxy Failover!


keepaliveDkeepaliveD


DB DB DB DB


VIP VIP VIP VIP

Clients

Everyone:Awesome!!




DB DB DB DB


VIP VIP VIP VIP

Clients

Everyone:Awesome!!

Keepalived:: I know right?!!


keepaliveD

Clients

Everyone: No Way!!!

Keepalived: I Rock!

keepaliveD

Active/Active Replicated DatabaseActive/Active Replicated Database

DB DB DB DB


VIP VIP VIP VIP


keepaliveD

Clients

keepaliveD


DB DB DB DB


VIP VIP VIP VIP

Everyone:Wait... Hold up


keepaliveD

Clients

keepaliveD


DB DB DB DB


VIP VIP VIP VIP

Everyone: But.... What actually happens to the “other” nodes?


Peripeteia - per·i·pe·tei·a

a sudden reversal of fortune or change in circumstances, especially in reference to fictional narrative.


keepaliveDkeepaliveD keepaliveD


DB DB DB DB


VIP VIP VIP VIP

ClientsKeepalived:What Other Nodes? Keepalived:What Other Nodes?




DB DB DB DB


VIP VIP VIP VIP

Clients

Clients: HEY?! How come the thing I just wrote isn't in the database?




DB DB DB DB


VIP VIP VIP VIP

Clients

Clients: HEY?! How come the thing I just wrote isn't in the database?

Clients:Yeah, what's going on?!




DB DB DB DB


VIP VIP VIP VIP

Clients

* Everyone: I've made a huge mistake....


The Missing Piece?



Pacemaker: System Level HA

● System level HA is holistic.

Pacemaker


DB DB DB DB


VIP VIP VIP VIP

Pacemaker


DB DB DB DB


VIP VIP VIP VIP


Pacemaker



DB DB DB DB


VIP VIP VIP VIP


● Defines the policy of how to recover a set of applications


Pacemaker



DB DB DB DB


VIP VIP VIP VIP


● Defines the policy of how to recover a set of applications

● Enforces the policy to achieve system wide deterministic behavior.


● We don't question what happened to the other nodes... We know.

Pacemaker Pacemaker


DB DB DB DB


VIP VIP VIP VIP

Back to the Story... How does Pacemaker Help?


● We don't question what happened to the other nodes... We know.

● And how do we know this?

Pacemaker Pacemaker


DB DB DB DB


VIP VIP VIP VIP

Back to the Story... How does Pacemaker Help?


Introducing STONITH


Introducing STONITH

Shoot the Other Node in the Head


STONITH = Pacemaker's Fencing Daemon.

● Pacemaker knows the state of lost/misbehaving nodes

Pacemaker Pacemaker


DB DB DB DB


VIP VIP VIP VIP


STONITH = Pacemaker's Fencing Daemon.

● Pacemaker knows the state of lost/misbehaving nodes

● Because with fencing via STONITH... that state is dead.

Pacemaker Pacemaker


DB DB DB DB


VIP VIP VIP VIP


Quick Recap...


Without Pacemaker+STONITH

keepaliveDkeepaliveD keepaliveD


DB DB DB DB


VIP VIP VIP VIP

ClientsKeepalived:I dunno. Who cares? Keepalived:I dunno. Who cares?


With Pacemaker+STONITH...

Pacemaker Pacemaker


DB DB DB DB


VIP VIP VIP VIP

Pacemaker:They are dead because I killed them..

Clients


The Takeaway.

● Pacemaker and load balancing are NOT mutually exclusive.


The Takeaway.


● Pacemaker and HAProxy are meant for one another.


The Takeaway.


● Pacemaker and HAProxy are meant for one another.

HAProxy

Pacemaker


PacemakerThe Distributed PID 1


Modern PID 1's role.

● SystemD:

● Launch services parallel● yet observe strict ordering between dependent services.● Monitor/recover failed resources.

systemd

galerarabbitmqStart Order Unrelated dependencies

start in parallel.

Ordering is enforced. These services can start in parallel only after their dependencies start.Nova

redis

ceilometer


The Problem

● OpenStack services are not isolated to a local machine.

systemd

Galera ClusterForm Galera cluster

Then Start Nova.

Nova

NODE1 NODE2 NODE3 NODE4

systemd systemd systemd


The Problem

● OpenStack services are not isolated to a local machine.

● SystemD Can't coordinate this.

systemd


systemd systemd systemd


Then Start Nova.

Nova


The Fix.

● But Pacemaker can

● because Pacemaker is distributed.

Pacemaker



Then Start Nova.

Nova


The Fix.

● Pacemaker, just like systemd, can...

● Launch services parallel● yet observe strict ordering between dependent services.● Monitor/recover failed resources.

Pacemaker

galerarabbitmqStart Order Unrelated dependencies

start in parallel.

Ordering is enforced. These services can start in parallel only after their dependencies start.Nova

redis

ceilometer


The Fix.● Except pacemaker can coordinate this across any number of nodes.

Pacemaker


RabbitMQ Cluster

NODE5 NODE6 NODE7 NODE8 NODE9

Galera Cluster Redis Cluster

Ceilometer ClusterNova Cluster


The Fix.● Except pacemaker can coordinate this across any number of nodes.

● With any number of resources

Pacemaker

NODE1 NODE2 NODE3 NODE4 NODE5 NODE6 NODE7 NODE8 NODE9

HAProxy HAProxy HAProxy

VIP-Galera VIP-RedisVIP-Rabbit VIP-Nova

HAProxy

VIP-keys

HAProxy

VIP-cinder VIP-celio

Keystone ClusterCeilometer Cluster

Nova Cluster

Cinder Cluster

VIP-glance

HAProxy

VIP-neutron

HAProxy

Glance Cluster

Horizon Cluster

HAProxy HAProxy

RabbitMQ ClusterGalera Cluster Redis Cluster

Neutron Cluster

Swift Cluster

Heat Cluster


Resource Constraints

● Pacemaker has unique capabilities for managing resources and modeling complex resource dependencies.

● Examples:

● Start resource X then start resource Y● Colocate resource X with resource Y● Resource X prefers node A over node B● Resource X prefers node A between 8am-5pm


ArchitectureHA OpenStack Controller Nodes


How it works... in one sentence

Pacemaker managing Virtual IPs + Load Balancers + Controller services to maximize the availability of OpenStack APIs


Pacemaker

HA-proxy

VIP

Service Service Service

NODE1 NODE2 NODE3

Each distributed service has a front end Virtual IP tied to a Load Balancer.



HA-proxyHA-proxy



Each Load Balancer routes VIP traffic to its respective Active/Active service instances



Pacemaker

HA-proxy

VIP


NODE1 NODE2 NODE3

HA-proxyHA-proxy


Pacemaker

HA-proxy

VIP


NODE1 NODE2 NODE3




HA-proxy


HA-proxy




Active/Active OpenStack Controller service... like Nova, Glance, Keystone, Galera, Redis, Rabbitmq, ect...



Pacemaker

HA-proxy

VIP


NODE1 NODE2 NODE3

HA-proxyHA-proxy


PROXY CLONE

SERVICE CLONE

Pacemaker

HA-proxy

VIP


NODE1 NODE2 NODE3

Scaling with Resource Clones

● Pacemaker's ability to clone services makes scaling trivial.

● Want more instance?

HA-proxy HA-proxy


PROXY CLONE

SERVICE CLONE

Pacemaker

HA-proxy

VIP


NODE1 NODE2 NODE3

Scaling with Resource Clones

NODE4 NODE5 NODE6


● Increment the number of clone instances pacemaker is allowed to run for a service to scale service instances.

HA-proxy HA-proxy HA-proxy HA-proxy HA-proxy


Pacemaker

HA-proxy

Galera VIP

Galera Galera Galera

NODE1 NODE2 NODE3

How it works, continued... ● Services interact with one another using each service's Virtual IP● Example: Both Glance and Nova need access to Galera... Galera is

accessed via the front end Virtual IP and those requests are distributed to the backend galera cluster.

Glance NovaDB Request DB Request

HA-proxyHA-proxy


Deployment StrategiesHA OpenStack Controller Nodes


Collapsed Architecture

PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

HA Proxy HA Proxy HA Proxy

Separate VIP per service

● All controller nodes run the same services.

● VIP+load balancers distribute access to APIs across cloned nodes.



PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S


NOVA VIP

Clients



PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S


Glance VIP

Clients


Collapsed Architecture Scaling

PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S



NODE4

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE5

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE6

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S



Startup Ordering Revisited.

PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S



Bootstrap and start Start Galera Cluster across multiple nodes.


Startup Ordering Revisited.

PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S



Bootstrap and start Start Galera Cluster across multiple nodes.

Then Start Nova instances, which depend on an active Galera cluster.


Stop ordering Ordering Revisited.

PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S



If we're shutting down galera cluster



PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S



Stop everything that depends on Galera. Like Nova...



PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S



Then Shutdown Galera cluster.


Complex Startup Ordering

PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera Redis Slave Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ


Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera Redis SlaveNeutron N

Swift

Heat

Neutron S



Start Redis Clone.



PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ


Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ


Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera Redis MasterNeutron N

Swift

Heat

Neutron S



Promote one instance of Redisclone to be Master Instance.



PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ


Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ


Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera Redis MasterNeutron N

Swift

Heat

Neutron S



Then start Ceilometer cluster whichdepends on Redis


Segregated Architecture

Pacemaker

● Each service runs on its own dedicated hardware.

● Scales much further

● Add capacity where capacity makes sense.

● Requires lots and lots of nodes.



Pacemaker

● Take a closer look.



PacemakerNODE1

Galera

NODE2

Galera

NODE3

Galera

● Possible to have have an entire set of nodes just for Load balancing

● Dedicated cluster for galera

NODE4

Proxy

NODE5

Proxy

NODE6

Proxy

NODE7

Nova

NODE8

Nova

NODE9

Nova VIP VIP VIP

More services that way.

Glance

NODE9

VIP VIP VIP VIP VIP VIP VIP VIP VIP VIP VIP VIP


Mixed Architecture

● Mixture of collapsed and segregated.

● Break some components into separate hardware

● Most of cluster is collapsed

PacemakerNODE1

Galera

NODE2

Galera

NODE3

Galera

NODE4

Proxy

NODE5

Proxy

NODE6

Proxy

NODE7 NODE8 NODE9

VIP VIP VIP VIP VIP VIP VIP VIP VIP VIP VIP VIP VIP VIP VIP

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Redis

Neutron N

Swift

Heat

Neutron S

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Redis

Neutron N

Swift

Heat

Neutron S

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Redis

Neutron N

Swift

Heat

Neutron S


Pacemaker Advantages● Automates bootstrap of services that previously required hand holding.

● Example: Automate Galera bootstrap

1. Find out which galera instance is most up-to-date

2. Bootstrap most current galera instance first.

3. Then sync other galera instances


Pacemaker Advantages. Continued... ● start/stop distributed services in a graceful ordered manner.

● Gracefully and controller node into standby for maintenance.

● Dynamically grow capacity by adding more pacemaker nodes

● Centralized view of distributed service state.


ArchitectureHA OpenStack Compute Nodes


HA for Cattle

● Both Pets and Cattle need High Availability.● Recognize the techniques used for each are different.


Pacemaker for Pets and small Herds.

● No limits in the number of resources.

● Pacemaker supports “n-node” clusters.

● Cluster are limited by the Corosync messaging layer to 16 nodes.

Pacemaker


Pacemaker Remote for the Cattle.

● Pacemaker Remote allows clusters to scale beyond corosync membership layer limitations.

● Pacemaker Remote can scale clusters to 100s possibly 1000s of nodes.

Pacemaker + Pacemaker Remote


The Solution: Pacemaker Remote

● Pacemaker Remote is a single daemon, pacemaker_remoted

Pacemaker Remote

Remote Node



Pacemaker

Node 2 Node 3Node 1

Pacemaker Remote

Node 5 Node 6Node 4Node 8 Node 9Node 7


Node 16

Remote Node

● Pacemaker Remote is a single daemon, pacemaker_remoted● This daemon is a lightweight way of integrating nodes into the cluster.



Pacemaker

Node 2 Node 3Node 1

Pacemaker Remote



Node 16

Remote Node

● Pacemaker Remote is a single daemon, pacemaker_remoted● This daemon is a lightweight way of integrating nodes into the cluster.

● Cluster services spread out across pacemaker and pacemaker_remote nodes as a single cluster partition.

Cluster Services


Pacemaker Remote use-case

Pacemaker

Node 2 Node 3Node 1

Pacemaker Remote



Node 16

Remote Node

● Management of services on Pacemaker Remote can work just like Pacemaker

● But thrives in the cattle use case where every remote instance is identical.

Cluster Services

Cloned service

Pacemaker RemoteRemote Node

Cloned service

Pacemaker RemoteRemote Node

Cloned service


Pacemaker Remote use case

Pacemaker

Node 2 Node 3Node 1

● Cloned services scale quite well on pacemaker remote

Cluster Services

Cloned services

Remote Node

Pacemaker Remote


Compute Node HA Strategy.

PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S




Compute Node HA Strategy.

Compute Service Group

Remote Node

Pacemaker Remote

PacemakerNODE1

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE2

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S

NODE3

Keystone

Ceilometer

Nova Cinder

Glance

Horizon

RabbitMQ

Galera

Redis

Neutron N

Swift

Heat

Neutron S


Separate VIP per serviceLibvirtdD

Neutron Open vSwitch

Ceilometer Compute Nova Compute


Why HA Compute Nodes?

● Maximize the Availability of Compute Instances.● detection of dead Cattle instances● Automate recovery of Cattle instances


Why HA Compute Nodes?

● Maximize the Availability of Compute Instances.● detection of dead Cattle instances● Automate recovery of Cattle instances

● Pacemaker Remote also has a secret weapon.


STONITH + Pacemaker Remote.


The Future


Limits?



How manyservices?




How manyservices?

How manynodes?


Questions?

Visit us at

clusterlabs.org

Pacemaker: OpenStack's Pid 1

Software