PuppetConf 2016: High Availability for Puppet – Russ Mull & Zack Smith, Puppet

High Availability for Puppet

Russ Mull - @mullr Senior Software Engineer

Zack Smith - @acidprime Principal Professional Services Engineer

Puppet Services related to High AvailabilityThe important bits of Puppet to make highly available

Enterprise Readiness: High Availability

Auto scaling Active/ Active

Robust Backup and Restore

tooling

Disaster recovery in Multi

datacenter/ Geo diverse

environments

Eliminate Single Points of Failure

Building capabilities that matter

Puppet Runs continue

High Availability for Puppet - Puppetconf 2016

Building a new catalogClassification, Exported Records, Hiera Data and puppet code being synced from version control

4

run

Puppet Code

Classifier


Two of Everything!don’t forget about your external services like git, ldap etc

5

Cl

Pup

Classifier

Pup

Classifier

Balancer“Load”

check

check backup

Lets start with the basics

Certificate Authority FilesPuppets SSL implementation

7


Puppet CA Replication Components

CA private key and cert

Signed Directory

Serial file

Certificate Revocation List (CRL)

8

serial

3E8

crl.pem

signed

101

ca_crt


CA Private key

ca

If you don’t care about revocation (security/revocation) this is the only file needed to replicate

9

ca_key.pem

ssl

ca_crt.pem


Signed Directory

signed

Used when checking for duplicate CN ( certs with the same name)

10

host1.company.com.pem

ca


Serial fileTracking the next numeric serial to be issued to new agent

11

serial

ca

3E8

decimal: 1000

decimal: 10013E9


Certificate Revocation ListTracking revoked certificates

12

crl.pem

cadecimal: 1000

decimal: 1001

1000

1001

serial3E9


Simply copy your ssldir ahead of the second installation

13

$ssldirscp -r

$ssldir

installer

CA

When using old versions of PE delete the pe-internal* certs post transfer , pre install

DR site

1.

2.

PostgreSQL ReplicationDatabase level synchronization

14


Streaming ReplicationThis happens as the postgres database layer

15

PostgreSQL PostgreSQL

PuppetDBPDB PuppetDBPDB

22

Read (Standby)Write

5432

write ahead logs

R

W1

2

16MB


Split Reads and writesCan survive temporary failures of the write master

16


PuppetDBPDB

Read (Standby)Write

5432

W R

WWW

queue


Promote Standby to WritableThis happens as the postgres database layer

17

PostgreSQL

WriteRead (standby)

PostgreSQL

PostgreSQL

Write

5432

PostgreSQL

Read (standby)

5432

Multi master PuppetDB BetaPuppet Enterprise Only

18


Master side FailoverThis is know as “terminus” failover as its handled in the puppetdb terminus package code

19

PuppetDBPDB

[main] server_urls = https://primary:8081, https://replica:8081

Primary

PuppetDBPDB

Replica1 2


Command Broadcastcommand_broadcast = true in puppetdb.conf

20

PuppetDBPDBPuppetDBPDB


PuppetDB Replication ReconciliationReconciliation happens on an interval

21

PostgreSQL

8081

Write

PostgreSQL

PuppetDBPDB

Write

PuppetDBPDB

Sync Interval

Puppet Enterprise HAComing soon…

22


Simple HAMonolithic master + Replica

23

P Primary R Replica


Large Environment InstallationMonolithic master + Compile masters + Replica

24

R Replica

BalancerLoad

P Primary

BalancerLoad


New: Agent Side Failover!Shipping in Puppet 4.6 and higher, PE 2016.4+

25Primary Replica

1 2


Use Cases

● Puppet runs keep working

● Promote replica to master

26

When master is unreachable


When the master is unreachable

● Run puppet

● Promote replica to master

27

You can:

You can’t● Change classification

● Deploy new puppet code

● Issue new certs

● Use the Puppet Enterprise Console

● Use Application Orchestrator

Provisioning Replica(monitoring replication)

28


Command Line Interface

29

puppet infra provision replica <hostname>replica.mycorp.net

puppet infra enable replica

replica.mycorp.net

puppet infra status. . .


1. Provision Replica

30

puppet infra provision replica <hostname>replica.mycorp.net


2. Monitor status of replication

31

puppet infra status> Per-service ‘alerts’> Visible in the UI as well


3. Enable replica

32

puppet infra enable replicareplica.mycorp.net

Replica ServicesServices on a provisioned replica

33


What’s a replica?

● Compile Master

● PuppetDB (r/w)

● RBAC, classifier, activity (r/o)

● Orchestrator data (not running)

● CA data (r/o using a proxy)

34

R ReplicaPuppetDBPDB

Console

File Sync ReplicationReplicate Code Directory and Certificate Authority Data

35


File Sync - Compile Masters

M

Master of Masters

MOM

Compile Master

COMC

Compile Master

COMC

Compile Master

COMC


File Sync - CA Replica data

Primary Master Replica Master

R ReplicaP Primary

8140

ssl ssl


Puppet Enterprise CA Proxy

Primary Master Replica Master

R ReplicaP Primary

ssl CSR

Database ReplicationReplicate the data used in your PE deployment

39


PGLogical Replication

PostgreSQL

RBAC

NCClassifier 5432

5432

PostgreSQL

RBAC

NCClassifier

Write Read (Standby)

PuppetDBPDB

PuppetDBPDBNot synced


PE HA - ReplicationPuppetDBPuppetDBPDB PuppetDBPDB


PGlogical

PGlogical

FileSync

Primary Replica

RBAC

NCClassifier

W

W

RBAC

NCClassifier

R

R


Puppet Enterprise HA - Beta Signup

Interested in what you heard? Please signup for our HA beta program through the Puppet Enterprise Support portal

42


43

https://goo.gl/Z85HLS

PE HA Beta Signup

Support Knowledge base

Z 8 5 H L S

PuppetConf 2016: High Availability for Puppet – Russ Mull & Zack Smith, Puppet

Technology