Service discovery and puppet

Service Discovery and PuppetMarc Cluet

Who am I?

@lynxman

Engineer based in London

Co-Founder of Ukon Cherry

Working at TrainLine

19 years of experience as a SysAdmin

Founding member of Juju and MAAS while at Canonical

Built a DevOps Engineering Team at Rackspace

Been DevOps’in for the last 7 years

2

Meet other people in London!

3

http://www.meetup.com/London-DevOps/

Running in the Cloud?

4

https://www.flickr.com/photos/pontla/11879018534/

The Problem

Cloud is Hard!

Old Style DNS

TTL a problem for auto-scaling

Non-reactive health checks

Where do I keep my metadata?

5

https://www.flickr.com/photos/qchristopher/5038229030/

The Solution!You were expecting this weren't you

6

Service Discovery

Automatically define your services

Active Health Checking

Dynamically updated service lists

Can be DNS accessible (if needed)

API accessible (win!)

7

https://www.flickr.com/photos/marknye/12992319144/

Service Discovery

8

Service Publication

Service Node A

Service Node B

Service Node C

Service Discovery

9

Service Publication

Service Node A

Health Check

Discovery Agent

Discovery Agent

Service Node B

Health Check

Service Discovery

10

Service Publication

Service Node A

Health Check

Discovery Agent

Discovery Agent

Service Node B

Health Check

Service Discovery

11

Service Publication

Service Node A

Health Check

Discovery Agent

Discovery Agent

Service Node B

Health Check

Service Discovery

12

Service Publication

Service Node A

Health Check

Discovery Agent

Discovery Agent

Service Node B

Health Check

Amazeballs!

13

https://www.flickr.com/photos/neilmartin/31519343/

Amazeballs!

14


Service Discovery

15

Service Node A

Service: web

10.10.10.1 10.10.10.2 10.10.10.3

Service Node B

Service Node C

Service Discovery

16

Service Node A

Service: web

10.10.10.1 10.10.10.2 10.10.10.3

Service Node B

Service Node C

Service Discovery

17

Service Node A

Service: web

10.10.10.1 10.10.10.2 10.10.10.3

Service Node B

Service Node C

Amazeballs!

18


Service Discovery

19

Agent Master Agent Agent Agent Agent

Agent Agent Agent Agent Master Agent

Agent Agent Master Agent Agent Agent

Agent Agent Agent Agent Agent Agent

Service Discovery

20





Service Discovery

21





Service Discovery

22



Agent Agent Master Agent Agent


Service Discovery

23



Agent Agent Master Agent Agent


! !

!

Service Discovery

24





Amazeballs!

25


Service Discovery SolutionsChoices!

26

Service Discovery Solutions

CoreOS Fleet Uses the sidekick

model, uses separate agents to orchestrate

service discovery.

Kubernetes Service All pods declare services to be

discoverable by the cluster.

27

Consul All services are declared by the

agents, the agents themselves are

responsible.


CoreOS Fleet API publication

k/v Strongly Consistent

Container checks

Kubernetes Service API publication

Container checks

Auto-Heal

28

Consul API + DNS publication

k/v Strongly Consistent

Host + Service checks

Puppet modulePuppet module Hiera access

Puppet module Hiera access


29

What is Consul?

It's a Service Discovery System (duh!)

Service Publications (DNS + API)

k/v Storage (strongly consistent)

Health Checks

With encryption! (whaaaaaat)

30

What is Consul?

31

Consul Concepts

Datacenter

Node

Service

Health Check

Watches

ACLs

32

Consul Architecture

33

Consul Master

Consul Master

Consul Master

Consul Architecture

34

Consul Master

Consul Master

Consul Master

Consul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul Agent

Consul Architecture

35

Consul Master

Consul Master

Consul Master

Consul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul AgentConsul Agent

Consul Master

WAN DC

Consul DNS publisher

Port 8600

Use dnsmasq:

server=/consul/127.0.0.1#8600

36

Consul DNS publisher

service name: web

zone: yourDC

<servicename>.service.<zone>.consul

web.service.yourDC.consul

37

Consul Catalog API

API http://localhost:8500

/v1/catalog/

/v1/catalog/datacenters/

/v1/catalog/nodes/

/v1/catalog/services/<servicename>

/v1/catalog/service/<name>

/v1/catalog/node/<nodename>

38

Consul Health Checks

{

"check": {

"id": "mem-util",

"name": "Memory utilization",

"script": "/usr/local/bin/check_mem.py",

"interval": "10s"

}

}

39


40

Server


41

Server

Memory Disk CPU Load Logs


42

Server

Service Service

Consul Restful API


/v1/kv/

/v1/agent/

/v1/catalog/

/v1/health/

/v1/session/

/v1/acl/

/v1/status/

43

Consul Restful API


/v1/kv/

/v1/agent/

/v1/catalog/

/v1/health/

/v1/session/

/v1/acl/

/v1/status/

44

Now comes the Puppet stuff!All of this applied!

45

You can do all this after the talk

https://github.com/lynxman/consul-first-steps

46

https://www.flickr.com/photos/hortlander/6245707871/

Puppet + Consul

47

https://www.flickr.com/photos/hortlander/6245707871/in/

You can integrate at several levels

Puppet Module

KyleAnderson-consul

Hiera

lynxman-hiera_consul

Puppet + Consul

48


You can integrate at several levels

k/v access

venmo-consulr

Templates with Consul

ghdbaston-consul_template

Puppet Module - Install

49


Puppet Forge makes it very easy to install

$ puppet module install KyleAnderson-consul

Puppet Module - Dnsmasq Install

50


$ puppet module install saz-dnsmasq

include dnsmasq

dnsmasq::conf { 'consul':

ensure => present,

content => 'server=/consul/127.0.0.1#8600',

}

Puppet Module - Health Checks

51


consul::check { 'disk_space':

script => 'check_disk -w 5% -c 1%',

interval => '30s',

}

Puppet Module - Services

52


consul::service { 'nginx':

port => '80',

checks => [

{

script => 'check_http -H localhost -w 20 -c 60',

interval => '30s',

},

],

}

Puppet Hiera

53

Puppet Hiera Consul

Hiera Yaml

Puppet Hiera Module - Install

54



$ puppet module install lynxman-hiera_consul

Puppet Hiera Module - Config

55


:backends:

- yaml

- consul

:yaml:

:datadir: /etc/puppetlabs/hieradata

:consul:

:host: 127.0.0.1

:port: 8500

:failure: graceful

:paths:

- /v1/catalog/service

- /v1/catalog/node

Puppet Hiera Module - Config

56


:consul:

:host: 127.0.0.1

:port: 8500

:failure: graceful

:paths:

- /v1/catalog/service

- /v1/catalog/node

Puppet Hiera Module - Arrays

57


notice('Generating rabbitmq cluster members based on Consul information')

$consul_service_array = hiera('rabbitmq',[])

$mq_cluster_nodes = consul_info($consul_service_array, 'Address')

notice("Result: ${mq_cluster_nodes}")


58


notice("Generating neo4j_ha cluster members based on Consul information")

$consul_service_array = hiera('neo4j_ha',[])

$consul_fields = [ 'Address', 'ServicePort' ]

$consul_ha_initial_hosts = consul_info($consul_service_array, $consul_fields, ':')

$ha_initial_hosts = join($consul_ha_initial_hosts, ',')

notice("Result: ${ha_initial_hosts}")

Puppet Hiera Module - Accessing the k/v

59


:consul:

:host: 127.0.0.1

:port: 8500

:failure: graceful

:paths:

- /v1/kv/


60


:consul:

:host: 127.0.0.1

:port: 8500

:failure: graceful

:paths:

- /v1/kv/mystuff/


61


:consul:

:host: 127.0.0.1

:port: 8500

:failure: graceful

:paths:

- "/v1/kv/%{env}/"

Puppet SecurityDon't get your a** on the line!

62

Puppet Security - Github

63


$ git commit -m "All my passwords"

$ git push


64


https://www.flickr.com/photos/tomukas/3554360505/

Puppet Security - Hiera Modules

65


hiera-eyaml

hiera-gpg

Puppet Security - Hiera

66

Puppet Hiera Consul

Hiera eyaml

Puppet Security - Module Install

67



$ puppet module install hiera-eyaml

Puppet Security - Hiera eyaml

68


$ eyaml createkeys

keys/public_key.pkcs7.pem

keys/private_key.pkcs7.pem


69


Usage: eyaml <subcommand>

Please use one of the following subcommands or help for more help:

createkeys, decrypt, edit, encrypt, recrypt, version


70


mysecret: DEC::PKCS7[mypassword]!


71


mysecret: ENC[PKCS7,MIIBmQYJKoZIhvcNAQcDoIIBijCCAYYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAAikKizydVh0wXQrvtoMC7vM9NxfksqwOX2jtajDYMMJwXXP/5zKHjnnGmr+LSXFVkL52FuGentCdityjF0zZEvbZ2D95TWnRTinO9htteC8ZiwqpTeuNJkTJikOEEZvHbNlp6eX381ppKoatV1k0EmIHwsnqeRJN5T9TVScoXOb/1Fre4H7TxSvvaFqo02MWUBaKkWECoEu2PLiuXWEoiLrkDq8pxhjYADvGUJLWC8PUSWT/94075z5UKHYBQgLlFrzG+89Rhm5keTy/cuHsOK9d0nUScjd4m6duCEsvRT5SG/n6GwTEk/cDMqIuvAwNETv2fdepu4z5nR383zlngDBcBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBAJCDkds8PbXeBUMZhFPxWTgDDH1pvUCbCLtWDNVFkW2yZ1NYF06RuqsSTxofHfMwajC+BSPcTu7heMKQnbKP/KE6o=]

Puppet Security Extra time!This is beyond awesome

72

Puppet Security - Extra

73


Puppet Security - Extra

Friday, October 21 • 11:15am - 12:00pm

Using HashiCorp's Vault With Puppet

Seth Vargo, HashiCorp

74

Service discovery and puppet

Technology