Bootstrapping Made Easy Matt Ullmer · Puppet Clusters and Roles Snapshot Server Smartstack Bootstrapping Made Easy: Tying it all together. Cluster Main Monitoring Roles Master Reporting

Matt Ullmermeu@yelp.com

Bootstrapping Made Easy

Yelp’s MissionConnecting people with great

local businesses.

Four DBAs: Jenni, Susanne, Patrick, myself

Hundreds of serversFully managed, configured, and maintained by usPercona Server 5.6Ubuntu 14.04

Who We Are

Basic Architecture

Tying it all together

Global Expansion

Bootstrapping Made Easy

Linux

Apache

MySQL

Python

Basic Architecture

Multiple clusters

Multiple services

Multiple datastores

Basic Architecture: Scaling Out

Time to Automate

Basic Architecture: When To Automate

Basic Architecture

Tying it all together

Global Expansion

Bootstrapping Made Easy

What is the goal of the system?Self ManagingSelf DeployingSelf TestingSelf Healing

Bootstrapping Made Easy: Tying it all together

New hosts automaticallyPick a source host in their clusterPull a backup in real timeReparent to an intermediate master databaseService Discovery

Bootstrapping Made Easy: Tying it all together

New hosts automaticallyPick a source host in their clusterPull a backup in real timeReparent to an intermediate master databaseService Discovery

Bootstrapping Made Easy: Tying it all together

Pick a source host in their clusterPuppet Clusters and RolesSnapshot ServerSmartstack

Bootstrapping Made Easy: Tying it all together

ClusterMainMonitoring

RolesMasterReportingRead

Bootstrapping Made Easy: Puppet Clusters and Roles

profile_mysql::instance::valid_roles: - master - read - reporting

profile_mysql::instance::backup_roles: - master - read - reporting

profile_mysql::read_only::master: '0'profile_mysql::read_only::read: '1'profile_mysql::read_only::reporting: '1'

How we classify environments

EcosystemSuperregionRegionHabitat

Bootstrapping Made Easy: Puppet Clusters and Roles

Pick a source host in their clusterPuppet Clusters and RolesSnapshot ServerSmartstack

Bootstrapping Made Easy: Tying it all together

Snapshot ServerListens on a portCompares incoming connections to expected clusterTriggers an xtrabackup_to_stream encrypted with KMS

Bootstrapping Made Easy: Tying it all together

class BackupStreamingRequestHandler(BaseRequestHandler): def handle(self): cluster_name = self.request.recv(4096).rstrip() if cluster_name != get_cluster(): return

os.nice(10)

self.request.setsockopt(socket.SOL_SOCKET, SO_PRIORITY, 1)

fileno = self.request.fileno()

os.dup2(fileno, sys.stdin.fileno()) os.dup2(fileno, sys.stdout.fileno())

try: os.execv('/usr/bin/xtrabackup_to_stream', ['xtrabackup_to_stream', '3306']) except ProcessExecutionError as e: if e.returncode == 255: print("This error typically occurs when qpress is called with a blank or corrupted stream") print("Try netcatting the snapshot server on the remote host to debug further") raise

Pick a source host in their clusterPuppet Clusters and RolesSnapshot ServerSmartstack

Bootstrapping Made Easy: Tying it all together

SmartstackZookeeper, Nerve, SynapseAdvertise thing 'X' at port 'Y' across the infrastructure on a common IP

Susanne is speaking about smartstack later today @ 3:50 in Ballroom D

Bootstrapping Made Easy: Tying it all together

New hosts automatically Pick a source host in their clusterPull a backup in real timeReparent to an intermediate master databaseService Discovery

Bootstrapping Made Easy: Tying it all together

Pull a backup in real time BackupsBackup TestingStreaming

Bootstrapping Made Easy: Tying it all together

Xtrabackup

Low impact

Compression

Streaming

Backups

xtrabackup.cnf:port = 3306socket = /mysqld/mysqld.sockdatadir = /mysqld/data/innodb_log_group_home_dir = /mysqld/data/open_files_limit = 2048

Wrapping innobackupex:/usr/bin/innobackupex --defaults-extra-file=xtrabackup.cnf --stream=xbstream --parallel=1 /xtrabackup/tmp --slave-info | nice pigz -p 4 > $OUTFILE

Crontab:45 0 * * * root local_backup_wrapper_script45 10 2-30/2 * * root encryption_and_s3_offsite_script

Pull a backup in real time BackupsBackup TestingStreaming

Bootstrapping Made Easy: Tying it all together

Scan for backups, restore, and verify restorability

Backup Testing

Pull a backup in real time BackupsBackup TestingStreaming

Bootstrapping Made Easy: Tying it all together

ssh -A db1 "sudo -H /usr/bin/innobackupex --stream=xbstream --parallel=1 --no-version-check --slave-info /databases/tmp | pigz -p 10" 2> xtrabackup.err | pv --name input -c | pigz -d | pv --name output -c | sudo xbstream -C /databases/mysql -x

Xtrabackup_to_streamAutomated version of the previous streamingAdvertises itself with a snapshot server

Streaming Xtrabackup

compressed_contents() { set +e innobackupex \ --defaults-file='/nail/etc/xtrabackup.cnf' \ --user xtrabackup \ --parallel=8 \ --stream=xbstream \ --slave-info \ --no-version-check \ --tmpdir "$tempdir" \ "$tempdir" 2> innobackupex_stderr innobackupex_retcode="$?" set -e echo "$innobackupex_retcode" > innobackupex_retcode xbstream -c innobackupex_stderr innobackupex_retcode return "$innobackupex_retcode"}

2016-07-20 09:41:43,377 INFO:root:Connecting to snapshot server on 192.168.0.12016-07-20 09:41:43,378 INFO:root:Unpacking backup

New hosts automatically Pick a source host in their clusterPull a backup in real timeReparent to an intermediate master databaseService Discovery

Bootstrapping Made Easy: Tying it all together

Reparent to an intermediate master databaseIntermediate MasterReparenting

Bootstrapping Made Easy: Tying it all together

Intermediate MasterSmartstack DiscoverySaves BandwidthEases ReparentingDon't bootstrap from this host

Bootstrapping Made Easy: Tying it all together

Reparent to an intermediate master database Intermediate MasterReparenting

Bootstrapping Made Easy: Tying it all together

ReparentingDowntime sensu alertingCHANGE MASTER TO…

Bootstrapping Made Easy: Tying it all together

2016-12-15 11:23:27 33529 [Note] Event Scheduler: Loaded 0 events2016-12-15 11:23:27 33529 [Note] /usr/sbin/mysqld: ready for connections.Version: '5.6.32-78.1-log' socket: '/nail/databases/mysql/mysqld.sock' port: 3306 Percona Server (GPL), Yelp Release 12016-12-15 11:30:16 33529 [Note] 'CHANGE MASTER TO executed'.

New hosts automaticallyPick a source host in their clusterPull a backup in real timeReparent to an intermediate master databaseService Discovery

Bootstrapping Made Easy: Tying it all together

Service DiscoveryHealthchecksAuto TopologyJoin the Load Balancer

Bootstrapping Made Easy: Tying it all together

Service DiscoveryHealthchecksAuto TopologyJoin the Load Balancer

Bootstrapping Made Easy: Tying it all together

Service DiscoveryHealthchecksAuto TopologyJoin the Load Balancer

Bootstrapping Made Easy: Tying it all together

We now have all the pieces

Bootstrapping Made Easy: Tying it all together

Basic Architecture

Tying it all together

Global Expansion

Bootstrapping Made Easy

Snapshotting in a new region

How do you launch databases

How do you do service discovery

Global Expansion

Snapshotting in a new region

How do you launch databases

How do you do service discovery

Global Expansion

main: extra_advertise: region:old_region: - region:new_region

Cross region advertising unlocks automatic bootstrapping

Global Expansion

Snapshotting in a new region

How do you launch databases

How do you do service discovery

Global Expansion

How do you getdatabases setup?

Hand rolled?

Named?

Global Expansion

Advice: no host is special

Global Expansion

Amazon ASG

Schedules

Terraform

Global Expansion

Snapshotting in a new region

How do you launch databases

How do you do service discovery

Global Expansion

Discovery Service

Global Expansion

Jenkins builds new topology

Jenkins checks the topology is valid

Jenkins pushes it to all services

Global Expansion

Four DBAsMatt UllmerSusanne Lehmann (25 April - 3:50 PM - 4:40 PM @ Ballroom D)Patrick King (26 April - 1:00 PM - 1:50 PM @ Ballroom G)Jenni Snyder

Yelp DBAs

www.yelp.com/careers/

We're Hiring!

Questions

@YelpEngineering

fb.com/YelpEngineers

engineeringblog.yelp.com

github.com/yelp