Top Banner

Click here to load reader

Infrastructure Automation at Scale

Jul 19, 2015


  • Infrastructure Automation at ScaleWith Puppet and FriendsNick Jones, Senior Cloud Systems Engineer11th September, 2014

  • Introduction

  • ScalabilityScalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth.

  • Scale-abilityRepeatabilityFlexibilityDiversityTransparencyAutomate!

  • Configuration management and automationProgrammatically define server and service configurationRelationships between servicesDriven by dataDescribe and document infrastructureScalable


  • Puppet CommunityNeutron = 8422 LOCNova = 6678 LOCGlance = 3336 LOCCinder = 6379 LOCHorizon = 1681 LOCKeystone = 4020 LOCCeilometer = 3710 LOC~ 5000 CPUs, 1TB RAM

    }2478 LOC to deployOpenStack

  • Open Source

  • Puppet Code # Install the MIBS meta-package package { 'snmp-mibs-downloader': ensure => latest, }

    file { 'snmp.conf': require => Package['snmp-mibs-downloader'], path => '/etc/snmp/snmp.conf', content => 'com2sec local localhost public', }

  • Puppet Deployment PatternsA role is a high-level (business) description of a nodes functionRoles decompose into profiles that include configuration specificsProfiles include various modules and take care of declaring the necessary classes and resources

  • The ForemanLifecycle managementPhysical and virtualExternal node classification (ENC) for Puppet

  • The Foreman - Architecture

  • Foreman and Puppet

  • Foreman Dashboard

  • Developers, developers, developers, developers

  • Development ProcessGit for distributed version controlWork on feature branches locallyPush to local Git repo, mirrored onto GitHubGitHubs social aspects for code reviewMerge into master branch on team consensus

  • Development ToolsPackerVirtual machine image creationTemplate stored in ForemanAutomatically creates Vagrant boxes

    VagrantDefine virtual environmentsPuppet provisionerPortable

    *Introduction Here to talk about infrastructure automation at scale and what that meansDefine scalability with Infrastructure contextTechnology at DataCentredWhats in our toolbag for managing scalabilityPuppet and Friends

    Quick demo*What does scale in the context of infrastructure automation

    Its coping with the ever expanding business requirementsAny business that relies on IT will have expanding requirementsLooking to IT to deliver capability as and when its requiredFlexibility new business directions and capabilitiesAdding new servers and new servicesDoing more with lessBeing cost effective

    Infrastructure automation = laziness

    *Repeatability - Lazy sysadmins would rather put the work into doing things right once and then having whatever it is look after itselfFlexibility Comprehensive toolset that is able to handle almost any relevant taskDiversity Infrastructure perspective - support across a number of platforms and operating systemsTransparency Want to understand how a given part of our infrastructure operates? Look at the Puppet code! Complex systems. Well defined and understood deployment patterns.

    *Traditionally, automation has meant using shell scriptsInflexible, very fragile, hard to accommodate for various edge casesMany, many different shells and quirks in syntaxMention time at Sun Grid put together with shell, expect, and bit of string

    *What is Puppet?Used for configuration management and automation tasksAllows us to programmatically define server and service configurationWhy choose Puppet?DC chose Puppet as its the most mature, huge amount of community and commercial support, massive library of existing modules for us to useUsed by companies like Google, Vmware, Red Hat, othersData-driven infrastructure example defining a new subnet, automatically rolls out changes to network hardware, DNS (PTRs), and so on.Support - not limited to OS platforms big buy in from network hardware vendors for exampleCross-platform, open-source works on Windows, Linux, OpenBSD, etc.The other good thing is that its automatically documenting your infrastructureSingle source of truthResource Abstraction LayerInfrastructure these days can be very variedTeams expected to look after many platformsDoing more with less - teams being scaled downScalable designed to work across thousands of machines

    *Huge amount of support and a massive ecosystem of existing modulesMajority are mature, well maintainedLot of people have done the hard work for you

    Neutron module is ~ 8K LOCNova module is ~ 8K LOCGlance is ~ 1K LOCDCs OpenStack code < 2K LOC

    ~ 5000 CPUs, 1TB RAM, PB storage

    Without even talking about all the depndancies Galera, RabbitMQ, haproxy, keepalived, etc. etc. etc.

    *Open source is very close to the hearts of everyone that works at DataCentred, and weve also tried to contribute back to the community whereever possibleEither new modules where weve found a gap such as our LDAP oneOr contributing code back upstream*Probably the most boring slide on here!

    Example Puppet code for deployment of a package containing SNMP MIBS and the subsequent configuration (or not) of snmp.confVery simple example, possible to handle multiple package names based on OS distribution type mention programming familiarities such as case statementsEstablish a relationship here that says before we can populate snmp.conf, we have to make sure that package is installed


    Roles and Profiles introduction

    Layers of abstraction that help with making Puppet code more maintainableHow do we make sure our code is understandable as we scale

    Modules can usually be considered atomic, that is theyre only responsible for the software theyre written forCan combine modules together into a profileExample above one profile for configuring nova-compute, one for configuring neutron-agent, and one for neutron-common (which is monitoring)Profiles include and manage modules to define a logical technical stack as well as define relationships between modulesA role includes one or more profiles that define the server and its service

    Comes back to transparency and ease of maintainability and hence scalability

    Switch to Terminal and show some profile code

    *Introduction to ForemanLifecycle management provisioning of new operating system installationsAuto-discovery of new machines operating system instances, can be physical or virtualAligns very closely with Puppet providing a way of classifying nodes based on a very flexible set of criteria could be network, could be hardware typeAllows us to assign roles and profiles to a given machine business role of machine (webserver) combined with packages and configuration (profiles)*Combination of Puppet and Forman allows us to automate vast majority of infrastructure-related tasksBetween Puppet and Foreman we have everything we need to automatically accommodate for and provision any variety of operating system or platform we choose (within reason)This is the sort of thing were doing at DataCentred fully populated racks that we can plug in, have them be automatically discovered by Foreman, and then provisioned based on specification.Combination of these tools (plus underlying supporting pieces of technology) allows us to do some pretty cool stuff, limited only by your imagination!*From bare-metal provisioning through to application configuration and patch managementForeman provides a centralised console for monitoring and control, all API-accessible for further automation and extensionAnd of course, open-source*Behind the scenesAs a wise man once said.

    Infrastructure as codeDevOps culture, this is the essenceOnce you're managing your infrastructure as code, lends itself to coding techniques and methodologies*Infrastructure deployment using Puppet code lends itself to software development methodolgiesWe use a combination of git for distributed version controlEverything backed up to GitHub and encrypted where necessaryRepository mirrored onto our central Puppet serverCombined with ease and portability of spinning up and down self-contained virtual development environments consistency ensured means its a very iterative and rapid development approach for this kind of work.

    Going further - CI and CD, automated testing

    *Development platform how do we ensure consistency, repeatability, portability?We use Packer to create virtual machine imagesAnd Vagrant to define virtual environments tool for building and distributing virtual environments, developer focussedEnsures consistency across the team right the way up to the serverIntegrates nicely with Puppet for provisioning and testing Puppet code


Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.