PuppetConf 2016: Implementing Puppet within a Complex Enterprise – Jerry Caupain, KPN

Ready, set, go! Implementing Puppet in a Complex

Enterprise

[email protected]

Who is this guy

@mistajaydawg

[email protected]

3

About this Talk

WhatDoesComplexityEvenMean?

4

Connectedandfree

About the company

5

Foraconnectedsociety

About the company

About the company Timeline: A very brief overview

1852

1881

Telegraphy

NationWideTelephoneNetwork

1962

ManualSwitching

FullyAutomatedTelephoneNetwork.

1989

Privatized

2007

CombineTelcoandICTservices

Mobile

Internet

DigitalTelevision

BigData

ManagedServices

Cloud

Infrastructure

The KPN Technology Book The 13 generic technology trends

Technology trends that matter for KPN Outlook toward 2020

BusinessImpact

TechnologyImpact

2016 20202017 2018 2019

DataScienLst

OpenStandards 4.5G

Mobile

MachineLearning

5GMobileSmall

Cell

XpointMemory

ContainerApps

Developercentric

Photonics

IoT

3D

Raising the bar Taking successful strategy to the next level

Innovate Grow

Simplify Strengthen

Digital & simple service and delivery

Converged Telco & IT services

Excellent user experience

Flexible & simplified networks and operating model

Best-in-class secured integrated networks

Applying innovative technologies

Commercial

Operational

SIMPLIFY GROW INNOVATE

ITServices TIServices(BSS/OSS)

Services

Domain

IA-Cod

ing

Domain

Infrastructure

Domain

Pla\ormsPower/Sparc

Genericx86Pla\orms

Compute,StorageandNetworkingresources

Different views, different interest One common goal….. Composable and Automated

ConsulLng

HosLngServices

CloudNLVMware

CloudNLMicrosoa

KVM OpenStack

MANO

Corporatemarket ConsumermarketMSP1 MSP(X)

AWS/AzureSoalayer

Managedservices(SO)

ResourceorchestraLonWorkloadautomaLon

KPN

I(C)TDomain TI(ETSI)Domain

APIAPI API API API

B2C

B2B

LooselyCoupled

Business Operations

WE help take care of the Netherlands

Goals Changing the conversation with the customer.

Nice! What else do you have to offer!

What an Effective* IT Services Partner Looks Like

Commercialflexibilityandinnova>on

Automa>onappliedtomanagedoperaLons

Deepandfocusedindustrydomainexper>se

Flexibledeliverymodelablethatisbothscalableandagile

Appliesdataanaly>cstoenhanceoperaLons

@Bakesenterpriseclasscybersecurityacrossthewhole

por\olio

Contract

------------------------------------

Abilitytomanagerisk

!

*)CXPGroup,CXOscan2016

It isn’t just technology that matters!

Goals Improve the quality of our services, and become the best!

v Standardized services v Automate as much as possible v Re-shore our managed operations v Maximum visibility v Auditability v Increase Speed of Delivery v Predictability

HardWork!

It’s challenging but…… No Excuses!

You Still Gotta Do The Work!

Snowflakes In The Datacenter Are Not Cool

Challenges Low standardization ratio

Challenges Change is needed in a lot of different areas

Challenges v Manual way of working v Siloed Organization v Heterogeneous landscape v Low Standardization ratio v Managing Windows with Puppet v “Taming the Brownfield” v Lot’s of different tools

People

Process

Technology

Challenges Can we change human nature?.............

Agile Organization From silo to effective collaboration

Module Dev Team

Teams

v Full Stack Engineers v Fully responsible v Agile way of working v Infrastructure as code

Community Lead

Community Lead

v Get feedback v Way of working v Standards

v Guidance v Organize Meetups v Soft link between the

teams

Tooling Infra Team

Customer Teams

IaaS Teams

Toolbox Tools to automate the complete chain

GitHub Enterprise

Puppet

Splunk

Jenkins

CoreTools

vRealize

Process Automation

Puppet Eco-system It’s all about the integrations

•  Broad platform

support

•  Flexible

• Great support

• Ops Friendly

•  Unix, Linux and

Windows

• Desired State

WhyPuppet?

Configura>onManagement

Enterprise

Configuration Management Moving up the stack!

InstallVM

ConfigureAgent

Setcustomfacts

Applybaseprofile

Applycustomroles+profiles

NetworkingDisksServices+SeOngsBackupMonitoringHardeningComplianceTimeServicesEtc.

Platforms

Windows 2008 / R2

Windows 2012/ R2

Centos 5,6,7

Red Hat 5,6,7

Solaris, AIX (Roadmap)

Puppet Eco-system It’s all about the integrations

•  Stability

•  Predictability

• Multi-platform

•  High Quality

• Managed by

Puppet

Why?

AutomatedAcceptanceTes>ng

Enterprise

Puppet Eco-system Module CI/CD Pipeline

syntaxcheck

Auto

puppet-lint unittest

acceptancetest

MergetoTEST

Auto Auto Manual

Mergeto

PROD

ManualChangein

git

Periodictrigger

NoLfybymail&Hipchat

Fail Fail Fail Fail

1.  kpn_jenkins::pipeline::module{'kpn-puppet-forge/puppet-kpn-hosts':2.  test_platform=>['linux','windows'],3.  pe3_syntax =>false,4.  pe4_syntax=>true,5.  pe4_unit=>true,6.  pe3_unit=>false,7.  acceptance_test=>true,8.  acceptance_platform=>['centos-7-x64-pe4',9.  'windows-2008r2-x64',10.  'windows-2012r2-x64',11.  'windows-2008r2-x64-pe4',12.  'windows-2012r2-x64-pe4',13.  'windows-2016-x64-pe4',14.  ],15.  hipchat_notification=>true,16.  hipchat_room=>'Moduleteam',17.  email=>‘[email protected]',18.  }

25

Puppet Eco-system Adding a module to the pipeline

Whichtestsdoyouwanttorun?

Whichopera>ngsystemsdoyouwanttorunthecodeon?

Howdoyouwishtogetno>fied?

Puppet Eco-system It’s all about the integrations

•  Familiar Developer

Workflow

•  Easy to manage

•  Easy setup

•  Security (RBAC)

•  Support

•  Integrations

WhyGitHubEnterprise

VersionControl

Enterprise

Puppet Eco-system It’s all about the integrations

•  Nodes

•  Platforms

•  Applications

•  Modules

•  Module versions

•  Puppet run info

•  Nodes in NOOP mode

•  Puppet facts

•  Configuration Changes

•  GitHub commit details

•  Capacity Management

•  Patch Management

•  Compliancy

•  Real Time

Troubleshooting

About?

Opera>onalIntelligence

Enterprise

Benefits & Challenges Making progress…..but not there yet

Benefits Challenges v Operational Intelligence (Troubleshooting) v Standardizing the brownfield! v Auditability v PCI DSS compliancy v Data is far more reliable v Quickly modify generic configuration

changes v Growing community v Online Documentation v Etc.

v Adoption is still slow v Steep learning curve v Brownfield still painful

What did we learn

Current State

Technology v ~ 4000 nodes……and counting! v 30 Puppet Masters v 129 Puppet modules

87 in-house developed 42 Puppet Forge

v 22 teams on boarded People

v 170 Puppet Fundamentals v 100 In-house technical

training v 25 In-house non-technical

training

Lessons Learned

v Have infrastructure capacity for testing purposes v Training is good, just doing it is better v Get the right feedback fast v Monitor the change….closely v Don’t over-centralize v Start with Greenfield first. v You cannot please everybody

Roadmap Strengthen and improve

Roadmap Items

Provisioning v Infrastructure deployment with Terraform v Image management using Packer (In progress) Configuration Management v Application Orchestration v Microsoft DSC integration v Docker integration for automated testing Analytics v Improve Splunk integration Collaboration v Extend the use of GitHub v Implement ChatOps

KEEP INNOVATING!

JUST……........... GETTING STARTED!

Thank you team @KPN!