Puppet Labs EMC DevOps Day NYC Aug-2015

How to Build High-Performing

IT TeamsPresented by Alanna Brown

Agenda

• Build the case• Identify the desired state• Align incentives• Align your teams• Implement key technical practices

Build the Case

People don’t buy what you do, they buy why you do it.

Simon Sinekhttp://bit.ly/sinektedtalk

Reliability

Agility

Opposing Forces

High-performing IT orgs are more agile

30xMore frequent deployments

200xFaster lead times than their peers

Source: Puppet Labs 2015 State of DevOps Report

High-performing IT orgs are more reliable

60xChange success rate

168xFaster mean time to recover (MTTR)

Source: Puppet Labs 2015 State of DevOps Report

High-performing IT orgs are winning

1.5xMore likely to exceed profitability, market share & productivity goals

50%Higher market capitalization growth over 3 years.*Source: Puppet Labs 2015 State of DevOps Report

Learning is not compulsory, but neither is survival.

Edward W. Deminghttp://bit.ly/deming14pts

Identify theDesired State

Organization

Low trust culture High trust culture

Siloed teams Cross-functional teams

Lack of alignment Aligned around business goals

Processes

Lots of manual work Mostly automated work

Long cycle times Short cycle times

Poor visibility Fast feedback & insight

People

High burnout High job satisfaction

Stagnant Able to grow

Checking boxes Creative innovation

Continuous Delivery Practices

Lean Management Practices

Align Incentives

“Trying to effect process, people, technology and cultural changes across the entire application

portfolio, in a globally dispersed team and with a lot of associated technical debt, is an epic challenge.”

Jonathan Fletcher Enterprise Architect and Lead for Technology,

Platform and DevOps at Hiscox http://bit.ly/devopshiscox

Hiscox: Results

• Reduced cost per release on one application by 97%

• Reduced time per release by 89%• Reduced staff required to release by 75%• Automated testing reduced multiple man days

of effort down to an overnight hands-free process

Conflicting Incentives

Business Delivering value to customers

Dev teams Delivering new features

Ops teams Ensuring stability of systems

Quality teams

Ensuring quality of software releases

Everyone is responsible for quality and we’re all trying to

deliver the best solution for our customers.

Reena Mathew, Principle Architect Quality Engineering, Salesforce

http://bit.ly/sfdevops

Salesforce: Results

Aligned Incentives

Delivering value to

customers

Business

Ops teams

Quality teams

Dev

teams

We can’t do DevOps because our application is

________________________.

Architectural CharacteristicsSignificant affect on IT performance• Does not require integrated testing

environment• Can deploy/release app

independently of other apps/services it depends on

• Custom software with microservices architecture

No affect on IT performance• Packaged commercial software /

COTS.• Systems of record / systems of

engagement• New, not-yet-deployed systems.• Software w/ embedded component that

runs on manufactured hardware device / user-installed component that runs on user’s machine

Align Your Teams

Typical Enterprise Org StructureIT Operations

NOC

Commercial Banking

Business Units

Credit Cards

Mortgages

Investment Banking

Systems Engineers

Network Engineers

Storage Admins

DBAs

InfosecDev teams reside in BU

Pattern 1: Smooth Operations

Dev Ops

Recommended Reading: http://blog.matthewskelton.net/2013/10/22/what-team-structure-is-right-for-devops-to-flourish/

Pattern 2: Cross-functional teamCharacteristics• Consists of devs, testers, ops,

product owner, etc.• Focused on delivering a single

application• Self-sufficient• Optimized for throughput

Pattern 3: DevOps Team

Dev OpsDevOps

Characteristics• Consists ideally of devs with

systems experience, or sysadmins with programming experience

• Focused on automating pain points• Responsible for building a platform

that allows devs to self-service• Provides a toolchain to enable devs

to build, test and deploy their systems

• Coaches other teams

Roles & ResponsibilitiesRoles Responsibilities

“The Business” Understand market trends and identify customer needs

IT Manager Build trust with counterparts on other teams; create culture of learning and continuous improvement; delegate authority; remove roadblocks

Dev Manager Build trust with Ops counterpart; bring Ops into the planning process early

Systems Engineer Automate the things that are painful; help devs get feedback

QE Provide input into scale and performance; provide feedback on staging environments

Devs Plan for deployment as you’re planning new features; get feedback from ops and work with them on deployment process

Implement Key Technical Practices

Version Control

Configuration Management

Continuous Integration

DeploymentTools

Monitoring

And others…

DevOps Toolchain

Infrastructure as Code

Infrastructure as Code

Version Control

Peer Review

Continuous Delivery

Collaboration IterationFast

FeedbackVisibility

Version Control

Source: Puppet Labs 2014 State of DevOps Report

Peer-Reviewed Change Process

• Code can be contributed by anyone

• Code changes can be reviewed by anyone

• Code can be worked on as a team

Continuous Delivery

• Code is repeatable

• Code is sharable

• Code is promotable

• Code is testable

Measuring ResultsThroughput• Deployment frequency• Change lead time (from dev’s laptop to

production)• Cycle timeStability / Reliability• Change fail rate• Mean time to recover• Availability / downtime

Deployment Pain

Tomorrow, ask your team two questions: • How painful are your deployments?• What’s causing the pain?

Diversify Your Team

Teams with more women have higher: • Financial performance• Stock market performance• Hedge fund returns• Collective intelligence

Q&A

Resources• The 2015 State of DevOps Report is here! puppetlabs.com

/2015-devops-report

• The Phoenix Project by Gene Kim

• Continuous Delivery by Jez Humble

• PuppetConf 2015: http://2015.puppetconf.com/

• DevOps Enterprise Summit: http://devopsenterprise.io/