Why Everyone Needs DevOps Now: My Fourteen Year Journey Studying High Performing IT Organizations - Gene Kim, Author of The Phoenix Project
Post on 31-Aug-2014
880 Views
Preview:
DESCRIPTION
Transcript
Session ID:
@RealGeneKim, genek@realgenekim.me
Gene Kim
Why Everyone Needs DevOps Now:
My Fourteen Year Journey Studying High Performing IT Organizations
@RealGeneKim
Where Did The High Performers Come From?
@RealGeneKim
Visible Ops: Playbook of High Performers
� The IT Process Institute has been studying high-performing organizations since 1999
� What is common to all the high performers?
� What is different between them and average and low performers?
� How did they become great?
www.ITPI.org
@RealGeneKim
Act I: IT Ops Fixing Fragile Artifacts
@RealGeneKim
@RealGeneKim
The Product Managers
@RealGeneKim
Act 2: The Developers
@RealGeneKim
@RealGeneKim
@RealGeneKim
IT Ops And Dev At War
10
@RealGeneKim
Nothing Left For Infosec
@RealGeneKim
@RealGeneKim13
The Downward
Spiral…
@RealGeneKim
@RealGeneKim
So, CEOs Don’t Trust IT…
� “If IT fails I don't know why… and if IT succeeds I don't know why.”
� “By managing inputs and outputs, I can hold any area of the business accountable – except for IT…”
� “Large investments in IT projects that eventual fail, without warning. And the CIO is the first to say, ‘I told you so.’”
� “I can’t hold IT accountable – IT is way too ‘slippery.’”
15
Source: Gene Kim 2012
@RealGeneKim16
The IT Core Chronic Conflict
� Every IT organization is pressured to simultaneously:
� Respond more quickly to urgent business needs
� Provide stable, secure and predictable IT service
Source: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints and author of The Goal, has written extensively on the theory and practice of identifying and resolving
core, chronic conflicts.
@RealGeneKim
Every Company Is An IT Company…
� 95% of all capital projects have an IT component…
� 50% of all capital spending is technology-related
We are here…
Where we need to be…
IT is always in the way(again…)
@RealGeneKim
The Urgency Of This Business Problem
“Of the Fortune 500 companies in 1955, 87% are gone...
“In 1958, the Fortune 500 tenure was 61 years; now it’s 18 years…”
–Richard Foster, “Creative Destruction”
18
19 | Reimagining the Application Lifecycle
Obama campaign’s tech team beat Romney by using opposite strategy—“insourcing.”
Even taken with the software and Web hosting expenses, the Obama campaign spent a seventh of what the Romney campaign spent on digital….
In the end, the deciding factor wasn’t what the Obama campaign spent money on, but what it did with all that money. Insourcing gave the campaign a strategic flexibility that the Romney campaign lacked….
“This is the difference...between a well run professional machine and a gaggle of amateurs....I would be shocked if such a chasm exists next cycle between the parties—these aren’t mistakes to be repeated if you want to do things like win elections.”
http://arstechnica.com/information-technology/2012/11/how-team-obamas-tech-efficiency-left-romney-it-in-dust/
How Team Obama’s tech efficiency left Romney IT in dust
Technologies accelerate business-practice changes
Hired campaign staff engineers from Facebook, Twitter, Google, Microsoft, and technology startups.
“We ran the election 66,000 times
every night,” said a senior official,
describing the computer
simulations the campaign ran to
figure out Obama’s odds of
winning each swing state. “And
every morning we got the spit-out
— here are your chances of
winning these states. And that is
how we allocated resources.”
Surveys used live interviewers, very large sample sizes and very short questionnaires, which focused on vote preference and strength of support, with no more than a handful of additional substantive questions.
The massive scope of its polling effort helped guide the Obama campaign in ways that would be impossible with conventional polling…three-day rolling-average tracking in each state.
Build. Measure. Learn.
http://www.theatlantic.com/technology/archive/2012/11/when-the-nerds-go-marching-in/265325/
http://www.huffingtonpost.com/2012/11/21/obama-campaign-polls-2012_n_2171242.html
http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/
Act 3:There Must Be A Better Way…
21
@RealGeneKimSource: John Allspaw
@RealGeneKim
@RealGeneKimSource: John Allspaw
@RealGeneKim
@RealGeneKimSource: John Allspaw
@RealGeneKimSource: John Allspaw
@RealGeneKimSource: Theo Schlossnagle
@RealGeneKimSource: Theo Schlossnagle
@RealGeneKimSource: Theo Schlossnagle
@RealGeneKimSource: John Jenkins, Amazon.com
@RealGeneKim
@RealGeneKim
Who Is Doing DevOps?
� Google, Amazon, Netflix, Etsy, Akamai, Twitter, Facebook, Pinterest …
� BNY Mellon, Bank of America, World Bank, Paychex, Intuit…
� The Gap, Nordstrom, REI, Macy’s, GameStop, Target …
� Portland State University, Seton Hill University, Kansas State University…
� Who else?
33
@RealGeneKim
High Performing DevOps Teams
� They’re more agile
� 30x more frequent deployments
� 8,000x faster lead time than their peers
� They’re more reliable
� 2x the change success rate
� 12x faster MTTR
Source: Puppet Labs 2012 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
@RealGeneKim35
How Can We BetterSell DevOps?
36
@RealGeneKim
Eric Passmore, former SVP Global Engineering, AOL (2007)
37
The Downward SpiralOperations Sees…
� Fragile applications are prone to failure
� Long time required to figure out “which bit got flipped”
� Detective control is a salesperson
� Too much time required to restore service
� Too much firefighting and unplanned work
� Planned project work cannot complete
� Frustrated customers leave
� Market share goes down
� Business misses Wall Street commitments
� Business makes even larger promises to Wall Street
Dev Sees…
� More urgent, date-driven projects put into the queue
� Even more fragile code put into production
� More releases have increasingly “turbulent installs”
� Release cycles lengthen to amortize “cost of deployments”
� Failing bigger deployments more difficult to diagnose
� Most senior and constrained IT ops resources have less time to fix underlying process problems
� Ever increasing backlog of infrastructure projects that could fix root cause and reduce costs
� Ever increasing amount of tension between IT Ops and Development
These aren’t IT Operations problems…
These are business problems!
@RealGeneKim
Gene Kim, CTO, Tripwire, Inc. (2006)
39
The Downward SpiralOperations Sees…
� Fragile applications are prone to failure
� Long time required to figure out “which bit got flipped”
� Detective control is a salesperson
� Too much time required to restore service
� Too much firefighting and unplanned work
� Planned project work cannot complete
� Frustrated customers leave
� Market share goes down
� Business misses Wall Street commitments
� Business makes even larger promises to Wall Street
Dev Sees…
� More urgent, date-driven projects put into the queue
� Even more fragile code put into production
� More releases have increasingly “turbulent installs”
� Release cycles lengthen to amortize “cost of deployments”
� Failing bigger deployments more difficult to diagnose
� Most senior and constrained IT ops resources have less time to fix underlying process problems
� Ever increasing backlog of infrastructure projects that could fix root cause and reduce costs
� Ever increasing amount of tension between IT Ops and Development
These aren’t IT Operations problems…
These are business problems!
@RealGeneKim
Anonymous Product Manager / UX (2011)
41
The Downward SpiralOperations Sees…
� Fragile applications are prone to failure
� Long time required to figure out “which bit got flipped”
� Detective control is a salesperson
� Too much time required to restore service
� Too much firefighting and unplanned work
� Planned project work cannot complete
� Frustrated customers leave
� Market share goes down
� Business misses Wall Street commitments
� Business makes even larger promises to Wall Street
Dev Sees…
� More urgent, date-driven projects put into the queue
� Even more fragile code put into production
� More releases have increasingly “turbulent installs”
� Release cycles lengthen to amortize “cost of deployments”
� Failing bigger deployments more difficult to diagnose
� Most senior and constrained IT ops resources have less time to fix underlying process problems
� Ever increasing backlog of infrastructure projects that could fix root cause and reduce costs
� Ever increasing amount of tension between IT Ops and Development
These aren’t IT Operations problems…
These are business problems!
@RealGeneKim
Anonymous Infosec Officer (2012)
43
@RealGeneKim44
@RealGeneKim
@RealGeneKim46
“This book will have a profound effect on IT, just as The Goal did for manufacturing.” –JezHumble, co-author Continuous Delivery
“This is the IT swamp draining manual for anyone who is neck deep in alligators.” –Adrian Cockroft, Cloud Architect at Netflix
“This is The Goal for our decade, and is for any IT professional who wants their life back.” –Charles Betz, IT architect, author “Architecture and Patterns for IT”
@RealGeneKim
The First Way: Flow
@RealGeneKim
The First Way: Flow
� Understand the flow of work
� Always seek to increase flow
� Never unconsciously pass defects downstream
� Never allow local optimization to cause global degradation
� Achieve profound understanding of the system
@RealGeneKim
“Annual business planning sessions can be madding. They think IT Operations is an ‘all you can eat buffet.’”
-Ben Rockwood, Director Systems Engineering,Joyent
@RealGeneKim
Define The Work and Make It Visible
� Business projects (e.g., new order system)
� Internal IT projects (e.g., configuration management, automation, debt reduction)
� Changes (e.g., deploys, improve database performance)
� Unplanned work (e.g., site down, site impaired)
50
@RealGeneKim
Questions
� What is your lead time for changes? (i.e., how long does it take to go from “code committed” to “code successfully running in production”)
� How much of that is queue time vs. run time?
51
@RealGeneKim
@RealGeneKim
@RealGeneKim
Create One Step Environment Creation Process
� Make environments available early in the Development process
� Make sure Dev builds the code and environment at the same time
� Create a common Dev, QA and Production environment creation process
@RealGeneKim
If I had a magic wand, I’d change the Agile sprints and definition of “done”:
“At the end of each sprint, we must have working and shippable code, demonstrated in an environment that resembles production.”
@RealGeneKim
Deploy Smaller Changes, More Frequently *
� Decouple feature releases from code deployments
� Deploy features in a disabled state, using feature flags
� Require all developers check code into trunk daily (at least)
� Practice deploying smaller changes, which dramatically reduces risk and improves MTTR
56
@RealGeneKim
Breaking The Bottlenecks In The Flow
� Environment creation
� Code deployment
� Test setup and run
� Overly tight architecture
� Development
� Product management
57
58
How organizations achieve high performance
• 89% are using infrastructure version control
• 82% are using automated code deployments
Source: Puppet Labs 2012 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
@RealGeneKim
Why Dedicated Teams Vs. Shared Services
59
@RealGeneKim
@RealGeneKim
Leankit Kanban
61
@RealGeneKim
Blackboard Learn: 2005-Present
62
Source: David Ashman, Chief Architect, Blackboard, Inc.
@RealGeneKim
Blackboard Learn Building Blocks
63
Source: David Ashman, Chief Architect, Blackboard, Inc.
@RealGeneKim
The First Way: Outcomes
� Creating single repository for code and environments
� Determinism in the release process
� Consistent Dev, Test and Production environments, all properly built before deployment begins
� A continuous delivery pipeline that can be relied upon and daily Dev code commits
� Free ourselves from the learned behavior of catastrophic deployments
� Decreased lead time
� Reduce deployment times from 6 hours to 45 minutes
� Refactor deployment process that had 1300+ steps spanning 4 weeks
� Faster cycle time and release cadence
@RealGeneKim
The Second Way: Feedback
@RealGeneKim
The Second Way: Feedback
� Understand and respond to the needs of all customers, internal and external
� Shorten and amplify all feedback loops: stop the line when necessary
� Create quality at the source
� Create and embed knowledge where we need it
@RealGeneKim67
Source: John Shook
@RealGeneKim
“We found that when we woke up developers at 2am, defects got fixed faster than ever”
– Patrick Lightbody, CEO, BrowserMob
@RealGeneKim
Require That Devs Manage Their Own Code For 6+ Months
69Source: Tom Limoncelli, Google
@RealGeneKim
Test Whether Developers Qualify For IT Operations Resources
� Types/frequency of pager alerts
� Maturity of monitoring
� System architecture review
� Release process
� Defect counts and severity
� Production hygiene
70Source: Tom Limoncelli, Google
@RealGeneKim
Return Fragile Services Back To Dev
71Source: Tom Limoncelli, Google
@RealGeneKim
Feedback And Situational Awareness
“Having a developer add a monitoring metric shouldn’t feel like a schema change.”
– John Allspaw, SVP Tech Ops, Etsy
72
@RealGeneKim73
@RealGeneKim74
@RealGeneKim
Integrating Into Continuous Delivery
� The days of reviewing RFCs in Word docs in change management meetings are over
� Failures must result in automated tests in the continuous deployment pipeline (Release, Config, Change)
� Invite or embed Ops into Dev standups and the scrum teams (“hey, we can sprint and scrums, too!”)
@RealGeneKim
Embed Dev Into IT Ops
� Embed Dev into IT Ops incident escalation process
� Put production monitoring in pre-production environments
� Invite Dev to post-mortems/root cause analysis meeting
� Have Dev and Infosec cross-train IT Operations
� Ensure application monitoring/metrics to aid in Ops and Infosecwork (e.g., incident/problem management)
@RealGeneKim
What’s In It For Infosec And QA?
77
@RealGeneKim
The Second Way:Outcomes
� Defects and security issues getting fixed faster than ever
� Standardized and reusable Ops and Infosec user stories now part of the Agile process
� All groups communicating and coordinating better
� Everybody is getting more work done
@RealGeneKim
The Third Way:Continual Experimentation And Learning
@RealGeneKim
The Third Way:Continual Experimentation And Learning
� Foster a culture that rewards:
� Experimentation (taking risks) and learning from failure
� Repetition is the prerequisite to mastery
� Why?
� You need a culture that keeps pushing into the danger zone
� And have the habits that enable you to survive in the danger zone
@RealGeneKim
Break Things Early And Often
“Do painful things more frequently, so you can make it less painful… We don’t get pushback from Dev, because they know it makes rollouts smoother.”
– Adrian Cockcroft, Architect, Netflix
@RealGeneKim82
@RealGeneKim
Inject Failures Often
@RealGeneKim
You Don’t Choose Chaos Monkey…Chaos Monkey Chooses You
@RealGeneKim
Break Things Before Production
� Enforce consistency in code, environments and configurations across the environments
� Add your ASSERTs to find misconfigurations, enforce https, etc.
� Add static code analysis to automated continuous integration and testing process
@RealGeneKim
Reduce Technical Debt
� “The deal with engineering goes like this. Product management takes 20% of the capacity right off the top and gives this to engineering to spend as they see fit. Whatever is required to avoid, ‘we need to stop features to rewrite code.
“If you’re in really bad shape today, you might need to make this 30% or even more of the resources. I get nervous when I find teams that think they can get away with much less than 20%.”
– Marty Cagan, Inspired
@RealGeneKim
Allocate 20% Of Cycles To Technical Debt Reduction
@RealGeneKim
Recognize Compounding Technical Debt…
@RealGeneKim
That Gets Worse…
@RealGeneKim
And Fixing It…
Source: Pingdom
@RealGeneKim
An Innovation Culture
“By installing a rampant innovation culture, they now do 165 experiments in the three months of tax season.
Our business result? Conversion rate of the website is up 50 percent. Employee result? Everyone loves it, because now their ideas can make it to market.”
–Scott Cook, Intuit Founder
91
@RealGeneKim
Convergence And Evolution Of Ideas
� Four Steps To The Epiphany, Steven Blank (2005)
� Principles Of Product Development Flow: Second Generation Lean Product Development, Donald Reinertsen (2009)
� Lean Startup, Eric Ries (2011)
� Lean UX, Jeff Gothelf (2013)
92
93
Performance by DevOps maturity
Organizations that implemented DevOps practices over 12
months ago were 5x more likely to be high performing than
organizations that weren’t implementing DevOps at all. Source: Puppet Labs 2012 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
Why Do I Think This Is Important?
94
@RealGeneKim95
The Downward
Spiral…
@RealGeneKim
@RealGeneKim97
@RealGeneKim
If I Could Wave A Magic Wand, Everyone Will…
� See the suffering downstream, and have confidence that your intuitions and skills can make a profound and positive difference…
� Become conversant with DevOps and recognize the practices when you see them
� Be energized about how practitioners can contribute in this organizational journey
� Leave with some concrete steps to get some great outcomes
� Help create a team that starts putting DevOps practices into place
98
@RealGeneKim
If I Could Wave A Magic Wand, Everyone Will…
� Become conversant with DevOps and recognize the practices when you see them
� Be energized about how practitioners can contribute in this organizational journey
� Leave with some concrete steps to get some great outcomes
� Become a part of a team that starts putting DevOps practices into place
99
@RealGeneKim100
“Some books you give to friends, for the joy of sharing a great novel.
“Some books you recommend to your colleagues and employees, to create common ground.
“Some books you share with your boss, to plant the seeds of a big idea.
“The Phoenix Project is all three.”
–Jeremiah Shirk, Integration & Infrastructure Manager at Kansas State University
@RealGeneKim
Our Mission: Positively Impact The Lives Of One Million IT Workers By 2017
� Free 170 page excerpt:http://itrevolution.com/the-phoenix-project-excerpt/
� http://slideshare.net/realgenekim
� DevOps Defensive Audit Toolkit
� Enterprise DevOps Case Studies
� Early draft of upcoming “DevOpsCookbook” (Allspaw, DeBois, Edwards, Humble, Kim, Orzen)
� Email me at genek@realgenekim.me
top related