© 2015 IBM Corporation Using Lean Thinking to Identify and Address Delivery Pipeline Bottlenecks
Jul 14, 2015
© 2015 IBM Corporation
Using Lean Thinking to Identify and Address Delivery Pipeline Bottlenecks
Who’s that guy?
Eric Minick
DevOps Evangelist
@EricMinick
Background as a
developer, tester and tools
guy
Helping clients implement
build, deploy & release
tooling for the past 10
years
Systems of Record
Systems of Interaction
Continuous client experience
Partner value chain
Cloud-based Services
Systems of Engagement
Inefficient software delivery impacts the entire business
of customers experience
production delays
>45%
of outsourced projects fail to meet
objectives
>50%
of budgets devoted to maintenance and
operations
>70%
to deliver even minor application changes to
customers
4-6 weeks
DEVELOPMENT/TEST
Speed mismatch between faster moving front office and slower
moving back office systems, delaying time to obtain feedback
SUPPLIERS
Delivery in the context of agile
OPERATIONS
Rapid app releases impact system stability and
compliance
LINE-OF-BUSINESS
Takes too long to introduce or make changes to
mobile apps and services
CRM HR
DB ERP
4
Customers now control the pace of technology adoption cycles
Days
<10%
Quarters
35%
Months
40%
Weeks
15%
6-12 Month Delivery Cycles Are Still the Norm
Delivery cycle profile across 600 business enterprises
Feedback cycles
Source—Forrester: “The New Software Imperative: Fast Delivery with Quality”
Moving from Months to Weeks is a challenge
Relatively small practice changes required to go
from 18 month cycles to 3 month
Huge practice changes require to go from
months to weeks or days
Speed is a Capability
7 Source: Forrester report, “The New Software Imperative: Fast Delivery with Quality” Oct 2014
DevOps approach: Apply Lean principles accelerate feedback and improve time to value
Line-of-
business
Customer
1
3
2
1. Get ideas into production fast
2. Get people to use it
3. Get feedback
Non-Value-added waste Value-added production work
Lean
Transformation
http://ibm.co/devopsfordummies
IBM DevOps Adoption Model Practices, tools and services to plan and execute a staged adoption of DevOps to improve business outcomes
Feedback
Cycles
Productive Waste
Efficiency
Steer Product-based
Agile
Automated
Collaborative
Optimizing
More
Predictable
More
Transparent
More
Continuous
Process-based
Process-heavy
Manual
Silo-ed
Develop/Test
Deploy
Operate
Inefficient Leaner Leaner and Smarter
10
The Big Sources of Wasted Efforts: Find the Hidden Factory
Type of Waste Create Feature Deliver Feature
Unnecessary
Overhead
Communicating ideas/knowledge Communicating between development
and operations
Unnecessary
Re-work
Tasks assigned back to developers
from testing and usage
Tasks assigned back to developers from
production rollbacks
Over-production
Unnecessary functionality produced Unnecessary hardware, data center,
personnel
Non-Value-added waste Value-added production work
Lean
Transformation
What is Overhead? vs What is Productive?
Fat efforts to minimize
Waiting
Training
Reporting
Traceability
Late rework
Duplicate efforts
Metrics collection
Regression testing
Change propagation
Document generation
Meetings/Checkpoints
System administration
Resource accounting
Human inspections
Streamline or automate
More valuable efforts to improve
Scoping
Learning
Feedback
Refactoring
Designing
Teaming
Coding
Testing
Planning
Engineering
Empowering
Prediction
Deciding
Steering
Facilitate or smarten
Priorities of Indian Global System Integrators
Fat efforts to minimize
Late rework
Waiting
Regression testing
Duplicate efforts
Reporting
Document generation
Training
Metrics collection
Change propagation
Traceability
Human inspections
Meetings/Checkpoints
System administration
Resource accounting
Streamline or automate
More Valuable efforts to improve
Scoping
Designing
Planning
Testing
Reusing
Deciding
Steering
Feedback
Coding
Prediction
Engineering
Learning
Teaming
Refactoring
Facilitate or smarten
Eight DevOps Macro Practices
1. Deliver in small increments of functionality.
2. Form dedicated, cross-functional teams.
3. Use loose coupling between applications.
4. Automate environment provisioning.
5. Continuously integrate code.
6. Continuously test.
7. Continuously fund.
8. Provide real-time transparency.
Throughput of each process must be the equal in order to avoid backlogs.
When preceding process is upgraded to a higher throughput, subsequent
processes must be upgraded to the same higher throughput in order to maintain
balance.
Optimize pipeline with an even flow end to end
Steer the Product Pipelines
IBM Confidential
Value accrues as product artifacts evolve, NOT by effort
expended or by progress in supporting artifacts
Ideas
Tradeoffs
Delivered
Simple Instrumentation: Time: Volume: Cycle time Batch size Change speed Queue size Queue time Throughput
• Minimize waiting in backlog queues • Optimize work in progress • Adjust capacity and flow
Validate
Build
Proposed
Understood
Verified
Certified
Use spaghetti diagrams show flow of work
1. PM emails Dev manager
2. Dev Manager emails PM
3. PM emails QA Manager
4. QA Manager emails PM
5. PM emails Operations
6. Operations emails PM
Value stream map shows delays
---
30
30
30
30
30
1
1
1
1
1
1
Waiting
Working
150
6
1. PM Dev
2. PM Dev
3. PM QA
4. PM QA
5. PM Ops
6. PM Ops
Value stream map after process change
---
---
30
---
30
30
1
---
1
---
1
1
Waiting
Working
90
4
1. PM Dev
2. ----------
3. Dev QA
4. ----------
5. QA Ops
6. Ops PM
19
Stop and Go Deeper
Map your Bottlenecks
Idea/Feature/Bug Fix/
Enhancement Production
Development Build QA SIT UAT Prod
PMO
Requirements/
Analyst
Developer
Customers Line of Business
Build
Engineer
QA Team Integration Tester User/Tester Operations
Artifact Repository
Deployment Engineer
Release Management
Code Repository
Deploy
Get Feedback
Infrastructure as Code/
Cloud Patterns
Feedback
Customer or
Customer Surrogate
Metrics - Reporting/Dashboarding
Tasks
Artifacts
Map your Bottlenecks
Idea/Feature/Bug Fix/
Enhancement Production
Development Build QA SIT UAT Prod
PMO
Requirements/
Analyst
Developer
Customers Line of Business
Build
Engineer
QA Team Integration Tester User/Tester Operations
Artifact Repository
Deployment Engineer
Release Management
Code Repository
Deploy
Get Feedback
Infrastructure as Code/
Cloud Patterns
Feedback
Customer or
Customer Surrogate
Metrics - Reporting/Dashboarding
Tasks
Artifacts
The adoption of DevOps ==
increased velocity of application
delivery
Puts pressure on the
infrastructure to respond more
quickly
Software Defined Environments
enable you to capture
infrastructure as a software
artifact
Deploying Infrastructure is the biggest bottleneck for the Delivery Pipeline
Application
Changes
Infrastructure
Changes
Deployment Automation
Blueprint
Application Resource
Template
Continuous Delivery to Cloud
Capture cloud pattern to be used for creating an Environment
Incremental deployment of application builds to cloud environments
Map the application to multiple cloud patterns
The freedom to provision a version of a full stack or incrementally deploy an
application version into an already provisioned environment
Environments | Processes | Configurations
Create env
from pattern
Deploy app
QA PROD DEV DEV DEV DEV
Utilize Cloud
Environment
pattern
Cloud Platform
24
Building a DevOps Culture
• Everyone is responsible for Delivery
• Common measures of Success
• Right People are needed
Product Owner
Team Member
Team Lead
Team Member
Team Member
Senior Executives
Users Domain Experts
Auditors
Gold Owner
Support Staff
External System Team
Operations Staff
Above all - it’s all about the People/Culture
Culture:
The words we use and how we behave
Tools impact behavior and vocab
25
Lifecycle Measurements 2008 2010 2012 – 2014 Total
Improvement
Project Initiation 30 days 10 days 2 days 28 days
Groomed Backlog 90 days 45 days On-going 89 days
Overall Time To Development 120 days 55 days 3 days 117 days
Composite Build Time 36 hours 12 hours 5 hours 700 %
BVT Availability N / A 18 hours < 1hour 17 hours
Iteration Test Time 5 days 2 days 14 hours 4 days
Total Deployment Time 2 days 8 hours 4 hours -> 20
minutes
2 days
Overall Time To Production 9 days 3 days 2 days 7 days
Time Between Releases 12 Months 12 Months 3 Months 9 Months
Innovation / Maintenance 58% / 42% 64% / 36% 78% / 22% +20% / -20%
Double-digit revenue growth, increased client adoption, improved client satisfaction
26
How IBM Rational Cloud Hosted Products have improved!
Summary
Look for Queues (there are measures to help)
Check for how Downstream Processes change when something is fixed
Question the real business value of overhead processes like approvals
Use tools to help nudge culture
27
Additional Resources
Visit UrbanCode Solutions on the Web:
– https://developer.ibm.com/urbancode/
Learn about IBM DevOps:
– http://www.ibm.com/ibm/devops/us/en/
28
ibm.co/ARDfordummies
IBM UrbanCode Deploy ROI Calculator
https://ibm.biz/sdepaper
29 © 2015 IBM Corporation
© Copyright IBM Corporation 2014. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
www.ibm.com/devops