DevOps Year One

DEVOPS: YEAR ONEMagnus Hedemark

Principal Software EngineerBronto Software

@Magnus919

WHAT SHALL WE COVER?• We’re trying to stuff a year’s worth of material into an hour. This won’t be a deep dive.

• We’ve got people from many different roles here (workers, line managers, executives, etc)

• This will be a very high level overview. This is meant to inspire DevOps uptake under generalized conditions. Take it and make it work for you.

• We’ll briefly touch on some of the larger guiding philosophies, where you can opt to dig deeper.

• Some guidance will be given on how to measure success as you go.

IN THE BEGINNINGEvery server was a special snowflake. So was every piece of software.

Workers pretty much kept to their own tribe. Talking to anyone else was risky.

BLAH BLAH BLAH

• You’ve heard all of this part before.

• You’ve bought into the idea of DevOps.

• You’re here because you want to know how.

STAKEHOLDERS• These transformations happen from the top down, from the bottom up, and sometimes even

across the board.

• Top-down can succeed through virtue or through brute force. If it’s virtuous, it’ll stick when you’re not even looking.

• Bottom up is hard. Not impossible, but damn hard.

• This works best when you’ve got passionate people from across the business working together, in alignment with a common vision & values, to affect change.

• Having executive sponsorship changes everything.

PREPARE TO FAIL…• You’re about to try a lot of brave, new things.

• Change is hard.

• Some things aren’t going to work as expected, or even at all.

• Be patient. Learn from your mistakes. In DevOps, even failure is success. (SCIENCE!)

• As changes become smaller and better understood, confidence goes up, impact of failures goes down.

• How you handle failure will become a crucial part of your new DevOps culture… and its success.

LIKE A FOX.

HOW DO WE FAIL?• One important transformation that many shops must make is changing how they view failure.

• Failure is seldom an individual problem at which to cast blame, or even punishment.

• Failure is often more complex than a single human being or root cause. Digging often exposes systemic defects or waste that can be reduced or eliminated.

• A person who executed a plan and failed has probably learned something more valuable for the organization than somebody who executed a plan and succeeded.

• Punishing failure discourages a culture of continuous experimentation and learning. The organization must earn trust.

• Scientific curiosity FOR THE WIN!

HOW DO WE WORK?• There are many paths to DevOps

• Agile is not a bad way to go. But partner with a good coach, and the whole organization needs to be aligned.

• Be wary of “Scrummerfall”

• Lots to learn from Lean, and Toyota Production System.

• One Piece Flow

TPS OBJECTIVES

• Design overburden (muri) out of the system

• Design inconsistency (mura) out of the system

• Eliminate waste (muda)

• Eliminate waste? Isn’t that kind of hand-wavy?

ENTER THE MUDA1. Waste of over production

2. Waste of time on hand (waiting/idle)

3. Waste of transportation

4. Waste of processing itself

5. Waste of stock at hand

6. Waste of movement

7. Waste of making defective products

8. (added later) failure to utilize human intellect or skill

9. (added later) Producing something the customer does not use

FIND AND ELIMINATE WASTE• It’s not “technical debt” if you’ve never logged

it and made a scheduled plan to pay it off.

• DevOps & Lean practice will surface a lot of these technical debts defects

• “We’re an infrastructure as code shop, except these couple of tools are special.” Waste.

• Getting rid of waste hurts. It’s scary. It’s hard to get leadership buy-in. It’s hard to get team buy-in. But life is much better on the other side.

THE TOYOTA WAY• Culture is baked into the way Toyota works. Very DevOps-y.

• The Toyota Way establishes a number of key principles and behaviors that underlie Toyota’s leadership structure and manufacturing process.

• These principals can be applied directly to industries other than auto manufacturing.

• Such as… software development manufacturing

FOUR GROUPS OF PRINCIPLES OF TPS

1. long-term philosophy

2. the right process will produce the right results

3. add value to the organization by developing your people

4. continuously solving root problems drives organizational learning

RECURRING MEETINGS

• Retrospective, or Hansei-Kai

• “Standup”

• Standup², or “Scrum of Scrums”

• Work Planning

• Stakeholder demos, lightning talks, workshops, sprint kickoff, lunch-n-learn, etc.

HANSEI-KAI, OR AGILE RETROSPECTIVE• TPS and Agile have different words for similar customs.

• Teams need organizational investment in time for reflection

• Opportunities for improvement should lead to actionable recommendations.

• XP teams do this often, maybe weekly. Scrum teams usually do this every sprint (bi-weekly, typically).

• Doing this less frequently than monthly risks loss of expensive lessons learned.

• These can run a long time. Be patient. It’s a good investment.

• This is the most important team-level recurring meeting for organizational learning. #kaizen

RETROSPECTIVE STRUCTURE• Laptops closed. Phones silenced/away. Be here, and only here.

• How did we do with the action items from the last retrospective?

• What new things are we doing that have been going well for us, that we wish to keep doing?

• What things could we be doing better at? And how will we do better?

• Agree on action items for next retrospective.

RETROSPECTIVE ATTENDEES• This needs to be a place for respectful but direct honesty and group introspection.

• Consider whether or not the manager’s presence might squelch honesty.

• If there is an embedded Product Owner, Project Manager, they should be there, too.

• Consider carefully what meeting minutes will be taken and published. This could impact honesty. I like to have an agreement that the discussion is private, but the action items are to be published.

• If the manager is not included, set some time aside following the retrospective to invite them to discuss the action items together.

OVERBURDEN (MURI)• Don’t present your team with more work than

they can reasonably accomplish in a near-horizon time.

• Context switching produces the appearance of being busy, but little gets completed. Also produces more defects.

• Someone being tasked with something far outside of their comfort zone will be over-burdened.

• Can your systems or work process handle the loads expected of them?

UNEVENNESS IN DEMAND (MURA)• Feast or famine

• Work harder

• Work longer hours

• Find something useful to do

• Furlough

• Creates muda and muri

SOFTWARE DEVELOPMENTBECOMES

SOFTWARE MANUFACTURING• DevOps extends the role of the engineer

beyond merely building a product or a feature.

• The engineering team gains responsibilities across the product lifecycle.

• Waste of handoffs becomes more obvious.

• Suddenly we can learn from a whole different industry!

KANBAN• Kanban describes the card with a task

on it, not the board it’s pinned to.

• Concept adapted from Toyota’s one piece flow system.

• Practitioners often borrow from Agile.

• Start by visualizing what you already have, without changing it.

• Limit your work in progress.

TOYOTA’S SIX RULES FOR KANBAN

1. Customer (downstream) processes withdraw items in the precise amounts specified by the Kanban.

2. Supplier (upstream) produces items in the precise amounts and sequences specified by the Kanban.

3. No items are made or moved without a Kanban.

4. A Kanban should accompany each item, every time.

5. Defects and incorrect amounts are never sent to the next downstream process.

6. The number of Kanbans is reduced carefully to lower inventories and to reveal problems.

KANBAN EXAMPLE

We’re going to come back to this. :)

SUCCESS ACCORDING TO AGILE• “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”

• “Working software is the primary measure of progress.”

• Ergo…

• customer satisfaction

• continuous delivery

• value of delivered software

• quality of working software

MEASURING TEAM PERFORMANCE

• maturity in “the practice”

• Continuous Delivery maturity model

• Kanban metrics include Lead Time, Cycle Time, cumulative flow

• Visual controls on everything, small batch sizes, bring waste to the surface

• DANGER! You can measure a team effectively this way, but not an individual.

LEAD TIME

• This is what the customer sees.

• Customer asks you for something.

• How long does it take you to give it to them, starting from the moment they asked for it?

CYCLE TIME• From the moment an engineer begins

working on a kanban card, how long does it take for them to finish?

• Over time, as standards develop and automation is leveraged, this time can become predictable.

• Great variances can reveal many opportunities for learning and growth.

CUMULATIVE FLOW• How many Kanban cards are

waiting in each state before being moved downstream?

• A granular workflow will help to reveal bottlenecks and inventory.

• Can also reveal overburden on the team.

But does it work?

OH. HELL. YES.• Lead time couldn’t even be

measured 3 months ago• High of 101 day rolling avg, 122 day

standard deviation• Low of 7 day rolling avg, 18 day

standard deviation

• Cycle time was high of 34 days rolling average, 77 days standard deviation.

• Down to under a day rolling average, standard deviation of about a day.

HOW DO I CHANGE CULTURE?

• “You can’t directly change culture. But you can change behavior, and behavior becomes culture” – Lloyd Taylor VP Infrastructure, Ngmoco

HOW DO I CHANGE BEHAVIOR?

How not to change behavior

HOW DO I CHANGE BEHAVIOR?

• Define values, principles. Don’t just pay lip service to these. Must be a real part of everyday decision making and evaluations.

• Define the goal of the organization. (See Eli Goldratt’s book “The Goal”)

• Organizational alignment. All leaders in line behind the goal. Understand their contribution to it.

• Reward your agents of change that embrace and embody the change you want to see.

• Look out for workers and leaders who are looking out for themselves, or their own little silo, instead of looking out for the organizational goal.

EMPLOYEE ENGAGEMENT = $$$

• Quick mention, Gallup poll last year showed about 70% of American workers were either not engaged at work or actively disengaged.

• Estimated half a trillion dollars lost to the US economy

• DevOps model, putting culture first, places a high value on engagement

• Job satisfaction is #1 indicator of organizational performance.

• Don’t just pay lip service to culture and engagement; there’s $$$ at stake. Get this right.

AWESOME BOOKS

• “The Phoenix Project”

• “Continuous Delivery”

• “The Toyota Way”

• “The Goal”

• “Out of the Crisis”

• …so many more. Start here. Tweet me if you get bored and need more. ;-)

CONTINUOUS DELIVERY MATURITY MODEL

source: “Continuous Delivery” by Jez Humble, David Farley

ORGANIZATIONAL CONSIDERATIONS• Are your teams functional or cross-functional?

• Functional teams don’t make sense in a one-piece flow model.

• Cross-functional teams eliminate costly handoffs, provide low-cost high-value communication.

• Does your reporting structure deliver direct value at every level?

• If it helps… SOA your org structure.

THE DEVOPS TEAM

• There is no such thing.

• If you have one of these, you’ve already failed.

• PuppetLabs and I disagree on this, respectfully. Don’t silo your comprehensive business transformation.

REQUISITE ORGANIZATION

• Theory by Elliot Jaques. Book by same name ($$ and hard to get, but look for it).

• Organizational dysfunction mostly comes from poor structure & systems, not bad employees

• Fix the organization instead of the employees

RO: EXAMPLE INTERVENTIONS• match employee capability to role complexity

• appropriately space employee and manager

• have the right number of layers

• define managerial authority and accountability

• define managerial leadership processes

• define cross-functional working relationships

• match compensation to job complexity (felt fair compensation)

ENGINEERS: DO THIS• Standardize your working practices

• Swarm on bigger tasks. Break it down and flow it through the team.

• Figure out your main types of work (new feature, defect process, etc)

• Create a workflow for each type of work, which your kanban cards will transition through. A hybrid physical board for fine-grained work flow with coarse changes in JIRA or some other tracking tool might be best today. Alternatives?

• Mantra: “Every git merge represents a potentially shippable change in my product.”

• Meet regularly with peers on other teams, even clandestinely, to start breaking down silos, share, and learn from each other.

FRONT LINE MANAGERS: DO THIS• Get aligned with the leadership above you on the goal, principles, values, behaviors. Then get the people reporting to

you aligned.

• Learn and embody the values and desired behaviors of your organization.

• Learn and relentlessly pursue the goal of your organization.

• Pursue heijunka for your team.

• Support, challenge, and reward them for eliminating muda. They need your help with this, especially mura (unevenness) and muri (unreasonableness/overburden).

• Seek coaching by your superiors in your organization’s practice, and be a coach to your direct reports in-kind.

• Support regular retrospectives (hansei-kai) so your team can learn and improve from reflection.

MIDDLE MANAGERS: DO THIS• Consider whether or not your teams are organized well for success (functional vs cross-

functional, staffing levels vs workload)

• Inject kaizen challenges to your organization. Once waste is brought to the surface, challenge them to apply the practice to reducing or eliminating the waste.

• Understand the goal and the principles and desired behaviors of your organization, align your line managers, and be their compass when they are making tough decisions.

• Translate the strategy from your executive to tactics for your line manager and workers. You are the glue between the strategy and tactics of your organization to achieve the goal.

EXECUTIVES: DO THIS• Translate the goal from your CEO to a strategy for success.

• Work with your peers, your CEO, and with the leadership elsewhere in the org (including workers in influential roles), and distill your core values and desired behaviors. Keep your organization in alignment on these regularly.

• Consider the appropriateness of your organizational design. Are the right people in the right places to succeed at the goal?

• Grow leaders in your organization. The desired behaviors, principles, and values of your practice should be pervasive and well-understood by leaders below you.

• Establish reasonable indicators of success, and recognize achievement. This is hard, it’s scary, and it takes a long time.

LET’S KANBAN

• I’ll need some volunteers.

FIN

@Magnus919