Fail Fast, Fail Often

FAIL FAST, FAIL OFTEN

Gordon Haff @ghaff, Technology EvangelistWilliam Henry @ipbabble, DevOps Strategy Lead13 July 2016

FAILURE

2

3

FAILURE

4

FAILURE

ALSO FAILURE

5

FAILURES HAVE CONSEQUENCES

6

THE INESCAPABLE CONCLUSION?

7

DON’TFAIL

8

DON’TFAIL

9

FAILWELL

10

11

Experiment by Peter Skillman, former VP of design at Palm

12

WHAT HE LEARNED

• Kindergarteners do not spend 15 minutes in a bunch of status transactions trying to figure out who is going to be CEO of Spaghetti Corporation.

• They don’t sit around talking about the problem. They just start building to determine what works and what doesn’t.

SOFTWARE = GREAT MATCH FOR

FAILING WELL

13

14

FIVE PRINCIPLES:

THE RIGHT

scopeapproachworkflowincentivesculture

15

THE RIGHT SCOPEConstrain the impact of failure

• Enable experimentation

• Stop cascading of failures

• Make deployments incremental, frequent, and routine events

• Generally decouple activities and decisions from each other

• Small, autonomous, bounded context services

16

SMALL

• “Two pizza teams”

• Well-defined functional units

• Organized around business capabilities (Conway's Law)

17

AUTONOMOUS

• Implementation changes can happen independently of other services

• Data and functionality exposed only through service calls over the network

• Designed to be externalizable

• No back-doors

18

THE RIGHT APPROACHContinuously experiment, iterate, and improve

• It’s about the process

• Identify mistakes early

• Establish safety nets

• Fail and move on

19

THE PROCESSInvolves people and communication

• The most effective process have continuous communication - think scrums and kanban

• Allows for collaboration that can identify failures before they happen

• Allows for feedback to continuously improve and cultivate growth

• Provides transparency

20

DEV LESSONS: BREAKING CODE VIOLENTLYBuild in violent failures to highlight issues

• C/C++ lessons:

• Sanity check using assertions

• Invariant checks

• If ever I’m here in the code and these conditions aren’t met, then I have no business being here. Something is wrong and I should fail violently.

• Involves tracing through the failure

21

AUTOMATED REGRESSION TESTING

• As products and services evolve we discovered that maintaining and incrementally adding new tests became valuable

• These tests were/are most often based on experienced failures and bugs

• Scripts were developed to run nightly builds against various developer changes to test for regression

• Testing tools evolved - proprietary and open source

22

OPS LESSONS: CHAOS MONKEYTest robustness of recovery using failure

• Platform should provide uninterrupted services to the customer

• Therefore:

• Should always recover in acceptable amount of time

• We should have random failures to ensure that changes have not regressed or caused new recovery problems

http://understeer.hatenablog.com/entry/2012/02/29/224629

23

THE RIGHT WORKFLOWRepeatably automate for consistency

• Goal is repeatable automation

• Toyota’s yellow cord

• Initially pipelines may be very different

• Different tools

• Traditional vs. “cloud native”

• It’s a journey• Consolidation evolves naturally

24

DESIRABLE ENTERPRISE CI/CD WORKFLOW

myRepo ProjectRepo

CI

Commit Push

Pass/Fail

Local Test

BuildRepo

CD

ReleaseRepo

Monitor

Build Test Review/Appr Deliver Deploy

3rd Party

25

CI/CD PIPELINE TOOLSET

CI/CD Workflow UI

gerrit

26

OPS LESSONS: RED/GREENConfiguration as code has built in failure

Continuous Integration / Continuous Deployment

Image & Package &Metadata Repository

src repo

Dev./Build QA Productionin OHC

Events

27

THE RIGHT INCENTIVESAlign rewards and behavior with desirable outcomes

• Incentives (advancement, money, recognition) need to reward trust, cooperation, and innovation

• Peer reward systems also valuable

• Individual has control over their own success

• But people still have responsibility for their actions

28

THE RIGHT CULTUREBuild systems and organizations that allow for failing well

• Transparency

• Even good decisions can have bad outcomes

• Innovation inherently risky• Cut losses (avoid sunk cost fallacy)

This is why open source is so successful!

29

30

BUT CULTURE ISN’T SOMETHING YOU JUST CHANGE

• Lack of agreed-to model of what “right” culture looks like

• Different organizations require different behaviors

• Culture change is difficult to measure and quantify

• Culture is very hard to impose

• Culture is an output, not an input

31

CULTURE IS:

emergentpervasivethe keystone

plus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHatNews

THANK YOU

CREDITS

33

Tacoma Narrows Bridge: Barney Elliott; The Camera Shop - Screenshot taken from 16MM Kodachrome motion picture film by Barney Elliott.

Time cover: Time, Inc.

Wipeout, Flickr/CC: https://www.flickr.com/photos/andymorffew/15843725192

Marshmallow challenge: http://marshmallowchallenge.com/Welcome.html

Linux Collaboration Summit: Linux Foundation.

Two pizzas: Flickr/CC https://www.flickr.com/photos/dongkwan/283076601

Frog: Kathy CC/Flickr https://flic.kr/p/b9fFV

Square peg Flickr/CC: https://www.flickr.com/photos/epublicist/3546059144/