DevOps Culture Shift: Expanding On-Call Responsibilties

DevOps Culture Shift: Expanding On-Call Responsibilities

Todd Vernon - CEO, VictorOpsKurt Bittner - Research Analyst,

Forrester ResearchNick Goodman - Director of Platform Engineering, BunchballPaul Beltrani - TechOps, Onshape

#DevOpsCulture

Moderator

Todd Vernon, CEO VictorOps

Industry veteran of SaaS business

CTO, Raindance Communications CEO, Lijit NetworksCEO, VictorOps@toddvernon

#DevOpsCulture

Yesterday (Developers Developed)

We spent a long time writing product requirements

We spent a longer time writing software

We pushed new code every 3-6 months to production

When it worked (12 hours later), we locked the data center

Reliability was obtained by NOT CHANGING THINGS

#DevOpsCulture

Today (Development is the new NOC)

Agile replaced Waterfall

Virtual replaced Physical

Puppet and Chef replaced System Admins

Continuous Delivery replaced lack of delivery

DevOps is replacing Operations

Developers are replacing the NOC #DevOpsCulture

Joining Us Today (Smart People)

#DevOpsCulture

Housekeeping (What you need to know)

• Half of today’s webinar will be a panel discussion seeded by questions from attendees

• Chat in your questions as we go. Two ways.– Chat directly into the platform– Twitter @VictorOps or #DevOpsCulture

• Today’s webinar will be recorded and the slides made available

• All attendees will receive a copy of the recent Forrester brief “Putting Devs On The Front Lines”

#DevOpsCulture

Poll (Audience Participation)

• How many currently include developers in their on-call rotations?

#DevOpsCulture

Kurt BittnerPrincipal Analyst, Forrester@ksbittner

#DevOpsCulture

© 2015 Forrester Research, Inc. Reproduction Prohibited

Putting Devs On The Front LinesDevOps Culture Begins With Visibility and Responsibility

#DevOpsCulture

© 2015 Forrester Research, Inc. Reproduction Prohibited 10© Golden Gate Bridge, Highway and Transportation District

Making Software Is Nothing Like Building A Bridge

http

://b

etan

ews.

com

/201

5/02

/26/

agile

-ver

sus-

wat

erfa

ll-de

velo

pmen

t-th

e-ca

se-f

or-a

gile

/

http://1

08

0pa

nim

ew

allpa

pers.co

m/n

etw

ork/ne

two

rk-op

eratio

ns-ce

nter.h

tml

Dev and Ops live in different worlds, with different cultures and values

© 2015 Forrester Research, Inc. Reproduction Prohibited 11© Golden Gate Bridge, Highway and Transportation District

Making Software Is Nothing Like Building A Bridge

http

://b

etan

ews.

com

/201

5/02

/26/

agile

-ver

sus-

wat

erfa

ll-de

velo

pmen

t-th

e-ca

se-f

or-a

gile

/

http://1

08

0pa

nim

ew

allpa

pers.co

m/n

etw

ork/ne

two

rk-op

eratio

ns-ce

nter.h

tml

Dev and Ops live in different worlds, with different cultures and values

Innovation (Encourage Change)

Stability(Prevent Change)

© 2015 Forrester Research, Inc. Reproduction Prohibited 12© Golden Gate Bridge, Highway and Transportation District

Making Software Is Nothing Like Building A Bridge

http

://b

etan

ews.

com

/201

5/02

/26/

agile

-ver

sus-

wat

erfa

ll-de

velo

pmen

t-th

e-ca

se-f

or-a

gile

/

http://1

08

0pa

nim

ew

allpa

pers.co

m/n

etw

ork/ne

two

rk-op

eratio

ns-ce

nter.h

tml

Dev and Ops live in different worlds, with different cultures and values

Innovation (Encourage Change)

Stability(Prevent Change)

Customer Success Requires Both

© 2015 Forrester Research, Inc. Reproduction Prohibited 13

Measure everyone the same way.

Reward improved customer experiences.

http://e-strategyblog.com/2011/05/daily-numbers-killing-time-on-mobile/

© 2015 Forrester Research, Inc. Reproduction Prohibited 14

Walk in the other person’s shoes

http://newsroom.blogs.cnn.com/2009/11/12/shoes-for-africa-sat-nov-14-at-10-p-m-et-on-cnn/

© 2015 Forrester Research, Inc. Reproduction Prohibited 15http://www.cnn.com/2013/11/27/world/gallery/most-beautiful-libraries-in-the-world/

Expand Application Support Knowledge

© 2015 Forrester Research, Inc. Reproduction Prohibited 16http://www.huffingtonpost.com/2013/05/22/blended-family-advice_n_3322007.html

“If you write code, you’re on call.

If you want to take vacations, write good tests.”

- Mario Cruz, CIO at Choose Digital

© 2015 Forrester Research, Inc. Reproduction Prohibited 17http://aragec.com/winding+roads.html#prettyPhoto[galeri]/1/

Think End-to-End

“The more time that developers spend

working in production on operations

issues with operations staff, …

… the more that they will learn about

what it takes to design and build a real-

world system.”

© 2015 Forrester Research, Inc. Reproduction Prohibited 18https://leadershipfreak.files.wordpress.com/2015/01/winston-churchill-the-price-of-greatness.jpg

Nick Goodman Director of Platform Engineering, Bunchball

#DevOpsCulture

DEVOPS is a Culture of InnovationCulture of Creation

Everybody creates something Technology choices driven by values

Culture of Empowerment It's all code, so everybody can work on it Developers feel responsible for their code's behavior

Product-Centric Culture Ignore IT Our roles exist to deliver a product to the customer Focus on the things we want to be good at

DEVOPS is HARDConvincing People is Hard

“I'm not a sys admin, and I don't want to be one” “I didn't write that, so I can't fix it”

DEVOPS Technologies Create Different Problems Biggest benefit comes from wide adoption within the code Such problems require a DEVOPS team to address

Paul BeltraniTechOps at OnShape

#DevOpsCulture

Onshape• https://www.onshape.com/• Founded by same people who founded

Solidworks• Disrupted the workstation era with CAD on MS Windows• Disrupting desktop era CAD with Cloud based CAD

•Many, smart developers who are new to WebOps

• Challenge, foster a DevOps culture• Started by getting developers involved with support rotations (on-

call)

How we brought Dev to Ops• Developers participate in the on-call rotation

• Change the traditional role of on-call• Incident manager VS repair person

• Lots of support and guidance• Paired on-call, person from Dev and a person from Ops

• Knowledge transfer in both directions

• Documentation and run books

• Group / User autonomy• Teams manage their own rotation

• Users manage their own alerting

Primary On-call• Responsible for making sure the service is

available• keeps an eye on the service and supporting infrastructure• responds to alerts 24/7*

• Manages incidents• Doesn’t have to fix the issue but is responsible for coordinating

other resources and managing the incident to resolution

• Point of contact• Keeps others informed, e.g. Customer Service• Provides updates to the teams. (Has authority to delegate if

they're busy.)

Panel (Q&A)

• Why change the status quo?• How do I align two disparate groups?• How do I remove the obstacles that

separate Dev and Ops teams?• Will customers see a difference if devs are

on call? • Can IT start to think differently?

#DevOpsCulture

Panel Discussion (Chat/Tweet in your questions)

#DevOpsCulture

Thank you (Further Resources)

• Visit the VictorOps Knowledge Drop for more resources on this topic.

• Get started. Try VictorOps FREE for 14 days and

take your webinar learnings for a test drive.

#DevOpsCulture