@nicolefv What I Learned from Four Years of Sciencing the Crap Out of DevOps Nicole Forsgren, PhD Director, Organizational Performance and Analytics
@nicolefv
WhatILearnedfromFourYearsof Sciencing theCrapOutof
DevOps
NicoleForsgren,PhDDirector,OrganizationalPerformanceandAnalytics
@nicolefv
Outline
Howtomakeyourdatasuckless• Writinggoodsurveyquestions• Makingsurethesurveyquestionsaregood-withSCIENCE
• (Thesemethodsapplytoyoursystemandlogdata)
Whatwefound…thatwedid(ANDdidn’t)expectThingsaboutContinuousDeliveryThingsaboutManagement
@nicolefv
Notalldataiscreatedequal
Whoherethinkssurveysaresh*t?
@nicolefv
Notalldataiscreatedequal
Whoherethinkssurveysaresh*t?
WhohereLOVESthedatafromtheirlogfiles?
@nicolefv
WhatisaLatentConstruct?
@nicolefv
Weuse
PSYCHOMETRICS
tomakeoursurveydatagood*
*orgiveusareasonableassurancethatit’stellinguswhatwethinkit’stellingus(&someofthiscanalsoapplytoyourlogdata)
@nicolefv
Psychometricsincludes:
Constructcreation(manual)• Whenpossible:usepreviouslyvalidatedconstructs• Basedondefinitionsandtheory,carefullyandpreciselyworded,cardsortingtask,pilottested
Constructevaluation(statistics)• EstablishingValidity:discriminantandconvergent• EstablishingReliability
@nicolefv
PsychometricsWritingExample:Culture
• Doesitmattertoourstudy?• Morethanjustintuition?
• WhatKINDofculture?• National identityandnorms• Adaptiveculture• Value learning(2014study)• Value informationflowandtrust(2014and2015
studies -- Westrumculture)
@nicolefv
PsychometricsWritingExample:Culture
• Doesitmattertoourstudy?• Morethanjustintuition?
• WhatKINDofculture?• National identityandnorms• Adaptiveculture• Value learning(2014study)• Valueinformationflowandtrust(2014and2015
studies-- Westrumculture)
@nicolefv
WestrumtypologyPathologicalPower-oriented
BureaucraticRule-oriented
GenerativePerformance-oriented
Low cooperation Modest cooperation High cooperation
Messengers shot Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to scapegoating
Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to problems Novelty implemented
Try writing items
yourself!
Use strong statements with clear language.
Westrum, R. (2004). A typology of organisational cultures. Quality and safety in health care, 13(suppl 2), ii22-ii27.
@nicolefv
WestrumCultureItems
• Onmyteam,informationisactivelysought.• Onmyteam,failuresarelearningopportunities,andmessengersofthemarenotpunished.
• Onmyteam,responsibilitiesareshared.• Onmyteam,cross-functionalcollaborationisencouragedandrewarded.
• Onmyteam,failurecausesinquiry.• Onmyteam,newideasarewelcomed.
Found to be valid & reliable
Predictive of IT Performance & Organizational Performance
@nicolefv
PsychometricsAnalysisExample:NotificationofFailureAtmyorganization…
• Weareprimarilynotifiedoffailuresbyreportsfromcustomers.
• WeareprimarilynotifiedoffailuresbytheNOC.• Wegetfailurealertsfromloggingandmonitoring systems.• Wemonitorsystemhealthbasedonthresholdwarnings(ex.
CPUexceeds 100%).• Wemonitorsystemhealthbasedonrate-of-changewarnings
(ex.CPUusagehas increasedby25%overthelast10minutes).
Original in 2014, but there was a surprise.
Can you spot it?
@nicolefv
PsychometricsAnalysisExample:NotificationofFailureAtmyorganization…
• Weareprimarilynotifiedoffailuresbyreportsfromcustomers.
• WeareprimarilynotifiedoffailuresbytheNOC.• Wegetfailurealertsfromloggingandmonitoring systems.• Wemonitorsystemhealthbasedonthresholdwarnings(ex.
CPUexceeds 100%).• Wemonitorsystemhealthbasedonrate-of-changewarnings
(ex.CPUusagehas increasedby25%overthelast10minutes).
Notification from
NEAR
Notification from FAR
@nicolefv
Moredatatests!
Plus,wetesttomakesurethesurveydoesn’thaveotherproblems.• Commonmethodvariance(CMV)(akaCMBforBias)• Earlyvs.lateresponders• Surveydrop-offratesandbias
@nicolefv
OkayNOW wecanlookatthedataandhowitrelatestoeachother
@nicolefv
Anoteaboutanalysismethods
Oneofthreeconditionsmustbemet:
1. Longitudinal(no,thisiscross-sectional)
2. Randomized,experimentaldesign(no,thisisanon-experimental)
3. Theory-baseddesign
Whenthisconditionwasnotmet,onlycorrelationsweretestedandreported
@nicolefv
KEYFINDING:ITPerformance anditsbehavior
Acombinationofthroughput andstability
leadtimeforchangesreleasefrequency
timetorestoreservicechangefailrate
Forsgren, N., J. Humble (2016). "DevOps: Profiles in ITSM Performance and Contributing Factors." In the Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV.
@nicolefv
KEYFINDING:ITperformancematters!“Firmswithhigh-performingITorganizationsweretwiceaslikelytoexceedtheirprofitability,marketshareandproductivitygoals.”
ITPerformanceispredictiveoforganizationalperformance.http://bit.ly/2014-devops-report/
http://bit.ly/2015-devops-report/
Forsgren, N., J. Humble (2016). “The Role of Continuous Delivery in IT and Organizational Performance.” In the Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV.
@nicolefv
Continuousdelivery
OkayNOWwecanlookatthedata.
@nicolefv
Andnowwithaddedmath
https://devops-research.comForsgren, N., J. Humble (2016). "The Role of Continuous Delivery in IT and Organizational Performance." In the Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV. Available at SSRN: http://ssrn.com/abstract=2681909
@nicolefv
@nicolefv
somesurprises
@nicolefv
What’snot stronglycorrelatedw/ITPerf?
Third-partyscriptsHomegrownscriptsCommercialconfigurationmanagementtoolsOpensourceGoldenimagesManualconfigurationmanagement
@nicolefv
What’snot stronglycorrelatedw/ITPerf?
Third-partyscriptsHomegrownscriptsCommercialconfigurationmanagementtoolsOpensourceGoldenimagesManualconfigurationmanagement
@nicolefv
Whichofthesemeasureeffective testpractices?
Developersprimarilycreate&maintainacceptancetests
QAprimarilycreate&maintainacceptance tests
Primarilycreated&maintainedbyoutsourcedparty
Whenautomatedtestspass,I’mconfidentthesoftwareisreleasable
Testfailuresarelikelytoindicatearealdefect
It’seasyfordeveloperstofixacceptance tests
Developersshareacommonpooloftestserverstoreproducefailures
Developerscreateondemandtestenvironments
Developersusetheirowndev environmentstoreproducefailures
@nicolefv
Whichofthesemeasureeffective testpractices?
Developersprimarilycreate&maintainacceptancetests
QAprimarilycreate&maintainacceptance tests
Primarilycreated&maintainedbyoutsourcedparty
Whenautomatedtestspass,I’mconfidentthesoftwareisreleasable
Testfailuresarelikelytoindicatearealdefect
It’seasyfordeveloperstofixacceptance tests
Developersshareacommonpooloftestserverstoreproducefailures
Developerscreateondemandtestenvironments
Developersusetheirowndevenvironmentstoreproducefailures
@nicolefv
Changemanagement
Allproductionchangesmustbeapprovedbyanexternalbody(e.g.changeapprovalboard,manager,etc.)beforedeploymentorimplementation(R)
Onlyhigh-riskchanges,suchasdatabasechanges,requireapproval
WehavenochangeapprovalprocessWerelyonpeerreviewtomanagechanges
@nicolefv
Changemanagement
Allproductionchangesmustbeapprovedbyanexternalbody(e.g.changeapprovalboard,manager,etc.)beforedeploymentorimplementation(R)
Onlyhigh-riskchanges,suchasdatabasechanges,requireapproval
WehavenochangeapprovalprocessWerelyonpeerreviewtomanagechanges
@nicolefv
SurpriseswithCulture
•Wewantedtoaddadditionalmeasuresofculture
• Googlestudy• Identity• RetainWestrum culture
@nicolefv
IdentityandGoogleitems• Iamglad Ichosetoworkforthisorganizationratherthananothercompany.
• Italkofthisorganizationtomyfriendsasagreatcompanytoworkfor.
• Iamwillingtoputinagreatdealofeffortbeyondwhatisnormallyexpected tohelpmyorganizationtobesuccessful.
• Ifindthatmyvaluesandmyorganization'svaluesareverysimilar.
• Ingeneral, thepeopleemployedbymyorganizationareworkingtowardthesamegoal.
• Ifeelthatmyorganizationcaresaboutme.AdaptedfromadaptedfromAtreyi Kankanhalli,BernardC.Y.Tan,andKwok-Kee Wei(2005),“ContributingKnowledgetoElectronicKnowledgeRepositories:AnEmpiricalInvestigation,“MISQuarterly,29,113-143.
@nicolefv
IdentityandGoogleitems• Iamglad Ichosetoworkforthisorganizationratherthananothercompany.
• Italkofthisorganizationtomyfriendsasagreatcompanytoworkfor.
• Iamwillingtoputinagreatdealofeffortbeyondwhatisnormallyexpected tohelpmyorganizationtobesuccessful.
• Ifindthatmyvaluesandmyorganization'svaluesareverysimilar.
• Ingeneral, thepeopleemployedbymyorganizationareworkingtowardthesamegoal.
• Ifeelthatmyorganizationcaresaboutme.AdaptedfromadaptedfromAtreyi Kankanhalli,BernardC.Y.Tan,andKwok-KeeWei (2005),“ContributingKnowledgetoElectronic KnowledgeRepositories:AnEmpiricalInvestigation,“MISQuarterly,29,113-143.
Westrum items
@nicolefv
Nowformanagementstuff
WeallknowmanagingWIPisimportant,right?
@nicolefv
Nowformanagementstuff
WeallknowmanagingWIPisimportant,right?CorrelationbetweenWIPandITPerfisnegligible
@nicolefv
Nowformanagementstuff
WeallknowmanagingWIPisimportant,right?CorrelationbetweenWIPandITPerfisnegligibleWhat’sgoingon?
@nicolefv
LeanmanagementSEM
@nicolefv
Alsoleanproductmanagement
@nicolefv
Conclusions
• Evenifyouthinkit’sobvious,TESTWITHDATA.• (iftheresults don’tsurpriseyou,you’redoingitwrong)• (ifyoudon’talsoconfirmsomethingsyouexpected, you’redoingitwrong)
•WeCANhaveitall,oratleastthroughputANDstability.
• ITmatters(butyouhavetodoitright)• DevOpsculture&practiceshaveameasurableimpactonIT&orgperf
@nicolefv
SignupforourROIwhitepaper&getpeer-reviewedresearchdevops-research.com
Formorescience-ing…
@nicolefv
Thankyounicolefv.com