Agile Analytics: A GQM Approach to Enterprise Metrics

© 2012 Fiserv, Inc. or its affiliates.

The GQM Approach to Enterprise Agile Metrics

Version 1.42

© 2012 Fiserv, Inc. or its affiliates.

JohnTannerSVP&EnterpriseTransformationConsultant,LeadingAgile

Ø 20+yearsinSoftwareDevelopmentØ RolesinindustryasConsultant,Engineer,

Architect,Product&ProjectManagerØ ImplementedAgileinalltheenvironments:

o Non-profito Corporateo Government

[email protected] www.linkedin.com/in/tannerjs www.facebook.com/leadingagile www.twitter.com/leadingagile

© 2012 Fiserv, Inc. or its affiliates.

Defining the Problem

© 2012 Fiserv, Inc. or its affiliates.

Threeviewsonmetrics

• Weneedmaximummetrics,becausewejustdon’tknowwhatwedon’tknow

• Weneedminimalmetrics,becausetheyjustdon’taddvalue

• Weneedsomemetrics,becausesomethingisbetterthannothing

© 2012 Fiserv, Inc. or its affiliates.

Weneedall ofthemetrics!

“Metricsarehowwemeasuresuccess.Solet’scapturemetricsforeverything.”

• Metricsdashboardsbecomeprolific,andsomeonehastomaintainthem

• EachTeamwindsupwithauniquesetofmetricstocapture,butwithnocommonexpectationforusingthem

• Theorganizationspendsmoretimecreatingmetricsroll-upsthantheydocreatingproduct

• Thereisnocohesivestrategyfortyingthedatatogethertomaketargetedimprovements

© 2012 Fiserv, Inc. or its affiliates.

Weneednone ofthemetrics!

“Metricsdonotaddvalue,sotheyarewaste.Let’seliminatethemaltogether.”

• Assumesallteamsarecurrentlyproducingworking,testedproductathighlevelsofefficiency

• Onlyworkswhentheorganizationdoesn’tcareiftheyeverimprovethesystem…

• …doesn’tcareiftheyaren’tproducingvalue• …doesn’tgethungupondetailslikemakingpayroll

© 2012 Fiserv, Inc. or its affiliates.

Weneedsomemetrics…

“Metricsmighthelpusimprove,soweneedmeasuresofsomesort.”

• Withoutclarity,weoftencapturemetricsformetrics’sake• Whenwegathermetricswithoutclearexpectationsaroundtheirusetheycanbewasteful,orevenharmful

• Wefocusonmetricsinawaythatultimatelydrives“localoptimization”attheexpenseoftheoverallsystem

• Weforgetthatthe“why”andthe“how”matters

© 2012 Fiserv, Inc. or its affiliates.

The “why” and the “how”

© 2012 Fiserv, Inc. or its affiliates.

Let’stalkaboutwhy,first…

Backlogitem

Backlogitem

Backlogitem

Backlogitem

Backlogitem

Backlogs

Team

Teams

WorkingTestedProduct

WorkingTestedProduct

© 2012 Fiserv, Inc. or its affiliates.

Howweoftenapproachtheproblem

InAgile,wefeelprettyconfidentweknowhowtomeasuresuccessbecause…

...isMakingandMeetingCommitments

...isProducingQualitySoftware

TheTeam...hasaStableVelocity

© 2012 Fiserv, Inc. or its affiliates.

Team

Backlogitem

Backlogitem

Backlogitem

Backlogitem

Backlogitem

WorkingTestedProduct

Makes&MeetsCommitments

Metric:StoryPoint

Completion%

HasaStableVelocity

Metric:VelocityVariance

ProducesQualitySoftware

Metric:EscapedDefect

Count

Howweoftenapproachtheproblem

© 2012 Fiserv, Inc. or its affiliates.

ProblemSolved?

Yay!Wediditteam!

No,butseriously,manypeoplethinklikethat…

© 2012 Fiserv, Inc. or its affiliates.

Howdoesthatworkatscale?

Team Team Team Team

TeamTeamTeam

Delivery Tier

Team Team TeamProgramTier

TeamPortfolioTier

© 2012 Fiserv, Inc. or its affiliates.

Goal-Question-Metric

© 2012 Fiserv, Inc. or its affiliates.

WhatisGQM?

TheGoal-Question-Metricapproachisasimplemodelwhereweapproachmeasuresfromthreedistinctlevels:

Conceptuallevel(Goal)Operationallevel(Question)Quantitativelevel(Metric)

© 2012 Fiserv, Inc. or its affiliates.

WhatisGQM?

Conceptuallevel(Goal)Agoalisdefinedforateamorsystem,tomeetaspecificpurpose,withafocusondemonstratingameaningfuloutcomeforthebusiness.

© 2012 Fiserv, Inc. or its affiliates.

WhatisGQM?

Operationallevel(Question)Asetofquestionsiscraftedtodefineanobjectivemodelforassessingtheachievementofourgoals.

© 2012 Fiserv, Inc. or its affiliates.

WhatisGQM?

Quantitativelevel(Metric)Asetofmetrics,basedupontheobjectivemodel,isassociatedwitheveryquestioninordertoansweritinameasurableway.

© 2012 Fiserv, Inc. or its affiliates.

• Identifytherightgoalsforateamorsystem• Clarifyourgoalsbyaskingtherightquestions• Identifymeasuresandmetricsthatcananswerthequestions

• Findwaystocollectthedatanecessarytorealizeourmeasuresandmetrics

• Analyzethedatatoassesswhetherweareansweringourquestionsandachievingourgoals

• Usewhatwediscovertoimproveourgoals,questions,andmetrics

HowdoweuseGQM?

© 2012 Fiserv, Inc. or its affiliates.

Example:Mygoalfortoday

Goal

Myaudiencedoesn'tfallasleepinthenexthalf

hour

Question

Didtheyevenshowupinthefirstplace?

Havetheybeenthoroughlycaffeinated?

Aretheypayingattentionsofar?

MetricEmptySeatvsSoldSeat(ActualvsExpected)

SnoretoSilenceRatio(SnoreComplete/SilenceCommitted)

EyeContactVariance

HecklingDensity

© 2012 Fiserv, Inc. or its affiliates.

Redefining the Problem

© 2012 Fiserv, Inc. or its affiliates.

RedefiningtheProblem

InAgile,wefeelprettyconfidentweknowhowtomeasuresuccessbecause…

...ismeetingCommitments

...isProducingQualitySoftware

TheTeam...hasaStableVelocity

© 2012 Fiserv, Inc. or its affiliates.

RedefiningtheProblem

Let’sfigureitout,GQMstyle

InEnterpriseAgile,wefeelprettyconfidentweknowhowtomeasuresuccessbecause…

© 2012 Fiserv, Inc. or its affiliates.

GQMAreastoDiscuss

• DeliveryTeamHealth(Scrum)• TechnicalQuality• ProgramHealth• ProductQuality• PortfolioHealth• PortfolioFinancials

© 2012 Fiserv, Inc. or its affiliates.

GQM at the Delivery Tier

© 2012 Fiserv, Inc. or its affiliates.

Teamscanplan,coordinate,anddeliverpredictablyenoughtomeetarelease

levelcommitment

Hastheteamestablishedastablevelocity?

Doestheteamhavetheresourcesitneedstobesuccessful?

Doestheteamdeliverthecommittedfunctionalityeachsprint?

StoryPointCompletion%(SPsComplete/SPsCommitted)

VelocityVariance

TeamStabilityIndex

Isanythingpreventingtheteamfromdeliveringoncommitments? Blockers

Doestheteamdeliverthroughoutthesprintoratthe“lastminute”?

SprintCompletionRatio(%SPsAcceptedbySprintEnd- 2)

DeliveryTeamHealth(Scrum)Goal MetricQuestion

UserStoryCompletion%(USsComplete/USsCommitted)

Towhatdegreeistheteamencapsulated(i.e.make/meetcommitmentsonitsown)?

%UserStoriesw/outsidedependencies

© 2012 Fiserv, Inc. or its affiliates.

Teamscanfrequentlydeliverworking,tested

softwareofahighlevelofquality

Istheteamabletomaintaintheoveralltechnicalhealthoftheproduct?

Canthequalityoftheproductbeingshippedbeverified? OpenDefects

TechnicalDebt

Aretheteamsabletoaddresstechnicalproblemsinatimelymanner? DefectAging

Aretechnicalproblemsgettingpastourqualitychecks? EscapedDefects

Aretechnicalproblemsbeingdiscoveredinproduction? LatentDefects

Goal Question Metric

TechnicalQuality

© 2012 Fiserv, Inc. or its affiliates.

GQM at the Program Tier

© 2012 Fiserv, Inc. or its affiliates.

ProgramHealth

ProgramManagementcanmakedeliverycommitmentsandsuccessfullymanage

incomingrequests

IstheProgrameffectivelymanagingorchestration/mitigatingdependencies?

Istheprogramdeliveringcommittedfunctionalityeachrelease?

Istheroadmapoffeaturessufficienttoeffectivelymaintainareadybacklog? FeatureRoadmapVisibility

%ReadyBacklogUserStoriesw/opendependencies

FeatureCompletionRatio

HowlongdoesittakeforanOpportunitytomovefromIdeatoDelivery? FeatureLeadTime

Goal MetricQuestion

IstheReadyBacklogsufficientlygroomedtobepredictablyconsumedbytheDT? %ReadyBacklog

IstheProgram(POTeam)focusedondeliveringvalue? %ValueinSprint

© 2012 Fiserv, Inc. or its affiliates.

Goal Question Metric

Theorganizationdeliversanintuitiveexperienceand

productfeaturesthatdelightourcustomersandproducethehighestbusinessvalue

possible

Didthedeliveredsolutionmeetthebusinessneed?

Product/MarketFit(ActualvsExpected)

DoestheOverallProductDelighttheCustomer? NetPromoterScore

DidtheFeatureDelighttheCustomer?PirateMetrics

(Acquisition,Activation,Retention,Revenue,Referral)

ProductQuality

© 2012 Fiserv, Inc. or its affiliates.

GQM at the Portfolio Tier

© 2012 Fiserv, Inc. or its affiliates.

PortfolioHealth

32

Theorganizationdeliversproductfeaturesand

capabilitieswiththebestabilitytodeliverearlyROI

Howlongdoesittakeforanopportunitytomovefrom“goodidea”to”done”?

Isthesystemdeliveringconsistently?

EpicLeadTime

DueDatePerformance

Areweinvestingintherighttypeofwork? PortfolioInvestmentMix

Arewefocusingonlyonthemostimportantthings? ValueStreamWIP

AbandonedWorkArewestartingontoomanythings?

Goal Question Metric

© 2012 Fiserv, Inc. or its affiliates.

PortfolioFinancials

33

Goal Metric

The Portfolio is meeting its strategic, financial

obligations and objectives

Are we getting the right return on our investments?

Are we spending too much just to keep the lights on?

Value to Cost Ratio

CAPEX to OPEX Ratio

Are we getting the expected results from this Portfolio?

Variance to Results($ or % Market Share)

Question

Are we meeting our targeted spend rates? Variance to Spend Target

Are we hitting our targeted value and maintenance blend?

Variance to Value/Maintenance Target

($ or % Spend)

© 2012 Fiserv, Inc. or its affiliates.

Redefining the Problem (Solved)

© 2012 Fiserv, Inc. or its affiliates.

AttheEnterpriseLevel,wefeelconfidentweknowhowtomeasuresuccessbecause…ThePortfolio...consistentlydeliversproductfeatures&capabilitieswithafocusontimetomarketandearlyreturnoninvestment(PortfolioHealth)...consistentlymeetsitsstrategicfinancialobjects(PortfolioFinancials)TheProgram...canmakedeliverycommitmentsandsuccessfullymanageincomingrequests(ProgramHealth)...canensuretheorganizationdeliversthemosteffectiveexperienceandproductfeaturestodelightourcustomersandproducethehighestbusinessvaluepossible(ProductQuality)TheDeliveryTeams...canplan,coordinate,anddeliverpredictably,inordertomeetareleasecommitments(TeamHealth)...canfrequentlydeliverworkingtestedcodeofahighlevelofquality(TechnicalQuality)

© 2012 Fiserv, Inc. or its affiliates.

So now what?

© 2012 Fiserv, Inc. or its affiliates.

üIdentifytherightgoalsforateamorsystemüClarifyourgoalsbyaskingtherightquestionsüIdentifymeasuresandmetricsthatcananswerthequestions

• Findwaystocollectthedatanecessarytorealizeourmetricsandmeasures

• Analyzethedatatoassesswhetherweareansweringourquestionsandachievingourgoals

• Usewhatwediscovertoimproveourgoals,questions,andmetrics

NextSteps

© 2012 Fiserv, Inc. or its affiliates.

Final Thoughts