An Agile approach to Business Metrics

An Agile approach to Business Metrics

by @Pablo_Valcarcelpvalcarcel@wemanity.com

Product Owner’s Adventures in

Dataland

Who is this guy?@Pablo_Valcarcelpvalcarcel@wemanity.com

‘Conventional’ Agile Metrics

Lead Time*Defect count Work in ProgressCode coverageUnplanned ChangesVelocity (Stpts per sprint)Return on investment*Innovations per sprintArtifacts generatedSlack timeFailure Load (firefighting time)

Iteration Burn-DownUnfinished StoriesCustomer Satisfaction*LOC (lines of code)Un-deployed Stories# BlocksBudget/Schedule ComplianceFlow Efficiency (lead time / touch time)Release Burn-Up

*Italics: Connected to business value.

Your average Sprint Review...

KPIKPI KPI

KPI

Stakeholder

Scrum MasterProduct Owner

Is this ‘Agile’ enough?

KPIKPI KPI

KPI

Stakeholder

Scrum MasterProduct Owner

● Does it help increase customer satisfaction?● We deliver business value on every iteration,

but how do we measure it?● Shouldn’t the definition of metrics be an

iterative+feedback-based process on itself?● Priorities and markets change all the time.

Enterprise Metrics: The Ocean of Data

The Startup Way: Go Pirate

‘AARRR!’-Dave McLure500 Startups

The Lean Metrics Process (Croll & Yoskovitz)

The Lean Sprint (Ash Maurya)

The Thesis: The Analytics Sprint

Agile discipline and ‘Lean Analytics’ thinking can provide a better method to measure value.

The Analytics Sprint

Planning:

At the beginning of each sprint discuss: ● With stakeholders: What are the OKRs/key metrics? What’s the

KPI we should work on the next iteration? What’s the baseline?● With team: Hypothesis are defined on why/how to tackle that

KPI.

All of this: KPI, baseline, hypothesis and experiments are documented in writing.

The Analytics Sprint

Reviewing:

At the end of each sprint: ● There’s an assessment. Did we hit the goal? If not pivot into

another experiment. If we did we persevere onto the next KPI.● Data is turned into validated learning. Even failed experiments

provide learning (think Google with OKR).

The Analytics Sprint: The Metrics Master

Metrics Master: Role responsible for the definition, revision and accounting of the key metrics.● Facilitates discussion with stakeholders on

business value and key metrics.● Discusses hypothesis and experiments with

team.● Tracks and does innovation accounting.● Reports back on the results and either pivots or

perseveres on the next metric.● It can be the Product Owner (business value) or

a dedicated team-member (e.g. data-scientist)..

But wait, I already do that!Even if you’re already measuring a business value metric every sprint: ● When was its value defined?● Who defined it?● How’s the reporting being done to

stakeholders and customers? (Theatre of success or. customer discovery?).

Case Study: Spotify Metrics-Driven Development

Every retrospective they measure squad performance, and they refine the metric.

They also set goal targets at the start of every integration project we do (which may span multiple sprints). For example: “we will know we’re successful when this integration brings us x-amount of users.”

Source: Lynn Root

Case Study: Spotify Metrics-Driven Development

● They consciously avoid waste by measuring everything.

● They also set baselines with historical data.

● Goals are a shorter decision-making cycle as well as make more informed decisions about strategy and partnerships.

Some pro-tips

● Write it down. Prevents biases.

● Always define a baseline or benchmark.

● Avoid ‘bad metrics’.● Results will be murky.● Conversation with

stakeholders, customers and team is a value on itself.

Further Reading:Spotify: Metrics Driven Development by Lynn Root.Lean Analytics by Alistair Croll & Ben Yoskovitz.Appropriate Agile Measurement by Deborah Hartmann & Robin Dymond.Five Tips For Testing Your Idea by Anita Newton, Cindy Alvarez & Alistair Croll.Scaling Lean by Ash Maurya.Managing Metrics in an Iterative Incremental Development Environment by John D. McGregor.Agile Metrics, post by André Dhont.

A call to action:We like thinking of Agility not as a cargo cult, but as a set of values and principles that we adapt to the specifics of every single case.

So, we need you to learn.Please, hit me with your ideas, case studies and questions at pvalcarcel@wemanity.com

Thanks for your attention!!!

@Pablo_Valcarcelpvalcarcel@wemanity.com

Reasons for an Agile approach● We don’t know what we should be measuring.● We discover a better way to measure it.● Business priorities and political priorities change.● The product’s incremental development requires an incremental

an adaptative approach to its metrics.● Project (or transition) is too big for our resources and we have

to pick our battles.

We want to measure true business value as early as possible!

Challenges with this approach

From a business point of view:

● First, what information is available early in the process and how reliable is that information?

● Second, how can we use the early estimates to predict the actual values?

● The incremental aspect of the process requires that there be a plan for combining the values of certain metrics in order to obtain project-wide values (e.g. Addition, Averaging, Min-Max values).

Startup Vs. Intrapreneurial metrics (by A. Croll)