Agilelessons scanagile-final 2013

Pieces of the Big Agile Puzzle -problems and solutions

11.11.2013

ANTTI VIRTANEN+358 44 507 0050 [email protected]

Agenda

1. The context and background1. Ammunition for ad hominem arguments2. The projects we will talk about3. Lean and agile4. What is a big agile project (our context)5. Some relevant human limitations6. Things they didn‟t teach you at the university

2. Fundamental principles to follow1. The foundation you need2. Some unsatisfactory solutions

3. Actual problems and applying the principles4. QA and throwing stones

Who? What experience?

Short background

About meProgrammer and a jack of all trades for some time.

Hobbyist -> Professional developer -> Researcher -> Developer -> Architect

About Solita250 employees. Enough to make serious things happen.

Growing, succeeding, profitable, constantly improving.

Expert at making complex tailored business software.

Serious capacity also at BI/DW, integration, consulting and UX.

The projects (in TOP-3 for Solita)

ERP (2007-2010) Kirre (2010-2013)

Multiple teams Yes. Also multi-site. Yes. (multi-site)

Long build times Hours :-( Semi-solved.

Quality pressures Yes. Delayed much. Extreme. Delayed a a bit.

DB tables (oper.) Approx. 800 Approx. 300

Codebase LOC About 1 million About 300k

Parallel work? No. Yes,after a fashion.

Happy developers? No.. :-( Yes.. mostly

Complex domain? No, but unfamiliar. Yes. And unfamiliar.

More about the ERP project

Solita‟s “death march” projectUnrealistic timetable.

Arrogance on our part.

A Pyrrhic victoryWe possess a certain amount of sisu.

There were no good books or guides to follow.

My reason for being here today.

Picture: Sisukas Nainen @ youtube

Kirre, the second big one..

Kirre keeps track of all land property ownership in Finland.Problems would make Big Money very angry. Failure in this project would have been fatal to Solita as a company.

Keys to successSisu and stubborness again. Defeat does not exist in this dojo. Less arrogance. Hired professional coaches.Books had been written. Studied a lot.Not brute-forcing. Reshaped “process” totally.

This feels like a real victory.There were issues, but this is something I can be proud of.

Agile and lean.

“Lean architecture and Agile feature development are not about working harder. They are about working „smarter‟ in the academic or traditional computer science senses of the word „smart.‟”

Coplien, Lean Architecture

Big project

What is different?Why can‟t humans cope with it?What new skills are required

What is a big project? How is it different?

Bigger system poses technical challengesLonger build times, slower test runs, complex deploymentsToo many details.Some documentation is required.

People issuesMultiple teams -> management and communication challenge.Treating people as resources does not work.Training new people is seriously expensive. Motivation will falter after three years with no productiondeployment.

Human limitations

Some well-understood limitsContext-switching is not free.Eight hours per day available for work. At most.Communication is unreliable, unpredictable and time-consuming. There are limits to handling complexity

Social limitations and issuesThere is no ”team” of 25 people.Team formation is tricky

We are quite irrationalRational choice is an illusionWe are geared towards short-term profits and instant rewards.

Engineering is not enough

Programming

UI design

Testing

Marketing Systems thinking

Psychology Mathematics

Philosophy Drama and acting

We value things on the left.. in a big project things on the right even more!

Fundamental principlesThese are simple principles. But not easy.

Decoupling Composition over monolithic ideasAvoid up-front decisionsSeparation of concernsCreating MVPs

Parallelism is the paramount concern

Teams move simultaneously toward a shared goalA team must have a subgoal (purpose)

Do not hinder with accidental obstacles

Pull value, don‟t push (be lean)Single PO or synchronization is push.

Pushing does not truly scale.

Central authority is also a risk factor.

Avoid monoliths,favor composition

You can build a huge monolith in parallel…

It‟s a design and engineering decisionFunctional programming

OS products and frameworks often

Decoupling functional specifications in a moment..

Avoid up-front decisions

When is the ”last responsible moment”?Who knows.. but choosing technology up-front is too early

Pushing the moment further away:1. Paper prototypes. (sketching function)2. Initial boundaries and API (Form follows function)3. Iterate and write some code. (live form)

(For this path I recommend Coplien‟s book Lean Architecture.)

Separation of concerns

Avoid leaking implementation detailsExternal API – UI – data model

Tool selectionA big project does have very special needs!Automate everything (that DevOps thing). Composable tools.

Rethinking toolsMaven is a repository, not a build tool.Jenkins is UI for cron, not a CI tool.

Aim for MVP at all levels

You absolutely must postpone features.But which ones are nice-to-have ?

You need feedback early onCreate a minimal PoC implementationApply this to programming, design, specifications etc.

A ”big picture” is too big to seeThe MVP/prototype makes it clear as a side-effect!

Be wary of easy solutions

”Solutions” that don‟t address the fundamentals

Technological miraclesDecoupling and separation of concerns (SOA, REST , ESB)Opinionated frameworks (Rails)Management Tools

”Hiding” complexity (BPML, TOGAF, MDM)Simplified recipes

You can‟t ”Teach Yourself Lean in 21 Days”! Preposterous!

To be fair, some of these have value.. But you really can’t solve a human problem with technological magic tricks!

Principles in real world(What did we actually learn?)

Conditions of greedy selection no longer hold for the backlogDecoupling of functional specifications Normal form in relational database considered harmfulFocus on results, not tasksTeams around functionality, not layers and technologyLeadership is not management

How big is the elephant?

Three ways to search the problem space:Depth-first (to measure complexity and risk)

Breadth-first (to get an overview)

Explorative mapping (black art, you may get interestingfindings)

I highly recommend balancing between all threein a big project. Early on!

Backlog priorization

What is the most valuable item?The most risky one?

-> reduce risk

The one with most direct value to end-user (lean maxim)?-> customer value

Getting a core workflow done in it‟s entirety? -> feedback and reduction of risk

In a big project it is a good to rethink the criteria.

Decoupling functional specs (case Kirre)

Three features: Work queue, search and application handlingSeparate from the user‟s point of viewSo parallel work comes naturally?

Why not? Some bad reasonsThe database is a monolithic normal form design.The dev tools do not support this.The build pipeline tool doesn‟t allow it.The specification team (the ”architects” or ”PO-team”) is a separate silo.

We had limited success decoupling these things. But it is definitely the right direction to go!

The relational database posse is lacking

The RDBMS mindset (for waterfalls)A static view of the end state. The ”normal form” is by definition a monolith!

Relational model is not broken per se But ”agile developer” is not served by the vendors.Normal form doesn‟t deal with uncertainty and agile.Big projects are especially vulnerable to these issues!

Part of the NoSQL hype is a result of this attitude.

All models are wrong. Some are useful.(actually we retain normal form here)

Application

Queue Assignee

Applicationgroup

DecisionApplication

typeProperty

Queue_view

UI for queue

handling

Views are great!• Early feedback• Parallel development• Hidden complexity• Decoupling

Everyone. Must. Focus. On. Results.

Bad projects (tasks) Good projects (goals) The elite (results)

What is the next task today? What are you going to work on today?

What are you going to accomplish today?

How many tasks left in the backlog? Is the invoice integration ready?

No - when will it be ready?

Does the user have all essential invoicing functionality?

No - when will it be deployed?

Are we doing what was specified in the contract?

Did we bill all the hours?

Are we heading to the right direction?

Is the customer happy?

Are we using the resources (time and money) in the best possible way?

What is the most valuable thing to do next?

The amazing Kirre project tool (data migration)

Team‟s purpose

Short term plan (Kanban style)

backlog

Test plan

The purpose statement helped maintain focus and prioritize backlog.

About team organization

Tempting the dark side isIntegrations – back-end –UI…

Where‟s the end-user? Where‟s the use-case?

Ok, use-case/scenario based then?You need that cross-functional team. Got one?

You cannot force team formation!

The use-case based view is the right way. But not easy.After many changes we reinstated the integration team.

Leadershipment (case Kirre)

A single project manager, about twenty peopleA bottleneck, not parallel! A risk!

When faced with a serious situation. (final deadline)Divided the responsibility (leadership) to five individualsNo reports (or management duties) requiredNo holds barred, bare fist prison rules.. Just make it happen

Did it work? It was total pwnage and salvation of the project! The Great Enablers

For each team: Autonomy – mastery - purposeSupport and coaching. A scrum master who was not part of the team.Big picture priorization over teams. Together with everyone.

To “make it so”, important considerations

Voluntary. If a sane person commits, it‟s a good sign.

You must delegate power and freedom too. You need a manager who is not insecure!

Leader must have some respect from peers.Leadership is based on ”social influence” by definition.

A title is not related to this.

It’s about getting things done.It‟s not a ”tech lead” role.

Summary

To summarize.. How to make it alive..

Flexibility in scope, budget and timetables are required.Also the customer must accept this and trust you.

Respect the issues related to sizeMotivation, complexity, human limitations.

No single architect, PO, manager to make all decisions.

Do not ”manage” and control, lead insteadGood employees don‟t need ”management”

Get real cross-functional teams. Motivated & autonomous.

Stuff that victories are made of

Lean Agile

Continuous feedback

HonestyTransparency TrustMotivation

Empowered teams

Parallel development

End-user needs

MVP first Form follows function

Compositional designContinuous Integration

Continuous Delivery

Realistic shared vision and plan (backlog)

Continuous innovation Continuous flow

Automated tests

Use-cases in backlog

Resources and further reading

Even more booksLean From The Trenches

Lean Architecture

Lean Thinking

Lean Software Development: An Agile Toolkit

The Toyota Production System

Rational Choice in an Uncertain World: The Psychology of Judgment and Decision Making

Nonviolent Communication: A Language of Life

Systems Thinking

THANK YOU.

Antti Virtanen | [email protected]

@SolitaOy