Who are these guys, anyway?
Vibhu SrinivasanAgile Coach, Technologist.Doing XP since 1999
Ed KraayScrum Master, CSPAgile Coach
Doing scrum and XP since 2004
Hypothesis: Scale hurts due to increased communication cost
• Communication increases geometrically, while team members are added linearly.
• Open environment was not enough to solve the issue.
• Teams locally optimize
The context: An MMO game project0 Identity Management
Suite
0 Content Management System
0 Social Networking
0 3D Game
0 Back-end game infrastructure
The service platform
0 Technologies in Play
0 Java, Windows Communication Foundation , C# .NET 2.0, .NET 3.0, ASP.NET
0 PHP
0 A packaged product for content management
0 LDAP
EnvironmentScrum + XP
•Scrum Process•All 3 roles, artifacts, and ceremonies
•XP Practices•Paired Programming•Continuous Integration•Test Driven Development
•Open environment
5 teams, 9 projects, ~50 people
0 Multiple Team Gaming Program
Customer Team
7/18/2010All Rights Reserved – Vibhu Srinivasan
http;//agilefaq.net 8
How we organized our Meetings, and how that mapped to the 5 levels of
planning
TimelineTeam Forming:
March 07
First Release, Infrastructure:
August 07
Game Infrastructure
Teams: August 07
Program Retrospective
June 08
Planned Releases:
September 08 –December 08
The Timeline
How did we generate these insights?
• Held a cross team program retrospective
• 1 day facilitated event
• Set the Stage, Built a 1+ year timeline, generated insights and defined actions
• This was in addition to project, iteration retrospectives
Hurt: Undone work at the end of a sprint
0 Caused the next sprint to reduce velocity, reduced quality
0 Helped: Tracking the Right Metrics: Committed vs actual points, and worked to even them out.
Test Coverage
Committed vs
Completed Points
Hurt: Defects slowed velocity
0 Helped: Test Driven Development
0 Hypothesis: TDD reduces defects.
0
100
200
300
400
500
600
700Defects and Tests
Unit Tests
Acceptance Tests
Defects
Hurt: Multiple Backlogs0 Priorities handled on
the back end, mid sprint
0 Caused injection of priorities
Many Teams, many backlogs
Helped: chief product owner
Hypothesis: product definition team
0 A team of product owners, UX, stakeholders, working ahead of the teams, defining and feeding the backlog.
Hypothesis: Use one product backlog to reduce pain of multiple backlogs
Hurt: Multiple business priorities
0 Caused shifting focus on team.
0 Helped: Vision Statement
Hurt: Work inserted in the middle of the sprint
0 Would break the sprint, or cause delays
Helped: Synchronizing our sprints
Hurt: lack of cross team culture 0 Caused silo effect,
even when teams were feet apart from each other.
0 Helped: X-Team Events: Retrospectives, Lunch, Release Planning, Celebrations.
Team A
rules!
Team B
rules!
SwarmingHelped: Collaborative Environment
Helped: X-Team Release Planning
Helped: Ensuring all customer stakeholders are represented by
Product Owners
Hurt: Impediments
0 Hardware Requisition Process
0 Network Issues
0 Team Issues
Stop the line
7/18/2010All rights reserved Vibhu Srinivasan
26
Services are painful to consume
0 Teams are working in silos and when services are built, they are too painful to consume
Service Versioning
Well defined contracts
Service level agreements
Lack of design caused too much technical debt
0 Agile Myth = There is no design and architecture in Agile development
0 Going from story to code causes a lot of technical debt
0 UML is not enough to convey details
0 Teams are required to white board designs one day after the sprint planning with others
0 Use more spikes to understand a design before it can be used
0 More emphasis on pairs to talk about what they are going to do, before changing code
UML does not convey the details of the complex models
0 Some services had too many complicated scenarios that could not be communicated easily in pictures.
0 Use Fundamental Modeling Concept ( FMC ) models to convey the overall intent of the system0 Color Coded by Release
linehttp://www.fmc-modeling.org/
0 Invest in a great digital camera and a large whiteboard
SERVICE
Services Architecture
PrivisionCollection
Package services
Goals Accounts
Uniquq Rights
Client Systems
Database
RSERVICE API
R
Manage
CollectionR
Purchase
R
Open
R
Combine/
ConvertR
Unique
R
Use
In
Game
R
View
Administration
RSERVICE API
R
Redeem
R
Escrow
R
Destroy
R
Share
Security
R
Define
R
Package
R
Publish
R
Create
Collation R
Recipe
R
Consume
RLog R
AuditR
Manually
Override
Ownership
RSet
Inventory
Levels
RTrack
Ownership
Patterns
R
Audit
R
Repository
Feed
R
Health
Check
Release Line 1
Release Line 2
Release Line 3
Game
New Game
Game2
Game 3
Game 4
R
Trade
R
Run
Recipe
More Engineering Pains0 Common Engineering
Practices Missing
0 Code was growing too fast in too many ways across teams
0 Duplicated code
0 Customers had a core architecture group that had less and less visibility as the teams grew and this caused unease.
One solution to these pains
0 Introduce a cross team architect and embedded agile coach0 A servant thought
leader
0 Mentor and the readily available to pair
0 Shields teams from technical bombshells
0 Active listener and helper
0 Help inject technical stories
More People = Less Quality
0 Some teams were reporting lower bugs than others
0 Unit testing was not consistent
0 Acceptance testing was missing
0 Team did not see value in TDD
0 More debates and arguments were leading to unhappy developers
0 Bad smells in code
Hurt: Defects slowed velocity
0 Helped: Test Driven Development
0 Hypothesis: TDD reduces defects.
0
100
200
300
400
500
600
700
Defects and Tests
Unit Tests
Acceptance Tests
Defects
Better Smells
0 Teams were asked for mandatory testing . They could choose to implement the tests any way they want, but should have unit tests
0 Build systems were made smarter to publish test results
0 Category tags were used to group tests for quicker running.
0 Inject one or two senior developers who knew TDD well in every team- Peer pressure = great code
7/18/2010All Rights Reserved – Vibhu Srinivasan
http;//agilefaq.net 36
It works for me syndrome
0 One teams code does not interface with the other team well
0 Code working on local build fails to integrate
0 Its not easy to deploy the code to customer
Solution – Integrated build and deployment strategy
0
Odds and Ends
0 Mock objects were mockingly simplistic
0 Acceptance tests if not automated does not provide a whole lot of value
0 MOB Programming helps for cross team communication
0 A team agreement document is a must have tool.