EXPERIENCE REPORT: LARGE SCALE AGILE DEVELOPMENT PRACTICES Daphne Chong (@daphnechong) Niall Connaughton (@nconnaughton)
May 13, 2015
EXPERIENCE REPORT: LARGE SCALE AGILE DEVELOPMENT PRACTICES
Daphne Chong (@daphnechong) Niall Connaughton (@nconnaughton)
What was the project?
C# Winforms SQL Server database Large CRUD system App deployed to client sites
Started in 2001 Market leading product
What was the project?
Low volume Performance important, but not critical Goal was a stable, consistent, feature-rich app Rapid release cycle (after any successful build)
In 2006…
60 Developers 5 million lines of code (including tests) ~100 solutions, 250 projects 300+ db tables 160,000 automated tests
Today…
60 Developers ~10 million lines of code 485 solutions 470 db tables 343,090 automated tests
The Pit of Success
We want our customers to simply fall into winning practices by using our platform and frameworks
– Rico Mariani
Build platforms [where] developers just fall into doing the “right thing”... Types should be defined with a clear contact that communicates effectively how they are to be used (and how not to)
– Brad Abrams
Pit of Success – Technical Factors
Inheritance Template Method pattern Strict naming conventions Common reusable controls Own type system Single generic repository – handled common
scenarios in CRUD, transactions, referential integrity, caching, persistence
Pit of Success
Tried to avoid too much divergence in code Made it difficult for people to stray from the “true
path”
Typical Application Layers
Typical Application Layers
Finding the value
Consistency
70 % of a dev’s time is understanding existing code – Peter Hallam [2]
If you can reduce that even by a small amount, it’s significant productivity gain
Consistency
Consistent approach to everything Naming conventions – project, solutions, file locations,
namespaces, class names, methods, database tables and columns
Formatting UI look and feel (achieved through base controls) Patterns of design
Consistency
Standardised on a core set of technologies that best fit the application
Avoided “technology soup” support and maintenance
Reduced learning curve for new developers
Insane things we did
60 devs, no feature branches! SourceSafe Highly sensitive to the build breaking True continuous integration Mitigated by clients opting in to “alpha”, “beta” or
“stable” versions Two large re-writes of architecture
Result was more suited to our needs than if we had kept the first and refactored it progressively
How did we build?
Full build from batch file Update source files Solutions built in dependency order defined in xml All binaries output to one folder Build on dev machine = 20-25 minutes Build server with RAID = 4 minutes
Quick build tool (written in c#) Only build solutions that have changes
Development - Architecture
Built our own architecture Re-usable controls Comprehensive base classes Own type system
Primitive types as well as Custom types Could define helper
methods .IsEmpty, .SubstringSafe() , IsInRange()
Development - Architecture
Architecture constantly extended Approach was always unified though the
architecture Enabled good ideas to have widespread effect
Development – Technical debt
Management support for improving bad code Aggressive attitude to technical debt Very little dead code
Example: methods with 7 overloads that can’t be deleted because don’t know who’s consuming them
Development – Technical debt
Constant small amount of pain Rare occurrences of large amounts of pain Once a hard cost has been paid, make sure that
nobody ever has to pay the cost again
Development - Custom Tools
Automated repetitive and error prone tasks Custom ORM generator Database upgrading tool GUI binding helpers to ensure targets exist Build monitor system tray application Code Generators for new modules Automated deployment tool
Testing
Two types of tests Business Tests: user functionality Meta Tests: code design and contracts
Testing
Most of our tests were actually integration tests of varying scope
Didn’t use a mocking framework (not yet well known)
Created our own stubs by inheriting from the concrete class and overriding any methods that used resources, etc.
Business Tests
Generally state based logic Easy to read and write Fairly resilient throughout refactoring Derived from a base TestCase with ~25 basic tests
Array properties don’t return null Properties don’t throw NullReferenceExceptions Don’t alter the state of the object before shown in GUI
Business Tests - Example
Meta Tests
Don’t test direct functionality Enforced consistency, patterns and code contracts Implemented using reflection and attributes Slow to run, but very valuable
Meta Tests – Examples
Check code conforms to conventions and contracts Public constructor with a specific parameter exists Assemblies are strong named Check for memory leaks
Check that each model had an associated test
[TestsSubclassesOf(typeof(BusinessBase))]public abstract class BusinessBaseTest{
// contract tests here}
Meta Tests
The Meta Tests help to enforce the rules, rather than having a “bad cop” in your team Not political or personal Once established, it gave power to the rest of the
team to contribute new tests and rules
Meta Tests
Allowed the architecture to grow rapidly Already know the code conforms to a particular
standard
How do you run 343,000 tests?
Initial approach similar to CruiseControl One machine building and running all tests Results emailed after completion
… stalled once we hit 25,000 tests … Took 4 hrs to run Feedback cycle wasn’t fast enough
Growth of Tests
Distributed Testing Framework
Highly customised Tests distributed to up to 60 agents Build time around 1 hour
if built on single agent = 30-40 hours Package on successful build Agents ran on idle developer machines Ran 24/7
Distributed Testing Framework
Distributed Testing Framework
Developer submitted builds could be processed (i.e. pre-checkin)
Task tray to signal progress Red light = broken tests, broken build or checkin
embargo Green light = OK to check in
Team
Very Flat structure 6-8 devs in architecture team (“Blue Code”)
Business & GUI architecture Development tools Testing framework Database development Upgrading code
Most other developers building business rules into software (“Red Code”)
Architecture vs Business teams
Team - Architecture
Made up less than 20% of team Improved productivity by much more than 20% Allowed devs to focus on smaller slice of stack and
get quicker feedback cycles
Team - Business
Low barrier to entry for development Junior or new developers could be productive very
quickly Varied skill sets in the team were utilitsed
accordingly Technically minded people drawn to architecture Business minded people focused on implementing
customer requirements
Team – Sharing Information
Free flowing collaboration Some development overhead in helping other
people Some information silo-ing
Team – Pair Programming
Initially strict pairing Became “organic”
Not enforced Never discouraged
Team – Work Ethic
People took a lot of responsibility and accountability of their own accord
Boy scout rule – leave things cleaner than when you found it
Always fix broken windows
Work satisfaction
Low turnover rates for first 4 years Lots of smart people to work with More meritocracy than political Developers took pride in their work Ability to implement large scale change open to
everyone
Work Satisfaction – flip side
Hard to have interesting work for 60 developers Could be pigeon-holed into the business layer More stress than an average agile environment Long work hours implied because there’s only one
build – responsibility to keep the dev machine going. No leaving the build broken or failing tests throughout lunch or overnight.
Conclusion
Team made a lot of investment in design and conformity
Easy to do with a CRUD system, harder with other types of business software
Project has expanded and moved on, but the architecture has evolved with it and is still in use
Thanks to…
Zubin AppooSenior Manager, Product Development
www.cargowise.com
References
[1] Brad Abrams, The Pit of Success http://blogs.msdn.com/b/brada/archive/2003/10/02/50420.aspx
[2] Peter Hallam, What do Programmers Really do Anyway? http://blogs.msdn.com/b/peterhal/archive/2006/01/04/509302.aspx