Continuous Delivery with Smarter Testing
May 14, 2015
Continuous Delivery with Smarter Testing
Me
Dave Hart – Lead Developer in Test @ IBM
@DeveloperInTest
http://developerintest.blogspot.com/
Agenda
• Continuous delivery
• Smarter story level testing
• Smarter product level testing
• Automation
• One team
• Stories +
• Conclusion
• Questions
Continuous Delivery
The Agile ideal
The twelve principles of agile software:
1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
7. Working software is the primary measure of progress.
Where we can fall short
Test in agile teams– Can still be silo’d
– Let test lag behind development
Story focused– Our week by week work is blinkered – strong focus on individual
stories
– Lots of testing not necessarily story related gets neglected• System Quality Testing (Installers, performance, deployment,
compatibility, stress, release hardening, globalisation etc etc)
What can we do?
• Develop a ‘Test Runway’
• Be a team
Smarter TestingStory Level
Smarter testing – story level
• Start testing earlier– Requirements and design– TDD, BDD– Alongside development
Smarter testing – story level
• Minimise test documentation– Guidelines over precise steps– Completed stories and running tests primary
measure of testing performed
• Know your features– Understand the purpose of the story deliverable– Know how it fits in the wider product– Be involved in the design
• Prioritise the scenarios– Attempt to understand how a feature will be used– Break it down into common, occasional and possibly
never scenarios, and prioritise accordingly
Smarter testing – story level
• Reduce test duplication– Know who is testing what– What is tested at the unit level, integration
system etc
• Utilise test metrics– Coverage statistics– Defect route cause analysis– In sprint defects raised
Smarter TestingProduct Level
Smarter testing – product level
• Minimise Test documentation– Approach and strategy over detailed plan– Guidelines for common areas– The backlog state is our measure of progress
• Know your customers– Understand how and why they use your
software– Understand the day to day routes through
your software– Prioritise the bits that are important over the
rarely used (test for the 80%)
Smarter testing – product level
• Have an idea of all the parts of your release
• Know what is tested where– Reduce duplication– Identify bits that fall between teams/stories– Define a test strategy / approach so we have
an idea of the confidence we can derive from the various testing activities
• Choose your test technologies carefully
Smarter testing – an example
Web Service Application
Client
Unit
UnitUnit
Unit
Unit
Unit
Component
Component
Unit
Unit
Unit
Component
Smarter Testing - Automation
Automation
• Regression, regression, regression– Continuous feedback of continued operation– Run often (continuous integration builds ideal)
• Build in testability– At code level– At UI level– Architect and model for code re-use
• Automate what you can
Automation
Diagram from Agile Testing – Lisa Crispin & Janet Gregory
One Team to Rule them All
One team
• Commit as a team, deliver as a team
• Support specialism's
• Develop a quality culture
Stories +
Stories + - infrastructure
Infrastructure is Key!
Build systems– Fast feedback– Track failures back to source easily– Configurable triggers
Stories+ - DevOps
• Automate your deployments– Link it into your build system– Make it available– Consider build frequency
Web ServicesApplication
CustomCode
Web Server
HardwareConfiguration
Environment
CustomerConfiguration
Deployment
Stories + - DevOps
• Enables– Cutting edge software for demonstration– Fast, usable feedback – at all levels
Stories + - DevOps
Testability
CodeCodeTestsCodeTests
Deployment
ConfigurationTesting
PerformanceTesting
User testingAPI testingGlobalisation
Testing
Enables
Smarter Testing – An example
Web Service Application
Client
Unit
UnitUnit
Unit
Unit
Unit
Component
Component
Unit
UnitUnit
Component
DeployedSmoke test
Deploy
PerformanceFully System
UI test
Conclusions
Conclusions
• Start at the design
• Automation and regression
• Testing Runway
– Step back– Inspect– Plan
• Know your product
• Invest
Conclusions
Recommended Reading
• Agile Software Requirements – Dean Leffingwell
• Agile Testing – Lisa Crispin, Janet Gregory
• Continuous Delivery – Jez Humble, David Farley
• Cartoon tester: http://cartoontester.blogspot.com/
Any Questions?