Transcript
End-To-End TestingConsidered Harmful
Steve Smith
@AgileSteveSmith
www.alwaysagileconsulting.com
Steve Smith
Continuous Delivery consultant
Co-organiser of LondonCD and PIPELINE
Co-author of Build Quality In on Leanpub
b: www.alwaysagileconsulting.com/blog
l: www.linkedin.com/in/agilestevesmith
t: @AgileSteveSmith
Always Agile Consulting Ltd
Continuous Delivery consultancy
Continuous Delivery training courses
e: hello@alwaysagileconsulting.com
t: @AlwaysAgile
w: www.alwaysagileconsulting.com
Associated with Equal Experts
End-To-End Testing
Home Move
A Home Move Story
A Home Move Story
End-To-End Testing
End-To-End Testing
End-To-End Testing
High Coverage?
High Coverage?
Low Coverage
Low Coverage
End-To-End Testing
Low Investment?
Low Investment?
High Investment
High Investment – Response Times
High Investment – Non-determinism
End-To-End Testing
Continuous Testing
High Coverage
High Coverage
Low Investment
Continuous Testing
Unit Testing
Name Test Scope Quantity Frequency Duration Env
UnitTesting
Intent vs. impl Units of service
100 → 1000+
Per build < 30s total
● Local● Build
Name Test Scope Quantity Frequency Duration Env
Acceptance Testing
Impl vs. reqs Service 10 → 100+
Per build < 10m total
● Local● Build
Acceptance Testing
Name Test Scope Quantity Frequency Duration Env
Exploratory Testing
Explore impl vs. reqs
Service 10 → 100+
Per build Timebox per test
● Local● 3rd party
Exploratory Testing
Name Test Scope Quantity Frequency Duration Env
Contract Testing
3rd party impl vs. reqs
3rd party service(s)
~20 Per 3rd party deploy
< 1m total
● 3rd party● Build
Contract Testing
Contract Testing
Name Test Scope Quantity Frequency Duration Env
Smoke Testing
Impl vs. reqs Entire system
~5 Per deploy < 5m total
All
Smoke Testing
Monitoring
Name Test Scope Quantity Frequency Duration Env
Monitoring Impl vs. reqs Entire system
10 → 100
Always < 10s total
All
Monitoring
Continuous Testing
Name Test Scope Quantity Frequency Duration Env
UnitTesting
Intent vs. impl Units of service
100 → 1000+
Per build < 30s total
● Local● Build
Acceptance Testing
Impl vs. reqs Service 10 → 100+
Per build < 10m total
● Local● Build
Exploratory Testing
Explore impl vs. reqs
Service 10 → 100+
Per build Timebox per test
● Local● 3rd party
Contract Testing
3rd party impl vs. reqs
3rd party service(s)
~20 Per 3rd party deploy
< 1m total
● 3rd party● Build
Smoke Testing
Impl vs. reqs Entire system
~5 Per deploy < 5m total
All
Monitoring Impl vs. reqs Entire system
10 → 100+
Always < 10s total
All
Caveats
End-To-End Testing
“Not only won’t system testing catch all the bugs, but it will take longer and cost more – more than you save by skipping effective acceptance testing”
Gerald Weinberg
End-To-End Testing looks good, but is invariably bad
Continuous Delivery needs rapid, reliable feedback
Continuous Testing is a worthwhile investment
End-To-End TestingConsidered Harmful
Steve Smith
@AgileSteveSmith
www.alwaysagileconsulting.com
top related