Tes$ng Lies (The &tle is the first lie) Hannover, Germany, 2014 Cur$s “Ovid” Poe h5p://www.allaroundtheworld.fr/ Photo CC license: h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128h.jpg Copyright 2014, All Around The World March 26, 2014
May 08, 2015
Tes$ng Lies
(The &tle is the first lie) Hannover, Germany, 2014
Cur$s “Ovid” Poe h5p://www.allaroundtheworld.fr/
Photo CC license: h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128-‐h.jpg
Copyright 2014, All Around The World March 26, 2014
But first …
March 26, 2014 Copyright 2014, All Around The World
h5p://www.slideshare.net/gvwilson/bits-‐of-‐evidence-‐2338367
Bits of Evidence by Dr. Greg Wilson
March 26, 2014 Copyright 2014, All Around The World
• Overwhelming evidence that evidence works • Science demands evidence
• Programmers provide anecdotes
• (… and defend them vigorously)
Evidence
• Absence of evidence is not evidence • … but evidence of absence is
March 26, 2014 Copyright 2014, All Around The World
Absence of Evidence
• There is no evidence that gods exist • There is no evidence that gods don’t exist
March 26, 2014 Copyright 2014, All Around The World
Evidence of Absence
• Belief: small subs more maintainable/correct • Reality: these proper$es are absent
March 26, 2014 Copyright 2014, All Around The World
Evidence of Absence
• Belief: small subrou$nes/methods “be5er” • Inverse correla$on (Bell, Perricone 1984) • No correla$on (Shen, et al. 1985) • No correla$on and larger is cheaper (two studies) • Small rou$nes had more errors (Selbi, Basil 1991) • Less maintenance for larger rou$nes (Lind, Vairavan 1989)
• 500+ lines had more errors (Jones 1986)
(cita$ons via: Code Complete, Steve McConnell, Microsoh Press, 2nd Edi$on)
March 26, 2014 Copyright 2014, All Around The World
P.O.P. Copyright 2013, Cur$s "Ovid" Poe
Reality trumps opinion
3/26/14
Yeah, yeah, let’s get started
March 26, 2014 Copyright 2014, All Around The World
Caveats
March 26, 2014 Copyright 2014, All Around The World
• Some points self-‐evident • Some points maddening
• Not many studies
• Always be skep$cal
Photo CC BY-‐SA 2.0 h5p://www.flickr.com/photos/busbeytheelder/515542705/
Code Coverage
March 26, 2014 Copyright 2014, All Around The World
100% Code Coverage
March 26, 2014 Copyright 2014, All Around The World
100% Code Coverage
• What if we pass zero? • … or a string? • … or no argument?
• … or a filehandle? • … or type when reciprocals don't make sense?
• … or, or, or …
March 26, 2014 Copyright 2014, All Around The World
Expecta$ons
• Tes$ng should be an act of aggression • Programmers tend to test against expecta$ons
• False confidence
March 26, 2014 Copyright 2014, All Around The World
Code Coverage of Dead Code
March 26, 2014 Copyright 2014, All Around The World
Unit Versus Integra$on Tests
• Unit tested subs don’t talk to other subs • But your code certainly does! • Dead code is cogni$ve overhead • Dead code can’t be tested via your interface
March 26, 2014 Copyright 2014, All Around The World
Code Coverage of Dead Code
March 26, 2014 Copyright 2014, All Around The World
March 26, 2014 Copyright 2014, All Around The World
March 26, 2014 Copyright 2014, All Around The World
Covered but not tested
March 26, 2014 Copyright 2014, All Around The World
Devel::Cover Recap
• 100% code coverage doesn’t handle data • Don’t cover dead code • Covered but not tested • Don't blindly trust your tools
March 26, 2014 Copyright 2014, All Around The World
Kwalitee
• Lie: Perl community is the most test-‐infected • Truth: Quan$ty ain’t quality
March 26, 2014 Copyright 2014, All Around The World
CPAN Testers (March 6, 2014)
March 26, 2014 Copyright 2014, All Around The World
Versions and Plauorms
March 26, 2014 Copyright 2014, All Around The World
CPAN Testers
March 26, 2014 Copyright 2014, All Around The World
Perl Isn’t Stealing Enough
We need quality to go with quan$ty
March 26, 2014 Copyright 2014, All Around The World
All Pairs Tes$ng
March 26, 2014 Copyright 2014, All Around The World
The Coupling Effect
• Inves$ga$ons of the Sohware Tes$ng Coupling Effect — A. Jefferson Offu5
• Competent programmers write nearly correct programs
• Most faults are simple faults
• Simple faults can be easily detected
March 26, 2014 Copyright 2014, All Around The World
All-‐Pairs Tes$ng
• 10 arguments == 10 billion combina$ons • Combined pairs == 54 combina$ons
• Extend each argument with reasonable values
March 26, 2014 Copyright 2014, All Around The World
All-‐Pairs Tes$ng
First pairs Second Pairs Third Pairs Fourth Pairs And so on
arg1, arg2 — — — …
arg1, arg3 arg2, arg3 — — …
arg1, arg4 arg2, arg4 arg3, arg4 — …
… … … arg4, arg5 …
arg1, arg0 arg2, arg0 arg3, arg0 … …
March 26, 2014 Copyright 2014, All Around The World
All-‐Pairs Tes$ng
March 26, 2014 Copyright 2014, All Around The World
• Fast genera$on of tests • Likely to catch many more bugs
• Built in to many tools (for sta$c languages)
• For Perl: ALLPAIRS Test Case Genera$on Tool
Documenta$on
• Lie: Tests are documenta$on • Truth: I wish
March 26, 2014 Copyright 2014, All Around The World
March 26, 2014 Copyright 2014, All Around The World
March 26, 2014 Copyright 2014, All Around The World
March 26, 2014 Copyright 2014, All Around The World
Tests As Documenta$on
• Tests are ohen obfuscated • Ohen wri5en as an aherthought • Missing or non-‐descrip$ve test names
• Describe code logic instead of business logic
March 26, 2014 Copyright 2014, All Around The World
Making Tests Documenta$on
• Write them to be read • Describe use cases with: – test fixtures names
– subtest names – test method names
March 26, 2014 Copyright 2014, All Around The World
March 26, 2014 Copyright 2014, All Around The World
March 26, 2014 Copyright 2014, All Around The World
March 26, 2014 Copyright 2014, All Around The World
BDD for Test Documentaiton
• Test::BDD::Cucumber • Perl Behavior Driven Development (slideshare)
March 26, 2014 Copyright 2014, All Around The World
h5p://commons.wikimedia.org/wiki/File:Og%C3%B3rki...jpg
Test Driven Development
• Lie: TDD is a far superior tes$ng methodology • Truth: Prove it.
March 26, 2014 Copyright 2014, All Around The World
A Typical TDD Study
• Evalua$ng the Efficacy of Test-‐Driven Development: Industrial Case Studies
• Compared to non-‐TDD
• TDD increased development $me ~15%
• TDD had far fewer defects than non-‐TDD
March 26, 2014 Copyright 2014, All Around The World
What is “non-‐TDD”?
• No tes$ng? • Test last? • Test as you go? • QA-‐only tes$ng? • It doesn’t say!
March 26, 2014 Copyright 2014, All Around The World
Same Study
“An (sic) year-‐long empirical study performed at IBM using professional programmers found that the TDD prac?ce helps programmers produce higher quality code.”
March 26, 2014 Copyright 2014, All Around The World
Same Study
“An (sic) year-‐long empirical study performed at IBM using professional programmers found that the TDD prac?ce helps programmers produce higher quality code.”
Higher quality code than what?
And what does “quality” mean?
March 26, 2014 Copyright 2014, All Around The World
Pro-‐TDD Studies
• Comparisons against straw men common • Ohen use students • Some$mes use single developers
• Some$mes aware they’re being studied
• Or new to TDD (may be more mo$vated)
March 26, 2014 Copyright 2014, All Around The World
A TDD Enthusiast Looks At Evidence
• The benefits of TDD are neither clear nor are they apparent — Kane Mar – Part 1 – Part 2 – Part 3
March 26, 2014 Copyright 2014, All Around The World
Kane Mar – Aher Reading Studies
“[Claims about TDD] such as beHer design, beHer APIs, simpler design, lower complexity, increased produc?vity, more maintainable code etc., are simply not supported.”1
1. A cherry-‐picked, out-‐of-‐context quote.
March 26, 2014 Copyright 2014, All Around The World
The Reality (vis-‐à-‐vis other tes$ng strategies)
• TDD tends to generate more tests • Higher quality not proven • Absence of evidence, not evidence of absence • Don’t stress about it
March 26, 2014 Copyright 2014, All Around The World
Summary
• Evidence trumps anecdotes • We rely on anecdotes
• That’s sort of OK if we don’t have evidence˜ • Code coverage is only part of the solu$on • We need to escape our bubble
• Be skep$cal of hype • Be skep$cal of me
March 26, 2014 Copyright 2014, All Around The World
Summary for TDD Fana$cs
• Don't sweat anecdotes for casual ideas • Don’t accept anecdotes from dogma$sts
March 26, 2014 Copyright 2014, All Around The World
March 26, 2014 Copyright 2014, All Around The World
Image by Hay Kranen
Bonus Slides!
March 26, 2014 Copyright 2014, All Around The World
Basis Path Tes$ng
March 26, 2014 Copyright 2014, All Around The World
Cycloma$c Complexity
March 26, 2014 Copyright 2014, All Around The World
Cycloma$c Complexity (McCabe)
= (edges – nodes) + ( 2 * exit nodes ) = ( 19 – 14 ) + ( 2 * 1 ) = 7 = minimum number tests
Calcula$ng Cycloma$c Complexity
March 26, 2014 Copyright 2014, All Around The World
Cycloma$c Complexity and Bugs
• Some studies show a posi$ve correla$on • Other studies are not conclusive • May make bugs harder to fix
• May lead to side effects when making changes
• A Complexity Measure — McCabe • A Cri$que of Sohware Defect Predic$on Models — Fenton, Neil
• Cycloma$c Complexity Metrics Revisited — Gill, Kemerer
March 26, 2014 Copyright 2014, All Around The World
Perl
March 26, 2014 Copyright 2014, All Around The World