Testing
Jan 18, 2016
Testing
2
Today’s Topics
• Why Testing?• Basic Definitions• Kinds of Testing• Test-driven Development• Code Reviews (not testing)
3
Why Testing?
• Ideally, we’d prove that the code was correct using formal mathematics
• This is extremely difficult even for trivially small programs
• Not practical in most cases• Not even for mission critical or safety code
4
Why Testing?
• Near ideal: use symbolic or abstract model testing • Prove that model is correct• Automatically extract mathematical model from code• Prove properties of model for all possible cases
• In practice, can work well for simple properties• E.g., “System will not crash in this way”
• Doesn’t work well for complex systems / data structures
5
Why Testing?
• We can actually run code to see if it works• This is software testing
• Always necessary, even when you can prove correctness• Proof seldom tied to actual code
• Testing is NOT a lost resort; it is critical part of sw dev
6
Why Testing?
• Testing all too often an afterthought
• Test before faults become too expensive to fix or pressing deadlines
• The more code written before fault is detected, the more code needs to be fixed…
• Better to detect fault(s) on a small prototype or “finished” product?
7
Waterfall
Requirements analysis
Design
Implementation
Operation
Testing
Prototyping
8
Spiral
Draft a menu ofrequirements
EstablishrequirementsPlan
Analyze risk &prototype
Draft a menu ofarchitecture designs
Analyze risk &prototype
Analyze risk &prototype
Draft a menu ofprogram designs
Establisharchitecture
Establishprogram design
Implementation
TestingOperation
Plan
9
Agile
Customer provides “stories”(short requirement snippets)
System and acceptance tests
Do “spike” to evaluate & control risk
Prioritizestories and plan
Implement
OperationWrite/run/modify
unit tests
10
Testing Saves Lives and Money
• NIST Report: “The Economic Impact of Inadequate Infrastructure for Software Testing”
• Inadequate software testing costs US $22 - $59 billion annually
• Better approaches could cut this is half
11
Testing Saves Lives and Money
• Major failures: • Ariane 5 explosion• Mars Polar Lander• Intel Pentium FDIV bug• THERAC-25 radiation machine: killed 5
Ariane 5- exception-handling bug forced self destruct on maiden flight (64-bit to 16-bitconversion: about $370 mill lost)
12
Today’s Topics
• Why Testing?• Basic Definitions• Kinds of Testing• Test-driven Development• Code Reviews (not testing)
13
Definitions: Testing
• Running a program to find faults (bugs)• … in code• … in specs• … in documentation• … in tests
14
Definitions: Test Case, Test Suite, Tester
• Test case: one execution of the program that may find a bug
• Test suite: a set of program executions, grouped together• Made of up test cases
• Tester: program that generates tests
15
Definitions: Coverage
• Measures or metrics• Abstraction of “what a test suite tests”
• Common measures:• Statement coverage• Decision coverage• Path coverage• Mutation coverage
16
Definitions: Coverage
• Statement coverage:• aka: line or basic block coverage• Which statements execute in test suite
• Decision coverage: • Which boolean expressions in control structures evaluate to true and
false during suite execution.
17
Definitions: Coverage
• Path coverage:• Which paths through a program’s control flow graph are taken in the test
suite
• Mutation coverage:• Ability to detect random variations in the code
18
Today’s Topics
• Why Testing?• Basic Definitions• Kinds of Testing• Test-driven Development• Code Reviews (not testing)
19
Kinds of Testing
• White box• Black box• Unit • Integration• System • Acceptance
20
White Box Testing
• Opens up the box• aka glass box, clear box, structural testing
• Use source code to design test cases
21
Black Box Testing
• Testing does NOT look at source code or internal structures
• Sends program stream of inputs, observes outputs
• Abstracts away the internals• Useful perspective for system and integration testing
• Sometimes you don’t have access to the source code
22
Testing Stages
• Unit testing: first phase; done by module develops
• Integration testing: combines unit tested modules, tests interactions
• System testing: test whole program to make sure reqs met
• Acceptance testing: tests by users to see if system meets reqs
23
Unit Testing
• First phase; mostly done by module developers• Typically earliest type of testing• Unit could be as small as single function or method• Often relies on “stubs” to represent other modules or
incomplete code• Tools to support unit tests available for most popular
languages, e.g., Junit ( http://junit.org )
?
24
Integration Testing
• Combines unit-tested modules and tests interactions• Relies on completed units• After unit testing, before system testing• Test cases focus on interfaces between components and
assemblies of multiple components• Often more formal than unit testing• More detailed test plan(s)
??
?
25
System Testing
• Tests whole program; make sure reqs met• After integration testing• Focuses on “breaking the system”• Finds defects in completed product, not just component
interactions• Checks quality of requirements as well as system• Often includes stress testing: beyond bounds of well-defined
behavior
26
System Testing: Functional Testing
• Developer tests program from “user’s” perspective• Different mindset than unit testing
• Analogy: home inspection vs. walk-through • inspector => software engineer; looking for “hard facts”• buyer => system user; looking for functionality
27
Acceptance Testing
• Tests by user to see if system meets use requirements
• Form of black box testing
• Done by end-users to determine if system really meets needs
• May revise requirements
• May find bugs in the system
28
Today’s Topics
• Why Testing?• Basic Definitions• Kinds of Testing• Test-driven Development• Code Reviews (not testing)
29
Test Drive Development
• Write test cases before code• Make sure code is tested as early as possible• Idea arises from Extreme Programming• Often used in agile development
• Write automated test cases first• Then write code to satisfy tests
30
Test Driven Development
• How to add a feature:
1. Add a test case that fails (but succeeds with new feature)2. Run all tests, make sure only new test fails 3. Write code to implement new feature4. Rerun all tests, make sure new test succeeds (no others fail)
31
Test Driven Development Cycle
32
Test Driven Development Benefits
• Results in lots of useful test cases• A very large regression set
• Forces attention to controllable and observable sys behavior • Only write code as needed to pass tests• May get good coverage of paths through program• Reduces temptation to tailor tests to idiosyncratic behaviors
• Testing is a first-class activity in this kind of development
33
Problems with Test Driven Development
• Needs institutional support• Difficult to integrate with waterfall process• Management may wonder why time is spent writing tests, not code
• Many test cases may create false confidence• If developers have written all tests, there may be blind spots due
34
Exhaustive vs. Representative Testing
• Can we test everything? (exhaustive)• Probably not! Permutations / combinations are exponential!
• Need to be thoughtful about creating representative tests• Create tests at boundary conditions• Want to “represent” the range of input values
35
Today’s Topics
• Why Testing?• Basic Definitions• Kinds of Testing• Test-driven Development• Code Reviews (not testing)
36
Code Reviews
• Code Review ≠ Testing
• Method for finding bugs and determining code quality
• Code walkthrough: developer leads review team through code• Informal
• Code inspection: review team checks against concerns• Prepared offline (in many cases)• Led by a team moderator
37
Code Reviews
• Code inspections: found to be one of the most effective practices for finding faults• Some experiments show removal of 67-85% of defects via inspections
• Some consider XP’s pair programming as a kind of “code review” process
• Can review/walkthrough requirements and design docs, not just code
38
Next Steps
• Homework 5 due Friday!!
• Vision paper due 11/21!!
• Midterm due 11/25!!