Testing Overview - users.ece.cmu.edu · Testing finds bugs but why are you finding bugs? Testing Low Quality Software: Find bug, remove bug, iterate until out of time How many bugs
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
"In September of 1962 … an $18 million rocket had been destroyed in early flight because "a single hyphen was left out of an instruction tape."... The nature of programming being what it is, there is no relationship between the "size" of the error and the problem it causes. Thus, it is difficult to formulate any objective for program testing, short of "the elimination of all errors" - an impossible job. "
“Smoke Test” Quick check to see if the software runs at all Valuable pre-test if it exercises major functions
Exploratory Testing Expert tester beats on the system Can find low hanging fruit to fix if tester is good
Problems with unstructured approaches Hard to know how much you’ve tested Test coverage is often subjective Effectiveness heavily dependent upon tester skill
Black Box Testing Use requirements to guide test design Exercise all code functions
White Box Testing Use structure of code to guide test design Exercise all paths through code
Need both black box and white box testing Black box testing can miss implementation corner cases White box testing doesn’t exercise missing functionality
Coverage: how completely have you tested? White box coverage: fraction of code exercised in test Black box coverage: fraction of requirements exercised Other coverage: e.g., fraction of state chart arcs traversed
Testing within V process Unit test: single executable procedure Integration test: do modules play nice? Software test: exercise system or subsystem Acceptance test: whole-product test with
customer use scenarios Other types of testing Beta test: see if representative users discover defects Regression test: test to see if a previously fixed bug comes back Performance testing: measure capacity and identify bottlenecks Robustness testing: what happens with invalid inputs? Security testing: can an attacker penetrate the system? Fault injection: what happens when a component fails?
Good starting point for more info: http://www.softwareqatest.com/index.html
Useful references: Kaner, Cem, Jack Falk, and Hung Quoc Nguyen.
Testing Computer Software, Second Edition.International Thomson Computer Press, 1993. ISBN 1-85032-847-1. OR John Wiley & Sons, Inc., 1999. ISBN 0-471-35846-0.
Beizer, Boris. Black Box Testing. John Wiley & Sons, 1995, ISBN 0-471-120904-4.