SASQAG $99 day, October 21 st Seattle Area Software Quality Assurance Group Acceptance Test Driven Development (ATDD) in Practice This one-day class covers the whole ATDD cycle, including Discussing the new feature or story, Distilling acceptance criteria into acceptance tests, Developing the feature and automating the acceptance tests, and Demonstrating the resulting feature to the business stakeholder(s) responsible for accepting it. Elisabeth Hendrickson has over 20 years software industry experience and has been working with Agile teams since 2004. 2010-10-05 1
31
Embed
Seattle Area Software Quality Assurance Group Acceptance Test Driven Development (ATDD) in Practice This one-day class covers the whole ATDD cycle, including.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
SASQAG $99 day, October 21st
Seattle Area Software Quality Assurance GroupAcceptance Test Driven Development
(ATDD) in PracticeThis one-day class covers the whole ATDD cycle, including Discussing the new feature or story, Distilling acceptance criteria into acceptance tests, Developing the feature and automating the acceptance tests, and Demonstrating the resulting feature to the business stakeholder(s) responsible for accepting it.
Elisabeth Hendrickson has over 20 years software industry experience and has been working with Agile teams since 2004.
Over test: For each operation, repeat the same operation on a side “test” implementation.
Test system must be as performant as Real system (to keep same level of concurrency)
2010-10-05
7
Banking Transactions
Transfer $100 from checking to savings1. Add money to savings or Remove
from checking2. Remove money from checking or Add to
savings 2 Steps, both or neither must occur
for consistency.
Checking Savings
$100 $100
Checking Savings
$100 $100
2010-10-05
8
Simple Transaction Oracle
Maintain a consistency invariant outside of transactions (but violated inside)1. Randomly choose first operation and value:
Transfer Savings -> Checking 1,234.572. Choose last operation to keep balances
/1,000A. Add more: Savings -> Checking 765.43
(2,000 total transfer)B. Add less : Checking -> Savings 234.57
(1,000 total transfer)2010-10-05
9
Simple vs Accurate
Pros: Simple fast Check
(can be made post production)Cons: Difficult Failure Diagnosis
(just like with real customers)Related notions: Metamorphic testing Sampling & Consistency Oracles
lesson #125 “Avoid Complex logic in your Scripts.”2010-10-05
Metamorphic Relation (MR) Reuse existing test cases to generate
more tests. (Existing tests as Oracles) MR: Any relation among the inputs
and the outcomes of multiple executions of SUT.
SqRt(X)*SqRt(Y) == SqRt(X*Y) Test 1: SqRt(9) == 3 Gen Test 2: SqRt(9)*SqRt(2) ==
SqrRt(9*2)
Shortest Path examples
• ShortestPath(G, a, b)find the shortest path between vertices a and b in graph G and also output its length, where G is an undirected graph with positive edge weights. (dijkstra’s algorithm)
• Reverse: The shortest path between B and A should be the reverse of the shortest path between A and B.
• Prefix: For any vertix, V, on shortest path between vertices A and B, the Shortest path between A and V must be the same sequence as the start of the path between A and B.
Test Boyer-Moore example
Boyer-Moore algorithmreturns the index of the first occurrence of a specified pattern within a given text.
Metamorphic Relations Reverse: if string X exists in string
Y, then the reverse of Y exists in the reverse of X.
13
Agenda
Lessons Learned in Software Testing Using Simple OraclesTest Case size and Known Errors Never be Gatekeeper – know release
goals Testing Exception handling Alternatives to execution Testing Combinations
2010-10-05
14
Pass Rate by 25% jumps
1 3 5 7 9 11 13 15 17 19 21 23 25 27 290
25
50
75
100Keith ProjectA ProjectB
Test Cycle (or week)
% Pass Rate
lesson #280 “How to lie with
test cases”
2010-10-05
15
Test in the Presence of Failures Spec Explorer “long test”
Ideal: Single test for all transitions in a model
Pass/Fail test == Pass/Fail product Simplest way to design test cases is
to fail fast. As soon as an error is detected, note the error and stop any further processing (Assert)
Information loss when many Asserts.Symptom: Many “blocking” failures
lesson #34 “It Works” really means it appears to meet some requirement to some
degree”2010-10-05
16
Test Case Size
“Appropriately complex” [Kaner 2003] Towards smaller tests: Fewer blocking failures (less dependency/commonality) Parallel execution. Test Prioritization for regression (running < whole set) Automatic Failure Analysis tools
Towards larger tests: Reduced redundant setup/cleanup time Chance of inadvertently implementing a test idea that
finds a bug. [Marick 2000]
Systemmedium-large?
Integration
medium
Functional
small-med
Unitsmall
2010-10-05
17
Agenda
Lessons Learned in Software Testing Using Simple Oracles Test Case size and Known ErrorsNever be Gatekeeper – know
release goals Testing Exception handling Alternatives to execution Testing Combinations
2010-10-05
18
Lesson #12, “Never be the gate keeper”
New Java Transaction Api (JTA) [Part of J2EE]
Built on top of existing Transaction engine.
1 Week before Beta: Multi-threaded testViolated Isolation! (I in ACID)
I and Dev mgr think must slip to fix Project Mgr understands meets
release goals!Just document in release notesSet customer expectations
#205: “Don’t sign-off to approve the release of a product”2010-10-05
19
Agenda
Lessons Learned in Software Testing Using Simple Oracles Test Case size and Known Errors Never be Gatekeeper – know release
goalsTesting Exception handling Alternatives to execution Testing Combinations
2010-10-05
20
Old: Returning Error codes
2010-10-05
If (!ret) return ret;
21
Modern: Exception Propagation
2010-10-05
22
Normal/Exception States
Start
Norm
al
retru
n
Previous Return
Exception
Exc
epti
on
Exception
Normal return
Previous Return
Normal
Exception
Normal return
2010-10-05
Lesson #25, “All testing is based on models”
23
Pattern to Exercise Transitions
Start
2. N
orm
al
retru
n
Previous Return
Exception
3.
Exc
epti
on
1. Exception
Previous Return
Normal
4. Exception
5. N
orm
al
retru
n
2010-10-05
24
Agenda
Lessons Learned in Software Testing Using Simple Oracles Test Case size and Known Errors Never be Gatekeeper – know release
goals Testing Exception handlingAlternatives to execution Testing Combinations
2010-10-05
25
Avoiding hard stuff?
2010-10-05
26
Agenda
Lessons Learned in Software Testing Using Simple Oracles Test Case size and Known Errors Never be Gatekeeper – know release
goals Testing Exception handling Alternatives to executionTesting Combinations
2010-10-05
27
Testing Combinations
2010-10-05
In the code,the BypassProxyOnLocal property on the WebProxy object does not reflect the fact that it is checked
30% fewer test cases vs. minimal test suite created manually: 60 tests from PICT vs. 80-90 manual out of 200 possibilities (exhaustive). Testers had high confidence in quality
235 tests generated for a domain with over 1M possible combinations. All these tests implemented in less than one week; traditional methods: 15-30 tests per week. An order of magnitude of a difference.
Games testing: Asheron’s Call testing: the team was able to cut test
resources by half due to use of PICT Vanguard testing: 4 man-months to build and
implement test casesvs. 44 man-months during last release. Process more efficient by an order of magnitude
Windows command-line tools testing: (findstr.exe &) Attrib.exe testing: 13 test cases generated by PICT
achieved identical block coverage as 370 exhaustive test cases
2010-10-05
29
Summary
Testing is a business activity with a cost. Understand the product and release
goals, Deal with common issues like known
errors, Anticipate likely errors Choose the right evaluation methods
2010-10-05
30
Remember
Use Simple Oracles Allow for Known Errors Choose “appropriately complex” Test
Case size Know release quality goals
GateKeeeper Test Exception handling Consider Pairwise Testing of
Combinations Consider Alternatives to Execution2010-10-05