Feedback-Based Specification, Coding and Testing… …with JWalk Anthony J H Simons, Neil Griffiths and Christopher D Thomson
Jan 06, 2016
Feedback-Based Specification, Codingand Testing…
…with JWalk
Anthony J H Simons, Neil Griffiths and
Christopher D Thomson
Overview
Lazy Systematic Unit Testing JWalk testing tool (Simons)
The JWalkEditor tool Integrated Java editor and JWalk (Griffiths)
Feedback-based methodology Prototype, validate, specify, test
Evaluation of cost-effectivness Testing effort, time, coverage (Thomson)
http://www.dcs.shef.ac.uk/~ajhs/jwalk/
Motivation
State of the art in agile testing Test-driven development is good, but… …no specification to inform the selection of tests …manual test-sets are fallible (missing, redundant cases) …reusing saved tests for conformance testing is fallible –
state partitions hide paths, faults (Simons, 2005) Lazy systematic testing method: the insight
Complete testing requires a specification (even in XP!) Infer an up-to-date specification from a code prototype Let tools handle systematic test generation and coverage Let the programmer focus on novel/unpredicted results
Lazy Systematic Unit Testing
Lazy Specification late inference of a specification from evolving code semi-automatic, by static and dynamic analysis of code
with limited user interaction specification evolves in step with modified code
Systematic Testing bounded exhaustive testing, up to the specification emphasis on completeness, conformance, correctness
properties after testing, repeatable test quality
http://en.wikipedia.org/wiki/Lazy_systematic_unit_testing
JWalk Testing Tool
Lazy systematic unit testing for Java static analysis - extracts the public API
of a compiled Java class protocol walking (code exploration) - executes all interleaved
methods to a given path depth algebraic testing (memory states) - validates all observations
on all mutator-method sequences state-based testing (high-level states) - validates all state-
transitions (n-switch coverage) for inferred high-level states
http://www.dcs.shef.ac.uk/~ajhs/jwalk/
JWalkEditor
Integration of JWalk with a Java-editor
Editing features Java-sensitive Integrated with JDK
compiler tools – track exceptions to source
Testing features Invokes JWalk (6 validation/testing modes) Confirm/reject key results via dialogs Browse test result-sets via tabbed panes
Snapshot – Editing a Stack
Syntax sensitive text highlights
Full build cycle,multiple sources
Set JWalk test parameters
Snapshot –Testing a Stack
Tabbed pane with test results for the Empty state, for all
paths of depth 2
Colour-coded test outcomes
Colour-coded test sequences
Dynamic Analysis of States
Memory states generate all interleaved method paths to depth 1..n prune sequences ending in observers from the active edges,
preserving mutator sequences distinguish observer/mutator methods by empirical low-level
state comparison (extracted by reflection)
High-level states generate all mutator-method paths (as above) evaluate all state predicates, eg: isEmpty(), isFull() seek states corresponding to the product of boolean
outcomes: viz: {Default, Empty, Full, Full&Empty}
Test Result Prediction
Strong prediction From known results, guarantee further outcomes in the
same equivalence class eg: test sequences containing observers in the prefix map
onto a shorter sequence target.push(e).size().top() == target.push(e).top()
Weak prediction From known facts, guess further outcomes; an incorrect
guess will be revealed in the next cycle eg: methods with void type usually return no result, but may
raise an exception target.pop() predicted to have no result target.pop().size() == -1 reveals an error
Feedback-based Methodology Coding
The programmer prototypes a Java class in the editor Validation
JWalk systematically explores method paths, providing useful instant feedback to the programmer
Specification JWalk infers a specification, building a test oracle based on
key test results confirmed by the programmer Testing
JWalk tests the class to bounded exhaustive depths, based on confirmed and predicted test outcomes
JWalk uses state-based test generation algorithms
Example – Library Book
Validation surprise: target.issue(“a”).issue(“b”).getBorrower() == “b” violates business rules: fix code to raise an exception
Testing all observations on chains of issue(), discharge() n-switch cover on states {Default, OnLoan}
public class LibraryBook { private String borrower; public LibraryBook(); public void issue(String); public void discharge(); public String getBorrower(); public Boolean isOnLoan();}
Extension – Reservable Book
Validation surprise: target.reserve(“a”).issue(“b”).getBorrower() == “b” violates business rules: override issue() to refuse “b” here.
Testing all obs. on chains of issue(), discharge(), reserve(), cancel() n-switch cover on states {Default, OnLoan, Reserved,
Reserved&OnLoan}
public class ReservableBook extends LibraryBook { private String requester; public ReservableBook(); public void reserve(String); public void cancel(); public String getRequester(); public Boolean isReserved();}
Evaluation
User Acceptance programmers find JWalk habitable they can concentrate on creative aspects (coding) while
JWalk handles systematic aspects (validation, testing) Cost of Confirmations
not so burdensome, since amortized over many test cycles metric: measure amortized confirmations per test cycle
Comparison with JUnit propose a common testing objective for manual and lazy
systematic testing; evaluate coverage and testing effort Eclipse+JUnit vs. JWalkEditor: given the task of testing the
“transition cover + all equivalence partitions of inputs”
Amortized Interaction Costs
number of new confirmations, amortized over 6 test cycles con = manual confirmations, > 25 test cases/minute pre = JWalk’s predictions, eventually > 90% of test cases
Test class a1 a2 a3 s1 s2 s3
LibBk con 3 5 7 0 0 5
LibBk pre 2 8 18 18 38 133
ResBk con 3 14 56 0 11 83
ResBk pre 6 27 89 36 241 1649
eg: algebra-test to depth 2, 14 new confirmations
eg: state-test to depth 2, 241 predicted results
Comparison with
JUnit manual testing method Manual test creation takes skill, time and effort (eg: ~20 min
to develop manual cases for ReservableBook) The programmer missed certain corner-cases eg: target.discharge().discharge() - a nullop? The programmer redundantly tested some properties eg: assertTrue(target != null) - multiple times The state coverage for LibraryBook was incomplete, due to
the programmer missing hard-to-see cases The saved tests were not reusable for ReservableBook, for
which all-new tests were written to test new interleavings
Advantages of JWalk
JWalk lazy systematic testing JWalk automates test case selection -
relieves the programmer of the burdenof thinking up the right test cases!
Each test case is guaranteed to test a unique property Interactive test result confirmation is very fast (eg: ~80 sec in
total for 36 unique test cases in ReservableBook) All states and transitions covered, including nullops, to the
chosen depth The test oracle created for LibraryBook formed the basis for
the new oracle for ReservableBook, but… JWalk presented only those sequences involving new
methods, and all interleavings with inherited methods
Speed and Adequacy of Testing
Test goal: transition cover + equiv. partitions of inputs manual testing expensive, redundant and incomplete JWalk testing very efficient, close to complete
eg: wrote 104 tests, 21 were effective and 83 not!
eg: JWalk achieved 100% test coverage
Test class T TE TR Adeq time min.sec
LibBk manual 31 9 22 90% 11.00
ResBk manual 104 21 83 53% 20.00
LibBk jwalk 10 10 0 100% 0.30
ResBk jwalk 36 36 0 90% 0.46
Conclusion
Performance of JWalk testing clearly outperformed manual testing coverage based on all states and transitions input equivalence partitions are not yet handled
Performance of JWalkEditor unexpected gain: automatic validation of prototype code c.f. Alloy’s model checking from a partial specification
Moral for testing just automatically executing saved tests is not so great need systematic test generation tools to get coverage automate the parts that humans get wrong!
Any Questions?
http://www.dcs.shef.ac.uk/~ajhs/jwalk/
Bibliography•A J H Simons, JWalk: a tool for lazy systematic testing of Java classes by introspection and user interaction, Automated Software Engineering, 14 (4), December, ed. B. Nuseibeh, (Springer, USA, 2007), 369-418. SpringerLink: DOI 10.1007/s10515-007-0015-3, 8 September, 2007. Final draft version also deposited with White Rose Research Online.
•A J H Simons and C D Thomson, Lazy systematic unit testing: JWalk versus JUnit, Proc 2nd. Testing in Academia and Industry Conference - Practice and Research Techniques, 22-24 September, eds. P McMinn and M Harman, (Cumberland Lodge, Windsor Great Park: IEEE, 2007), 138. See also the A1 poster presenting this result.
•A J H Simons and C D Thomson, Benchmarking effectiveness for object-oriented unit testing, Proc 1st. Software Testing Benchmark Workshop, 9-11 April, eds. M Roper and W M L Holcombe, (Lillehammer: ICST/IEEE, 2008).
•A J H Simons, N Griffiths and C D Thomson, Feedback-based specification, coding and testing with JWalk, Proc 3rd. Testing in Academia and Industry Conference - Practice and Research Techniques , 29-31 August, eds. L. Bottacci and G. M. Kapfhammer and M. Roper, (Cumberland Lodge, Windsor Great Park: IEEE, 2008), to appear.
•A J H Simons, A theory of regression testing for behaviourally compatible object types, rev. and ext., Software Testing, Verification and Reliability, 16 (3), UKTest 2005 Special Issue, September, eds. M Woodward, P. McMinn, M. Holcombe, R. Hierons (London: John Wiley, 2006), 133-156.
•A J H Simons, Testing with guarantees and the failure of regression testing in eXtreme Programming, Proc. 6th Int. Conf. on eXtreme Programming and Agile Processes in Software Engineering (XP 2005), eds. H Baumeister et al., Lecture Notes in Computer Science, 3556, (Berlin: Springer Verlag, 2005), 118-126.
•Wikipedia entry for JWalk
•Wikipedia entry for Lazy Systematic Unit Testing