Designing Software For Testability A forgotten design pattern Rohit Nayak Talentica
Jan 15, 2015
Designing Software For Testability
A forgotten design pattern
Rohit Nayak
Talentica
Agenda Background and Motivation DFT in VLSI and hardware design Issues in Software Testing DFT for Software Development Automation and Testability Some Examples Cons, Review
The Anti-Pattern
If it works, its the developer, if not blame QA!
VLSI/PCB Testing Issues Mass produced Each piece needs to be validated Complexity
Number of subsystems Amount of logic
Access to internal logic Testing costs Cost of recall
Testability: “Effort required to test a product to
ensure that it performs its intended
function”
DFT Principles Controllability Observability
Test Point Insertion Built-In Self Test (BIST) Fault Modelling Fault Simulation Test Pattern Generation
Software Testing“Testability is a design issue and needs to
be addressed with the design of the rest of the system. Projects that defer testing to the later stages of a project will find their programmers unwilling to consider testability changes by the time testers actually roll onto the project and start making suggestions. Testability takes cooperation, appreciation and a team commitment to reliability.”
- Bret Pettichord
Cost Of Bugs
Where Found
Where Introduced
Requirements Architecture Coding System Test Production
Requirements 1x 3x 5-10x 10x 10x-100x
Architecture 1x 10x 15x 25-100x
Coding 1x 10x 10-25x
Software Testability Suitability: clarity of specifications QA has Observability: we can only test what is visible Simplicity: easier design/UI makes testing easier Controllability: better the control, better the coverage
and automation Stability: how often it changes Performance: how fast it works Diagnosability: writing effective bug reports!
Software DFT Patterns BIST
Automation suite w/ oracle
Unit Tests Assertions
Fault Simulation Mock/Stub Invalid params
Controllability Decoupling Bypassing Mock/Stub Set/Reset
Observability Logging Reporting Test Interfaces
Good Automated Tests are
Repeatable Easy to Write Easy to Understand Fast Way Easier with a Testable Software
When/How to Automate? Manager & Team are commited Testers/Dev either experienced or interested
in learning to script Product release cycles managed well Functionality / UI changes under control Early/Incremental approaches work best Build integration, reporting Each bug results in a automated test
Automation As Coding Automation scripts and frameworks are CODE! Use tools and scripting languages Evaluate tools on real problems Follow development processes
Spiral/Agile
Use Source Control same as rest of code These will have bugs as well!
Test “oracle” Expected Result
Generated once from previous run or Manually specified or Legacy system
Time adjustment Results
Database File Inline with code
Test Environment Multiple VMs Automated installs/images Automation tools Bug reporting tools
Unit Tests Each module has independent set Developer written/maintained setup, teardown xUnit, TestNG Runs on build Can run sub-set on install TDD
Diagnosability All environments: Test / Staging / Production
Errors result in visual messages Errors are raised where they occur Errors can be localized Details are sufficient to fix issue Ability to send diag data to dev
Logging Module Log levels (VERBOSE, INFO, WARNING, ERROR, CRITICAL, TIME) Define CRITICAL/ERROR levels well and use them! These should result in urgent notifications Lower levels in Production Time, App, Component, ThreadId, Message
8/2/2010 14:22pm: Notifier: Pop3: 8242: Sending welcome email to userid 334
Delimited columns for import Rotation based on Time, Size
Built-in Self Test Inserting test code/interfaces Set/Reset to bring state to known value Reporting to get current state Assertions about values/state
assert(order.billed==true)
“oracle” based regression
Some Examples Web Application
Software As A Service Browser Only UI Ajax / Dynamic HTML
Banking (Client-Server) Installed Application Desktop Client Server API
Web Application HTML Page Title Id values for important divs/controls Hidden values (non-textual, graphs, tables) Measurable Ajax responses Tools: Selenium, Sahi, Watir, WebTest, curl Logging incl. browser/ip/session cookie Ability to simulate time zone, language
Web Application - 2 Bypassing Captcha Mock TP APIs eg. Facebook, Google, OpenId SSL bypassing Transactions to be voided Multiple runs Always initialise controls
Browser autofill
Desktop Unique identifiers to GUI controls Key Replay tools, SendKeys API test suites bypassing UI Client logs, ability to email Ability to run db queries in scripts
Desktop - 2 Automation Friendly Third Party Controls
Ability to select cell Copy-enable text fields Key shortcuts to Forms UI for control focus,
clicking, navigation Access by value (tree/list controls)
OCR for images/text consoles
Bypassing Credit card payment
Test cards Mock object Dev auto-approval bypass code
Order placement Dummy users, auto-fulfill Dummy vendors (email order) Admin screens move
Dates Never use system date directly Config override
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<add key=“SystemDate" value=“$now"\>
</appSettings>
</configuration>
Nant: XMLPoke<xmlpoke file="App.config"
xpath="/configuration/appSettings/add[@key = 'SystemDate']/@value"
value="1-1-2009" />
Sending Emails Email subsystem should be a stub in dev Bypass SMTP: log to file … Email should be sent only to a whitelist Use Gmail Id & Atom API Use Unique Id/Timestamp to distinguish
Mocking / Stubbing Replace module during testing Different kinds:
Sub-system (Email, Print) Webservice (Credit Card, Flight Booking) Hardware device (Biometric, CNC Machine)
Return default values, implement simple/fixed logic
Mocks - 2
Simulate Errors and Exceptions Provide more logging Jmock, Dependency Injection Con:
Code level Needs to change with API
Anti-Testability Viewpoints Security compromise by testability interfaces,
logging
Remove or lock down in prod Extra coding time
Much lower testing costs, better quality Privacy issues in logging
Show only partial data Performance
Almost never an issue in practice!
Review BIST
Automation suite w/ oracle
Unit Tests Assertions
Fault Simulation Mock/Stub Invalid params
Controllability Decoupling Bypassing Mock/Stub Set/Reset
Observability Logging Reporting