Mobile App Testing: Design Automation Patterns You Should Use

6/2/15

1

MOBILE APP TESTING: DESIGN AUTOMATION PATTERNS YOU SHOULD USE (OR CONSIDER USING)

Jon Duncan Hagar, Grand Software Testing

Grand Software Testing (GST) Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”

Where is the App World Today?

Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”

6/2/15

2

What do you think of when we say mobile test automation?

3 Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”

Many Automation Tools

Definitions: Tool - any aid for doing your job (not just software) Automation - Hardware and software that helps get the job done Examples


6/2/15

3

•  Introduction

•  Problem

•  Automation you may not have thought about

•  Implications, Conclusions and Recommendations

•  Summary/Conclusions

•  References

Agenda


•  Software is now in most systems •  CPUs, microprocessors, FPGAs, etc. •  Provides features, flexibility and “smarter systems”

•  V&V/Test of software has been advocated for years, but •  Requirements verification-checking is necessary, but not sufficient o  Continued existence of fielded “issues” can be COSTLY

•  Numerous concepts, ideas, and tools exist o  Project context determines how to mix and match - Context includes: budget, schedule, skills, regulations, and domain

o  There is NO best practice, tool, or single technique

There is NO MAGIC

Problem: Mobile System-Software Quality Is Important

6 Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”

6/2/15

4

Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices” 7

Hard situation - We need to find bugs, but where? - We need good user experiences, but how?

Understanding an Issue Taxonomy


Taxonomy (researched) Super Category

Aero-‐Space Med sys Mobile General Time 3 2 3 Interrupted -‐ SaturaQon (over Qme)

5.5 Time Boundary – failure resulQng from incompaQble system Qme formats or values

0.5 1 Time -‐ Race CondiQons

3 1 Time -‐ Long run usages 4 1 20 Interrupt -‐ Qming or priority inversions

0.7 3 Date(s) wrong/cause problem

0.5 1 Clocks 4 2 ComputaQon -‐ Flow 6 23 19 ComputaQon -‐ on data 4 1 3 1

6/2/15

5


Taxonomy part 2 Super Category

Aero-‐Space Med sys Mobile General Data (wrong data loaded or used) 4 5.00 2 IniQalizaQon 6 2.00 3 5 Pointers 8 2.00 18 10 Logic and/or control law ordering

8 43 3 30 Loop control –Recursion

1 Decision point (if test structure) 0.5 1 1 Logically Impossible & dead code

0.7 OperaQng system – (Lack of Fault tolerance , interface to OS, other) 1.5 2 6 Software - Hardware interfaces

16 13 SoCware -‐ Software Interface

5 2.00 3 SoCware -‐ Bad command- problem on server 3 5 UI -‐ User/ operator interface

4 5.00 20 10 UI -‐ Bad Alarm 0.5 3 UI -‐ Training – system fault resulQng from improper training

3 Other 10.6 9.00 5 5

Note: one report on C/C++ indicated 70% of errors found involved pointers

LET’S CONSIDER POSSIBLE AUTOMATED TOOL OPTIONS

10 Copyright 2015 Jon D. Hagar – SoCware Test AFacks to Break Mobile and Embedded Devices

6/2/15

6

AUTOMATED MOBILE TESTING TO IMPROVED THE USER EXPERIENCE

Developer testing Exploratory Testing

with capture playback and regression automation Usability Checklist

11 • Copyright 2015 Jon D. Hagar – Software Test Attacks to Break Mobile and Embedded Devices

Unit – Developer Testing Attacks


Example Automation Concepts Static Code Analysis Tool Review/Inspection Tool Databases Online Modeling – classes and boundary Combinatorial tool Data generation tool Data dictionary database Code Coverage Tool

- Levels Stub-Drivers Metric analysis

6/2/15

7

Most developers love to Automate (it is their job) Agile and TDD expect it Continuous integration

What can testers do?


Mobile Capture Playback Tooling Supporting Exploratory Testing

What? Impossible?

Inconceivable?


6/2/15

8

•  Verification checking (tests) of requirements is most common

•  Additionally, successful teams practice concepts such as risk-based and exploratory attack-based testing

• ISO 29119 is a risk-based testing standard • Whittaker, Hagar, and others advocate • Attacks are design patterns which can include many test techniques • Allows rapid test exploration due to lack of highly scripted tests • Requires “skilled” test teams

•  Exploratory testing must be balanced with other V&V

Attack and Risk-based Software Test Planning with Exploration Concepts and Automation


Conduct exploration while running a capture-playback tool Plan exploration with a risk-based analysis tool Design exploratory test data with a combinatorial test tool

- Boundary value analysis - Equivalence classes

Emulators and simulators to support exploratory testing “Rack” or cloud hardware device testing

Mobile Exploration with Automation (Examples)


6/2/15

9

The Simple UI Checklist “Tool”


Database of what has been checked On-line Reviews Pop-up reminders on work flow

MATH -BASED TESTING WITH AUTOMATION Combinatorial and others


6/2/15

10

Underused Math-based (Tool) Concepts for System and Software Level Testing

19

Math-based Concepts for System and Software Level Testing

General Technique Concept Tool Examples Examples where techniques can be used

Specific sub- technique examples

Combinatorial Testing

ACT [4], Hexawise[5] Medical, Automotive, Aerospace, Information Tech, avionics, controls, User interfaces

Pairwise, orthogonal arrays, 3-way, and up to 6-way pairing are now available

rdExpert [6]

PICT[7]

Design of Experiments DOE ProXL[8] Hardware, systems, and software testing where there are "unknowns" needing to be evaluated

Taguchi [12]

(DOE) DOE++ [9] JMP [10] DOE

Random Testing Random number generator feature used from most systems or languages

Chip makers, manufacturing quality control in hardware selection

Testing with randomly generated numbers includes: fuzzing and use in model-based simulations

Statistical Sampling SAS [10] Most sciences, engineering experiments, hardware testing, and manufacturing

Numerous statistical methods are included with most statistical tools

Software Black box Domain Testing

Mostly used in manual test design, though some tools are now coming available [11]

All environments and types of software tests. These are “classic” test techniques, but still underused

Equivalence Class, Boundary Value Analysis, decision tables

Copyright 2015 Jon D. Hagar Mobile-‐Embedded Taxonomies from “SoCware Test AFacks to Break Mobile and Embedded Devices”

•  Not clear if engineers know about these math-based concepts or if they are restricted due to budget/schedule constraints

•  Tools and training are improving for many of these concepts

•  NIST – Book and ACTS tool

•  Kaner et. al. – Advanced book and training on domain testing

•  Management, tool and expert support for these concepts needs to continue

Underused Math-based Concepts


6/2/15

11

Detailed Example from Attack 32: Combinatorial Tests

21

When to apply this attack? •  There are numerous related variables and

variable values which interact, while there maybe larger numbers of combinations that other math based approaches can easily handle

What faults make this attack successful?

•  An organized and minimized selection approach but with “coverage” of pairings, e.g. 1600 samples reduced to 64

Who conducts this attack? •  Tester, analyst

Where is this attack conducted? •  Tool running in the lab or field

How to determine if the attack exposes failures?

•  A test fails to meet success criteria

How to conduct this attack • Identify combinatorial situation • Identify combinatorial tool • Identify variables • Identify values • Identify constraints on values • Enter variables and values into tool with constraints • Exercise resulting combinations in usage scenario tests or automated tests • Look for failure • Repeat and refine as needed Note: may not be this simple

Copyright 2015 Jon D. Hagar – So7ware Test A=acks to Break Mobile and Embedded Devices

SAE INTERNATIONAL

A Possible Cool Future

Combinatorial testing with high numbers of cases based in equivalent

classes or edge bases

and

no oracles needed to find bugs 22

Copyright 2015 Jon D. Hagar – Software Test Attacks to Break Mobile and Embedded Devices

6/2/15

12

MORE POSSIBILITIES

MODELING

WITH TOOLS TO SUPPORT MOBILE APPS

Copyright 2015 Jon D. Hagar – Software Test Attacks to Break Mobile and Embedded Devices 23

•  Interest and use of model-based testing is growing in industry segments •  Telecom, finance, automotive, aero, space •  European and U.S. interests •  UML and UML testing profile (UTP)

•  Model-based testing with tool automation can support (examples): •  Generation of test cases from models into test automated execution

engines directly using scripts or through the use of keywords •  Improved understanding of the system and risks •  Use of models to support simulations to drive test environments •  Verification via compares between development and test models •  Generation of test result oracles or judges •  Support of independent testing such as Independent V&V (IV&V) •  Model analysis

Model-based Testing in Mobile Apps


6/2/15

13

25

An Example Test Flow with Modeling


•  OMG UTP and ISO Standards currently in place

•  Tools to support second “test” analysis model •  Produce test automation •  Graphic views aid understanding •  Serve as an oracle

•  Aids in avoidance and/or identification of issues early in lifecycle

•  Considerations for growth and continuing usage •  N-version problem •  Self-checking problem if only one model is created •  Skilled modelers and testers needed •  Correct development/test environment must be place

Model-based Test Advantages and Considerations


6/2/15

14

FASTER DEV-OPS/AGILE FEEDBACK FROM TOOL AUTOMATION


THIS IS THE END MY FRIEND


6/2/15

15

Software Attacks with Automated Exploratory Test

29

Software Test Attack Type Attack Finds Tool-Automation Notes on the Attack

Developer level attacks Code and data structure problems Automate, Automate, Automate

Control system attacks Hardware and software control system errors Modeling Automation

Hardware-software attacks Hardware and software interface issues Automate in cloud or “rack”

Communication attacks Digital communications problems Emulator and simulators

Time attacks Time, performance, sequence, and scenario errors

Load and stress testing using captured scenarios

User interface attacks Problems between man and machine Checklist

Smart/Mobile attacks Issues specific to smart device configurations including cloud issues

Lifecycle and move to left may mean automation

Security test hacking attacks

Software errors that can expose devices to security threats

Fuzzing, Pen-attacks, and identity spoofing

Generic functional verification attacks

Requirements and interoperability errors Modeling, mind mapping, and combinatorial testing

Static code analysis attacks Hard to find errors that classic testing often misses

Testers run the static code analysis tools


•  When done correctly and continuously with automation V&V- testing will be valued

•  Activities will vary from project to project since there is no one “best” •  Combination of automation, V&V, and testing actions needed

• Developers, support, testers, engineers, and customers

•  Specifically underused test bases: • Exploratory testing supported by attacks • test automation (more than just execution of tests) • mathematical techniques • model-based

•  Testers and project staff should increase their knowledge and skill

Summary and Conclusions


6/2/15

16

Notes: Thank You (ideas used from)

•  James Whittaker (attacks) •  Elisabeth Hendrickson (simulations) •  Lee Copeland (techniques) •  Brian Merrick (testing) •  James Bach (exploratory and tours) •  Cem Kaner (test thinking) •  Jean Ann Harrison (her thinking and help)

•  Many teachers •  Generations past and future •  Books, references, and so on

Copyright 2015 Jon D. Hagar – SoCware Test AFacks to Break Mobile and Embedded Devices

Book Notes List (my favorites) “Software Test Attacks to Break Mobile and Embedded Devices”

– Jon Hagar “How to Break Software” James Whittaker, 2003

And his other “How To Break…” books “A Practitioner’s Guide to Software Test Design” Copeland, 2004 “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993 “Computer Related Risks”, Neumann, 1995 “Safeware: System Safety and Computers”, Leveson, 1995 Honorable mentions:

“Systems Testing with an Attitude” Petschenik 2005 “Software System Testing and Quality Assurance” Beizer, 1987 “Testing Computer Software” Kaner et. al., 1988 “Systematic Software Testing” Craig & Jaskiel, 2001 “Managing the Testing Process” Black, 2002


6/2/15

17

More Resources

•  www.stickyminds.com – Collection of test info •  www.embedded.com – info on attacks •  www.sqaforums.com - Mobile Devices, Mobile Apps -

Embedded Systems Testing forum •  Association of Software Testing

–  BBST Classes http://www.testingeducation.org/BBST/

•  Your favorite search engine

•  My web sites and blogs (listed on front page)


1.  IEEE 1012, Standard for System and Software Verification and Validation- http://standards.ieee.org/findstds/standard/1012-2012.html, IEEE press, 2012 2.  ISO 29119, Software Test Standard - http://www.softwaretestingstandard.org/ 3.  Hagar, J. Software Test Attacks to Break Mobile and Embedded Devices, CRC press, 2013 4.  Kuhn, Kacker, Lei, Introduction to Combinatorial Testing, CRC press, 2013 (includes the tool ACTS) 5.  Tool: Hexawise - app.hexawise.com/ 6.  Tool: rdExpert – www.phadkeassociates.com/ 7.  Tool: PICT – msdn.microsoft.com/en-us/library/cc150619.aspx 8.  Reagan, Kiemele, Tool: DOE Pro XL - Design for Six Sigma, Air Academy Associates, self publish, 2000 9.  DOE++ - www.reliasoft.com/ 10.  SAS - www.sas.com/ 11.  Kaner, Hoffman, Padmanabhan, The Domain Testing Workbook, self publish, 2013 12.  Bailey, Design of Comparative Experiments. Cambridge University Press, 2008 13.  Kacker, Kuhn, Hagar, Wissink, "Introducing Combinatorial Testing to a Large System-Software Organization,” scheduled-2015, IEEE Software 14.  Whittaker, James 2003, How to Break Software, Pearson Addison Wesley 15.  Whittaker, James and Thompson, Herbert, How to Break Software Security, Pearson Addison Wesley, 2004 16.  Andrews, Whittaker, How to Break Web Software, Pearson Addison Wesley, 2006 17.  Levy, Tools of Critical Thinking: Metathoughts for Psychology, 1996 18.  Bach, Bolton, “Testing vs. Checking,” www.developsense.com/blog/2009/08/testing-vs-checking/ 19.  Hagar, “Why didn’t testing find the embedded GM Truck fire system error?”- www.breakingembeddedsoftware.wordpress.com/ 20.  OMG UTP 1.2, www.omg.org/spec/UTP/1.2/ 21.  Baker, Dai, Grabowski, Schieferdecker, Williams, “Model-Driven Testing:Using the UML Testing Profile,” 2008 22.  Green, Hagar, “Testing Critical Software: Practical Experiences,” IFAC Conference 1995 23.  Boden, Hagar, “How to Build a 20-Year Successful Independent Verification and Validation (IV&V) Program for the Next Millennium,” Quality Week Conference 1999 24.  Port, Nakao, Katahira, Motes, Challenges of COTS IV & V, Springer press, 2005

34

References

Copyright 2015 Jon D. Hagar “SoCware Test AFacks to Break Mobile and Embedded Devices”

Mobile App Testing: Design Automation Patterns You Should Use

Software

software automation

mobile systemsoftware

socware software interface

software hardware interfaces

mobile test automation

mobile app testing

automation tools definitions

design automation patterns