Testing Types and Testing Paradigms

© 2015 Carnegie Mellon University

Testing Types and Testing ParadigmsNext Generation Testing ConferenceLondon, England

Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

Donald G. Firesmith, Principle Engineer24 June 2015 (updated 1 July 2015 )

Presenter

Presentation Notes

2Testing Paradigms and Testing TypesDonald G. Firesmith, 1 JUly 2015

Copyright 2015 Carnegie Mellon University

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This material has been approved for public release and unlimited distribution except as restricted below.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].

DM-0001886.


TopicsGoals of PresentationWhat is Testing?Presentation ScopeRelevant Testing ChallengesTesting TypesTesting ParadigmsConclusion


Goals of the Presentation

Make it clear that:• There are many different types of testing.• Testing is a complex discipline with its own technical jargon.• There is a lot of overlap between different classes of testing types.

– Think multiple classification (object-oriented design)or multiple inheritance (object-oriented programming).

Become aware that there multiple testing paradigms.Get you to take a look at your:• Test strategies and ask yourselves “Are they sufficiently complete?”• Testers and other testing stakeholders and ask yourselves “Do they need

additional training in testing types and paradigms.


What is Testing?

TestingThe execution of an Object Under Test (OUT) under specific preconditionswith specific stimuli so that its actual behavior can be compared with its expected or required behavior– Preconditions: pretest mode, states, stored data, or external conditions– Stimuli:

• Calls, commands, and messages (control flows)• Data inputs (data flows)• Trigger events such as state changes and temporal events

– Actual Behavior:• During Test:

– Calls, commands, and messages (control flows)– Data outputs (data flows)

• Postconditions: post-test mode, states, stored data, or external conditions

Presenter

Presentation Notes

An OUT can be: A test tool or environment An executable requirements, architecture, or design model A system or subsystem (containing HW, SW, data, personnel, facilities, etc.) A software application, component, or unit Testing requires: Controllability to set up preconditions and provide stimuli Observability to verify preconditions, outputs, and postconditions


PresentationScope

by Object Under Test (OUT)

by Reason

by Level of Automation

by Role

by Quality Characteristic

by Phase

by Built-In-Test (BIT)

by Technique

by Level of Scripting

by Organizational Distribution

by Collaboration

Test and Evaluation (T&E)Quality Control (QC)

Verification and Validation (V&V)Methods

Testing Evaluation

Walk-Through

Inspection

Analysis (Static + Dynamic)

Review

Warantee

Reuse

Certification

Interview

Observation

Demonstration

← In Scope ← Out Of Scope

How

How

How

How Well

What

When

When

Why

Who

Who

Where

by Physical DistributionWhere

Presenter

Presentation Notes

The traditional verification techniques are analysis, demonstration, inspection, and test. The term static testing (e.g., static analysis, reviews) is a misnomer caused by the realization within the testing community that testing by itself is not adequate.


Relevant Testing Challenges

Types of Testing:• Many testers are only aware of a minority of types of testing, let alone know

how to perform them.• Test managers and developers are aware of even fewer testing types.• The test strategies, project test plans, and test sections of system/software

development plans tend to identify a very small number of types of testing (e.g., unit, integration, system, and acceptance testing).– Not planned → Not performed

Test Paradigms (overall approaches):• Many testing stakeholders are unaware that there are general testing

paradigms.• Many project and testing managers are not aware of the incompatibilities

between these different paradigms.• They may verbally back a transition to a new testing paradigm while

mandating past practices that are incompatible with the new paradigm.– Paradigm Shift– Paradigm Paralysis


Types of Testing – Overview

A type of testing is:• A specific way to perform testing• A class or subclass of testing• Much narrower in scope than a testing philosophy

There are relationships between the various types of testing.Most testers know:• A lot about a few types of testing• A little about some additional types of testing• Very little about a sizable number of testing types


Types of Testing – Answering 5W+H Questions

How:• by Technique vs. by Level of Automation vs. by Level of Scripting

How Well:• by Quality Characteristic

What:• by Object Under Test (OUT)

When:• by Phase vs. Built-In-Test (BIT)

Why:• by Reason

Who:• by Role vs. by Collaboration

Where:• By Physical Location vs. by Organizational Location


How: by Technique

Blackbox Testing

Whitebox Testing

Experience-Based Testing

Graybox Testing

Cause and Effect Testing

Classification Tree Testing

Combinatorial Testing

Decision Table Testing

Fuzz Testing

Requirements Testing

Risk-Based Testing

Scenario Testing Smoke Testing

State Transition

TestingSyntax Testing

Boundary Value Testing

Equivalence Class Testing

Control Flow Testing

Data Flow Testing

Branch Testing

Condition Testing

Statement Testing

All Definitions Testing

All Uses Testing

Error Guessing Testing

Exploratory Testing

Procedure Testing

Bug Hunt Testing

User Interface Navigation

Testing

Random Testing

Testingby Technique

Galumphing

Patterns-Based Testing

Monkey Testing

Fuzz Testing

Cat-on-the-Keyboard

Testing


How: by Level of Automation

Testingby Level of Automation

Manual Testing Automated Testing- Test Script Generation- Test Data Generation- Test Script Execution- Test Report Generation

Data-Driven Testing

Action-Keyword Testing

Script-Based Testing

Record-Playback Testing

Model-Based Testing


How: by Level of Scripting

Testingby Level of Scripting

Unscripted TestingScripted Testing


How Well:by QualityCharacteristicBased on the associatedquality characteristic and its associated quality attributes:• Uncover related defects• Determine level of quality

A/B Testing

Capacity Testing

Compatibility Testing

Consistency Testing

Data Migration Testing

Functionality Testing

Performance Testing

Configuration Testing

Reliability Testing

Robustness Testing

Safety Testing

Security Testing

Usability Testing

Load Testing

Stress Testing

Volume Testing

Endurance (Stability) Testing

Reliability Mechanism Testing

Encryption Testing

Testingby Quality Characteristic

EMSEC Testing

Accessibility Testing

Alpha Testing

Content Usage Testing

Correctness Testing

Interoperability Testing

Blue Team Testing

Red Team Testing

Access Control Testing

Infrastructure Testing

Penetration Testing

Scalability Testing

Backwards Compatibility Testing

Infrastructure Compatibility Testing

Anti-Tamper Testing

Anti-Spoofing Testing

Flexibility Testing Internationalization Testing

PersonalizationTesting

Beta Testing (Closed/Open)

(Flash) Mob Testing


What: by Object Under Test (OUT) Testing

by Object Under Test (OUT)

Model Testing

Requirements Model Testing

Design Model Testing

Architecture Model Testing

Software Testing

System Testing

System Integration Testing

SoS Integration Testing

SoS Testing

Subsystem Testing

System Testing

Hardware Testing

HW Qualification Testing

Software-in-the-Loop (SIL) Testing

Hardware-in-the-Loop (HIL) Testing

Processor-in-the-Loop (PIL) Testing

Tool/Environment Testing

SW Unit Component Testing

SW Application Testing

SW Integration Testing

Human-in-the-Loop (HIL) Testing

Data CenterTesting


Network Traffic Testing

Failover and Restore Testing

Test Environment Testing

Test Tool Testing


System Lifecycle Phases and Environments

DT (unit)

Development Environment

Build Environment

Test Environment

Test Environment

Staging Environment

Production Environment(s)

Low Fidelity High FidelityHigh Fidelity

DT OTAT + OTDT

•••

Continuous Testing

Continuous Integration

← Shift Left Testing

Continuous Delivery / Fielding

← Less Fidelity – More Fidelity →

Development OperationAcceptance


When: by Phase

Beware ofSynonyms!

Testingby Phase

Developmental Testing (DT)

Acceptance Testing (AT)

Business Acceptance Testing (BAT)

Contract(ual) Acceptance Testing (CAT)

Operational Acceptance Testing (OAT)

Regulation Acceptance Testing (RAT)

Customer Acceptance Testing (CAT)

Site Acceptance Testing (SAT)

Factory Acceptance Testing (FAT)

Production Acceptance Testing (PAT)

Operational Testing (OT)

Initial Operational Testing (IOT)

Follow-on Operational Testing (FOT)

Qualification Operational Testing

(QOT)

Many Types of Developmental

TestingOperational

Effectiveness Testing

Operational Suitability Testing

Data CenterTesting


Failover and Restore Testing

Network Traffic Testing

User Acceptance Testing (UAT)


When: by Built-In Testing (BIT)

Testing

by Built-In Testing (BIT) Execution

PeriodicBuilt-In Testing

(PBIT)

Power-UpBuilt-In Testing

(PupBIT)

Interrupt-driven Built-In Testing

(IBIT)

User-initiated Built-In Testing

(UBIT)

ShutdownBuilt-In Testing

(SBIT)


Why: by Reason

Smoke Testing: Is it good enough to test?Initial Testing: Does it have defects?Retesting: Have defects been fixed?Regression Testing: Have fixes introduced new defects?Error-Seeding: How many defects remain?

Testingby Reason

Initial Testing Regression TestingRetesting

Partial Regression Testing Complete Regression Testing

Smoke Testing Error Seeding

Presenter

Presentation Notes

Smoke Testing : Is it good enough to test? Initial Testing: Does it have defects? Retesting: Have defects been fixed? Regression Testing: Have fixes introduced new defects?


Who: by RoleTesting

by Role

Security Testing

Safety Testing

Developer

performs

Safety Engineer

performs

Usability Testing

Human Factors Engineer

performs

User Testing

Tester Testing

User

performs

Tester

performs

Security Engineer

Code Testing

Programmer

performs

Architecture Model Testing

Architect

performs

Requirements Model Testing

Requirements Engineer

performs

Developer Testing

Design Model Testing

performs


Who: by Collaboration

Testing

by Collaboration

Individual Testing

Pair TestingBuddy Testing

Group Testing


Where: by Organizational Location

Outsourced Testing

Insourced Testing

Testing

by Organizational Location


Where: by Physical Location

Testing

by Physical Location

Distributed Testing

Cloud Testing

Centralized Testing


Testing Paradigms – Overview

A Testing Philosophy is:• One of a small number of very general, fundamentally different ways of

approaching testing• Based on a cohesive set of general principles • Much broader in scope than individual types of testing

Most testers fall into one – or at most two – of these camps and many know little about the others.Trends in development (e.g., Agile, DevOps) often imply corresponding trends in test paradigms.


Spectrum of Testing Paradigms

Testing Philosophies

Ad HocTesting

Best-Practices-Based Testing

AgileTesting

Process-Driven Testing

Standards-Driven Testing

Model-Based Testing (MBT)

Context-Driven Testing

DevOps Testing

Risk-BasedTesting

More Formal (Heavy-Weight) → ← More Informal (Light-Weight)

Test paradigms lie along a spectrum from very informal (light-weight) to quite formal (heavy-weight).Test paradigms overlap.Location along spectrum is controversial.Testing paradigms can be like:• Paradigms• Armed camps• Religious sects


Conclusion

Testing Types:• Most testers are not aware of all of the different types of testing.• If you are not aware that it exists, then you don’t know whether you need it.• Most systems require multiple types of testing.

Testing Paradigms:• There are 8 major testing paradigms.• They divide into two major categories:

– More complete, rigorous, and standardized– More incomplete, informal, and flexible

• Many testers follow more than one related paradigms.• Some types of testing are more popular in some paradigms than others.


Contact Information Slide Format

Donald G. FiresmithPrincipal EngineerSoftware Solutions DivisionTelephone: +1 412-268-6874Email: [email protected]

U.S. MailSoftware Engineering InstituteCustomer Relations4500 Fifth AvenuePittsburgh, PA 15213-2612USA

Webwww.sei.cmu.eduwww.sei.cmu.edu/contact.cfm

Customer RelationsEmail: [email protected]: +1 412-268-5800SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257

Testing Types and Testing Paradigms

Documents