Control model testing

Matthew Sul l ivan Scot t Barber Sof tware Test Profess ionals Conference Fal l 2011

MANAGING RISK FOR SOFTWARE PRODUCTS

“STATE OF THE S/W TESTING PRACTICE”

• Find bugs (identify risks) OR • Check for compliance (V&V) “Role” of

QA/Testing

• Appears undervalued, BUT •Doesn’t provide nearly the value it

could “Value” of

QA/Testing

• Business goals & value propositions • Business risks & risk controls • Executive information needs

QA/Testing is “out of sync” with

“THE UNDER-INFORMED DIRECTING THE UNDER-TRAINED TO DO THE UNIMPORTANT”

Artifacts (the Unimportant)

•Bugs no one wants to fix •Metrics no one

understands •Documents no one

Testers (the Untrained)

•Don’t know what the executives need, SO

•They do what they are asked to

Executives (the Uninformed):

•Don’t know how to ask for what they need, SO

•They ask for what they know

IMPROVING THE SITUATION (PART 1)

Focus on: •Delivering business value •Reducing business risk

At every business layer, identify & balance: •Responsibility •Accountability

Get your superiors to read Ch 16:Rightsizing the Cost of Testing: Tips for Executives of How to Reduce the Cost of Software Testing; CRC Press 2011

IMPROVING THE SITUATION (PART 2)

Businesses reduce allocation of resources to testing because of a perception of diminished value.

FEELING UNDER SIEGE?

WHAT DIMINISHES VALUE FOR TESTING?

1. Lack of insight into future

2. Redundancy 3. Specification

blocks 4. Lack of

independence 5. Scope

reserved.

LACK OF INSIGHT INTO THE FUTURE

Why didn’t this come up in

testing!

REDUNDANCY

Sign here, and then sign the next box attesting to

the authenticity of the previous signature.

SPECIFICATION BLOCK

Honestly I’d love to start testing today, but first I need detailed requirements. VERY

detailed requirements

LACK OF INDEPENDENCE

Its not fun being the captain’s “no-

man”.

SCOPE CONSTRAINT

Someone else was supposed to be watching

for icebergs.

REQUIREMENT-DRIVEN APPROACH

The purpose of testing is to reduce uncertainty about the future impact of technology.

THE MEANING OF LIFE (FOR TESTERS)

ALTERNATIVE APPROACH

RISK AS A COMMON LANGUAGE

Security

Functional

Performance Usability

Compliance

Whether explicitly or implicitly, all forms of testing revolve around the reduction and management of risk.

To effectively manage risk, you must effectively manage knowledge.

THE SECRET TO MANAGING RISK

Control Model Testing is a business-aligned approach to software testing that derives “test cases” from knowledge models of the system based on a risk-based taxonomy .

WHAT IS CONTROL MODEL TESTING?

WHAT IS OUR TAXONOMY BASED UPON?

COSO Enterprise Risk

Management Integrated Framework

The Open Group Technical Standard on

Risk Taxonomy

PerfTest Plus Taxonomy Extensions for

Control Model Testing

WHAT ARE THE BASIC ENTITIES?

THE OPEN GROUP’S RISK ASSESSMENT FRAMEWORK

Business •Financial •Legal •Brand or Reputation

Product •Security •Performance •Usability •Other Qualities

Project •Budget •Schedule •Communication

RISK LAYERS

UNADDRESSED RISK

Controls prevent or mitigate risk which may impact business objectives. Control Model Testing helps identify and assess these controls.

HOW CAN TESTS ADDRESS THREATS AND LEVEL OF RISK?

Systems • Firewalls • Encryption • Load Balancing

Preferences • Settings • Security and Access Model

Policies • Code Standards • Monitor and Response • HR

TYPES OF CONTROLS

Development • Development and Test Tools • Code standards • Software components

Implementation • Checklists • Installation scripts

Maintenance • Alerts and Triggers • SOPs • Configuration Management

CONTROLS CONTEXT

“SAMSARIC” TEST LIFECYCLE

Knowledge

Effort

Analyze

Assess

Evaluate

Report

Examine • System • Users • Environment

Identify • Objectives • Processes • Threats • Controls

Output • Initial Control Model

ANALYSIS

INITIAL CONTROL MODEL

Activities •Identify authorities •Solicit opinions •Evaluate exposure •Determine impact

Outcomes •Risk assessment •Assessed Control Model •Test plan

ASSESSMENT

ASSESSED CONTROL MODEL

Activities •Execute planned and

derivative tests • Identify discrepancies •Determine capability

Outcomes •Tested Control Model •Test results • Issues /

recommendations

EVALUATION

EXECUTED CONTROL MODEL

Activities •Communicate •Recommend •Respond

Outcomes •Implementation plan •Knowledgebase update •Confirmation of or

revisions to test plan

REPORTING

Leader

Manager

Coordinator

Tester

THE FOUR ROLES IN CONTROL MODEL TESTING

Responsibilities: •Representation •Roadmaps

Interests • Information • Certainty

Talents • Communication • Vision

Typical Business Titles •Director of Testing or Quality Assurance • Chief Audit Officer (or Assistant to..) • Principle Consultant

LEADER

Responsibilities: •Organizing •Developing

Interests •Capability •Consistency

Talents •Understanding •Motivating

Typical Business Titles •Test Manager

MANAGER

Responsibilities • Planning •Oversight

Interests • Successful outcome • Thoroughness

Talents • Teamwork • Attention

Typical Business Titles • Test or QA Lead or Senior • Analyst or Engineer Level 2 or 3 •Manager 1

COORDINATOR

Responsibilities •Execution •Analysis

Interests •Discovery •Experimentation

Talents •Curiosity •Skepticism

Typical Business Titles •Test or QA Analyst or Engineer •Analyst or Engineer Level 1 or 2

TESTER

Business

Test Leader

Product

Test Manager

Project

Test Coordinator Tester

RISK LAYERS AND ROLES

Testing should be an indispensible advisor for leadership Testing should not be a convenience or scapegoat for

development All types of testing revolve around risk management The key to managing risk is managing knowledge Testing needs to be a learning discipline in the context of risk

taxonomy The test process should be a continuous cycle reducing effort

through increased knowledge Testing roles should correlate to management or risk, not

resources

SUMMARY

matthewgsullivan@hotmail.com sbarber@perftestplus.com

QUESTIONS?

The Open Group (http://www3.opengroup.org/): Risk Taxonomy Technical Standard - https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?publicationid=12156

The Committee of Sponsoring Organizations of the Treadway Commission, or COSO (http://www.coso.org/)

Enterprise Risk Management-Integrated Framework - http://www.coso.org/ERM-IntegratedFramework.htm

PerfTestPlus, Inc. (http://www.perftestplus.com/) Control-Model Testing – (http://www.perftestplus.com/control-model-testing) Rightsizing the Cost of Testing: Tips for Executives of How to Reduce the Cost of Software Testing; CRC Press 2011

RECOURCES

Matthew Sullivan Quality Control Engineer CCH TeamMate Wolters Kluwer

Test and Support Engineer for PricewaterhouseCoopers for 10 years

Extensive experience in audit and risk management industry

Specialist in testing Microsoft .NET, MS SQL Server, and Lotus Notes applications

MS in Sof tware Engineering from Regis University

Scott Barber CTO, PerfTestPlus, Inc Widely regarded exper tise in

per formance. Contributor to:

Performance Testing Guidance for Web Applications– Microsoft Press

Beautiful Testing - O’Reilly Press How to Reduce the Cost of Testing -

Taylor and Francis

Executive Director of the Association for Sof tware Testing

Co-Founder of the Workshop of Per formance and Reliabil ity

ABOUT US

Control model testing

managing risk

rights reserved

rights reserved

rights reserved

rights reserved

rights reserved

rights reserved

rights reserved

Technology

Modification and Testing System Control and Swing Model...

Challenges in testing Model...

MODEL BASED TESTING DEMYSTIFIED...MODEL‐BASED TESTING...

Improving Model-Based Design for Automotive Control ... ›....

TESTING & QUALITY CONTROL TESTING METHODOLOGY … · ·.....

Chapter 0: Model Building, Model Testing and Model...

Quality Control Testing

Attitude Control System testing SPACE HAUC Control...

Model-based Testing: Next Generation Functional Software...

SOFTWARE TESTING METHODOLOGIES - DNRCET · 2020. 4. 24. ·...

Testing Hollnagel’s Contextual Control Model: … · A...

Testing & Quality Control

Chapter 11, Testing: Model-based Testing

Ship Model Testing

Model Based Testing of Automotive Control Functions ·...

Groundwater Model Testing