Matthew Sul l ivan Scot t Barber Sof tware Test Profess ionals Conference Fal l 2011
MANAGING RISK FOR SOFTWARE PRODUCTS
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
“STATE OF THE S/W TESTING PRACTICE”
• Find bugs (identify risks) OR • Check for compliance (V&V) “Role” of
QA/Testing
• Appears undervalued, BUT •Doesn’t provide nearly the value it
could “Value” of
QA/Testing
• Business goals & value propositions • Business risks & risk controls • Executive information needs
QA/Testing is “out of sync” with
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
“THE UNDER-INFORMED DIRECTING THE UNDER-TRAINED TO DO THE UNIMPORTANT”
Artifacts (the Unimportant)
•Bugs no one wants to fix •Metrics no one
understands •Documents no one
reads
Testers (the Untrained)
•Don’t know what the executives need, SO
•They do what they are asked to
Executives (the Uninformed):
•Don’t know how to ask for what they need, SO
•They ask for what they know
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
IMPROVING THE SITUATION (PART 1)
Focus on: •Delivering business value •Reducing business risk
At every business layer, identify & balance: •Responsibility •Accountability
Get your superiors to read Ch 16:Rightsizing the Cost of Testing: Tips for Executives of How to Reduce the Cost of Software Testing; CRC Press 2011
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
IMPROVING THE SITUATION (PART 2)
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Businesses reduce allocation of resources to testing because of a perception of diminished value.
FEELING UNDER SIEGE?
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
WHAT DIMINISHES VALUE FOR TESTING?
1. Lack of insight into future
2. Redundancy 3. Specification
blocks 4. Lack of
independence 5. Scope
constraint Copyright © 2011 PerfTestPlus, Inc. All rights
reserved.
LACK OF INSIGHT INTO THE FUTURE
Why didn’t this come up in
testing!
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
REDUNDANCY
Sign here, and then sign the next box attesting to
the authenticity of the previous signature.
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
SPECIFICATION BLOCK
Honestly I’d love to start testing today, but first I need detailed requirements. VERY
detailed requirements
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
LACK OF INDEPENDENCE
Its not fun being the captain’s “no-
man”.
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
SCOPE CONSTRAINT
Someone else was supposed to be watching
for icebergs.
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
REQUIREMENT-DRIVEN APPROACH
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
The purpose of testing is to reduce uncertainty about the future impact of technology.
THE MEANING OF LIFE (FOR TESTERS)
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
ALTERNATIVE APPROACH
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
RISK AS A COMMON LANGUAGE
Risk
Security
Functional
Performance Usability
Compliance
Whether explicitly or implicitly, all forms of testing revolve around the reduction and management of risk.
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
To effectively manage risk, you must effectively manage knowledge.
THE SECRET TO MANAGING RISK
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Control Model Testing is a business-aligned approach to software testing that derives “test cases” from knowledge models of the system based on a risk-based taxonomy .
WHAT IS CONTROL MODEL TESTING?
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
WHAT IS OUR TAXONOMY BASED UPON?
COSO Enterprise Risk
Management Integrated Framework
The Open Group Technical Standard on
Risk Taxonomy
PerfTest Plus Taxonomy Extensions for
Control Model Testing
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
WHAT ARE THE BASIC ENTITIES?
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
THE OPEN GROUP’S RISK ASSESSMENT FRAMEWORK
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Business •Financial •Legal •Brand or Reputation
Product •Security •Performance •Usability •Other Qualities
Project •Budget •Schedule •Communication
RISK LAYERS
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
UNADDRESSED RISK
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Controls prevent or mitigate risk which may impact business objectives. Control Model Testing helps identify and assess these controls.
HOW CAN TESTS ADDRESS THREATS AND LEVEL OF RISK?
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Systems • Firewalls • Encryption • Load Balancing
Preferences • Settings • Security and Access Model
Policies • Code Standards • Monitor and Response • HR
TYPES OF CONTROLS
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Development • Development and Test Tools • Code standards • Software components
Implementation • Checklists • Installation scripts
Maintenance • Alerts and Triggers • SOPs • Configuration Management
CONTROLS CONTEXT
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
“SAMSARIC” TEST LIFECYCLE
Knowledge
Effort
Analyze
Assess
Evaluate
Report
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Examine • System • Users • Environment
Identify • Objectives • Processes • Threats • Controls
Output • Initial Control Model
ANALYSIS
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
INITIAL CONTROL MODEL
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Activities •Identify authorities •Solicit opinions •Evaluate exposure •Determine impact
Outcomes •Risk assessment •Assessed Control Model •Test plan
ASSESSMENT
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
ASSESSED CONTROL MODEL
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Activities •Execute planned and
derivative tests • Identify discrepancies •Determine capability
Outcomes •Tested Control Model •Test results • Issues /
recommendations
EVALUATION
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
EXECUTED CONTROL MODEL
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Activities •Communicate •Recommend •Respond
Outcomes •Implementation plan •Knowledgebase update •Confirmation of or
revisions to test plan
REPORTING
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Leader
Manager
Coordinator
Tester
THE FOUR ROLES IN CONTROL MODEL TESTING
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Responsibilities: •Representation •Roadmaps
Interests • Information • Certainty
Talents • Communication • Vision
Typical Business Titles •Director of Testing or Quality Assurance • Chief Audit Officer (or Assistant to..) • Principle Consultant
LEADER
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Responsibilities: •Organizing •Developing
Interests •Capability •Consistency
Talents •Understanding •Motivating
Typical Business Titles •Test Manager
MANAGER
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Responsibilities • Planning •Oversight
Interests • Successful outcome • Thoroughness
Talents • Teamwork • Attention
Typical Business Titles • Test or QA Lead or Senior • Analyst or Engineer Level 2 or 3 •Manager 1
COORDINATOR
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Responsibilities •Execution •Analysis
Interests •Discovery •Experimentation
Talents •Curiosity •Skepticism
Typical Business Titles •Test or QA Analyst or Engineer •Analyst or Engineer Level 1 or 2
TESTER
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Business
Test Leader
Product
Test Manager
Project
Test Coordinator Tester
RISK LAYERS AND ROLES
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
Testing should be an indispensible advisor for leadership Testing should not be a convenience or scapegoat for
development All types of testing revolve around risk management The key to managing risk is managing knowledge Testing needs to be a learning discipline in the context of risk
taxonomy The test process should be a continuous cycle reducing effort
through increased knowledge Testing roles should correlate to management or risk, not
resources
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
SUMMARY
[email protected] [email protected]
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
QUESTIONS?
The Open Group (http://www3.opengroup.org/): Risk Taxonomy Technical Standard - https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?publicationid=12156
The Committee of Sponsoring Organizations of the Treadway Commission, or COSO (http://www.coso.org/)
Enterprise Risk Management-Integrated Framework - http://www.coso.org/ERM-IntegratedFramework.htm
PerfTestPlus, Inc. (http://www.perftestplus.com/) Control-Model Testing – (http://www.perftestplus.com/control-model-testing) Rightsizing the Cost of Testing: Tips for Executives of How to Reduce the Cost of Software Testing; CRC Press 2011
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.
RECOURCES
Matthew Sullivan Quality Control Engineer CCH TeamMate Wolters Kluwer
Test and Support Engineer for PricewaterhouseCoopers for 10 years
Extensive experience in audit and risk management industry
Specialist in testing Microsoft .NET, MS SQL Server, and Lotus Notes applications
MS in Sof tware Engineering from Regis University
Scott Barber CTO, PerfTestPlus, Inc Widely regarded exper tise in
per formance. Contributor to:
Performance Testing Guidance for Web Applications– Microsoft Press
Beautiful Testing - O’Reilly Press How to Reduce the Cost of Testing -
Taylor and Francis
Executive Director of the Association for Sof tware Testing
Co-Founder of the Workshop of Per formance and Reliabil ity
ABOUT US
Copyright © 2011 PerfTestPlus, Inc. All rights reserved.