Geoff Thompson - Why Do We Bother With Test Strategies

Listen | Challenge | Understand | Interpret | Create

Private & Confidential Experimentus Ltd 85 Tottenham Court Road London W1T 4TQ T: +44 (0)870 770 6099 www.experimentus.com

Why do we bother with Test Strategies?Why do we bother with Test Strategies?

Presented by:Geoff Thompson

EuroSTAR 200813th November

Agenda

Drivers for change?

What had to change?

What we did first

The problem

How we made it work

© Experimentus Ltd 2008

2

Drivers for change


3

4


The Business was demanding a higher

standard to remain competitive

The Business wanted clearer

identification of Risk to quality and Risk

Mitigation

The IT department was under pressure

to do more for less

There was a continual drive to

increase in productivity

- Cost Reduction

- Risk Management

- Improving Quality

- Revenue Enhancement

5© Experimentus Ltd 2008

So they were looking for change to avoid the

wrong solution going live late!

They wanted software that just worked!


What had to change?


How did we find out?

We reviewed many assessment methods

We looked at TPI ® and TMM

Selected TMM because:

– It was a research project developed by Illinois Inst. of Technology

– It describes the processes, which ensure a well planned and controlled test project

– Has very clear 5 levels of test maturity

– Is based on the evolutionary testing model

– Has been empirically validated

– Inspired by the CMM (complementary)


TMM


The 5 levels of TMMi

10

How does it work?

Maturitylevel

Maturitylevel

Indicates Testing Processcapability

Indicates Testing Processcapability

Contains Key process

areas

Contains Key process

areas

Achieves (Sub) GoalsAchieves (Sub) Goals Contains Key practices

Contains Key practices

Describe ImplementationDescribe Implementation


For more information see

www.tmmifoundation.org


The recommended improvements

Needed a Test Policy

Needed a Testing Strategy/Framework

Needed a test process definition

Needed a defined Risk Based Approach to Testing

Needed a career path for testers

Needed to complete the roll out of TestDirector

Needed to implement reviews and static analysis

12

So what happened next?


What we did first


TESTING POLICY

TESTING DEFINITION Risk Based Testing is the process by which we explore and understand the status of the business benefits and the business risks associated with a release of software or of new or upgraded hardware, by :-

1. Making a prioritised list of risks,

2. Performing tests that explore each risk,

3. As risks evaporate and new ones emerge, adjust the test effort to stay focused on the current crop.

TESTING SYSTEM Development and execution of a test plan in accordance with The Legal & General Test Strategy :-

i. Testing will be organised in line with the Testing V model,

ii. The different levels (phases) of testing will be clearly defined and understood,

iii. Testing will be an integral part of the Project Lifecycle,

iv. Formal test techniques will be used at all levels (phases) of testing,

v. Appropriate training in testing will be provided,

vi. Defined roles and responsibilities and Career paths will be available,

vii. Appropriate Entry and Exit criteria will be defined, communicated and enforced for all test levels (phases),

viii. The Test environment requirements,

ix. Test tools will be used where appropriate,

x. All testware will be maintained under Configuration Management, to enable audit and reuse,

xi. Metrics and measures will be recorded and used to enable test improvement and better estimating for test activity,

xi As well as the process for Incident Management , the definition of fault priority and severity rating will be defined.

MEASUREMENT OF TESTING EFFECTIVENESS Test Effectiveness will be measured by comparing the volume of Priority 1, 2 and 3 faults found in the first 6 months of live running versus the volume of priority 1, 2 and 3 faults found during testing - DEFAULT DETECTION RATE (DDR).

MEASUREMENT OF TEST EFFICIENCY Test Efficiency will be measured by determining the total cost of testing (both Business and BIS time – both planning and execution but excluding fix costs) as a ratio of the total design and build costs (including Project Management) – TEST EFFICIENCY RATE (TER).

TESTING STANDARDS DDR of less than XXXX TER of XXXX or less

TEST PROCESS IMPROVEMENT Quality reviews will take place at key stages in the ‘test projects’ lifecycle.

Specific testing focused Post Project reviews will be held after every release of software or of new or upgraded hardware. ------------------------------------------- ---- -------------------------------------------

We agree to the general principals outlined above, and changes or additions made will be subject to change control.

Manager )________________

Dated 1 st


14

Test Tools

Reviews

Generic Test Process

Measures and Metrics

HR

Implementation and

Communications

Risk Based Testing

Corporate Test Strategy


15



Test Tools HR Reviews

Implementation steps




Pros- Quicker delivery- Early buy-in- Earlier benefit realisation

Cons- Rework- Harder to manage Strategic vision- Delivery costs are larger- Loss of confidence due to rework- Support costs are higher

Delivery choices - Vertical



16



Test Tools HR Reviews




Cons- Slower delivery- Needs commitment and trust

Pros- Minimal rework- Clear strategic view- Clear understanding of impacts between deliveries- Greater acceptance as full picture understood- Quick wins giving early buy-in

Delivery choices - Horizontal


17

The problem


Up to this point we had tremendous support

Particularly from the Project and Programme Management Community

BUT

The process and templates just didn't get used to provide the greatest benefit so were in danger of being bypassed by everyone

WHY?


There were two problems in the

implementation of the process improvements:

Testers

Project and Programme Managers


They couldn't plan!

Common issues

– Requirements

– Estimating

– Activity plans

– Resource requests

– Budgeting

Planning was traditionally successful but not effective, when undertaken two weeks before test execution was due to start, when everything was known!


Lessons learnt

Test resources were traditionally put in place long after the project has started, and often just before test execution starts when...

– All requirements are defined and almost built

– Budget has been given rather than asked for

– Resources are set

– Timescales are set, and probably squeezed

– Basically everything is in place to enable accurate plans to be drawn up!

Development Managers provided mentoring to Test Managers using their experience of planning when all that was available was a high level requirement

Test Managers were taught

– High level estimating

– Early Project Planning22© Experimentus Ltd 2008

The Project and Programme Managers

Couldn't understand the deliverables from testing

– Test Strategy?

– Test Plans?

They knew they needed a Test Strategy but didn't know why – ‘we always have one’

They were not clear when each deliverable should be produced

They still didn't invite testing in at the start of a project


Lessons learnt

Align Test deliverables with Project Deliverables

We reintroduced the section for an overview of testing that had been removed from the Prince2 template Quality Plan

Test Policy remained – Test Strategy abolished and replaced with a documented Test Process, and a Master Test Plan

Where appropriate rename Test Deliverables to match Project Deliverable

So lets see what we did to achieve this


Testing Deliverable Breakdown


Master Test Plan

Component Test /Component

Integration Test

WorkpackagePlan

SystemIntegration Test

System Test /Business

Acceptance Test

OperationalAcceptance Test

Co-ExistenceTest

ProductionAcceptance Test

TestSpecification

Tested/IntegratedComponents

Actual Results

WorkpackagePlan

TestSpecification

IntegratedSystems

Actual Results

Test ClosureReport

Test Incidents

WorkpackagePlan

TestSpecification

Tested Solution

Actual Results

Test ClosureReport

Daily IncidentReport

Daily ProgressReport

Test Incidents

Team Plan

TestSpecification

OperationallyTested Solution

Actual Results

Test ClosureReport



Test Incidents

Team Plan

TestSpecification

Co-ExistenceTested Solution

Actual Results

Test ClosureReport



Test Incidents

Team Plan

TestSpecification

TestedImplementation

Actual Results

Test ClosureReport

Test Incidents

Handover Report

Weekly TestSpecificationStatus Report

Early Test Planning – the inputs and outputs

Customer’s QualityExpectations

Acceptance Criteria

Quality Approachto Project Deliverables

Quality Plan Summary

Approved Testing Budget

Project Plan

Project Controls

Project Risks

Inp

uts

to

Tes

t P

lan

nin

g

Testing Budget

Test Stages forProject Plan

Project Controls

Testing Risks

Original idea and estimate produced

High Level Terms of Reference

Project Initiation Document

Quality Plan

ApprovedBusiness Case

Business Requirements

Testing EstimateIn

pu

t fr

om

Tes

tin

g

Test Policy

Test Managers Roles and

Responsibilities

EstimatingMetrics


26

Master Test Planning - the inputs and outputs

MTP or PID

PID

Master Test Plan

Acceptance Test Work Package

Team Plan

OperationalAcceptance Test Work Package

System Integration Test Work Package

System Test Work Package

Component Integration TestWork Package

Component Test Work Package

Team Plan

Team Plan

Team Plan

Team Plan

Team Plan

Risk Based TestingAssessment

Customer’s QualityExpectations

Acceptance Criteria

Quality Approachto Project Deliverables

Quality Plan Summary

Approved Testing Budget

Project Plan

Project Controls

Project Risks

Inp

uts

to

Tes

t P

lan

nin

g

27


Learning styles

Having a process isn't the full answer though!

Humans will initially learn the process verbatim, and will expect significant help to be able to adapt the processes to their environment

We first have to learn, before we can be agile with our use of the process

This takes time, which is why the process improvement experts suggest an 18 month timeline before process and culture come together to be at their most efficient

28


Reflect ive Observation Understanding

Concrete Experience

Following the rules

Active Experimentation

Doing

Abstract Conceptualisat ion

Thinking

Learning styles

This is where no real evaluation of whether the

process is right or not occurs

At this point the process usage is

second nature, with all users being able to evaluate what is

right and wrong.

29


No matter how adaptable to change people are, there will always be cultural issues which if not managed will inhibit the benefits of the change being realised

Remember...


That to make any process change work, agility in its delivery is the key – its no good enforcing change that no one can understand or is able to use

En

thu

siasm

Maximum Hype

Initial Invention

ActualUse

Dis -illusion-

ment

PracticalBenefit

RIP

30

Thank You Any questions?


31

Geoff Thompson - Why Do We Bother With Test Strategies

Software

change experimentus

maturitylevel experimentus

levels of tmmi experimentus

test levels phases

test policyneeded

test plan

test effort

test tools