Swiss Testing Day 2013 - How to avoid the testing swiss cheese syndrome

How to avoid the Testing Swiss Cheese Syndrome

Marc Rambert

@MarcRambert

- 11/02/2008 2Marc Rambert - Kalistick

Our journey

New testing challenges

Testing Swiss Cheese Syndrome

Medication: Test/Code Coverage

How to cure this syndrome

Conclusion & Q & A

- 11/02/2008 3Marc Rambert - Kalistick

New Testing Challenges?

- Time 2 market: from 1 release per year to 1 per day/month/quarter

- Adapt testing to Speed & Quality

- Being Agile

- From one QA team to a whole team doing testing

- Adapt to time & cost constraints

- While managing risks

- 11/02/2008 4Marc Rambert - Kalistick

From 1 testing team to a whole team doing testing

Mike Cohn: Test PyramidFounding member of the Scrum alliance

- 11/02/2008 5Marc Rambert - Kalistick

Application

Version X

Manuel Automated

Unit

Integration

Functional

Regression

User AcceptanceExploratory

The “Testing Swiss Cheese Syndrome” !

Testing ‘holes’ / Gaps

modificationsmodifications

Bug catched !

High Risks!

“Testing is an investment and the investment we make at one layer should be influenced by how well testing has been done at the other layers” Mike Cohn

- 11/02/2008 6Marc Rambert - Kalistick

Where are the testing gaps?

Application – release X

Done by dev Tested

Risks ? Ok Effectiveness ?

8% to 16% of all bugs are created by fixes

Requirements

Capers Jones, “Software Quality in 2011: a survey of the state of the art”

“Dev are from Mars & Test are from Venus”

- 11/02/2008 7Marc Rambert - Kalistick

Application V x.1.11.1.2

Lot of questions.Few information

Modifications?

Impacts? Tests relevance?

‘Build’Tests coverage?

Risks? Risks Coverage?

Test Focus?

How much testing?

Go Live?

- 11/02/2008 8Marc Rambert - Kalistick

Changes

Application

Medication:Capture & Aggregate Test Coverage

Unit Tests

Automated Functional Tests

Integration Tests

Holes = Risks

=> Which manual tests are relevant?

- 11/02/2008 9Marc Rambert - Kalistick

What’s Test/Code coverageor more effective: Test Footprint

Test Team

Manual & automated

tests

Test 1

Application build/Version X

Test 2

Test 3

Test 1 - footprint

Test 2 - footprint

Test 3 - footprint Detect Code Changes

Testing Holes

Tests/Code Coverage

- 11/02/2008 10Marc Rambert - Kalistick

Improve your Testing Strategy

Automated tests executed

Code modifications

Functional tests executed

Functional tests not yet executed

Overall application code base

1

1

2

3Functional tests not yet executed

1

1

- 11/02/2008 11Marc Rambert - Kalistick 11

A quick way to get it

- 11/02/2008 12Marc Rambert - Kalistick

How to cure this syndrome

Functional tests

Unit tests

Application

AggregatedTest

Coverage

Softwarechanges

Integration tests

Testing Gaps

Identify relevant tests to fill the gaps

- 11/02/2008 13Marc Rambert - Kalistick

Impact on Testing Effectiveness

– Team 1• After executing planned campaigns• Selected 10 tests with the highest impact (Test Scoring) 3 regressions were found.

Best ratio ever: Bugs/Tests executed

– Team 2• Adopted Test Coverage & Test Scoring to improve its testing activities KPI: number of bugs found within 3 months after deployment - 50%

- 11/02/2008 14Marc Rambert - Kalistick

Conclusion + Q&A

– Different kind of testers with different tools

– Adopt an aggregated view of all testing activities spread along the application life-cycle

– Nurture collaboration between Dev & Test

Code Coverage is not only for developers

Test Footprint is more than coverage

More questions: @MarcRambert