1 © Andy Redwood 2003-2006 20 Golden Rules for Quality Testing Andy Redwood.

1

© Andy Redwood 2003-2006

20 Golden Rules for Quality Testing

Andy Redwood

2

What’s In?

Risk Assessed Testing How to save money Capability Maturity Testing Requirements Test Estimating Quick wins with

Strategies Marrying documentation

and traceability Philosophy and

psychology of testing

Test only what you need to

Can you learn from your defects?

Reusing Test Assets Test Environments Measures and metrics Use of Tools Importance of sign off

A selection of these -

3

Golden Rule Number 1

Testing is a Risk Management exercise. Is anybody worried?

why expend valuable resources searching for defects that are not important or viable to find?

Ensure all the Requirements are risk-assessed. to the best method that works for you or your

organisation. prioritise all the testing to target the most risk with the

least test effort. when there are only minor risks outstanding, an informed

decision can be made on when to stop testing.

4


Training in Testing reduces long term costs.

Budget for Test Training. less than 1% of an organisations training budget

dedicated to training in testing is not unusual. Training in Best Practice.

formal training in Test Management, Test Design, Static and Dynamic techniques.

ISEB Foundation and Practitioners Certification. Spread the Message. Integrate testing with other disciplines.

5


The Testing Culture in an organisation can only mature at the rate of its capability.

Don’t try too much too soon.Establish all the perceived problems.Create a balanced economy of scale.What can be achieved will vary from one

organisation to another.

6


The greater the number of late changes to Requirements, the more rework is required.

Firm Requirements? requirements are not just changing but new ones are being

created, even when you’re in Test Execution?

Adopt the “Bus to Bus Stop” rule. requirements get on the Bus. nothing else gets on and nothing gets off (ring-fenced scope).

requirements may change seats (change control). new requirements get on the next Bus coming along behind (good

release management).

7


Testing estimates should be based on Business Risk.

Do you guess? collect testing measures and metrics. use previous project information to guide you.

weight the risk with a ‘time taken to test’. improve with each proceeding project. predict with ever increasing accuracy.

Warning. ‘similar’ is not the ‘same as’. you will need to evaluate any new ‘hotspots’. please don’t estimate by habit!

8


Drafting a Test Strategy from a facilitated workshop is a Quick Win.

Strategy Workshops. confirm the status known testing requirements very early

in the life cycle. unknown activities surface early, creating contingency

within the programme. my last 100 Test Strategy workshops have raised an

average 36 previously unknown high risk issues. some of these would not otherwise have been spotted

until system test execution.

9


demonstrate the relationship – ‘what’ vs ‘how’.

documentation is what we as testers must test back to.

in many cases this relationship is implied and not explicit.

it’s not in the methodologies.

Linking Development documents to Test Documents improves Linking Development documents to Test Documents improves understanding.understanding.

Test Strategy

User Acc.Test Plan Ops. Accept.

Test Plan SystemTest Plan Integration

Test Plan UnitTest Plan

RequirementsDefinition

System DesignDocument

OperationalRequirements

Detailed DesignDocument

User Acc.Test Spec.

Ops. Accept.Test Spec.

SystemTest Spec.

IntegrationTest Spec.

UnitTest Spec.

ProjectDefinition

FeasibilityStudy

KEYDevelopmentDeliverable

TestingDeliverable

10


All test conditions should have a parent requirement.

Traceability. between Requirements and Test Conditions within a Test

Repository. all Requirements are tested. aids Test Progress reporting.

Eye of a Bird. scan back and forth looking for requirements that have no

conditions, also look for test conditions that do not map to a requirement.

you may identify missing business processes or find an item that requires a Business decision.

11


Always test the Test Environment. A few common phrases.

“The log-ons were all wrong”. “The interface link was down”. “We were pointing at the wrong files”.

Why is this? lack of ‘pipe-cleaning’ of the test environment prior to

starting the real testing. insufficient check-lists. not reusing previous knowledge.

12


Tests should be executed to prove that the item under test doesn’t work, rather than that it does work.

Is this the mindset? still a radical concept in some quarters? psychological conflict with reporting a high number of errors

and making good progress to plan. high number of errors found in test must remain an important

test objective. ensures a high quality and robust systems in Production.

tests should be designed to challenge high-risk attributes at the earliest opportunity.

13


It is better to test the errors out in early Project phases as opposed to having a failure in Production.

Origination. evidence suggests that 50% of all error found in

system testing originate before coding (I.e. in requirements definition or design).

Cost. it can cost hundreds of times more to fix a problem in

production than in the Requirements phase.

14


It is better to test the errors out in early Project phases as opposed to having a failure in Production.

Focus. on static testing techniques, for example,

walkthroughs, reviews, workshops and formal inspections will raise early incidents.

Warning. it is still radical for testing to be perceived as a full life-

cycle activity.

15


Each test phase should attempt to test something different with as little duplication as possible.

duplication costs. Some duplication is unavoidable. testing is more efficient and effective if each test phase is

defined with minimal duplication, within the Test Strategy or Approach document.

details for each test phase will appear in phase specific test plan documents.

question if a particular test phase is actually required. base this decision on risk and viability. prime objective for undertaking the phase, in some

cases, has already been proven in previous phases.

16


An incident is not a error or a fault, but is anything that was not expected.

The Nature of Incidents. incidents can occur in any Programme phase. incidents need not be testing specific. an incident that results from human error or a component fault

will be logged as a problem.

Evaluation of incidents. categorise them under strategically defined incident types. allow sufficient time to conduct root cause analysis. gather meaningful measures, establish trends, calculate

metrics.

17


All incidents should be traceable to a Test Condition and a Requirement.

This is fundamental if you are to - mitigate all the risks (against requirements). know you can stop testing. evaluate data on types of incidents raised against

requirements (or test conditions). use data to make judgement decisions on likely stability

and appropriate amounts of testing required. point at the high-risk areas in the production system to

monitor in the first few weeks post implementation.

18


Test Assets and Project Histories must be considered for Re-use.

Project Histories. previous documentation can

reduce analysis, design and build costs. allow better understanding of the existing functionality. allow more accurate project estimates. enable earlier test preparation .

testware is worth an average 40% of every project budget.

19


Test Assets and Project Histories must be considered for Re-use.

Re-use of Assets. test packs run as regression tests –

provide the status of changes since the previous release.

can be constructed at all test levels to enable maximum flexibility for testing large or small projects.

form a baseline from which to commence preparation of new tests.

20


If you can’t measure, you will not know if what you achieve is good or bad.

Why collect? measures lead to metrics lead to trends. metrics form Management Information. appropriate MI allow informed decisions. measures make perceptions factual.

Does it matter what you collect? objective of the measure may vary. only consistent measures will allow meaningful metrics.

21


Measures point to root causes. Fix the root cause and you improve the quality .

fix to ensure the error will not recur will reduce costs and improve quality of delivery.

Warning. care is required, as it is easy to mistake analysis for the

root cause for a witch-hunt. the root cause could be anywhere; it is unlikely to be in

the phase in which the incident was identified. the important thing is to fix it and learn from it, not

apportion any blame.

22


Tools can only automate existing process; they require support.

Are you ready? Existing process needs to be in a state that allows it to be

automated. A good way to do it!

run tests manually follow by independent automated test runs - less risk, more control.

The Cost… “the cost of the tool is not the cost of the tool”.

Help, I need somebody, help… user groups, limited (sometimes free) on-site vendor support. help from the many consultancies.

23


If you don’t ask a 3rd Party Supplier the question, you may never be told the answer.

Some of the most common questions. how did you test this package before you handed it over to us? do you have any test documentation I could look at? what errors do you know to be outstanding that are in this

release? does this release include changes for any other client and if so,

how does it affect us? what’s the turnaround time for fixing our priority one problems?

Want more? 71 weighted questions [email protected]

24


Sign off Criteria should never be a surprise to Management.

Review Points. when criteria are agreed in advance and reviewed at set points in

the life cycle, sign off into Production is the norm, not the exception.

nothing escapes into production with high-risk caveats or with insufficient testing.

Who Signs? list all interfacing managers that need to always be informed. recommend if sign off is ‘ok to go’ or ‘what the deviation is’ from

the standard approach. managers will make proactive decisions rather than have to wait

until it has finished to react.

25

© Andy Redwood 2003-2006

More Questions?

[email protected]

1 © Andy Redwood 2003-2006 20 Golden Rules for Quality Testing Andy Redwood.

Documents

test training

psychology of testing

test management

testing measures

golden rule number

testing estimates

testing culture

test design