_experience the commitment TM Why is Test Planning so Hard? Elizabeth DAngelo, PhD CGI Group Inc. September, 2008 © CGI GROUP INC. All rights reserved.

_experience the commitment TM

Why is Test Planning so Hard?Elizabeth D’Angelo, PhD

CGI Group Inc.

September, 2008

© CGI GROUP INC. All rights reserved

2

Confidential

Agenda

• Background• Test Planning: Three Issues• Test Planning: Three Solutions• Practical Tips to Help you in Test Planning• Conclusions• Q&A

3

Confidential

Reasons Why Test Planning is Difficult

Planning Expertise

Application Expertise

Forecasting Expertise

Testing Expertise

4

Confidential

Test Planning Issue 1: Knowledge (1)

• Right Mix is Critical!

• Requirements• Requirements: Implicit vs. explicit knowledge (people vs. docs)• Expertise

• Thorough knowledge of applications and flows• Breath vs. depth

Testing Domain

Application Domain

5

Confidential

Test Planning Issue 2: Project Management (1)

• Test Planning is Project Management within Project Management!

• It is NOT optional• Be prepared to articulate WHY is needed

• Plan – Prepare – Execute motto

• Take time to do proper planning • “Vanilla flavour” plan is useless

6

Confidential

Test Planning Issue 2: Project Management (2)

Environment

Data

Tests

7

Confidential

Test Planning Issue 3: The “Gruyere” paradigm

• How many holes are in this cheese?• What is actually delivered vs. what was planned to be

delivered?• Are all the features truly complete?

8

Confidential

Test Planning Solution 1: The Expert (1)

• KNOW the WHAT…• Industry Knowledge• Application/Environment Knowledge• Testing Knowledge

9

Confidential

Test Planning Solution 1: The Expert (2)

• Become the testing guru• Help your team to become the testing experts• Some Ideas:

• Formal training/certification• Attend conferences• But there are also cheaper ways:

• Lunch and Learn sessions • Organize workshops with presentations

• Test Planning• Risk-based approaches• Test terminology• Test case design• Metrics• Test tools

• Create a testing library

10

Confidential

Test Planning Solution 1: The Expert (3)

• Build your credibility by becoming the industry/application expert• Read ALL documentation

• Review the requirements, design, architecture

• Talk to users, developers• Don’t be afraid of asking questions

• Ensure to communicate your status properly• Assume responsibility when needed

11

Confidential

Test Planning Solution 1: The Expert (4)

• And there is more… Soft Skills!• Communications (defect reporting)• Detective• Negative thinking or “What can go wrong”• Time management• Discipline and Tenacity• Creativity

12

Confidential

Test Planning Solution 1: The Planner (1)

• KNOW the HOW• Testing scope• Discuss under/over testing• Who will be doing what• What needs to be in place before test starts• Test case design guidelines• Tool usage• Metrics to be collected• Entry/Exit criteria• …

13

Confidential

Test Planning Solution 1: The Planner (2)

• You will reap what you sow

• Planning is the most essential activity!PLAN, PLAN, PLAN ….

You cannot prepare what you have not planned for

You cannot execute what you have not prepared for!

14

Confidential

Test Planning Solution 1: The Planner (3)

• Make your Master Test Plan the Bible• Document ALL your assumptions• Think of ALL risks• Use ALL information available to you• Establish in/out scope INCLUDING

• Test boundaries

• Brainstorm and define what will be tested when!• Phased approach

• Don’t forget • Environment• Test Data• Access/Security

15

Confidential

Test Planning Solution 3: The Weather-Man (1)

• Look at signs to forecast quality issues• Previous history (application, business function, etc.)• Get to know the team• Collect and store metrics you will need them!

• Clearly communicate deviations from plan based on unexpected quality issues

16

Confidential

Test Planning Solution 3: The Weather-Man (2)

• Establish and Communicate Forecast for your Plan:• Failure rate• SLAs for defect fixes• Environment availability

Confidential

Practical Tips to Help you in Test Planning

18

Confidential

ALWAYS Use Risk-Based Approach! (1)

• Spend your effort where is worth it,…Unless you have unlimited time/budget/resources

• Risk-based approach will allow you to test• More comprehensibly what really matters• First what is more important• If needed, know where to cut testing

• NOT an isolated task, you need to socialize approach and get agreement on risk areas

19

Confidential

ALWAYS Use Risk-Based Approach! (2)

• Technical risks assessment should include:• New and immature technology, • Brand new system or enhancement• New and difficult to understand business requirements, • Complexity of business logic, • Complexity of technology, • Size of the project, • Skill mix on the project,• Number of users (including customer facing or back office

considerations) historical data when appropriated (the risk is higher for modified functions that have frequently failed in the past, i.e., Do you have any historical data of defects by area?)

20

Confidential

ALWAYS Use Risk-Based Approach! (3)

• Business Impact assessment should include:• Importance of features from client/customer perspective• Criticality of features from the business perspective• Frequency of use

21

Confidential

ALWAYS Use Risk-Based Approach! (4)

• As a rule of thumb:• High Impact:

• Calculation• Validation• Legal• Front end (end user)• High frequency, High volume of users

• Medium Impact: • Data changes• Wrong information displayed• Medium frequency and groups of users

• Low Impact: • Display• Visual appearance• Used by a few users• Back office

22

Confidential

Risk-Based Approach Process

1. Organize test risk assessment workshop with business and technical stakeholders

2. Identify and list requirements, business functions

3. Evaluate technical and business impact (rate 1-3)

4. Evaluate probability of failure (rate 1-3)

You might want to read: http://www.satisfice.com/articles/hrbt.pdf

Risk Factor Testing Priority Testing Extent

7-9 High Extensive

4-6 Medium Comprehensive

1-3 Low Smoke

23

Confidential

Risk-Based Approach: Business/Requirements Map

High

Low

Low HighImpact

Lik

eliho

od

1 32

1

2

3

Accounting

Finance

Product Filing

Req n-7

Licensing

DistributionCommunications

Req n-4

Req n-1

Administration

Extensive Testing

High Degree of Testing

Less Testing Required

24

Confidential

Cover all aspects of testing

• When planning… make sure to cover the overall process:

Test Preparation Test ExecutionPlanning Reporting

Test Results

Problem / Defect Reports

Test Plan (s)

Update Test Project Plan

Create Regression Test Suite

Test Environment set up

Test Data set up

Test Cases

Create Automation Test Suite

Test Exit Report

Lessons Learned - Recommendations

Final Test Report

Strategy

Test Strategy

Test Project Plan (Schedule)

Test Coordination

25

Confidential

Schedule Templates Detailing All Activities (1)

• Have an overall schedule template that you can easily fill with specific data

Activity

Work-Effort

(in days)

Start Date End Date# of Res

Resource Details

Review Business, Technical Requirements 0 YY/MM/DD YY/MM/DD 0 Test Manager, Test Leads, Test Analysts

Create Test Strategy/Master Test Plan/Traceability Matrix 0 YY/MM/DD YY/MM/DD 0 Test Manager, Test Leads, BAs

Review Test Strategy/Master Test Plan 0 YY/MM/DD YY/MM/DD 0 Project Team, Stakeholders

Sign-off Strategy 0 YY/MM/DD YY/MM/DD 0  

Create detailed Test Schedule 0 YY/MM/DD YY/MM/DD 0 Test Manager

Review Test Schedule 0 YY/MM/DD YY/MM/DD 0 Project Team

Sign-off Test Schedule 0 YY/MM/DD YY/MM/DD 0  

Creation of System Test Cases 0 YY/MM/DD YY/MM/DD 0 Test Team

Review Test Cases 0 YY/MM/DD YY/MM/DD 0 BAs, BSAs

Update Test Cases 0 YY/MM/DD YY/MM/DD 0 Test Team

Create Test Data 0 YY/MM/DD YY/MM/DD 0 Data Analyst

Sign-off System Test Cases 0 YY/MM/DD YY/MM/DD 0  

Review Regression Test Bed 0 YY/MM/DD YY/MM/DD 0 Test Team

Update Regression Test Bed 0 YY/MM/DD YY/MM/DD 0 Test Team

26

Confidential

Schedule Templates Detailing All Activities (2)

ActivityWork-Effort

(in days)Start Date End Date # of Res Resource Details

Create/Select UAT Test Cases 0 YY/MM/DD YY/MM/DD 0 Business Reps

Review Test Cases 0 YY/MM/DD YY/MM/DD 0 Test Leads

Update Test Cases 0 YY/MM/DD YY/MM/DD 0 Business Reps

Create UAT Data 0 YY/MM/DD YY/MM/DD 0 Data Analyst

Sign-off UAT Test Cases 0 YY/MM/DD YY/MM/DD 0  

Execute System Test Cases 0 YY/MM/DD YY/MM/DD 0 Test Team

Review exit criteria and produce System Test Report 0 YY/MM/DD YY/MM/DD 0 Test Manager

Sign-off System Test Results 0 YY/MM/DD YY/MM/DD 0  

Execute UAT Test Cases 0 YY/MM/DD YY/MM/DD 0 Business Reps

Support UAT Execution 0 YY/MM/DD YY/MM/DD 0 Test Team

Produce UAT Test Report 0 YY/MM/DD YY/MM/DD 0 Business Lead, Test Manager

Sign-off UAT Results 0 YY/MM/DD YY/MM/DD 0  

27

Confidential

Systematic Entry/Exit Criteria

Entry Criteria Exit CriteriaUnit/Integration Testing exit criteria has been fulfilled and approval provided

System Test Plan is approved or Approach is documented in the Overall Test Strategy/Plan

All system test cases as per plan have been executed

System scenarios, conditions and cases are complete and approved

All detected fixed defects have been re-tested

Test data has been created and is readily available in system test environment

No outstanding Severity 1 or 2 defects

Test environment is set up and operational

System Test Report has been completed, reviewed and approved

Plan

Test Cases/Data Readiness

Environment Readiness

Planned Activities

Quality

Communication & Sign-off

Preconditions

28

Confidential

Forecasting with Metrics – Failure Rate/Env

• Document what is the expected failure rate• Existing application/delivery team: use previous data• Be as granular as possible

• By Functional Area• By Application

• If you don’t have data, rule of thumb is around 30% (i.e., 30% of test cases will fail during test execution)

• Incorporate retesting activities to schedule

# TestersPlanned

# TestersActual

Env # TestsPlanned

# TestsActual

Passed Failed FailureRate

DefectsNew

DefectsFixed

ExecutionVariance

RetestingVariance

Total Variance

Cycle 1 5 5 100% 50 50 30 20 40.00% 12 0 0 -20Day 1 5 3 90% 50 30 25 5 16.67% 4 4 -20 -5Day 2 5 7 100% 50 75 60 15 20.00% 10 10 25 -15Day 3 5 50Day 4 5 50Day 5 5 50Day 6-retesting 5 50Day 7-retesting 5 50

TOTALS 400 155 115 40 26 14 5 -40 -35

29

Confidential

Forecasting with Metrics – Defect Age SLAs

• Negotiate and document SLAs for defect fixing, then measure• How long defects remain in New state (i.e., not assigned?)• How long defects remain in Assigned state (Are too many defects assigned to

the same resource and therefore there is no progress?• How long defects remain in Fixed state (i.e., before being closed by Test

Team)? (Is testing team following up on defect fix verification? Are there competitive priorities (execution vs. verification)?

• A high number of defects in Need Info state could indicate that there are concerns with the requirements, lack of details on the defect (to reproduce).

30

Confidential

Conclusions

• DO• PLAN, PLAN, PLAN …. AND THEN PLAN!• Get buy-in into risk-based approach• Create templates to help you be efficient• Be systematic• Use historical information/metrics to predict future• Share your documents, status and metrics so stakeholders understand

challenges• Hope for the best, but plan and prepare for the worst!

• DON’T• Work in isolation• Over complicate your methodology• Think you don’t need to understand domain• Do ad-hoc testing (unless it is planned)• Be inconsistent• Keep issues, departures from assumptions to yourself• Be naïve and plan for the best!

_experience the commitment TM

THANKS!!

Questions??

32

Confidential

Contact Information

Elizabeth D’Angelo, M.Sc. PhDDirector, Quality COE150 Commerce Valley Drive WestMarkham, Ontario(905) 695-6513(416) 459-1257 [email protected]