TP PM Tutorial 10/1/2013 1:00:00 PM "Test Managers: How You Can Really Make a Difference" Presented by: Julie Gardiner The Test People Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected]∙ www.sqe.com
23
Embed
Test Managers: How You Can Really Make a Difference
When leading a test team or working in an agile team, becoming a trusted advisor to other stakeholders is paramount. This requires three key skills: earning trust, giving advice, and building relationships. Join Julie Gardiner as she explores each of these skills, describing why and how a trusted advisor develops different “mindsets.” Julie shares a framework of “quick-wins” for test managers and team leaders who need to show the value of testing on projects. To help provide timely, relevant information to stakeholders, she shares seven powerful monitoring and predicting techniques. Julie demonstrates three objective measures showing how testing adds value to organizations. To make sure that everyone is on the same page, Julie urges managers to establish a foundation for testing through well-defined policy statements, agreed to and sanctioned by senior management. Receive a set of spreadsheets and utilities to support your activities as a test manager who really makes a difference.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
TP PM Tutorial
10/1/2013 1:00:00 PM
"Test Managers: How You Can
Really Make a Difference"
Presented by:
Julie Gardiner
The Test People
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
Passionate about Testing, Excited about Technology
Building relationships
§ deal with senior management as “human beings” § formula to building a strong relationship
§ BSR = AQ + LTA § be open and honest – not a sycophant § when you need help – ask
– this can be a sign of strength rather than weakness
§ show appreciation
make it your goal to be someone managers turn to first when issues arise and advice is required.
building relationships
5
Passionate about Testing, Excited about Technology
Giving advice effectively § be prepared
– look beyond the “now” & understand the whole situation
§ advice is almost never a purely logical process – don’t exaggerate or lie – ever! (this destroys trust) – give good news before bad news (don’t hide bad
news) – be fair with information
§ know your audience – the advice we give must be helpful – different people need different information
giving advice
Passionate about Testing, Excited about Technology
Developing the right mindset
six key attitudes to consider § focus on the other person § being sincere § having confidence (in oneself) § adopting a blameless culture § taking responsibility § recognise strengths and weaknesses
– in you and your team
having a right attitude and mindset is a massive step
closer to becoming a
“Trusted Advisor” to Senior
Management
6
Passionate about Testing, Excited about Technology
What makes us valuable on project
How to keep testing valuable on projects
Showing the value of testing
Contents
Passionate about Testing, Excited about Technology
Passionate about Testing, Excited about Technology
Managing testing
Plan Measure
Review Goal
Policy overall approach to testing
Strategy
Passionate about Testing, Excited about Technology
Test policies • represents testing philosophy for organisation
– outlines the corporate goals for testing
• guides to stated goals and objectives • communicates good practice for the organisation
– approach, culture and standards
• developed by the IT, R&D or Product Development • apply to both new development and maintenance • short, static, high-level document • some areas covered include
– test process, levels of testing, success factors, measurements, test process improvement
8
Passionate about Testing, Excited about Technology
Test policy examples
• testing is to be risk-driven
• reviews will include testers
• auditable test documentation records will be kept
• entry and exit criteria will be specified and enforced
• the value of testing will be monitored and reported
What policy statements do you have in your organisation?
Passionate about Testing, Excited about Technology
Policy, Strategy and Plans Company Policy for Testing
Company/Project Strategy for Testing
Project (High Level) Test Plan
9
Passionate about Testing, Excited about Technology
Monitoring
• regular and timely supply of information • what is actually happening • to compare with the plan • right use of words, graphs and tables • interpretation & explanation is often required • enables effective and timely control • used to predict what is likely to happen
now to introduce some powerful monitoring techniques to display on your
dashboard…
Passionate about Testing, Excited about Technology
S-‐curves for: tests run/a@empted
tests passed/complete bugs found
bugs fixed
One: S-curves
Plot the points you know, and planned end point e.g. no. tests run so far & planned final no. at end date Plot trend line: 3rd order polynomial
Useful approximaPon: straight line
Pme
Source: Marnie Hutcheson
How to use S-‐curves
10
Passionate about Testing, Excited about Technology
Example using S-Curves
Source: Tim Trew, Philips
Passionate about Testing, Excited about Technology
Two: Defect Measure (DM) Rate
DM = 10*H+5*M+L DM Rate = DM/(hours of test effort)
11
Passionate about Testing, Excited about Technology
Three: System Availability
0 2 4 6 8 10
Hours available in a day
Mon
Tue
Wed
Thu
Fri
Sat
Sun
Mon
Tue
Wed
Days
Host System Availability
AvailableDown time
Based on 10 hour day we lose 2 days of down time!
Passionate about Testing, Excited about Technology
Four: Regression Test Coverage Release 1.0 Release 1.1 Release 1.2
Key: not run defects found no defects
Solo Personnel Solo Payroll Personnel Breadth Payroll Breadth Personnel Scenario 1 Payroll Scenario 1
Test Procedure Release 1.0 Release 1.1 Release 1.2
12
Passionate about Testing, Excited about Technology
Raw Data 10/23/2013 17-Feb-13 24-Feb-13 03-Mar-13Code Turmoil 1127 1409 1020 799Defects found this week 23 16 6 13Total Open Defects 24 32 7 12Test Success Rate 95.50% 98.70% 99.50% 99.80%Test Completion Rate 44.80% 70.00% 71.80% 78.90%
0 1 2 3 4 5 6
Code Turmoil(loc)
Defects found this week
Total Open Defects
Test Success Rate
Test Completion Rate
Source: Mike Ennis, Managing the End Game of a Software Project, STAREast
Passionate about Testing, Excited about Technology
Six: Outstanding Risks
Progress through the test plan
today end date
residual risks of releasing
TODAY Res
idua
l Ris
ks
start
Source: Risk Based E-Business Testing – Paul Gerrard & Neil Thompson
all risks ‘open’ at the start
13
Passionate about Testing, Excited about Technology
Benefit (or objective) based test reporting
Open
Closed
Ris
ks
Open
Open
Closed
Closed
Open O
bjec
tive
Obj
ectiv
e
Obj
ectiv
e
Obj
ectiv
e
Bene
fit
Bene
fit
Bene
fit
Bene
fit
Bene
fit
Benefits available for release Source: Paul Gerrard
Obj
ectiv
e
Bene
fit
Closed
Passionate about Testing, Excited about Technology
Seven: Reliability/Confidence Curve
Reliability/Confidence Curve
0
20
40
60
80
100
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1
Release
Tim
e (
min
ute
s)
Time to failure
alternative 1 alternative 2
R 1.0 R 1.1 R 1.2 R 1.3
14
Passionate about Testing, Excited about Technology
What makes us valuable on project
How to keep testing valuable on projects
Showing the value of testing
Contents
Passionate about Testing, Excited about Technology
Three key measures to consider
Defect Detection Percentage (DDP)
Pre-release Defect Fixed Percentage (DFP)
cost saving per bug
15
Passionate about Testing, Excited about Technology
“Testing is expensive”
• compared to what? • what is the cost of NOT testing, or of bugs missed
that should have been found in test? – cost to fix bugs escalates the later the bug is found – poor quality software costs more to use
o users take more time to understand what to do o users make more mistakes in using it o morale suffers o => lower productivity
• what does a bug cost in your organisation?
Red Use Des Test 1
10
1000
100
Passionate about Testing, Excited about Technology
How much does it cost (based on Boehm)
How much do defects cost?
02000400060008000
1000012000140001600018000
requ
irem
ents
spec
ificati
on
initia
l test
syste
m/acce
ptanc
e
live
runnin
g
FolksamBoehm
Source: Kristina Bihlar, Folksam BR
16
Passionate about Testing, Excited about Technology
How much do our defects cost Defect in: Manhour to fix Costrequirements 1 665specification 2.5 1663initial test 4 2660system/acceptance test 8 5320live running 15 9975
Cost per hour in SEK: 665
What if we found 100 defects during testing:During system/acceptance test 532,000what could have been found in initial test We saved: 266,000what could have been found in analysis We saved: 465,500
But it could have cost: 997,500 …Source: Kristina Bihlar, Folksam BR
Passionate about Testing, Excited about Technology
How expensive for you? • do your own calculation
– calculate cost to fix bugs missed by testing
– calculate cost to fix bugs found in testing
– calculate cost of testing o people’s time, machines, tools
• estimate if no data available – your figures will be the best your
company has!
17
Passionate about Testing, Excited about Technology
Questions you may be asked How good is the testing anyway? Can you prove you are doing a good job?
Your testing can still be just as good in less time, can’t it? (That deadline pressure really didn’t matter, did it?)
Is the testing any better for this release? (Have we learned anything?) (Have we really improved our testing?)
How many bugs have we missed?
Passionate about Testing, Excited about Technology
Some questions for you • do you keep track of defects?
– defects found in testing o different test stages (e.g. system test, user acceptance
test), o different releases (e.g. testing for an incremental
release) – defects found in live running
o reported by users / customers • can you find these numbers from a previous project and your current
project? • do you have a reasonable number of defects found?
if so, you can use DDP to measure your test effectiveness
18
Passionate about Testing, Excited about Technology
Defect Detection Percentage (DDP)
• "this" testing could be
– a test stage, e.g. component, integration, acceptance, regression, etc.
– all testing for a function or subsystem – all testing for a system
Defects found by this tesPng Total defects including those found aYerwards
Passionate about Testing, Excited about Technology