Isabel Evans - Quality In Use - EuroSTAR 2011

Quality in use: the beating heart of the

customer experience

Isabel Evans - Quality Manager

Dolphin Computer AccessAccessible document describing this presentation available from [email protected]

Dolphin’s Purpose

Noel DuffyManaging Director

Dolphin Computer Access

“It is our purpose

to deliver independence

to people with vision and print impairments all

over the World.”

“In an ideal world no-one would need our products – no-one chooses to be blind”

• The Dolphin Challenge:

• Even if the products are perfect no-one chooses to be print-impaired...

Dave Williams

Dolphin Blindness Product Owner and Q&D Tester

Quality in Use

Our journey

What?

Why?

How?

Our journey

Dolphin recognises an opportunity for improvement

• Takes on a quality manager

– To improve testing

• What needs improving?

– No specialist testers

– No reporting on testing

– No techniques

– No automation

Solution 1: more testing, better testing

• Improving test skills and motivation

• Night-siding to New Zealand

• Regression test packs, personas, user stories

• Technique of the Week & lunchtime briefings

• Automated some testing– CTE and Python

– Rhubarb

– Beetroot

– Sledgehammer

• Lots more to do...

Solution 2 - Improve test reporting

tests passed

tests failed

tests not yet run

Example snapshot for one product – simple reporting:

Solution 3: improve metrics

Release X

More defects found after release

than in test

More defects found in testing than after

release

Defects found in test Defects found after release

Example snapshot for one product – simple reporting:

Release Z

Improvement target we have set

for ourselves

Release Y

By July Aim for December

Side effects of the cure

• Disadvantages of improving testing and acceptance criteria

• Being wedded to your own solution

• The “new cleaner” syndrome

• Fear of change

• Level of maturity

• Loss of balance between pressures

• Cannot automate yet...

• TEST IMPROVEMENT IS NOT ENOUGH!

Project stopped

and restarted

Morale drop

Workloadincrease

As well as test improvement support the company vision and purpose:

Deliver appropriate solutionsBe loved by customers

Deliver commercial success

Solution 4: use ISO25000 - Quality in Use

Usability

Context conformity; Context extendibility;

Accessibility

Safety

Effectiveness; Efficiency;

Satisfaction

Flexibility

Commercial damage; Health & safety (operator & public);

Environment

Quality in Use (QiU) and Dolphin

U

F

S

Quality in Use – “My tasks” and “My money”

Dealers

Sales Marketing

Buyers

People with Visual Impairments (VI)

People are different...

Context of usePersonasQiU scoresUser stories

QiU scores for two Dolphin product users

0

2

4

6

8

10Effectiveness

Efficiency

Satisfaction

Conformity

Extendibility

Commercial

Health &Safety

Environment

VI - SuperNova

VI - Guide

Solution 5: don’t forget engineering detail

Security

Operability

Functional suitability

Compatibility

Maintainability

Portability

Reliability

Performance

Internal Quality (IQ) How was it built? Is it faulty? Were good techniques used?

Internal quality (IQ) – engineering detail

Engineers and technologists care about these...Other people only notice them if they go wrong...

Support

Development

Sys-AdsF_O

Po_Co_Ma

S_P_R

Quality in UseInternal Quality

UFS

F_O

Po_Co_Ma

S_P_R

Sales

Marketing

Support

Development

QiU and IQ – Dolphin Project Team

Includes People with VI

Includes People with VI

Quality in UseInternal Quality

UFS

F_O

Po_Co_Ma

S_P_R

Where are the

testers?

Testers for QiU and IQ

UFS

F_O

Po_Co_Ma

S_P_R

Market Readiness testingBeta testing

Research, Design, Development, Code

Review, Unit test, Product test

Testers for QiU and IQ

Market Requirements

Iterations

Post Implementation Review

Solution 6 – customer-centric SDLC

U FS IQ

Solution 7 - Project workshops with sales and marketing

• Marketing, Sales, Support

– Focus on emotional

– Focus on practical

– Focus on tasks

• Vision

• User stories

• Acceptance criteria

• Development, Technical testers

– Focus on measureable acceptance criteria

– Focus on architecture

U F S

F_O

Po_Co_Ma

S_P_R

w1 w2 w3 w4 w5 w6 w7

w6 w7

w1Choose

Contents

Release

Market Ready test

R&D&T for this iteration

R&D&T for multi iteration items

Release

Solution 8 – Sensible Iterations to focus the detail

Market readiness email• What is the verdict on this build?

• Action: verdict by (date):

– Yes

– No

– Don't know

– No opinion

• Proposed deadline for …

– Date for all market readiness feedback

– Date for release Market

Ready test

Iteration content email

• What is going to be in the next iteration?

• What about the issue with (....)?

• How did we come up with this content list?

• What is the proposed schedule?

• What is the contingency plan?

• What is your verdict on this plan?

• Response by (date)Choose

Contents

What have we learnt?

• Marketing, Sales and Support involvement

• Improving project & quality management

• Improving test skills and motivation

• Iterations forced out process weaknesses

• Night-siding to New Zealand

• Technique of the Week & lunchtime briefings

• It’s good when other people have “your” ideas

• Automation is hard sometimes

• Accessibility and white-boarding...

• Lots more to do...

The journey continues

“In an ideal world no-one would need our products

no-one chooses to be blind”

The journey continues...Thanks and goodbye

Isabel Evans – Quality Manager – Dolphin Computer Access

Quality in Use:

The beating heart of the customer experience

Usability SafetyFlexibility