Agile San Diego: Testing as Exploration (Continuous Delivery w/o Automation)

Testing as Explorationaka. Continuous Delivery without

Automation

Maaret Pyhäjärvi

Email: <[email protected]> | Twitter: maaretp

Maaret Pyhäjärvi

Nimeä | Attribution (Finland)

http://creativecommons.org/licenses/by/1.0/fi/

http://creativecommons.org/licenses/by/1.0/fi/deed.en

1. LET’S TALK ABOUT

EXPLORATORY TESTING

?

Things Can Look Different from

Different Perspectives

Realizations about Nature of

Testing

20

16

1639

5±24

Exploratory Testing:

Better tests, better testers! • An approach, not a technique

• Find unknown unknowns

• Disciplined

• Test is a performance, not artifact

– Artifacts support human

memory

– Many forms: e.g. checklists and

automation

• Exploratory performance testing,

Exploratory test automation,

Exploratory regression testingTest-related

learning

Design of new

tests

Test executionResult

interpretation

5

Exploratory Testing:

Learning & Modeling

”A day’s work”

Vision (“Sandbox”) Current Charter

Other Charters Details

Bug

Reports

Perception of

quality and

coverage

Quality

ReportDebriefing

Tester

Test

Manager

Past

Results

Obstacles

Outlook

Feelings

?

#

xCharter backlog of the future

testing

Out of

budget

Next in

importance!#, ?, x, +

20:20:60

Session sheets of the past testing

Idea of

exploration

Metrics

summary

Coachin

g

6

Playbooks

Coverage outlines

2. CONTINUOUS DELIVERY

WITHOUT AUTOMATION

Where

does this

say fast?

Where does

this say

automation

?

A Bit of Background

• 3 years into a web based (.NET/C#) product

• 1:8 tester:developer ratio

• Missing agile technical practices: test automation non-existent

• Scrum-like monthly releases for first 2 years– Releases late

– Releases not tested well enough

• Negative cycle of failing with estimates eating team morale

• Jira in a central role with estimates and burndown

Going for Continuous Delivery

The Main Driver for Change:

Testing

Scheduled release

Feature 1

Feature 2

Feature 3

Feature 4

Testing

Feature 1

Feature 2

Feature 3

Feature 4

R1 R2 R3

Done

includes

Tested

Schedule

over

Quality

SCRUM + SCHEDULED DELIVERY with continuous integration

KANBAN + CONTINUOUS DELIVERY

Enablers that Made Timing Just

Right

• Henri Karhatsu and #NoEstimates –

experience report

• Availability of Git in Visual Studio without

plugins

• Problems with schedules in Scrum

• Lean and value delivery focus in Product

Management

Changing How We Managed Our

Work: Setting up Jira Kanban

Removed hour &

story point estimates

Agreed in WIP for

each phase

Agreed on swarming

Updated issue types

Branches and Test

Environments

Feature Test Environment

Feature branches on-demand builds

“Developers think this can be tested”

Development Test

Environment

Integration branch build-on-checkin

“Developers think this can be released”

Acceptance Test

Environment

Master branch with fixes and merges from integration

on-demand builds

“Release next morning”

Continuous Deployment but

Significant Lead Times

Mon Tue Wed Thu Fri Mon Tue Wed Thu Fri

Change to

Master

Integration

to master

Feature to

integration

Developers drive the decision on what

they want feedback on

Exploratory Testing Enables

Continuous Deployment• Every Jira task gets planned for

– Sometimes we go with developer testing only

– Sometimes we test extensively

• Exploratory Planning– No set test cases

– Talking to developers and reflecting to a model of of use created through earlier explorations

– Actionable information first –principle

– Production monitoring is an option for getting information

Changes

• Active discussion about schedules and merging, and needs of testing in the branches

• More pairing for testing the features

• More group work on defining the features

• Introducing Flowdock due to increased need to collaborate; integrating logs, emails

• Focus on getting better (scope of test automation; refactoring; pairing and group work; individuals’ skills)

Testers don’t break

your code, they

break your illusions

about the code. -- adapted from James Bach