Agile Test Strategy @paul_gerrard Paul Gerrard [email protected] gerrardconsulting.com.

Agile Test Strategy@

paul_g

erra

rd

Paul [email protected]

gerrardconsulting.com

Overview

• What is Agile Test Strategy?• Project Profiling• (Test Strategy as) Agile Interventions• Test Automation• What’s Left?• Summary• Q&A

Intelligent Definition and Assurance Slide 2

What is Agile Test Strategy?

Agile Strategy – an oxymoron?

Agile Test Strategy

• What do we mean by this?1. AGILE Test Strategy – how to create a test

strategy in an Agile way?2. AGILE Test Strategy – a test strategy for

an Agile project?

• We’ll look at how we created an Agile approach to strategy, but we’ll spend more time on strategy for an Agile project.


Google “Agile Test Strategy”

• There are plenty of recipes out there• Most offer a selection of techniques but don’t

provide much guidance on how to blend them• We need to know how to make choices, not

just know what choices exist

• Strategy is a thought process, not a document– Although you might document the ideas for

reuse, as a reminder or ‘for the record’.


Agile governance

• Governance is the act of governing. It relates to decisions that define expectations, grant power, or verify performance

Wikipedia• Define expectations – DEFINITION of need• Grant power – DELEGATION to a project

team• Verify performance – ASSURANCE of

solution.Intelligent Definition and Assurance Slide 6

Strategy helps you decide what to doA. The strategy presents some decisions that can be

made ahead of timeB. Defines the process or method or information that

will allow decisions to be made (in project)C. Sets out the principles (or process) to follow for

uncertain situations or unplanned events

• In a structured/waterfall environment, most questions answered off-the-shelf – “A-style, ready for anything”

• In an Agile environment – might have some ready-made policies but we manage scope and adapt (mostly C?)


Contexts of Test Strategy

TestStrategy

Risks

Goals

ConstraintsHuman resour

ce

Environment Timescales

Process(lack of?)

Contract

Culture

Opportunities

User involveme

nt

Automation

De-Duplicati

on

Early Testing

Skills

Communication

Axioms

Artefacts

Traditional v Agile test strategy• Traditional – structured, goal/risk-driven

– Identify stakeholders; what are their goals?– Product risk analysis– Allocate risks/goals to test stages– Formulate test stage definitions (entry/exit criteria,

environments, tools etc. etc.

• Agile – interventionist, consensus-driven– Project profiling to set the testing theme– Identify testing interventions (perhaps better,

contributions) in the Agile process– Test policy overlays the process; catches exceptions.


Project Profiling

Select a profile for your project first, then choose the aspects of test strategy that

suite your project

Template-driven? Bah!

• So this is just a template copy and edit process?• Won’t you always end up with the same document?

• Profiling doesn’t need to be prescriptive– No need to write a document if you don’t need to– But if company policy or common sense dictates certain

approaches, save yourself some time– Create a set of deeper, more detailed questions to be

answered (Pocketbook)

• Profilers are really just checklists: heuristic guidelines designed to help you make choices and trade-offs.


12

Cerise

Orange

Green

Test Plan Items

Product Risks

Project Profiler

Risk Profiler

Project Plan

Test

Assurance

Project Manager

Business, Project Teamand Boards

Cons

ulta

tion

Blue

Unknowns

Using a Project Profiler to Derive a Test Strategy and Project Plan(A government client example)

The Project Profiler (with Test Assurance) helps Project Managers to:• Select a project style that fits (Waterfall or Agile)• Identify the product risks that need testing• Identify test activities to include in project plans• Carefully define the scope of the project

Environment

Story Guideline

Tools

Incident Mgt.

WaterfallTest Strategy

SCRUM/AgileTest Strategy

Intelligent Definition and Assurance

13

Project profiling processTask

1 Have the Information you need to hand2 Project Profiler (section 3):

Select the statements that best match your project context. The Blue column indicates that you need more information – consult your stakeholders, team or relevant Board(s).

3 Generic Risk Profiler (section 4):Consider the generic project risks – which are significant for your project? Can testing help?

4 Product Risk Profiler (Section 5):Consider the functional and non-functional risks that most concern your stakeholders – should they be tested?

5 Actions and Test Activities (Section 6):Consider the actions that prompt your ‘next steps’ and the test activities that should be incorporated into your project plan.

6 Create your Test Strategy from the Test Strategy Framework Template7 Incorporate the activities from stage 5 and identified in 6 into your Project

Plan.


14

Project Profiler (part of section 3)

Project Aspect Cerise Orange Green BlueResponsibility for Acceptance

Users will take responsibility for UAT; they have UAT experience

Users will be responsible for UAT but have no test experience

Users will take part in UAT or witness tests at critical periods, and will review the outcome

Users are unwilling/unable to take part in UAT; reluctant to make the acceptance decision or not known

Requirements (Sources of Knowledge)

New system replaces a well-understood existing system; users have clear vision of system goals and prefer to document their requirements up-front

Users want to collaborate to jointly define requirements and meet them incrementally or iteratively

Users put the onus of requirements elicitation on the project; requirements and the solution will evolve

Inexperienced users who are unable or unwilling to collaborate with requirements gathering

Requirements Stability New system is a functional replacement of an existing system or a well-defined process (requirements can be fixed early on)

New system replaces an existing system with enhancements or an established (but not necessarily documented process)

New system supports a new business need; business process exists but will change/evolve; users have experience of requirements

New system supports a new business need; business process is not yet known; users have no experience or requirements

Visibility, Formality High visibility/risk to general public; formal progress reporting required at board level; fixed scope and deliverables; formal approvals and sign-offs

High visibility/risk to business; formal progress reporting required; some defined deliverables, some deliverables will emerge/evolve; some approvals and sign-offs

Relatively low business-risk; informal progress reporting is acceptable; partial solution may suffice, incremental/iterative delivery

Potentially, high visibility, high risk project, uncertain impact on the business

External Dependencies More than one or new external suppliers responsible for development (and supplier testing)

Single, known supplier responsible for development (and supplier testing)

In-house development, no external dependencies

Dependencies on external suppliers, their responsibilities or competence not yet known

Etc. Etc. Etc. Etc. Etc.


15

Project types - examplesCerise Structured, waterfall style of project (and includes

COTS projects)Orange Iterative/prototyping style of project using SCRUM in

a formal way and having dedicated resources for the Business Analyst and Tester roles.

Green A project using SCRUM in a less formal way but not having dedicated resources for the Business Analyst and/or the Tester roles.

Blue Blue column statements describe where there is insufficient information available to identify the style of project and the recommendation must be that some further investigation is required.


(Test Strategy as)Agile Interventions

I’m using Scrum/Sprint terminology, but you don’t

have to of course

Interventions (government client example)

• On the following slides, we highlight 8 interventions

• Some are test phases, but some aren’t

No. Activity When?1 Story Challenge As stories are added to the

Product Backlog2 Story Definition As stories are added to a

Sprint BacklogThese activit

ies are

repeated for

each Sprint iterati

on

3 Daily Stand-Up Once per day during the Sprint

4 Story Refinement Occurs throughout the Sprint as new information emerges

5 Developer Testing Occurs throughout the Sprint as the developer codes the stories

6 Integration (and incremental System) Testing

During and at the end of each sprint, including the final sprint

7 System Testing At the end of each sprint, including the final sprint

8 User Acceptance Testing

At the end of each sprint, including the final sprint

9 Non-functional Testing and Pre-Production Testing

Expected to take place on an as-needs basis.


Integration into Existing Code baseAutomated testing

New Code

8. User Test

7. System Test

Sprint 1

Developed Stories Developed Stories Developed Stories

Sprint 3Sprint 2

Sprint Backlog Sprint Backlog Sprint Backlog

1. Story ChallengeSuggest ‘what-ifs’ to challenge new stories and define story headlines

Increasing Scope of Sys. Test and UAT

Increasing Scope of Integration, System and Users Testing

2. Story DefinitionIntroduce scenarios to enhance the Acceptance Criteria

Complete Tests after Final Sprint

Project Level Test Activities(This diagram shows three sprints, but there could be more or fewer)

6. Integration Test6. Integration Test6. Integration Test


New Code

8. User Test

7. System Test

Sprint 1


Sprint 3Sprint 2










New Code

8. User Test

7. System Test

Sprint 1


Sprint 3Sprint 2










New Code

8. User Test

7. System Test

Sprint 1


Sprint 3Sprint 2



Increasing Scope of Int. Sys. and UAT







New Code

8. User Test

7. System Test

Sprint 1


Sprint 3Sprint 2



Increasing Scope of Int. Sys. and UAT






23

Daily ScrumStand-UpMeeting 24

Hours

2-4 WeeksBacklog tasks

expandedby team

Potentially ShippableProduct incrementProduct backlog

As prioritised by Product Owner

Sprint Backlog

4. Story RefinementRefine scenarios to enhance story definition, create system tests as stories, as required

6) Integration/System TestingIncorporate automated unit tests into the CI regime. On weekly basis and at end of Sprint, deploy to System test environment and tester runs system tests.

3. Daily Stand-UpReport anomalies found, stories tested, amended, created

5) Developer TestingPrivate ad-hoc tests and build/run automated unit tests

Test Activities in the Sprint


24


Hours


expandedby team



Sprint Backlog







25


Hours


expandedby team



Sprint Backlog







26


Hours


expandedby team



Sprint Backlog







27

4. Story Refinement (example definition)Objectives To define acceptance criteria for all stories that are included in a Sprint as they are worked

on by development To define scenarios that describe the tests and expected behaviours of the System Improve understanding of the requirement and communicate anomalies to developers To identify System Tests that exercise functionality of multiple stories that can be system

tested in this sprint To assure the completeness for stories in the current Sprint

What’s being tested? Stories to be included in the current SprintDeliverables Refined story definitions with defined acceptance criteria and scenarios, where appropriate

System testsResponsibilities (Orange) Tester – challenges stories by suggesting potential scenarios, new stories, story merges and

splits; performs ad-hoc testing with/on behalf of developers; assures completeness of stories.

Developers – considers stories, evaluates impact on development Product Owner or Analyst – collates feedback and decisions on stories Product Owner – approves changes to stories, accepts completeness of stories Scrum Master – monitors progress; evaluates impact on resources and schedules

Responsibilities (Green) Not performed in Green projectsBaseline Story Guideline (reference 3)Entry Criteria On commencement of the SprintExit Criteria When all stories within a Sprint are considered complete

Queries, anomalies, discrepancies and inconsistencies have been eliminated or explained System Tests appropriate to the Sprint have been defined Definition of acceptance is agreed with Product Owner


Test Automation

Could you create an Agile Test Strategy without automation?

29

Brian Marick’s Testing quadrants


30

Test Automation Pyramid – Lisa Crispin’s version (Google others)

• Pyramid reflects the relative numbers of tests

• Focus on unit/component– Acceptance of

“Services”• GUI are end-to-end• Manual checking the

exception?Intelligent Definition and

Assurance

GUI Test Framework

GUI Test Tool

Browser

Inter/Intranet

HTTP/S

Web Server

App. Server

DB Server

HTTP/S

HTTP Driver

Stories/Scenarios

Unit Test Framework

Test Code

HTTP/S

Unit Test Framework

Test Code

API

4. Programmers write low level HTTP

GET/POST calls through a driver that simulates a browser

3. Non-Technical testers write scripts

Tools Experts write interface

2. Technical Testers code scripts directly 1. Programmers

write unit tests or execute embedded unit tests using a

unit test framework to test components

Where do you automate?

32

Distributed testing

• Use business stories and scenarios/acceptance criteria to validate requirements

• Reuse those stories to feed ‘Acceptance-Driven Development’ BDD/TDD process

• Automated tests are an Anti-Regression tactic

• Automated tests don’t replicate manual tests; think of them as a ‘change trip-wire’ that triggers an alarm, if tripped.


33

Deriving scenarios to test

• To understand feature scope?• To get stakeholder to accept?• To validate the requirement?• To estimate the work to build this

feature?• To system test this feature?• To unit test this feature?

Scenarios are created to meet several goals

Story Challenge

Story RefinementStory DefinitionIteration PlanningSystem Testing

Developer Testing


What’s Left?

Other aspects of test policy

• Definitions (of done etc.)• Incident management• Test automation• Story format e.g. Gherkin• Environment request and

management• Regression testing (at what levels)• Test deliverables and documentation.


The Three Amigos

• Business Analyst– Liaises and manages stakeholders and their

needs– Transforms business requirements into

specification (at multiple levels)

• Developer– Scopes, designs, builds, tests and delivers

features

• Tester– Challenges the thinking on the project– Performs ‘Assurance in the small’.


The tester’s contribution to testing• _________ feature/story acceptance criteria• _________ the developers to unit test (auto)• _________ feature/story acceptance• _________ acceptance test• _________ service/UI level automation• Scope: from low-level detail to system integration• Liaison with integration testers and feedback

• Fill in the blanks yourself; negotiate with your team.


Summary

Close

• Agile test strategy has its place and many aspects of test can be pre-defined

• Importantly, we use a principled approach to deal with the unexpected

• Project profiling can help• Testing as interventions, rather than test phases• The testing role is split at least three ways – the

tester doesn’t own testing – think TESTMASTER• Test automation in the context of Specification

by Example, requirements validation, BDD, TDD.


Agile Test Strategy@

paul_g

erra

rd

Paul [email protected]

gerrardconsulting.com

Agile Test Strategy @paul_gerrard Paul Gerrard [email protected] gerrardconsulting.com.

Documents

agile project

agile strategy

project slide

aspects of test strategy

google agile test strategy

project team

project style

project managers