Page 1
T4 Class
4/19/2012 9:45:00 AM
"Session-based Exploratory Testing
on Agile Projects"
Presented by:
Bob Galen
Deutsche Bank
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Page 2
Robert ‘Bob’ Galen
Deutsche Bank Global Technology
Bob Galen is an agile coach at Deutsche Bank Global Technology and director, agile solutions at Zenergy Technologies, a North Carolina-based firm specializing in agile testing and leading agile adoption initiatives. Bob regularly speaks at international conferences and professional groups on topics related to software development, project management, software testing, and team leadership. He is a Certified Scrum Master Practicing (CSP), Certified Scrum Product Owner (CSPO), and an active member of the Agile Alliance and Scrum Alliance. In 2009, Bob published Scrum Product Ownership–Balancing Value from the Inside Out, which addresses the gap in guidance toward effective agile product management. You can reach Bob at [email protected] or [email protected]
Page 3
1
Session Based Exploratory Testing on
Agile ProjectsLessons Learned at iContact from 2009 - 2012
Bob GalenVP & Agile Coach, Deutsche Bank
Global Technology
President & Principal Consultant
RGCG, LLC
[email protected]
Copyright © 2012 RGCG, LLC 2
Introduction
Bob Galen
� Somewhere ‘north’ of 20 years experience ☺
� Various lifecycles – Waterfall variants, RUP, Agile, Chaos, etc.
� Various domains – SaaS, Medical, Financial, Computer Systems, and Telecommunications
� Developer first, then Project Management / Leadership, then Testing
� ‘Pieces’ of Scrum in late 90’s; before Agile was Agile
� Agility @ Lucent in 2000 – 2001 using Extreme Programming
� Formally using Scrum since 2000
� Currently Agile Coach at Deutsche Bank Global Technologies in Cary, NC
� Connect w/ me via LinkedIn if you wish9
Bias Disclaimer:
Agile is THE BEST Methodology for Software Development�
However, NOT a Silver Bullet!
Page 4
2
Copyright © 2012 RGCG, LLC 3
Outline
� Introduction – Exploratory Testing and SBET
� iContact Background
� History of our use of SBET
� Some key pro/con of our evolution
� Release mapping
� Q&A
Copyright © 2012 RGCG, LLC 4
Exploratory Testing
Basics
� What is Exploratory Testing—
� Basic dynamics?
� Principles?
� Practices?
� How do you prepare?
� How do you document the testing?
� Pairing?
� Agile implications?
Page 5
3
Copyright © 2012 RGCG, LLC 5
Exploratory Testing
Session Based
� What are the dynamics surrounding sessions—
� What testing do you focus the sessions on?
� Time-boxes?
� Focus guidance – Charters?
� Guidelines?
� Documentation?
� De-brief, discovery sharing?
� Agile Implications?
Copyright © 2012 RGCG, LLC 6
Why is Session Based Exploratory Testing
(SBET) so applicable to agile teams?
� We’ll explore the travels we had at iContact in implementing and executing Exploratory Testing, then wrapping it with Sessions
� Nimble – handling the most critical aspects of the day; changing scope & requirements are relatively easily handled
� Reactive – always testing the “most important thing”
� Collaborative – focus on people and skill rather than completely defined test cases; lightweight; pairing; fosters a whole-team view
� Resilient – to change, to operating conditions (in development environment, interfaces, databases), to what’s working and what’s not
� Simple & Fast – to ramp-up, to begin testing, to train new hires, to document
Page 6
4
Copyright © 2012 RGCG, LLC 7
iContact
Background
� Company formed in late 2003, so approximately 9 years old
� SMB eMail Marketing
� Singular SaaS product; small vs. medium business delineation;
w/options
� LAMP technology stack; heavy PHP use in middle and lower tiers
� Added testers to the teams in early 2009
� So from 2003 – 2008, development-driven testing
� Scrum shop from late 2007
� Cross-functional agile teams introduced early in 2009
� Mid 2011- mature Scrum shop making monthly deployments of
customer value
� Early 2012 – Kanban shop focusing on Continuous Deployment
Copyright © 2012 RGCG, LLC 8
Initial steps…
Early 2009 – early 2010
� We started with a small, singular script “smoke test” that was being
run on a periodic basis
� Adding a few new, functional tests every sprint
� Both/all were manually executed
� Then we broke the smoke test up into Charters and began to run
SBET as a means of
� Distributing the testing beyond a lone tester, across teams, and to gain
transparency
� Isolating Charters towards specific functionally focused Scrum teams
(Create, Integrate, Track, Core, Contacts)
� Reduce execution time and foster test pairing
Page 7
5
Copyright © 2012 RGCG, LLC 9
Follow-on steps…
Early 2010 – Late 2010
� Charters and SBET were our primary testing vehicle; that and
collaboratively driven functional testing within our scrum teams
� The # of Charters began to grow
� Aligning towards each product team
� As we added / extended functionality
� Continued to expand our coverage
� Physically grow as a company
� We had always been a “paired” testing environment
Copyright © 2012 RGCG, LLC 10
Paired SBET
� From the beginning we felt that Exploratory Testing should be a
‘paired’ activity
� Developer + Developer
� Tester + Tester
� Developer + Tester
� Team member + Customer Support member
� Cross-team implications
� As we continued to leverage continuous SBET, the paths got somewhat
‘old’; we needed new eyes & new pairs
� Teams tested each others work; ran each others Charters
� Side-effect that teams needed to x-communicate more; heavily Wiki
based
Page 8
6
Copyright © 2012 RGCG, LLC 11
Paired SBET
Benefits
� The impact of inviting customer support (and other x-functional team
members) was HUGE
� In understanding customer usage
� In adding / changing Charters and focusing domain knowledge
� Developers increased their understanding & empathy for testing
� It’s Hard!
� It’s Long!
� We need to leverage our unit test investments
� It needed broader automation!
Copyright © 2012 RGCG, LLC 12
Early 2011 Evolution
� We’re de-emphasizing SBET as our “do all” testing method. Focus
it on:
� Smoke Tests, for example after a hot-fix
� Sprint-based “Test Fests” that are whole-team based
� Risk-based regression testing
� Starting to write our test cases in Cucumber step format; so that we
can easily convert them / run them as Cucumber tests
� So in each sprint we develop functional tests AND adjust Charters for SBET as
appropriate
� We do a Test Plan per team that shares / exposes the Sprint x Sprint and
Release level testing strategy
� Building a repository of these functional tests
Page 9
7
Copyright © 2012 RGCG, LLC 13
Late 2011 - 2012 Latest Evolution
� Converting our Charters to functional automation
� We’ve used the expanded Charters as a base for developing mostly Cucumber
based automation
� Found the expanded format helpful for the conversion
� Reducing the # of steps and information in the Charters
� Converting them back to a truer form of SBET Charter with little “leading”
information
� Couple with our Test Idea brainstorming / design sessions to drive more new
tests
� Re-analyzing some of our results; leading towards updates of our
testing approaches & artifacts
General Benefits of
SBET
� It was a quick way to begin testing in an agile context
� From scratch; w/o any formal test repository / base
� Inherently collaborative
� Inherently agile, context-based, risk-based; testing the most important
things of the moment
� Pairing worked wonders to foster a “Everyone Owns Quality/Testing”
mindset
� Empathy for testing challenges
� Instilled the right behaviors
� Pulled the client / customer facing folks into our teams
Copyright © 2012 RGCG, LLC 14
Page 10
8
Regression Dynamics of
SBET
� Two-edged sword when using it for Regression
� On the one hand, the Charters get beefier and more prescriptive
� Logging looks very similar code drop over code drop
� Can manage, report, and control x-team execution
� But
� It’s really not the core intent of ET – we’re controlling the exploration
� No matter what you do, it’s not “repeatable”
� Can de-rail your Endgame from a defect discovery perspective
� In hindsight, probably not the best use
Copyright © 2012 RGCG, LLC 15
Planning & Reporting
� Each team is responsible for planning their Charters for a given set
of sessions.
� Planning occurs within Sprint Planning
� Testers define a Test Plan per sprint – point is to drive strategic thinking
within the team
� Identify and x-team dependencies for testing
� Test Fests per sprint & per release
� Clearly defined strategies (why) behind the testing focus is team-driven
� Results simply visible on the wiki as charters / sessions are completed
� War room to coordinate items found, questions, sharing, issues, etc.
Copyright © 2012 RGCG, LLC 16
Page 11
9
Copyright © 2012 RGCG, LLC 17
The Agile Release Train
Synchronized
Iterate
Iterate
Team 1
Team 2
Team 3
Team 4
Iterate
Iterate
Harden Iterate Iterate Iterate
X-team
Harden
Harden
Harden
Harden
Iterate Iterate
Iterate Iterate
Iterate Iterate
Iterate Iterate Iterate Iterate
Iterate
Iterate
Iterate
Internal
Release
External
Release
Docs,
Training,
Support,
UAT,
Comp.
Team n
�
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Copyright © 2012 RGCG, LLC 18
The Agile Release Train
Example: iContact / SaaS Model
Team 1
Team 2
Team 3
Team 4
Iterate
Iterate
Harden
X-team
Harden
Docs,
Training
Harden
Harden
Harden
Iterate
Iterate
Production
Release
Team 10�
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
3 weeks / 15 days 4-5 days
Rinse
&
Repeat
Environment
Evolution Dev + QA QA -> Staging Production
SBET, Exploratory –
Regression Testing
Page 12
10
Retrospectives
� Leverage the teams’ Retrospectives as a place to discuss:
� SBET – testing strategy
� Timing
� Coverage
� X-team interactions
At a scrum team and release level.
� Reflect on our results and implications to
� Functional tests (additions, retiring, new suites, etc.)
� Extending our Charters
Copyright © 2012 RGCG, LLC 19
Test Ideas
� We’ve recently begun to use Rob Sabourin’s views towards Test
Idea generation - http://www.amibugshare.com/articles/
� Brainstorming
� Formation
� Prioritization
� Execution & Retrospective
� Very useful to dovetail the ideas to the execution within Charters
� Sort of an idea laboratory for your ideas
� Fast feedback for fine-tuning
Copyright © 2012 RGCG, LLC 20
Page 13
11
Wiki-based examples of our
SBET artifacts
� We leverage a wiki (Confluence) for our SBET charter storage,
session notes, defects, and repetitive run logs
� We build in some forward learning notes, where applicable, for our
retrospectives
� So here are some examples9
Copyright © 2012 RGCG, LLC 21
Overall Charter - Findings
Each charter has a wiki page
associated with tracking results.
Results would be captured per session
including very specific bugs, notes,
and concerns / questions.
Copyright © 2012 RGCG, LLC 22
Page 14
12
Charter Team Assignments
Team and individual assignments would
be made on a TestFest or event basis.
This page identified the notion of:
“Charter Owners” and identified groups,
individuals, pairs, that were
recommended to execute the charters.
This varied as our application risk-based
test planning drove us in different
directions. So very dynamic; but, with
some solid core of team familiarity per
charter�
Copyright © 2012 RGCG, LLC 23
Charter Test Index / Management
This wiki page rolls up all charters for overall test
execution.
We also used it to monitor execution and connect
to individual results.
You can see notes regarding “Automation
Assisted”. As we developed functional and other
automation, we “connected” it to the charters &
sessions.
So automation execution became “part of” the
sessions. Not so to de-focus the exploration, but
as a means of augmenting the focus and round-
trip evolution of automation on�
Charter-based boundaries.
Copyright © 2012 RGCG, LLC 24
Page 15
13
Individual Charter - Logging
Each charter has a wiki page associated with
tracking results. Results would be captured per
session including:
� Software version, release target
� Who executed the charter
� Bugs found
� Script of “travels” in executing this session and
specific guidance around
� (1) how to reproduce bugs and
� (2) what would be good to convert into
functional test cases.
Copyright © 2012 RGCG, LLC 25
Individual Charter - Logging
Each charter has a wiki page associated with tracking
results. Results would be captured per session
including:
� Another important aspect of our logging was that all
history of execution was in a single wiki page. Yes,
this could get unwieldy over time, but we didn’t find
that to be the case�yet
� What we found is that sessions could be “directed”
based on historical notes on
� Stability vs. flakiness
� Previous “coverage”
� Previous “discoveries”
Copyright © 2012 RGCG, LLC 26
Page 16
14
Individual Charter – Expanded Steps
Each charter has a wiki page associated with
tracking results. Results would be captured per
session including:
� Testing guidance including entry, exit
conditions and test steps
� We expanded on the steps NOT to inhibit
exploration,
� But as an effort to allow the charters and
execution notes to allow us to develop more
formal functional test cases
� That would be “extracted” from the SBET
efforts�
Copyright © 2012 RGCG, LLC 27
The Future – Next Steps
� We’re moving back to the traditional “exploratory” nature
of ET and Session-Based ET
� That and team-based Test Idea brainstorming and
� Agile test planning
� Are driving our testing creativity and value-added tests
� Kanban evolution – continuous flow
� Continuous Integration – @ every check-in
� Off of Trunk and notion of Feature Branches (MMF)
� Continuous Deployment – weekly deployments to
production
Copyright © 2012 RGCG, LLC 28
Page 17
15
The Future – Next Steps
� We’re realizing our test
automation strategy
and gaining significant
automation
coverage9
� Allowing to next focus
on agile performance
testing, UX integration,
and continuously
improving tests
Copyright © 2012 RGCG, LLC 29
Wrapping up…
� SBET can be a very effective agile approach
� Beginning / Building team new to agile
� Ongoing team maturity
� Be careful not to ‘overload’ your Charters; allow/support for true
‘exploration’
� Pairing can be powerful
� Create wiki-based, very simple management & tracking
� Leverage Charters for your automation development
� Prioritization & experimentation
� Driving time savings & risk mitigation
Copyright © 2012 RGCG, LLC 30
Page 18
16
Contact InfoBob Galen
Principal Consultant, RGalen Consulting Group, L.L.C.
Director of Agile Solutions, Zenergy Technologies,
Director, Agile Practices, iContact
Experience-driven agilefocused training, coaching
& consulting
Cell: (919) [email protected]
[email protected]
Blogs:
Project Times - http://www.projecttimes.com/robert-galen/
Business Analyst – BA Times -http://www.batimes.com/robert-galen/
My Podcast on all things ‘agile’ - http://www.meta-cast.com/
Scrum Product Ownership – Balancing Value From the Inside Out published by RGCG in 2009.
Copyright © 2012 RGCG, LLC 31