Page 1
MF AM Tutorial
4/29/13 8:30AM
Seven Keys to Navigating Your Agile
Testing Transition
Presented by:
Bob Galen
RGalen Consulting
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Page 2
Bob Galen
Bob Galen is an agile coach at RGalen Consulting and director of agile solutions at Zenergy Technologies, a North Carolina-based firm specializing in agile testing and leading agile adoption initiatives. Bob regularly speaks at international conferences and professional groups on topics related to software development, project management, software testing, and team leadership. He is a Certified Scrum Master Practicing (CSC), Certified Scrum Product Owner (CSPO), and an active member of the Agile Alliance and Scrum Alliance. Bob published Scrum Product Ownership–Balancing Value from the Inside Out, which addresses the gap in guidance toward effective agile product management. Contact Bob at [email protected] or [email protected] .
Page 3
1
Keys for Transitioning to Agile
TestingMyths & Realities from the Trenches
Bob GalenPresident & Principal Consultant
RGCG, LLC
[email protected]
Seven Keys for Navigating your
Agile Testing TransitionMyths & Realities from the Trenches
Bob Galen – RGCG
[email protected]
Mary Thorn – Deutsche Bank
[email protected]
Fifteen
Page 4
2
Copyright © 2013 RGCG, LLC 3
Introduction
Bob Galen
� Somewhere ‘north’ of 30 years experience ☺
� Various lifecycles – Waterfall variants, RUP, Agile, Chaos2
� Various domains – SaaS, Medical, Financial Services, Computer & Storage Systems, eCommerce, and Telecommunications
� Developer first, then Project Management / Leadership, then Testing
� Leveraged ‘pieces’ of Scrum in late 90’s; before ‘agile’ was ‘Agile’
� Agility @ Lucent in 2000 – 2001 using Extreme Programming
� Formally using Scrum since 2000
� Currently an independent Agile Coach (CSC – Certified Scrum Coach, one of 50 world-wide; 20+ in North America) � at RGCG, LLC and Director of Agile Solutions at Zenergy Technologies
� From Cary, North Carolina
� Connect w/ me via LinkedIn and Twitter if you wish2
Bias Disclaimer:
Agile is THE BEST Methodology for Software Development�
However, NOT a Silver Bullet!
Introduction
Mary Thorn
� A VP of QA and Test Manager at Deutsche Bank
Global Technologies in Cary, North Carolina,
� Mary Thorn has a broad testing background that spans
automation, data warehouses, and web-based systems
in a wide variety of technologies and testing
techniques.
� During her more than fifteen years of experience in
healthcare, HR, agriculture, and SaaS-based products,
� Mary has held manager and contributor level positions
in software development organizations.
� She has a strong interest in agile testing methodologies
and direct experience leading agile teams through
Scrum adoption & beyond.
Copyright © 2013 RGCG, LLC 4
Page 5
3
Copyright © 2013 RGCG, LLC 5
Outline – Myths & Realities
Introduction
1. Transforming your Team
2. Automation
3. Developers & Automation
4. Developers Testing
5. Test Planning & Scripts
6. Testing within the Sprint
7. Exploratory Testing
8. Role of Testers
9. Developer to Tester Workflow
10. Managing Agile Testers
11. Test Metrics
12. Retrospectives – The Secret Sauce
13. Continuous Improvement
14. The Customer
15. Agile Requirements – The Product Backlog
Copyright © 2013 RGCG, LLC 6
#1, Transforming your team
� Myth: You need all programmers or highly technical
testers when you move to agile
� Reality: A mix is best –
� Manual, domain-centric and technical skills
� Some programming / scripting skills
� Soft / collaborative skills
� Reality: And throw out all of that Developer-to-Tester
ratio ‘stuff’.
Page 6
4
Copyright © 2013 RGCG, LLC 7
#2, Automation
� Myth: You need 100% automation to start
agile testing.
� Reality: You simply need to have a strategy AND
doggedly pursue automation where it makes sense
� Make it part of the Backlog and work it every sprint
� Reality: There are some excellent Open Source tools
that supplement agile automation development
� Reality: The Agile Test Automation Pyramid is the right
overall strategy
Agile Test Automation PyramidMike Cohn; Lisa Crispin & Janet Gregory
http://behaviordrivendevelopment.wikispaces.com/Testing
Copyright © 2013 RGCG, LLC 8
Page 7
5
Copyright © 2013 RGCG, LLC 9
#3, Developers & Automation
� Myth: QA designs, writes & runs all of the test
automation
� Reality: Everyone should be responsible for automation
� Developers need to minimally attend to Unit Level
� Participate in any framework or re-use development
� Writing ‘glue’ code – fixtures, step files, etc.
� Reality: It also extends into your Build & Continuous
Integration systems
� All automation should be ‘wired’ into CI
� Dashboards, trending, lava lamps, etc. for all to see2
#4, Developers Testing
� Myth: Developers can’t test their own code—they’re not
independent enough nor skilled enough to do it properly.
� Reality: We need to stop stereotyping team members,
their strengths and their abilities.
� Developers can absolutely test their own code.
� Some are better at it than others
� Pair with them to help test appropriately
Copyright © 2013 RGCG, LLC 10
Page 8
6
Copyright © 2013 RGCG, LLC 11
#5, Test Planning & Scripts
� Myth: You don’t need to plan
� (it just happens2)
� and you don’t need functional test cases
� (automation takes care of everything2)
� Reality: Plans help the team focus on the risk-based
testing required within an iteration AND across a release
� Reality: Scripts (test cases) help formalize and drive
your testing;
� Absolutely required in regulatory environments
� Reality: You’ll never actually automate every test
Copyright © 2013 RGCG, LLC 12
#6, Testing within the Sprint
� Myth: You simply need to run 100%
of the tests within the constraints of the Sprint2that’s
“Agile”
� Reality: Rarely possible in most contexts.
� You first need a high-degree of automation and business support
(for example: equipment costs)
� Very mature test automation and CI / CD environments
� Reality: Most agile teams adopt some sort of risk-based
testing approach for within the sprints
� Dealing with Technical Test Debt
� Then leverage Hardening / Stabilization pre-release sprints
Page 9
7
Copyright © 2013 RGCG, LLC
The Agile Release Train
Synchronized
Iterate
Iterate
Team 1
Team 2
Team 3
Team 4
Iterate
Iterate
Harden Iterate Iterate Iterate
X-team
Harden
Harden
Harden
Harden
Iterate Iterate
Iterate Iterate
Iterate Iterate
Iterate Iterate Iterate Iterate
Iterate
Iterate
Iterate
Internal
Release
External
Release
Docs,
Training,
Support,
UAT,
Comp.
Team n
�
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
13
Copyright © 2013 RGCG, LLC
The Agile Release Train
Example: eCommerce / SaaS Model
Iterate
Iterate
Team 1
Team 2
Team 3
Team 4
Iterate
Iterate
Harden
X-team
Harden
Docs,
Training
Harden
Harden
Harden
Iterate Iterate
Iterate Iterate
External
Release
Team 8�
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
10 days 10 days 5 + 2 days
Rinse
&
Repeat
Environment
Evolution Dev + QA Dev + QA QA -> Staging Production
14
Page 10
8
Copyright © 2013 RGCG, LLC 15
The Agile Release Train
Example: iContact / SaaS Model
Team 1
Team 2
Team 3
Team 4
Iterate
Iterate
Harden
X-team
Harden
Docs,
Training
Harden
Harden
Harden
Iterate
Iterate
Production
Release
Team 10�
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
3 weeks / 15 days 4-5 days
Rinse
&
Repeat
Environment
Evolution Dev + QA QA -> Staging Production
SBET, Exploratory –
Regression Testing
Brainstorm…
“Your” Agile Release Train
� Either individually or in small groups from the same
company2
� Take a few minutes and think about your current release
constraints
� timing, customers, domain, competition, # of teams, technology,
etc.
� Design a release train model for your organization
� Overlay it with testing activities, plans, and milestones
� Present it to your larger table/group; gain feedback &
adjust
� Be prepared to share2
Copyright © 2013 RGCG, LLC 16
Page 11
9
#7, Exploratory Testing
� Myth: There is no place for Session Based Exploratory
Testing in agile contexts.
� Reality: ET and SBET are a beautiful complement to
agile testing.
� Helping nurture pairing & collaboration across teams and
functions
� Defining new (more valuable) test cases
� Quickly gaining quality & usability feedback
Copyright © 2013 RGCG, LLC 17
Copyright © 2013 RGCG, LLC 18
#8, Role of Testers
� Myth: That the testers alone own quality & testing
practices within each team and sprint
� Reality: The testers foster a “Whole Team” view
towards quality—focusing less on “Testing” and more on
“Quality Practices & the Customer”
� Serving as guides for the team; Testing the “hard bits”
� Facilitating exploratory testing sessions—finding more
interesting / valuable tests
� Working with the Product Owners—are we solving the
customers problems?
Page 12
10
Copyright © 2013 RGCG, LLC 19
#9, Developer to Tester Workflow
� Myth: There is always a hand-off from developers to
testers; usually quite late in the sprint. That’s simply the
“way of things” in software development.
� Reality: Scrummer-fall is alive and well2but, Wrong!
Teams need to swarm on their work, as flow &
throughput matter the most.
� WIP limits and close proximity / collaboration help establish a
healthy tempo of developer & tester pairing
� Micro-handoffs – testing as development progresses!
� Do you log bugs? Or do you fix bugs?
Copyright © 2013 RGCG, LLC 20
#10, Managing Agile Testers
� Myth: The functional test manager is
in charge of deciding how, who, when , etc. for the test
team.
� Reality: You still absolutely need functional leadership
within agile teams;
� However, it’s focused towards quality practices, strategy &
coaching, and handling impediments / escalations
� Encouraging transparency, transforming metrics & reporting
� Supporting & protecting the teams
� Encouraging risk-taking, innovation & creativity (Slack Time)
Page 13
11
Levels of Done-Ness Criteria
Activity Criteria Example
Basic Team
Work Products
Done’ness criteria Pairing or pair inspections of code prior to check-in; or
development, execution and passing of unit tests.
User Story or
Theme Level
Acceptance Tests
Development of FitNesse based acceptance tests with the
customer AND their successful execution and passing.
Developed toward individual stories and/or themes for sets
of stories.
Sprint or
Iteration Level
Done’ness criteria Defining a Sprint Goal that clarifies the feature
development and all external dependencies associcated with
a sprint.
Release Level
Release criteria
Defining a broad set of conditions (artifacts, testing
activities or coverage levels, results/metrics, collaboration
with other groups, meeting compliance levels, etc.) that IF
MET would mean the release could occur.
21Copyright © 2013 RGCG, LLC
Copyright © 2013 RGCG, LLC 22
#11, Test Metrics
� Myth: You can and should move
forward reporting everything exactly as
you have before.
� Including any ‘dysfunctional’ metrics that your process and/or
PMO dictates.
� Reality: The metrics should change immediately.
� From QA and Test centric towards Team-Centric metrics (Value,
Throughput, Quality, Team)
� Stop reporting out on “Testing”; it’s irrelevant!
� This effects planning as well—estimation, progress, risk, etc.
� Contribute quality-centric Information radiators to the mix
Page 14
12
Copyright © 2013 RGCG, LLC 23
Brainstorm…
Morphing your Metrics
� Get together in small groups of
of 4-6.
� What are you measuring today? Why?
� How are they driving your success and behaviors?
� As you move to agile, what can/should you be
measuring in the 4 key areas:
� Value, Quality, Throughput & Predictability, and Team
� How will you change your existing metrics? What
behaviors are you trying to inspire?
#12, Retrospectives:
The Secret Sauce
� Myth: Testers are “Second Class” citizens who don’t
play an active part in the project & team
� Reality: There are many places to “make a difference”
� Getting the 800 lb. Gorillas out on the table; Showing courage;
telling truth
� Fostering continuous improvement within the team
� Setting the example; showing vulnerability—admitting you’re
wrong
� Team listening; active planning; dependencies; pairing
� Risk-taking; Failure!
Copyright © 2013 RGCG, LLC 24
Page 15
13
#13, Continuous Improvement
� Myth: We’re generally ‘stuck’ in our
approaches so just accept them and do
the “best you can”.
� Reality: Continuous improvement is everyone’s
responsibility—to engage, suggest, take ownership of
current results, explore root causes, etc.
� Active participation in your teams Retrospectives is a key way to
guide quality, testing, and customer-centric improvements.
� Courage!
Copyright © 2013 RGCG, LLC 25
#14, The Customer
� Myth: Business Analysts capture
customer requirements and testers test them for
completeness.
� Reality: You need to begin to partner with the Customer
– Stakeholders – Product Owners to produce software
that solves the their problems.
� Move to the “front” and help define & refine User Stories with your
Product Owner
� Actively participate in Sprint Reviews
� Show value for automation; placing test investments in the
Backlog
Copyright © 2013 RGCG, LLC 26
Page 16
14
Copyright © 2013 RGCG, LLC 27
#15, Agile Requirements –
The Product Backlog
� Myth: We can’t start testing until the requirements are
finished or stable; no matter how ‘agile’ we are.
� Reality: Hogwash! Get over it2
� Ambiguity and incompleteness need to become your friend and
ally.
� As does working with your Product Owners and Customers to
help define the requirements
� Realizing that the requirements (User Stories) are only complete
at the end of each sprint.
Agile Test Transformation Strategy:
3 Pillars of Agile Quality
Copyright © 2013 RGCG, LLC 28
Page 17
15
3 Pillars of Agile Quality
Copyright © 2013 RGCG, LLC
29
Development &
Test Automation
• Pyramid-based
Strategy: (Unit +
Cucumber + Selenium)
• Continuous Integration
• Attack legacy technical
debt in the Backlog
• Visual Feedback –
Dashboards
• Actively practice ATDD
and BDD
Software Testing
• Risk-based testing:
Functional & Non-
Functional
• Test planning @
Release & Sprint levels
• Exploratory Testing
• Standards – checklists,
templates, repositories
• Balanced across
manual, exploratory &
automation
• Agile-centric Metrics
Cross-Functional
Team Practices
• Team-based Pairing
• Stop-the-Line
• Code Reviews &
Standards
•
• Active Done-Ness
• Aggressive Refactoring
• User Stories – 3 Amigo
based Conversations
• Building the ‘Right’
Solutions
3 Pillars of Agile Quality
Copyright © 2013 RGCG, LLC
30
Development &
Test Automation
• Pyramid-based
Strategy: (Unit +
Cucumber + Selenium)
• Continuous Integration
• Attack legacy technical
debt in the Backlog
• Visual Feedback –
Dashboards
• Actively practice ATDD
and BDD
A central part of agile adoption is focusing on CI, 3-
tiered automation development, and Dashboards to
begin incrementally building coverage for faster
feedback on changes.
In the interim, Hardening or Stabilization Sprints and
having a risk-based Release Train concept help
It’s important that Test or QA not ‘own’ the tooling or
all of the automation efforts. The strategy can come
from Test, but the automation development is best
left to the team.
Mature teams invest in automation as part of Done-
ness and continually on their backlogs
Page 18
16
3 Pillars of Agile Quality
Copyright © 2013 RGCG, LLC
31
Software Testing
• Risk-based testing:
Functional & Non-
Functional
• Test planning @
Release & Sprint levels
• Exploratory Testing
• Standards – checklists,
templates, repositories
• Balanced across
manual, exploratory &
automation
• Agile-centric Metrics
Exploratory Testing (Charter / Session based and
paired) can be an incredibly effective way to
establish a whole-team, collaborative view towards
quality and testing. It also emerges new tests.
Leverage ‘plans’ as a whole-team collaboration
mechanism2and do plan.
Do not measure testing or tester progress; instead,
measure throughput, output, sprint outcomes, and
done-ness escapes at a team level.
You need a balanced test team; not everyone needs
to be able to program. But everyone needs to be
skilled testers.
Agile testing is a Risk-Based play in every Sprint and
across a release sequence. Don’t forget your
techniques!
3 Pillars of Agile Quality
Copyright © 2013 RGCG, LLC
32
Cross-Functional
Team Practices
• Team-based Pairing
• Stop-the-Line
• Code Reviews &
Standards
•
• Active Done-Ness
• Aggressive Refactoring
• User Stories – 3 Amigo
based Conversations
• Building the ‘Right’
Solutions
One of the hardest areas to get ‘right’ culturally. It
needs leadership alignment from Quality/Testing to
Product to Development and a consistent voice of
whole-team approaches.
This is where LEAN lives, where whole-team
collaboration happens, where professionalism and
craftsmanship are held dear.
I like the view of testers becoming the VOC,
champions of quality, and consistent questioners of
what is being build. Are we solving the right
problems2as simply as possible.
MMF, MVP, MMP, etc.
And yes Virginia, there ARE standards and
consistency!
Page 19
17
Copyright © 2013 RGCG, LLC 3333
Software Testing
Strategies
� It ALL starts with empowering testers AND creating a
Whole-Team view towards Quality
� Critical Early Steps:
� Creating a sense of empowered Functional Team
� Applying Testing Standards across all teams
� Deploying Exploratory Testing across all teams
� Defining a core set of Agile KPI / metrics
� ACTIVE participants in Sprint Planning
Copyright © 2013 RGCG, LLC 3434
Cross-Functional Team Practices
Strategies
� Training
� Agile / Lean in general, Story writing, Acceptance, Unit testing,
etc.
� Teaming – for example: feedback or 5 Dysfunctions / Trust
� Critical Early Steps:
� Coaches & Scrum Masters to reinforce: Pairing / Swarming; WIP
Limits across teams
� Define prescriptive and aggressive Done-Ness for ALL teams
� Implement coding standards & Crucible / code reviews across the
center (appropriate for technology stacks)
� Release Planning BEFORE allowing a team to start Sprint #1
� Backlogs have Bug + Refactoring + Automation targets (20%)?
Page 20
18
Copyright © 2013 RGCG, LLC 3535
Organizational Quality
Strategies
� Continuously communicate your unified Vision
� Your strategy must be aligned/shared across:
� Development, Quality/Testing, and Product
� Keep working your strategy across the pillars
� Don’t get stuck with too narrow a focus (easy road)
� Make your strategy visible (Information Radiators)
� Show progress (Ex: burn up of test automation coverage2across tiers)
� Visualize organizational impediments to your Agile Quality
strategies
� Attack them!
� Quarterly read-outs on progress, plans and adjustments
� Listen to your teams
� Celebrate successes!
What will be (your) agile strategy when you get
back home?
� Either in groups or individually
� Consider the 3-Pillars discussion
� Consider your current team / organization agile ‘state’
� Define a broad, 3-pillar view towards some immediate
focus points when you get back into the office
� What will you focus on? Why?
� How will you communicate the need for change?
� How will you measure results?
� What will come immediately afterwards?
Copyright © 2013 RGCG, LLC 36
Page 21
19
Wrapping up…
Agile is the best thing that’s
happened to testers since�
The Great Depression
� Whole Team view
� Testing, Metrics, Automation
� Planning, Reporting, Quality
� Facilitate feedback
� Multi-tiered automation
� Just-in-Time, risk-based testing
� Continuous improvement
� Trust the Team
� Retrospective
Copyright © 2013 RGCG, LLC 37
Contact Info
Bob GalenPrincipal Consultant,
RGalen Consulting Group, L.L.C.
Director of Agile Solutions,
Zenergy Technologies,
Experience-driven agile focused training, coaching & consulting
Contact: (919) 272-0719
[email protected]
[email protected]
www.rgalen.com
Blogs
Project Times -
http://www.projecttimes.com/robert-galen/
Business Analyst – BA Times -
http://www.batimes.com/robert-galen/
My Podcast on all things ‘agile’ -
http://www.meta-cast.com/
38Copyright © 2013 RGCG, LLC 38
Page 22
20
Additional Topics
Copyright © 2013 RGCG, LLC 39
Two Pillars of Lean ‘Thinking’
Respect for
People
� Customer, Employees,
Vendors2
� Develop your teams
� Trust & coach
� No wasteful work
Continuous
Improvement
� Embrace change,
challenge everything
� Kaizen – small, incremental
change
� Kaikaku – larger scale,
fundamental
40
From http://www.leanprimer.com
Copyright © 2013 RGCG, LLC
Page 23
21
Agile Testing QuadrantsBrian Marick; Lisa Crispin & Janet Gregory
Copyright © 2013 RGCG, LLC 41
Exploratory testing
Scenarios
Usability testing
UAT
Alpha / Beta
Unit tests
Component tests
API tests
Functional tests
Story tests
Examples
Prototypes
Simulations
Performance testing
Load testing
Security testing
Non-functional requirements
Q1
Q2 Q3
Q4
Automated &
Manual
Automated &
Manual
Manual
Automation,
Tools, and
Manual
Business Facing
Technology Facing
Supporting the Team
Critiq
ue the Product
Copyright © 2013 RGCG, LLC 42
10 Tenets of Agile Testing Jean Tabaka, Rally Software
1. The system always runs
2. Stop the line, vs. logging
defects
3. If it’s not tested, it’s not
“Done”
4. Testing comes first, not last
5. Finding defects after
Development is “Done” is too
late
� Continuous Integration
� Lean – fix it now!
� Early feedback; Earned
Value
� Collaborative testing, focus
on building in quality
� Early feedback; fix it now!
Page 24
22
Copyright © 2013 RGCG, LLC 43
10 Tenets of Agile Testing Jean Tabaka, Rally Software
6. “Development Complete” is
meaningless
7. Use testing, not analysis, to
explore requirements
8. Automation is “how” not a
“whether” or “when”
9. Tests are your second most
detailed specification
10. Testers are Customer-
Developer liaisons
� Whole Team complete view
– no “partial credit”
� Executable requirements
� Automate all testing;
feedback
� Code is first; later is
traditional specifications
� VOC; guide effective team
collaboration; ask the right
questions
Copyright © 2013 RGCG, LLC 44
10 Commitments of Agile Testing Jean Tabaka,
Rally Software
1. We commit to not moving forward if a hole is found
through root cause analysis without first writing a test
2. We commit to not relying solely on just automated
testing or just manual testing
3. We commit to not sitting behind a QA wall (no
boundaries!)
4. We commit to not allowing a code complete without
test code harness complete
5. We commit to not waiting for a test phase but rather
working in smaller and smaller pieces, sooner and
sooner
Page 25
23
Copyright © 2013 RGCG, LLC 45
10 Commitments of Agile Testing Jean Tabaka, Rally Software
6. We commit to not testing one iteration after
development is “Done”
7. We commit to not allowing surprises to accumulate for
large end-to-end testing (“mock it now”)
8. We commit to not leaving the riskiest tests to the end
9. We commit to being an equal participant with the
customer and the developer in defining “Doneness”
10.We commit to not taking this oath lightly