Dare to Explore… Discover ET!
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Dare to Explore…
Discover ET!
Outline • What Exploratory Testing (ET) is and isn’t • Dissecting Exploratory Testing – Learning – Designing – Executing
• Fitting it within the Agile Context • Adoption Experience • Q & A
What does Exploratory Testing mean to YOU?
Testing Pyramid
Integration Tests
Unit Tests
Developer-centric Are we building the code right?
Business-centric Are we building the right code?
Adaptation of Mike Cohn's test automation pyramid
Exploratory Testing
GUI Tests
Acceptance Tests (API layer)
veri
fica
tion
What is Exploratory Testing?
Test-related learning
Test design
Test execution
Test result interpretation
Exploratory Testing is a mindset
Embrace continuous improvement of test quality
Enabling choice, not constraining it
Tests are experiments for learning
Push the boundaries, don’t execute routine checks
Embrace continuous learning
Exploratory Testing… IS NOT IS Doing random stuff to see what happens
A rigorous investigative practice
Repetitive Adaptive
Replacement for predesigned testing
Complimentary to conventional techniques
About producing comprehensive test cases for later use
Creative & more free-style
Why ET?
Two sides of Testing
Tested Checked
(scripted manual + automated)
Explored
“No matter how many tests we write, no matter how many cases we execute, we always find the most serious bugs when we go off the script” Explore It! Reduce Risk and Increase Confidence with Exploratory Testing by Elizabeth Hendrickson
Checking
• Designing scripts with expected results upfront
• Verifies what was working still works
• Well documented and repeatable
Exploring
• Learning, designing and executing at the same time
• Tests the system capabilities, boundaries
• Can be sparsely documented and hard to repeat
Exploration with a general mission without specific step-by-step instructions Effectiveness relies on the tester’s knowledge, skills, and experience Simultaneously learning about the software under test while designing and executing tests, using feedback from the last test to inform the next.
Characteristics of ET
ET is a fit where… • Need for manual testing approach that’s adaptable
because the SUT changes rapidly
• Requirements change often, and the specifications are vague or incomplete
• Writing detailed test scripts for a manual test effort doesn’t make sense
• Need to provide rapid feedback on a new product or feature
Key testing problem
• We must test a product whose purpose is emerging and/or changing
• Coded without fully specified requirements upfront
Agile fit?
ET: Nuts & Bolts
The Flow…
• Focus the exploration
• Learn about the system/features
• Design tests with interesting variations
• Attack and report
Unscripted doesn’t mean unprepared
Test charters bring Focus
Charters define the area(s) we are testing and the kind of vulnerabilities we’re looking for - Features - Suspected risky areas of the system - Coupling points
Meandering without direction/purpose is getting lost
Sample Charter
• Target: What are you exploring? Feature, component, interface etc. • Resources: Data set, technique, configuration etc. • Information: What are you hoping to find? Is it security
vulnerability, performance, capability, usability etc.
Explore <target> With <resources> To discover <information>
Explore It! Reduce Risk and Increase Confidence with Exploratory Testing by Elizabeth Hendrickson
Explore the Foo Form Fee Payment functionality using active accounts in the Demo environment to discover how the payment interface responds.
Template Example
Learn the general shape
What does the system do?
Input, Store, Compute, Output
“I know that I do not know” ~ Socrates
Design tests with interesting variations
The narrower the view, the wider the ignorance
• Varying sequences of actions and varying timing
• Test error handling by making required resources
unavailable or locked, or to break connections
• Inventing personae to generate extreme scenarios
• ...
Use heuristics to guide test design on the fly
• Test Heuristics Cheat Sheet: (Elisabeth Hendrickson, James Lyndsay, and Dale Emery)
http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
• How To Break Software: “17 Attacks” (Alan A. Jorgensen, James A. Whittaker)
Ideas for Executing & Recording All code is guilty, until proven innocent.
• Explore Early, Explore Often • Time-box the session • Line up the team • Surfacing potential problems/risks quickly is the goal • Work in pairs or groups, whenever possible • Be distracted by anomalies and new ideas • Avoid test repetition • Document your findings in an appropriate way (low
overhead is key)
Running ET Sessions
Moving Quality Concerns
Upstream
Previous State Bug bash and IV&V executed just before the release
Target State Using a whole team approach to exploring early and often, we hope to surface and address risks sooner, and that can lead to better software, fewer surprises, and greater confidence
Approach
• Synchronized Cadence across 8 Agile teams
• Test Charter Autonomy • Time-boxed Exploratory Sessions
- Explore & Record Observations - Summarize and Report
Current Enterprise Comprised of many disparate data sources and systems, which leads to:
Inconsistent Data Data is defined, captured and interpreted inconsistently throughout the Enterprise
Ad-‐Hoc and Near Real-‐Time Analysis Limita<ons Limited ability to conduct ad-‐hoc and near real-‐>me analysis, delaying >mely decision-‐making
Cross-‐Func<onal Domain Limita<ons Inability to provide a cross-‐func>onal view across data domains easily or completely
Lack of Agility in Response to Evolving Business Model Lack of agility to adjust data procurement and delivery in response to a rapidly evolving business model
Lack of Data Across the Enterprise Data gaps exist due to limited understanding of current and an>cipated data usage and applica>on across the Enterprise
Notes From the Trenches Challenges and solutions transitioning a traditional test team into an exploratory test team on a large, heavily interfaced system.
I’ve a feeling we’re not in Kansas anymore Challenge: Changing a mindset Solution: Framework, cadence, training, discussions, and metrics
Why do you want access to my system? Challenge: You need “visas” to explore Solution: You need allies, and strong ones
Yet another build!!!? Challenge: Continuous Delivery means continuous deployments Solution: Test in parallel with and after production deployments
What is truth? Challenge: Acceptance criteria vs. Desired Outcome Solution: Find someone who cares
Boldly going where no one has gone before Challenge: Indigenous population may be indigent Solution: “Marketing” brochure and chocolate chip cookies
Concluding thoughts
Benefits
Unleashes tester’s creativity and skill Ability to perform focused testing without comprehensive documentation Rapid flow of feedback to the team
Based on the summary by Itkonen and Rau>ainen
Challenges
Managing test coverage can be difficult Quality of testing highly dependent on expertise and skill of tester Repeatability of found defects challenging at times
Summary
• Surface and address risks sooner • Can help identify the big bugs • Adaptive • Complementary to scripted testing • Agile fit
30
Resources and References
• Bach, James. “What Is Exploratory Testing?” • Marick, Brian. “A Survey of Exploratory Testing” • Hendrickson, Elisabeth. “Explore It!” • Jorgensen, Alan. “How to Break Software (with examples)” • Kaner, Cam, Tinkham, Andy. “Learning Styles and Exploratory
Testing” • Bolton, Michael, et al. “Exploratory Testing Dynamics” • Itkonen, Juha. “Do test cases really matter? An experiment
comparing test case based and exploratory testing”
32
Related Documents
