Michael Bolton and James Bach Critical Thinking for Testers 1 Critical Thinking for Testers James Bach http://www.satisfice.com [email protected]Twitter: @jamesmarcusbach Michael Bolton http://www.developsense.com [email protected]Twitter: @michaelbolton Call this “Checking” not Testing Observe Evaluate Report Interact with the product in specific ways to collect specific observations. Apply algorithmic decision rules to those observations. Report any failed checks. operating a product to check specific facts about it… means
Critical thinking is the kind of thinking that specifically looks for problems and mistakes. Regular people don't do a lot of it. However, if you want to be a great tester, you need to be a great critical thinker. Critically thinking testers save projects from dangerous assumptions and ultimately from disasters. The good news is that critical thinking is not just innate intelligence or a talent—it's a learnable and improvable skill you can master. James Bach shares the specific techniques and heuristics of critical thinking and presents realistic testing puzzles that help you practice and increase your thinking skills. Critical thinking begins with just three questions—Huh? Really? and So?—that kick start your brain to analyze specifications, risks, causes, effects, project plans, and anything else that puzzles you. Join James for this interactive, hands-on session and practice your critical thinking skills. Study and analyze product behaviors and experience new ways to identify, isolate, and characterize bugs.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Michael Bolton and James Bach Critical Thinking for Testers
Interact with the product in specific ways to collect specific observations.
Apply algorithmic decision rules to those observations.
Report any failed checks.
operating a product to check specific facts about it…
means
Michael Bolton and James Bach Critical Thinking for Testers
2
Acquiring the competence, motivation, and credibility to…
Testing is…
create the conditions necessary to…
so that you help your clients to make informed decisions about risk.
evaluate a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference, including…
operating a product to check specific facts about it…
Michael Bolton and James Bach Critical Thinking for Testers
3
Bolton’s Definition of Critical Thinking
• Michael Bolton
Wait, let’s try something simple…
Can we agree? Can we share common ground?
“There are four geometric figures on this slide.” “There is one square among those figures.”
“The square is shaded in blue.”
Michael Bolton and James Bach Critical Thinking for Testers
4
“Pass Rate” is a Popular Metric
00.10.20.30.40.50.60.70.80.9
12
/1
2/3
2/5
2/7
2/9
2/1
1
2/1
3
2/1
5
2/1
7
2/1
9
2/2
1
2/2
3
2/2
5
2/2
7
3/1
3/3
3/5
Pass Rate
Pass Rate
The Beginning of Critical Thinking
Michael Bolton and James Bach Critical Thinking for Testers
5
Why Don’t People Think Well?
“Steve, an American man, is very shy and withdrawn, invariably helpful but with little interest in people or in the world of reality. A meek and tidy soul, he has a need for order and structure, and a passion for detail.”
Is Steve more likely to be
a librarian? a farmer?
Reflex is IMPORTANT But Critical Thinking is About Reflection
REFLEX
REFLECTION
Faster Looser
Slower Surer
get more data
System 2
System 1 See Thinking Fast and Slow, by Daniel Kahneman
Michael Bolton and James Bach Critical Thinking for Testers
6
Exercise: Calculator Test
“You are carrying a calculator.
You drop it!
Perhaps it is damaged!
What might you do to test it?”
Assumptions vs. Inferences • An inference is something we treat as true based on evidence.
• An assumption is a something that we treat as true regardless of evidence.
• A premise is an assumption that begins a chain of reasoning. All logic is based on premises.
• Testers question assumptions & premises and gather data for better inferences.
Ass
um
pti
on
Inferen
ce
weak
inference
plausible
assumption
Michael Bolton and James Bach Critical Thinking for Testers
7
Levels of Assumptions
Reckless Assumptions that are too risky regardless of how they are managed. Obviously bad assumptions. Don’t make them.
Risky Assumptions that might be wrong or cause trouble, but can be okay with proper management. If you use them, declare them.
Safe Assumptions that are acceptable to make without any special management or declaration, but still might cause trouble.
Obvious Assumptions so safe that they cause trouble only IF you manage them, because people will think you are joking, crazy, or insulting.
It is silly to say “don’t make assumptions.”
Instead say “let’s be careful about risky assumptions.”
What makes an assumption more risky? 1. Consequential: required to support critical plans and activities. (Changing the
assumption would change important behavior.)
2. Unlikely: may conflict with other assumptions or evidence that you have. (The assumption is counter-intuitive, confusing, obsolete, or has a low probability of being true.)
3. Blind: regards a matter about which you have no evidence whatsoever.
4. Controversial: may conflict with assumptions or evidence held by others. (The assumption ignores controversy.)
5. Impolitic: expected to be declared, by social convention. (Failing to disclose the assumption violates law or local custom.)
6. Volatile: regards a matter that is subject to sudden or extreme change. (The assumption may be invalidated unexpectedly.)
7. Unsustainable: may be hard to maintain over a long period of time. (The assumption must be stable.)
8. Premature: regards a matter about which you don’t yet need to assume.
9. Narcotic: any assumption that comes packaged with assurances of its own safety.
10.Latent: Otherwise critical assumptions that we have not yet identified and dealt with. (The act of managing assumptions can make them less critical.)
Michael Bolton and James Bach Critical Thinking for Testers
8
What are We Seeing Here?
• Mental models and modeling are often dominated by unconscious factors.
• Familiar environments and technologies allow us to “get by” on memory and habit.
• Social conventions may cause us to value politeness over doing our disruptive job.
• Lack of pride and depth in our identity as testers saps our motivation to think better.
Themes
• Technology consists of complex and ephemeral relationships that can seem simple, fixed, objective, and dependable even when they aren’t.
• Testers are people who ponder and probe complexity.
• Basic testing is a straightforward technical process.
• But, excellent testing is a difficult social and psychological process in addition to the technical stuff.
“A tester is someone who knows that things can be different.”
Jerry Weinberg
Michael Bolton and James Bach Critical Thinking for Testers
9
Don’t Be A Turkey
• Every day the turkey adds one more data point to his analysis proving that the farmer LOVES turkeys.
• Hundreds of observations support his theory.
• Then, a few days before Thanksgiving…
Based on a story told by Nassim Taleb, who stole it from Bertrand Russell, who stole it from David Hume.
Graph of My Fantastic Life! Page 25! (by the most intelligent Turkey in the world)
Well
Bein
g!
DATA
ESTIMATED
POSTHUMOUSLY
AFTER THANKSGIVING
“Corn meal a little off today!”
Don’t Be A Turkey
• No experience of the past can LOGICALLY be projected into the future, because we have no experience OF the future.
• No big deal in a world of stable, simple patterns.
• BUT SOFTWARE IS NOT STABLE OR SIMPLE.
• “PASSING” TESTS CANNOT PROVE SOFTWARE GOOD.
Based on a story told by Nassim Taleb, who stole it from Bertrand Russell, who stole it from David Hume.
Michael Bolton and James Bach Critical Thinking for Testers
10
The Regression Testing Fantasy
“I rerun my old tests to ensure that nothing has broken.”
The Regression Testing Fantasy
“I rerun my old tests to ensure that nothing has broken.”
Michael Bolton and James Bach Critical Thinking for Testers
11
The Regression Testing Reality
“We run a smattering of old checks to ensure that they still find no bugs... And we assume that any bug not found is also not important.”
How Do We Know What “Is”?
“We know what is because we see what is.”
We believe we know what is because we see what we interpret as signs that indicate what is based on our prior beliefs about the world and our (un)awareness of things around us.
Michael Bolton and James Bach Critical Thinking for Testers
12
How Do We Know What “Is”?
“If I see X, then probably Y, because probably A, B, C, D, etc.”
• THIS CAN FAIL: – Getting into a car– oops, not my car.
– Bad driving– Why?
– Bad work– Why?
– Ignored people at my going away party– Why?
– Couldn’t find soap dispenser in restroom– Why?
– Ordered orange juice at seafood restaurant– waitress misunderstood
– McDonald’s clerk told me her husband died.
Remember this, you testers!
Michael Bolton and James Bach Critical Thinking for Testers
13
Models Link Observation and Inference
• A model is an idea, activity, or object…
• …that represents another idea, activity, or object…
• …whereby understanding the model may help you understand or manipulate what it represents.
31
such as an idea in your mind, a diagram, a list of words, a spreadsheet, a person, a toy, an equation, a demonstration, or a program
such as something complex that you need to work with or study.
- A map helps navigate across a terrain. - 2+2=4 is a model for adding two apples to a basket that already has two apples. - Atmospheric models help predict where hurricanes will go. - A fashion model helps understand how clothing would look on actual humans. - Your beliefs about what you test are a model of what you test.
Models Link Observation & Inference
• Testers must distinguish observation from inference!
• Our mental models form the link between them
• Defocusing is lateral thinking.
• Focusing is logical (or “vertical”) thinking.
32
My model of the world
“I see…”
“I believe…”
Michael Bolton and James Bach Critical Thinking for Testers
14
Testing against requirements
is all about modeling.
“The system shall operate at an input voltage range of nominal 100 - 250 VAC.”
“Try it with an input voltage in the range of 100-250.”
Poor answer:
How do you test this?
The Nature of Critical Thinking
• We call it critical thinking whenever we systematically doubt something that the “signs” tell us is probably true. Working through the doubt gives us a better foundation for our beliefs.
• Critical thinking is a kind of de-focusing tactic, because it requires you to seek alternatives to what is already believed or what is being claimed.
• Critical thinking is also a kind of focusing tactic, because it requires you to analyze the specific reasoning behind beliefs and claims.
Michael Bolton and James Bach Critical Thinking for Testers
15
The Nature of Critical Thinking
• “Critical thinking is purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based.” - Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction, Dr. Peter Facione
(Critical thinking is, for the most part, about getting all the benefits of your “System 1” thinking reflexes while avoiding self-deception and other mistakes.)
Why You Should Care
Technology is way more tricky than regular life.
But testers are not supposed
to get tricked.
Michael Bolton and James Bach Critical Thinking for Testers
16
How many test cases are needed to test the product represented by this flowchart?
Michael Bolton and James Bach Critical Thinking for Testers
28
Visual Strategy Annotation
Web Server
Database Layer
App Server
Browser
Observation or Oracle
Control or Modify Data or Object
Activity or Tool
Force Fail Path
Web Server
Database Layer
App Server
Browser
error monitor
- error monitor
- coverage analysis
dropout test
dropout test
data generator
table
consistency
oracle
history
oracle
man-in-middle
performance
data
high ET
stressbots
Michael Bolton and James Bach Critical Thinking for Testers
29
Web Server
Database Layer
App Server
Browser
Beware Visual Bias!
• browser type & version
• cookies
• security settings
• screen size
• client-side scripts & applets
• usability
• specific functions
Example data extraction and transformation system
Michael Bolton and James Bach Critical Thinking for Testers
30
Example dual head radio testing
Example dual head radio testing
Michael Bolton and James Bach Critical Thinking for Testers
31
One way to cope with really complex diagrams
• Consider making a special diagram that includes only the things that are worth testing, then put the annotations as bullets on the bottom…
DB
!!Hook
PTTHead (P)
PCMic
Covert
DB
!!Hook
PTTHead (S)
PCMic
Covert
Splitter
(optional)
Power DB
Torso
(optional)
PTT
PCMic
DB != DB, DB == DB
Disconnect/Connect
Start/Stop/Restart/Reset
On hook/off hook
PTT Y/N
Signal arriving at antenna
No testing for extender box?!
Coverage Screen Match
Contrast/Volume Independence
Muted/Unmuted
Reset on Disconnect
Reset on System Error
Pops
Oracles Happy path
Spam test
Connection tests (failover)
DB interactions
Pairwise interactions
Head interactions
Time (leave it sitting)
Ideas
PTT Mic
Mic
PTT
Extender box Extender box
Extender box Extender box
Michael Bolton and James Bach Critical Thinking for Testers
32
Exercise: Overlapping Events Testing
• You want to test the interaction between two potentially overlapping events.
• How would you test this?
time
Event A
Event B
Critical thinking about practices: What does “best practice” mean?
• Someone: Who is it? What do they know?
• Believes: What specifically is the basis of their belief?
• You: Is their belief applicable to you?
• Might: How likely is the suffering to occur?
• Suffer: So what? Maybe it’s worth it?
• Unless: Really? There’s no alternative?
• You do this practice: What does it mean to “do” it? What does it cost? What are the side effects? What if you do it badly? What if you do something else really well?
Michael Bolton and James Bach Critical Thinking for Testers
33
Beware of…
• Numbers: “We cut test time by 94%.”
• Documentation: “You must have a written plan.”
• Judgments: “That project was chaotic. This project was a success.”
• Behavior Claims: “Our testers follow test plans.”
• Terminology: Exactly what is a “test plan?”
• Contempt for Current Practice: CMM Level 1 (initial) vs.
CMM level 2 (repeatable)
• Unqualified Claims: “A subjective and unquantifiable requirement
is not testable.”
Look For…
• Context: “This practice is useful when you want the power of creative testing but you need high accountability, too.”
• People: “The test manager must be enthusiastic and a real hands-on leader or this won’t work very well.”
• Skill: “This practice requires the ability to tell a complete story about testing: coverage, techniques, and evaluation methods.”
• Learning Curve: “It took a good three months for the testers to get good at producing test session reports.”
• Caveats: “The metrics are useless unless the test manager holds daily debriefings.”
• Alternatives: “If you don’t need the metrics, you ditch the daily debriefings and the specifically formatted reports.”
• Agendas: “I run a testing business, specializing in exploratory testing.”