6 Secrets of Exploratory Testing in Embedded Systems Shmuel Gershon Tel Aviv, July 2013
6 Secrets of Exploratory Testing in
Embedded Systems
Shmuel Gershon
Tel Aviv, July 2013
Who are you people?Testers
Explorers
Programmers
ManagersAficionados
In the wrong conference?
Curious
All of the others in one
Web
Embbeded
2
What am I doing here
3
Is this the tester you hire?
• PERSONAL FREEDOM
• RESPONSIBILITY of the individual tester
• OPTIMIZE THE QUALITY OF THE WORK
• LEARNING + DESIGN + EXECUTION + RESULT
INTERPRETATION...
• THROUGHOUT THE PROJECT
4
Remember our Super Tester?"A style of software testing that emphasizes
the PERSONAL FREEDOM
and RESPONSIBILITY of the individual tester
to OPTIMIZE THE QUALITY OF THE WORK by
treating test-related
LEARNING, test DESIGN, test EXECUTION, and
test RESULT INTERPRETATION as mutually
supportive activities that run in parallel
THROUGHOUT THE PROJECT" (Cem Kaner)
5
Exploratory Testing by Cem:"A style of software testing that emphasizes
the PERSONAL FREEDOM
and RESPONSIBILITY of the individual tester
to OPTIMIZE THE QUALITY OF THE WORK by
treating test-related
LEARNING, test DESIGN, test EXECUTION, and
test RESULT INTERPRETATION as mutually
supportive activities that run in parallel
THROUGHOUT THE PROJECT" (Cem Kaner)
6
Embedded Systems:• An embedded system is a computer system
designed for specific control functions within
a larger system, often with real-time
computing constraints. It is embedded as
part of a complete device often including
hardware and mechanical parts.== It is “hidden”, “untouchable”,
inside other systems!
7
Test This:
8
Now Test This:
9
Now Test This One:
10
And Finally Test This One:
11
Common Direction
Automated Tests
SDK
API
Automation Suite
Handless Operation
Seems as if reality forces you into it…
12
Why Breaking the Pattern?
• Stiffness
• Long Feedback Loop
• Borderless infrastructure
• Premature learning
• ...
• ...
• ...
13
Exploring the UntouchableRovers on celestial bodies (suchas the Mars Rovers) cannot be
easily remote-controlled since the
radio signal speed is slow (asignal from Mars to Earth takesbetween 3 and 21 mins). A rovers
is capable of operating
autonomously with little
assistance from ground control
for data acquisition, but requires
human input for identifying
promising targets, and
determining how to position
itself.14
Exploring the Hidden
15
16
How to Break the Pattern?
17
6 Principles
Principles
Probes
Verbs
Cut the Middleman
Judge Not
Stay in Control
• Simple Autonomy
Traduttore, Tradittore
• Caveat Testor
18
1) Probes
Have a system that gives you information, all
the information, unfiltered, from all the system.
You can filter it elsewhere.
- Signals
- Logs / States
- Results
- Timers / Watchdogs
Probes
19
2) Verbs
Name your actions: Put your hands inside the system.
- If it can do something, you can do it too.
- Not only direct inputs – controls the environment
- Sometimes the way to control it is to know how the system deals with environments and inputs- Your simulation + the system rules = state
Verbs
20
Cut the Middleman
3) Cut the Middleman
The system should allow tests to flow, without
stopping to re-automate.
- Short action and reaction cycles increase
engagement and cognition
21
Judge Not
4) Be cautious in judgment
Computers help get results, testers interpret
results.
- You can’t possibly teach a computer how an
unknown bug looks like
22
5) Stay in Control
If you are exploring, you need the steering
wheel.
- Cruise Control may be for known areas
- Automatic Pilot may be for getting started
Stay in Control
• Simple Autonomy
23
Traduttore, Tradittore
• Caveat Testor 6) Traduttore, Tradittore
We are building an abstraction, and all
abstractions are leaky.
“All models are false but some models are
useful.”George E. P. Box
24
The only summary slide you need to remember:
“It’s all about building an automated system for manual tests”
25
(Free!) Bonus Benefits!
• Find Limits
• When Automated Tests are needed they
can use the same Manual infrastructure
• Debug, Troubleshooting, Bug Isolation
26
More Bonus!Automation Layers
Test Structure
System Structure
Test Cases Automation
27
Credits, Questions, Answers
• Michael Stahl
• Gary Draishpits
• Cem Kaner
• Michael Bolton
• Doron Blazer
• Doron Maliach
• Yonatan Tidhar
• Bat-Sheva Honigstein
• Intel Sensors Teamo My Manager too!
• QA & Test 28