Page 1
K1
Keynote
5/7/2014 8:30:00 AM
Principles Before Practices:
Transform Your Testing by
Understanding Key Concepts
Presented by:
Randy Rice
Rice Consulting Services, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Page 2
Randy Rice
Rice Consulting Services, Inc.
A leading author, speaker, and consultant with more than thirty years of experience in the field of software testing and software quality, Randy Rice has worked with organizations worldwide to improve the quality of their information systems and optimize their testing processes. He is coauthor (with William E. Perry) of Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems. Randy is an officer of the American Software Testing Qualifications Board (ASTQB). Founder, principal consultant, and trainer at Rice Consulting Services, Randy can be contacted at riceconsulting.com where he publishes articles, newsletters, and other content about software testing and software quality. Visit Randy’s blog.
Page 3
4/29/2014
1
PRINCIPLES BEFORE
PRACTICES
HOW TO TRANSFORM YOUR TESTING PRACTICES WITH BASIC AND ADVANCED PRINCIPLES
RANDALL W. RICE, CTAL
RICE CONSULTING
SERVICES, INC.
© 2013, Rice Consulting Services, Inc.
2
A LITTLE BACKGROUND
• Many years ago, I heard a great
analogy about conveying and
applying knowledge.
• If I were to teach someone how to
wash dishes, I could:
• Teach all the variations needed for
every type of dish, cookware and
utensil, or&
• I could teach principles and let the
person adapt the methods to
whatever they needed to wash.
Page 4
4/29/2014
2
3
THINGS LIKE…
• Rinse off the big foodstuff first.
• Save the really messy dishes until the
end so you don't get everything else in
the sink messy too.
• Use hot water, but not too hot or else
you will scald yourself.
• Be careful with sharp knives in sudsy
water"
4
SIMILARLY, IN TESTING…
• Take some sample tests early and find
where the big problem areas seem to be.
• Don't test the really complex areas at first.
Get your bearings first.
• Have strong tests, but if you make every
test strong, you may not have time to
finish.
• Early testing is good, expect when the
thing you are testing isn't ready even for
early testing.
• Test automation is not automatic.
Page 5
4/29/2014
3
5
COMMON QUESTIONS
• Which tools do we use in
certain situations?
• Which techniques are “best” for
a project?
• How do we adjust tools and
techniques?
6
WHAT WE WANT
No Knowledge
Mastery
Page 6
4/29/2014
4
7
REALISTIC VIEW
No Knowledge
Mastery
“The Dip”
Years
8
TESTING IS VERY NUANCED
• There are small to large variables on any project that require
you to:
• Select certain techniques
• Apply a technique differently
• In essence, all testing is context-driven
• And, there are small things that really make a big difference in
how we may apply a technique.
Page 7
4/29/2014
5
9
FROM CLASSROOM TO PRACTICE
Learning Objectives
Retention
Instruction
Exercises
Visual Aids
Application
10
BUT, IT GETS WORSE
Learning Objectives
Instruction
Visual Aids
ExercisesRetention for multiple people
Application bymultiple people
Passage of time. Needs change, organizations change
Page 8
4/29/2014
6
11
WHY?
• Let’s look at the lowest four levels of the Bloom
Taxonomy:
K4 (Analyze) - AssimilateK3 (Do) - PracticesK2 (Understand) –
Principles, concepts, theory
K1 (Remember) - Facts
The great majorityof Foundation Levellearning objectives
4 Foundationlearning objectives
12
7 GENERAL TESTING PRINCIPLES
• Testing shows the presence of defects
• Exhaustive testing is impossible
• Early testing
• Defect clustering
• Pesticide paradox
• Testing is context dependent
• Absence-of-errors fallacy
Source: ISTQB Foundation Level Syllabus
Page 9
4/29/2014
7
13
THERE ARE MANY OTHER IMPLICIT
PRINCIPLES
• Not every failure is a defect.
• Not every test can or should be
automated.
• Test automation doesn’t replace the
need for manual testing
• It doesn’t matter how good your test
is, if you are testing the wrong
version.
14
CLIMBING K2
• People normally dislike this level of
learning.
• Conceptual
• Theoretical
• Hands-off
• Requires deeper understanding
• Takes longer to build the basis of
understanding
Page 10
4/29/2014
8
15
ANOTHER CHALLENGE
• “Just show us how to use the tool.”
• The students
• “We want a course with all practical
application and little or no theory.”
• Management
16
A LITTLE TIME GOES BY…
• “We don’t know really get how to
design tests for our applications. We
just learned on the case study
example.”
• Students
• “The students passed the exam but we
are still struggling in how to implement
the techniques.”
• Management
Page 11
4/29/2014
9
17
AN EXAMPLE FROM MUSIC
• It is possible to learn to play an
instrument from mimicking someone else.
• The Suzuki Method
• However, you are able to play only specific
songs.
• Professional musicians, even many rock
stars, understand music theory!
• Blues, heavy metal and other styles are all
based on different types of scales and
tunings.
18
WE SEE THE SAME
THING IN SPORTS
• Why do professional baseball
players, golfers, football players need
coaches?
• Objectivity
• Accountability
• Teaching from a broader background
of experience
Page 12
4/29/2014
10
19
WHAT SETS APART PROFESSIONAL
MUSICIANS AND ATHLETES?
• Practice!
• Professional golfers are on the driving
range every day and play most days.
• Football players practice in extreme
conditions.
• Musicians may practice hours for one
show.
20
HOW DO TESTERS
PRACTICE?
• Try new approaches
• Delve deeper into an approach or
technique you already know
• Beta test a new technology
• Play mental games
Page 13
4/29/2014
11
TWO EXAMPLES OF
TEST TECHNIQUES
THAT REQUIRE
DEEPER
UNDERSTANDING
22
EXAMPLE 1 – PAIRWISE TESTING
• Key Principles
• Used when there are distinct numbers of conditions that can be
tested together, but the testing of all combinations would be
infeasible.
• Instead of testing all combinations of conditions, all unique pairs
of conditions are tested.
• Finds defects when:
• The interaction between TWO conditions are incorrect
• The conditions make sense to relate to each other
• Primarily applied by using either orthogonal arrays or pairwise
tools.
• This can be a very interesting and effective way to deal with
testing high numbers of combinations.
Page 14
4/29/2014
12
23
Teacher, why do we need to know this?
24
AN EXAMPLE
The IE Tools>AdvancedOptions window
53 binary conditions1 condition with 3 options1 condition with 4 options
253 = 9,007,199,254,740,992
X 12
108,086,391,056,891,904
Possible combinations ofconditions!
Source: James Whittaker, How to Break Software
Page 15
4/29/2014
13
25
THIS WOULD REQUIRE…
At one second per test execution:
108,086,391,056,891,904
360300,239,975,158,033.067= hours
12,509,998,964,918.04 days
34,273,969,766.9 years
To test all possible combinations!
26
WITH TEST CASE OPTIMIZATION
• The number of tests that
represent unique pairings = 7
• Obtained from using the AllPairs
tool
Page 16
4/29/2014
14
27
WHAT RESEARCH INDICATES
• There have been many studies done
where defects were researched after
the fact.
• Research has shown that the highest
likelihood of an integration defect
relates to interaction between pairs of
items or parameters.
• www.pairwise.org
28
EXAMPLE OF RESEARCH FINDINGS
• [Evaluating FDA recall class failures in
medical devices we established that] [...]
out of the 109 reports that [were] detailed
[enough], 98% showed that the problem
could have been detected by testing the
device with all pairs of parameter settings.
• D.R. Wallace and D.R. Kuhn, 2001
• http://csrc.ncsl.nist.gov/staff/kuhn/final-rqse.pdf
Page 17
4/29/2014
15
29
THE LIKELIHOOD OF TRIPLE-
MODE OR HIGHER DEFECTS
Condition B Condition DCondition CCondition A Condition E
Double-mode defects are more likely&
than Triple-modeor higher defects.
30
EXAMPLE – AN OA.4.3.2.2 ARRAY
011Test case 4
101Test case 3
110Test case 2
000Test case 1
Condition 3Condition 2Condition 1
Page 18
4/29/2014
16
31
EXAMPLE – WITH CONDITIONS
PoolFarmRuralTest case 4
PondHighriseRuralTest case 3
PondFarmUrbanTest case 2
PoolHighriseUrbanTest case 1
Condition 3Condition 2Condition 1
Some paired conditions may make no sense, or may be infeasible
32
ONE MORE SPIN ON
THIS
• A client was using orthogonal arrays to optimize the testing of
stock trades.
• 15 broker types
• 3 trade types
• 2 customer types
• 7 amount levels (bands)
• 4 account types
• This would yield 15*3*2*7*4 = 2,520 tests for all combinations.
• We noticed that the 15 broker types could be safely divided into
5 equivalence classes.
• 5*3*2*7*4 = 840
• This resulted in a 67% reduction in test cases before any pairwise
analysis!
Page 19
4/29/2014
17
33
WHAT WE’VE SEEN
• Instead of going immediately to a tool or technique with a set of categories and conditions, let’s look first to see:
• If the technique makes sense in the situation
• How do we deal with the downsides?
• In this example, can they be used as negative tests?
• What do the stakeholders think of the technique?
• I’ve seen some get very excited
• I’ve seen some get very nervous!
• Are we willing to accept the trade-offs of pairwise testing?
• It becomes very important which categories and conditions we test together.
• We still need an oracle to know the correct outcome.
• Maybe there is a provision for a highrise building on a farm!
34
EXAMPLE #2 – RISK-BASED
TESTING
• Key Principles
• Used when you can’t test the basic layer of functionality and/or when you need to prioritize testing to meet delivery schedules
• The higher the risk, the more intense the test should be
• Helps by:
• Focusing on areas that might experience high levels of loss
• Giving lesser attention to trivial or less important functions
• Tracing and mitigating risks identified during testing
• Can be used in just about any application as long as you are able to intelligently identify and measure risk.
• It is possible to miss some risks and some defects.
• Risk is a possibility, not a certainty
• Our knowledge is limited
• Some things will likely not get tested
Page 20
4/29/2014
18
35
Teacher, isn’t all testing risk-based?
36
Universe of Test Cases
Case 1
Case 2
Case 3
Case 4
Case 5
Case n
Case 6
Case 7
Case 11
Case 8
Case 12
Case 13
Case 10
Case 14
Case 9
CONSIDER THE FOLLOWING
Page 21
4/29/2014
19
37
Universe of Test Cases
Case 1
Case 2
Case 3
Case 4
Case 5
Case n
Case 6
Case 7
Case 11
Case 8
Case 12
Case 13
Case 10
Case 14
Case 9
High-risk cases
Low-risk cases
Moderate-risk cases
THE TEST CASE UNIVERSE
SEGMENTED BY RISK
38
Universe of Test Cases
Case 1
Case 2
Case 3
Case 4
Case 5
Case n
Case 6
Case 7
Case 11
Case 8
Case 12
Case 13
Case 10
Case 14
Case 9
High-risk cases
Low-risk cases
Moderate-risk cases
TEST CASES THAT SPAN RISK LEVELS
In testing this transaction, youspan all levels of risk. How do youprioritize risk in this scenario?
Page 22
4/29/2014
20
39
HOW TO APPLY
• At various levels:
• Product risk
• Functional risk
• Attribute risk – Usability, performance, etc.
• Project risk
• But what about?:
• Residual risks
• Safety critical (when everything is a “high” risk)
• When risks are:
• Difficult to identify
• Change often
• Are an excuse for reduced testing
• Difficult to mitigate and track
This is the practice level!
40
A TIP TO HELP IDENTIFY YOUR
PRACTICES
Area/Principles Implications
Test Automation
• You can’t automate everything
• Automation doesn’t replace people
• The ROI of test automation is repetition
• Select tests carefully to automate
• Consider which skills will need to be built
• To get payback, start with simple tests
that are performed a lot.
Test Design
• Test conditions form the basis of test
cases
• Many things can serve as a basis for test
design, including experience
• Before writing test cases, have a smart
way to identify test conditions.
• If you don’t have a good written basis for
testing, you can create one.
Test Evaluation
• If you can’t observe the outcome of a
test, you can’t evaluate it
• Testing is measurement
• If we can’t define a way to see, report, or
measure the outcome of a test, it does
little good to plan it.
• Since measurement of something
(defects, performance, etc.) is the
deliverable of testing, we should do it
well.
Page 23
4/29/2014
21
HOW TO GAIN
UNDERSTANDING
GOING FROM PRINCIPLE TO PRACTICES
42
A FEW EXPECTATIONS
• This is not easy, automatic or instant
• This is a mental process more than
anything else
• This is also the result of practice and
experience
• This is not a classroom deliverable
• Mentoring and coaching can help you see
blind spots, but they can’t make you
change.
Page 24
4/29/2014
22
43
WHAT REALLY HAPPENS
Instructor says, “Do it like this&” You say&”OK”But think, “I’m not so sure” or“I like my way better.”
44
However, I know I need to improve my game. (I think)
Page 25
4/29/2014
23
45
ACTUALLY, THAT’S FINE
• Each of us know internally
the way we work best.
• An external coach can
provide tips and advice, but
you know what works for you.
• A great resource for this is
the series, “The Inner Game”
by Timothy Gallwey.
46
WHAT WE ARE TALKING ABOUT IS
CHANGE
No Knowledge
Mastery
“The Dip”
Years
Change is hard!
We often fail
Page 26
4/29/2014
24
47
COMPETENCE LEVELS
You do things badly but don’t know it.
You do things badly but do know it.
You do things well but have to think carefully about what you are doing.
You do things well and really don’t think much about the mechanics.
48
Source: Will Taylor
Page 27
4/29/2014
25
49
IN TAKING EXAMS
• Don’t overthink the questions.
• Read it carefully and think it through, but
don’t go back and forth too much on one
question.
• Don’t rely on practice exams.
• They can be HUGELY misleading.
• Instead, ask “Can I demonstrate the
learning objectives?”
• Read the book “Choke”.
• It explains why we get all flustered and
don’t do well on exams.
50
IMPLEMENTING PRACTICES
• Pilot projects are great.
• Learn in the small before going to
the larger scale
• Use the “planned organic” approach
• Have senior management support
• Let others copy your success!
Page 28
4/29/2014
26
51
FINAL THOUGHT
• “the 10,000hr rule is a definite key in
success” ― Malcolm Gladwell, Outliers: The Story of
Success
52
Page 29
4/29/2014
27
53
BIO - RANDALL W. RICE
• Over 35 years experience in building and testing information systems in a variety of industries and technical environments
• ASTQB Certified Tester – Foundation level, Advanced level (Full)
• Director, American Software Testing Qualification Board (ASTQB)
• Chairperson, 1995 - 2000 QAI’’’’s annual software testing conference
• Co-author with William E. Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems
54
CONTACT INFORMATION
Randall W. Rice, CTAL
Rice Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73170
Ph: 405-691-8075
Fax: 405-691-1441
Web site: www.riceconsulting.com
e-mail: [email protected]