Testing as a processg prequirements Planning&
Control
olog
i
specificationpreparation test plan
tekn
o
(abstract)Test
generation
abstract/logicaltest casesspecification
tions
t generation
Testimplemen-
tation
concrete/executableltest cases
orm
at
implementationTest
execution verdict/logs
Info
quality reportCompletion
CISS
Test Automationrequirements Planning&
Control
olog
i
specificationpreparation test plan
tekn
o
(abstract)Test
generation
abstract/logicaltest casesspecification
tions
t generation
Testimplemen-
tation
concrete/executableltest cases
orm
at
implementationTest
execution verdict/logs
Info
quality reportCompletion
CISS
1 Manual Testing1. Manual Testing+ easy & cheap to start+ flexible testing- expensive every execution
i i- no auto regression testing- ad-hoc coverage- no coverage measurement- no coverage measurement
M. Utting ”Practical Model Based Testing”
2 Capture Replay Testing2. Capture-Replay Testing+ flexible testing+ flexible testingexpensive first execution+ auto regression testing- fragile tests break easily- ad-hoc coverage- no coverage measurement
3 Script Based Testing3. Script-Based Testing+/- test impl. = programming+ automatic execution+ auto regression testing+ auto regression testingfragile tests break easily?
(depends on abstraction)(depends on abstraction)- ad-hoc coverage- no coverage measurement
4 Keyword Driven Testing4. Keyword-Driven Testing+ abstract tests+ automatic execution+ auto regression testing+ auto regression testing- robust tests- ad-hoc coveragead hoc coverage- no coverage measurement
5 Model-Based5. Model BasedTestingg
+ abstract tests+ automatic execution+ auto regression testing+ auto regression testing+ auto design of tests+ systematic coverage+ systematic coverage+ measure coverage of model and requirements- modelling overhead
Script based TestingScript-based Testing
Test inputs and observations (verdictassignment) programmed in scripts in g ) p g pdedicated or general-purpose languages.
+ Repeatable (for regression testing)+ Repeatable (for regression testing)+ Data driven testing: Fixed scripts, each parameterized
with different datad e e da a– Mixes test harness and logical test cases– Must be updated when specs (or impl) changeMust be updated when specs (or impl) change– Very lengthy– Good programmers may write well-structuredGood programmers may write well structured
reusable test code but ”it is just test code”
Keyword Driven Testingy g• Script based automation where test case
d i i t d f t tidesign is separated from automation– Focus on actions user/environment can do on
objects in SUT (at different abstraction levels)• ”Actions” appear in scripts,pp p• ”Action-code” implements the action
• aka “Action-word” testing (or table driven• aka Action-word testing (or, table driven testing), < 1994
• Black-box, subsystem, accept tests• Automated execution using a “framework”.Automated execution using a framework .
Keyword Driven Testingy g+ Concise, flexible, maintainable, + Read-/writeable by non-programmers
÷ Expressiveness of a scripting languageC t l t t d l t ti– Control structures and complex computations
– (Branching in test cases, complex data, matching of expected results)
Keyword Driven TestyObject Keyword Value
Eg. Sequence
LoginDialog Login ”badUserId”, ”GoodPass”
VerifyLogin ”Login failed”VerifyLogin Login failed
LoginDialog Login ”goodUserId”, ”badPass”
VerifyLogin ”Login failed”
LoginDialog Login ”goodUserId”, ”GoodPass”
VerifyLogin ”welcome”
W l Di l h L i ” U N ” ” d”WelcomeDialog changeLogin ”newUserName”,”newpwd”
confirm ”goodUserId”, ”newpwd”
confirmChange ”Change Failed”
KWD ProcessTest Expert Automation Expert
http://www.softwaretestinghelp.com/wp-content/qa/uploads/2010/01/keyword-driven-testing.pdf
Automation FixtureAutomation FixtureKeywod ScriptKeywod Script
Open(Scripts)
Engine/driverGeneric Libs
(fixtures)
While(!EOF){ Keyword, data=readFile();Case(keyword){L i id t > b it(d t )(fixtures) Login: widget->submit(data);
verifyLogin: …
Application Libs(keyword functions)
}Close()
SUT
}
Tools• SAFS, open source• EMOS Framework, open source• FIT/FitNesse open source• FIT/FitNesse, open source• Certify, Worksoft• Unified TestPro, SDT• TestFrame LogicaCMG• TestFrame, LogicaCMG• TestArchitect, LogiGear• Tools with built-in keyword support:
TestQuest Pro– TestQuest Pro,– QuickTest Professional
ExampleExamplehttp://localhost:8080/FitBookExamples.Chapter4TestingActions.Fig6TestChatServer
http://fit.c2.com/wiki.cgi?FitBookhttp://my.safaribooksonline.com/0321269349
ComparisonpCriteria Manual Script KWD Model-Based
Cost Development L M M LCost Development L M M L
Maintenance H? H L
Tools/equip L M M Hq p
Time Time to first test L M M M-H
Re-execution H L L L
Maintenance L H H L
Learn-ability
Skills L H M* H*ability Maturity L M M-H H
Readability L L M H*
I t t / Ch ll i L M M HInterest / Challenging L M M H
Effective-ness
Bug detection H L L H
Tracability L L M HTracability L L M H
Coverage L L M H
ReferencesReferences• http://en.wikipedia.org/wiki/Keyword-driven_testing• http://www.cs.waikato.ac.nz/~marku/mbt/• http://www.onestoptesting.com/automation-
framework/keyword-driven-testing/• http://www.softwaretestinghelp.com/wp-
content/qa/uploads/2010/01/keyword-driven-testing.pdf• http://www.musala.com/press/PR_2008_03_17/pr/Keyw
d d i %20T ti dford-driven%20Testing.pdf• Mark Fewster and Dorothy Graham, Software Test
Automation 1999 (Chapter 22)Automation, 1999 (Chapter 22) • M. Utting Practical Model-based testing