Model-Based Testing with Keywords

W9 Concurrent Class

10/2/2013 1:45:00 PM

"Model-Based Testing with

Keywords"

Presented by:

Hans Buwalda

LogiGear Corporation

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Hans Buwalda

LogiGear

An internationally recognized expert in testing, Hans Buwalda is a pioneer of keyword-driven

test automation, an approach now widely adopted throughout the testing industry. Originally

from the Netherlands, Hans is the CTO of LogiGear, directing the development of the successful

Action Based Testing™ methodology for keyword-driven test automation and its supporting

TestArchitect™ toolset. Prior to joining LogiGear, Hans served as project director at CMG (now

CFI).

9/20/2013

1

© 2013 LogiGear Corporation. All Rights Reserved

Hans Buwalda

LogiGear

STARWEST 2013, Presentation W9

Anaheim, Wednesday Oct 2nd, 2013

Model-Based Testing

with Keywords


Topics for this presentation

Action Words (Action Based Testing)

Testing with models

Using actions to describe models

Non-deterministic applications (like games)

Examples

9/20/2013

2


Action Words / Action Based Testing

Around since 1994 presented for the first time at Eurostar in 1996

Offers testers the ability to create a "language" to

express tests and expected results benefit: flexibility

Automation concentrates on implementing the

language, not the tests benefit: maintainability

The language concept works for straightforward

cases, but also for expressing and controlling

models


The test developer creates tests in "test modules", using actions

Each action consists of a keyword and arguments

The automation task focuses on automating the actions

Each action is automated only once

Action Based Testing

number name quantity

new product P-9009 Sledge Hammer 5

number quantity

add quantity P-9009 20



number quantity

check quantity P-9009 34

actions, each with a

keyword and

arguments

"34" is the

expected value

here

read from top

to bottom

fragment from a test with actions

9/20/2013

3


Example of business level test module

Consists of an (1) initial part, (2) test cases and (3) a final part

Focus is on business functionality, with a clear business scope

Navigation details are avoided

TEST MODULE Car Rental Payments

user

start system john

TEST CASE TC 01 Rent some cars

first name last name car

rent car John Doe Ford Escape

rent car John Doe Chevvy Volt

last name amount

check payment Doe 140.4

FINAL

close application


Example of an interaction level test module

Lay-out the same, with an initial part, test cases and a final part

Interaction details that are the target of the test are not hidden

Focus is not on business ("is the payment amount correct"), but on

interaction ("do I see the payment amount")

TEST MODULE Screen Flow

user

start system john

TEST CASE TC 01 Order button

window button

click main create order

window

check window exists new order

FINAL

close application

9/20/2013

4


Variables and expressions with keywords

This test does not need an absolute number for the

available cars, just wants to see if a stock is updated

As a convention we denote an assignment with ">>"

The "#" indicates an expression

TEST CASE TC 02 Rent some more cars

car available

get quantity Chevvy Volt >> volts




car expected

check quantity Chevvy Volt # volts - 2


Data driven testing with actions

The test lines will be repeated for each row in the data set

The values represented by "car", "first" and "last" come

from the selected row of the data set

DATA SET cars

car first last value

Chevvy Volt John Doe 40000

Ford Escape Mary Kane 22500

Chrysler 300 Jane Collins 29000

Buick Verano Tom Anderson 23000

BMW 750 Henry Smyth 87000

Toyota Corolla Vivian Major 16000

TEST CASE TC 03 Check stocks

data set

use data set /cars

car available

get quantity # car >> quantity


rent car # first # last # car

car expected

check quantity # car # quantity - 1

repeat for data set

9/20/2013

5


Identify modules

Test Development Plan

Test Cases

Test

Objectives

Test Module 1

Test Cases

Test

Objectives

Test Module 2

Test Cases

Test

Objectives

Test Module N

Actions

. . .

ACTION AUTOMATION

Break down

Automate

for each module

decide what approach

to use, for example

models


Models

You can make models for just about everything automated systems, business processes, organizations,

the universe, conference presentations, . . .

A model is an abstraction it represents certain aspects of a subject matter in a more

accessable form, it hides others

when using a model beware of what it is that is hidden

Examples of formats: picture, set of mathematical

formulas, a simulation program, toy, game

9/20/2013

6


Models for testing

For a system under test, a model can usually

describe: structure

behavior

data

rules, calculations

Scope of the modeling: subject matter under test

system under test

the test

Models and testing: models can be used to generate test cases

however, actions can be used to describe models


Model: Implicit Specification of a Test

Using a "model" to get some variation in an automated

test

Models can be simpler than you think you might have used them already

Examples: business rules, like "entitled for social security when 67"

formulas, like "sales tax is 0.09 times price"

state-transition diagrams

monkey tests: random or semi-random moves through a GUI to spot

issues

Models can have an "oracle" that can help verify if a test

result is ok example for a mapping application: driving times both ways are

normally not far apart*

*source: Harry Robinson, Microsoft Bing Maps, this model-based test was executed with LogiGear's TestArchitect

9/20/2013

7


State Models

Among the most commonly used models

Particular good to describe behavior of a system

A system is describes as a collection of "states", plus

input events that cause "transitions" between the states a transition typically has outputs that can be tested

states can be fixed or derived from equivalence classes, for example

including parameters to further details them out

example of a state: "monster attacks player", which monster can then

be a parameter

To apply a state model: your test system must be able to detect which state the system is in

be able to generate the events that migrate you through the states


State Transition Diagram

doors closed

first check

second check

alert state

self destruct doors open

initial

signal

correct

correct

incorrect

incorrect

signal

tampering

time-out

9/20/2013

8


State Transition Table

NR STATE EVENT OUTPUT NEXT

1 doors closed signal beep first check

2 doors closed tampering hi-lo alert state

3 first check correct code green doors open

4 first check incorrect code hi-lo second check

5 doors open close signal beep doors closed

6 second check correct code green doors open

7 second check incorrect code hi tone self destruct

8 self destruct anything boom oblivion

9 alert state time out hi tone self destruct

10 alert state signal beep first check

Car Safety System, Model 007, Luxury Version*

*certain insurance restrictions might apply, please check with your agent


When to use Models?

Functionality under test is complex like an airplane control system

The system under test has non-deterministic

(unpredictable) behavior like a game

Functionality is specified in a way suitable for models like business rules, formulas or tables/matrices

be aware whether the test model is independent enough from

anything used in the application under test

Look for clear benefits if you consider using a model worth the efforts and the relative high complexity

don't make it a hobby

9/20/2013

9


Using actions to create models

Actions can be a way to describe your intentions

quickly and clearly

One can also use models to generate action

based tests, however: such models will then be more static

tests will be longer and harder to read

Using actions for modeling can: be used as a quick ad-hoc technique for specific cases

don't require investment and buying and learning model-

based tooling

create models that dynamically respond to behavior of an

application under test


Example: An Insurance System

Optimism Insurances Inc. "Let us do your damage!"

Car Insurance Guidelines and Conditions

…

Article 11 – Policy Acceptance

11a Applicants with no more than one accident in the past three years are

accepted.

11b When two accidents are recorded in which the applicant was the guilty

party, the application is only accepted if the applicant is female.

11c An applicant with three or more accidents in the past two years is not

accepted.

11d Applications from testers are rejected (a car is not a system under test).

…

9/20/2013

10



A table to hold the business rules, mapping inputs to states.

start state accidents gender end state

application 0 --- accept

application 2 f accept

application 2 m refuse

application 3 --- refuse



Handle states. The check in the first state is an "oracle", it may not

be able to calculate the premium exactly, but rather verify that a

premium is within a certain tolerance:

name argument argument

state accept accidents car value

accidents car value

check premium # accidents # car value

end state

name

state refuse

text

check message Not eligible for insurance.

end state

9/20/2013

11



File with some input data (you could also consider monkey style random

data generation):

data set customers

name accidents gender car value

Owen 0 m 19500

Mary 3 f 32010

Kim 1 f 15322

Bob 2 m 21399

Niyati 0 f 27666



This scenario uses the data from the data set to try car insurance

applications. Based on the table "try state" will either jump to the

state or do nothing:

name accidents gender

enter customer # name # accidents # gender

owner value

enter car value # name # car value

state argument argument

try state accept # accidents # car value

state

try state refuse

9/20/2013

12



To start the test:

table scenario

run table action acceptance table test insurance


Non Deterministic Systems

Non-deterministic: difficult to determine/predict outcomes

Typical reasons for non-determinism: system depends on external events, like the weather

system has deliberate random behavior (games, security)

subject matter of the system is too complex, like rocket science

Non-determinism poses in particular a challenge for

automation manual testing usually is easier than automation to test non-

deterministic behavior

however, automation can exhaust more variations if alternative paths

are involved

9/20/2013

13


Handling Non Determinism

If possible still make your test predictable set the environment (or basic data) yourself

shut off the random generator (test it separately)

Select out the parts that are not non-deterministic and test

them in separately

Define what it is that you want to test and focus on that

(what is it that you do know): "when 10 monsters are shot you move on to the next level"

"the company logo should be in the upper left corner of the page:

within 10% of the page size"

Use models


Handling Multi-media, some strategies

Consider manual testing

Find criteria and objects that you CAN test colors, volume levels, text recognition, etc

Investigate: try to find handles that represent

what is going on file names, numeric handles, api calls

Use a "play list" approach to reduce human

testing

9/20/2013

14


The "Play List" Approach

The testers focus on a play list items are played/shown while the machine tells what it is

the tester should see/hear

the tester acknowledges/falsifies

the system stores information on the items that were ok

Next time only the changed items go on the play

list again in most systems this will only be a limited amount of items

Using this technique can free up the model, to

just work with handles


Example: a Quiz Game

The game asks questions and poses challenges a "challenge" for this example is a calculation (like "what is

32*3")

After each successful reply from the player it will

randomly either ask another question or pose

another challenge

The questions are randomly taken from a set of

available ones

The calculation questions are also non-

deterministic

9/20/2013

15


Example: a Quiz Game

Prompt: Here is a question:

Prompt: Name a leading West Coast event on testing.

Answer: StarWest

Prompt: Very good

Prompt: Let's calculate

Prompt: What is 23*2?

Answer: 46

Prompt: Very good

…


Example: a Quiz Game, a Table

Use of a table for questions and answers (in this example we simplify bye

presuming answers are given verbatim):

table questions

question answer

Name a leading West Coast conference

on testing.

Star West

What approach separates testing and test

automation.

Action Based Testing

Which city hosts the next Star conference Orlando

Which kind of models are mostly used for

model-based testing.

State Models

9/20/2013

16


Example: a Quiz Game, Making a Model

the question state:

state question

row column

get question >> q1 1

row column

answer question # q1 2

text

check prompt Very good

variable

get prompt >> prompt

prompt match state

move # prompt Let's calculate question

move # prompt Here is a question formula


Example: a Quiz Game, Making a Model

the challenge state (we also assume we have a parser to extract and resolve

the formula):

state challenge

challenge

get challenge >> chall

challenge result

resolve challenge # chall >> res

text

answer challenge # res

variable

get prompt >> prompt

prompt match state

move # prompt Let's calculate question

move # prompt Here is a question formula

9/20/2013

17


Dealing with non-determination

In this example we saw three ways to deal with

non-deterministic behavior a table for the questions

a parser to extract and calculate the formula challenges

a state machine to distinguish between question and

challenge states

You could typically let the test play until: all transitions between states have been traversed at

least once (including wrong answers etc)

all questions have been played

all possible kinds of formulas have been played • you have to define what that means, typically with equivalence

classes


Summarizing

Models can be used to: as a more efficient alternative to straightforward test cases

handle complex situations like non-deterministic system

behavior

Many models are based on state models

Models are not a panacea use them if there are clear benefits

Using actions can facilitate model-based and

multi-media testing providing means to describe and automate in an

accessible way

Model-Based Testing with Keywords

Technology

testing testing

test developer

test lines

actions test casetc

keywords test casetc

keyworddriven test automation

cto of logigear

keywordshans buwalda