Advanced Modeling Model Based Test Generation and Abstract state machine Language Keith Stobie Test Architect XML Web Services Microsoft Corporation.

Advanced ModelingAdvanced Modeling

Model Based Test GenerationModel Based Test Generationand and

AAbstract bstract sstate tate mmachine achine LLanguageanguage

Keith StobieKeith StobieTest ArchitectTest Architect

XML Web ServicesXML Web ServicesMicrosoft CorporationMicrosoft Corporation

22SASQAG 16Jan03SASQAG 16Jan03AsmL ModellingAsmL Modelling

© © 2003 Microsoft Corporation, All Rights Reserved 2003 Microsoft Corporation, All Rights Reserved

What’s a Model?What’s a Model?A model:A model:

Is an Is an abstractionabstraction or simplified or simplified representationrepresentation of the system of the system from a particular perspectivefrom a particular perspective

Supports investigation, discovery, Supports investigation, discovery, explanation, prediction, or explanation, prediction, or constructionconstruction

May be expressed as a May be expressed as a description, table, graphical description, table, graphical diagram, or quantitative diagram, or quantitative mathematical modelmathematical model

Is Is notnot necessarily necessarily comprehensivecomprehensive

From: PNSQC 2002 “From: PNSQC 2002 “Modeling: A Picture's Worth 1000 Words” Modeling: A Picture's Worth 1000 Words” Copyright (c) 2002, Quality Tree Software, Inc.Copyright (c) 2002, Quality Tree Software, Inc.



Models in Everyday LifeModels in Everyday Life

Inspired by :“Modeling: A Picture's Inspired by :“Modeling: A Picture's Worth 1000 Words” Copyright (c) Worth 1000 Words” Copyright (c) 2002, Quality Tree Software, Inc.2002, Quality Tree Software, Inc.



Examples of ModelsExamples of Models

Flow ChartsFlow Charts Data Flow DiagramsData Flow Diagrams Entity- Relationship DiagramsEntity- Relationship Diagrams State DiagramsState Diagrams Deployment DiagramsDeployment Diagrams Class DiagramsClass Diagrams Use CasesUse Cases Activity DiagramsActivity Diagrams State Transition TablesState Transition Tables

Inspired by :“Modeling: A Picture's Inspired by :“Modeling: A Picture's Worth 1000 Words” Copyright (c) Worth 1000 Words” Copyright (c) 2002, Quality Tree Software, Inc.2002, Quality Tree Software, Inc.



Model Based Test GenerationModel Based Test Generation

NOTNOT just model based input generators just model based input generators (pair-wise, random data, grammars, etc.)(pair-wise, random data, grammars, etc.)

NOTNOT modeling toolmodeling tools (e.g. UML)s (e.g. UML)

set of

Test Cases

• sequence of stimuli• expected responses

Formal Model

Test GenerationDirectives

Generator



ModelingModeling

Modeling

Validation

Refinement Verification

SpecificationAsmL

Implementation

C/C++/C#

Product Idea / Specification

Are you building the product right ?

Are you building the

right product?

What product are you building?



Benefits Benefits (GOTCHA)(GOTCHA)

Testing pilot amounted to Testing pilot amounted to 10 person months10 person months, , including the including the GOTCHAGOTCHA learning curve. learning curve. This time is This time is less thanless than the time used to the time used to originallyoriginally test the subsystem ( test the subsystem (12 person 12 person monthsmonths). ). revealed 2 documentation changes defects revealed 2 documentation changes defects 15 of the 18 defects found15 of the 18 defects found by the original by the original function test effort would also have been found function test effort would also have been found by this pilot test effortby this pilot test effort.”.”

From: From: Using a model-based test generator to test for sUsing a model-based test generator to test for standard conformancetandard conformance



Benefits Benefits (TestMaster)(TestMaster)

““In a typical application of this approach, test In a typical application of this approach, test engineer engineer productivity has increased by a productivity has increased by a factor of five to tenfactor of five to ten over conventional manual over conventional manual approaches (Figure 5)”.approaches (Figure 5)”.

From: From: Model Based TestingModel Based Testing



FSM: Start State & actionFSM: Start State & action

Start state (Entry Point)

FilesShown: 0 FileSelect: NotSelectedExitPoint: False

Invert_FileSelection

From: Intelligent Test Automation by Harry RobinsonSoftware Testing & Quality Engineering (STQE) magazine. Sept/Oct 2000

Action



other other StatesStates

State Invert_FileSelection

start

Create_File

FilesShown: 0FileSelected: NotSelectedExitPoint: True

End State (Exit Point)

FilesShown: 1FileSelected: Selected;ExitPoint: False

Delete_File



Simple FSM ModelSimple FSM Model



File-in-a-directory ModelFile-in-a-directory Model

exit




Delete_File

Delete_File

Create_File

FilesOpened 0FileSelected: NotSelectedExitPoint: False

FilesOpened: 1FileSelected: SelectedExitPoint: False

FilesOpened 1FileSelected: NotSelected;ExitPoint: False

FilesOpened 0FileSelected: NotSelectedExitPoint: True

start

Start State (Entry Point)

State

Action




CreateInvert SelectionInvert SelectionDelete

Walk Every State-ChangeWalk Every State-Change



File-in-a-directory ModelFile-in-a-directory Model

exit




Delete_File

Delete_File

Create_File

FilesOpened 0FileSelected: NotSelectedExitPoint: False

FilesOpened: 1FileSelected: SelectedExitPoint: False

FilesOpened 1FileSelected: NotSelected;ExitPoint: False

FilesOpened 0FileSelected: NotSelectedExitPoint: True

start

Start State (Entry Point)

State

Action


C2

C1C3

C4



A: Invert SelectionB: Create; DeleteC: Create; Invert Selection; Invert Selection; Delete

Test Every Walk < 5 StepsTest Every Walk < 5 Steps



Test Model ToolkitTest Model Toolkit

Source: Testing Internet Explorer:Source: Testing Internet Explorer: From Underdog To Best-In-Class Browser From Underdog To Best-In-Class Browser



Test Model Toolkit - ImpactTest Model Toolkit - Impact

Used by over 20 teams at MSUsed by over 20 teams at MS

Time to automate fell by as much as 88%Time to automate fell by as much as 88% BizTalk - Took 1 week to generate a set of test cases BizTalk - Took 1 week to generate a set of test cases

that took 8 weeks by handthat took 8 weeks by hand

Code coverage increased by as much as 50%Code coverage increased by as much as 50% One subteam in Shell – In 2 weeks increased code One subteam in Shell – In 2 weeks increased code

coverage from 20% to 75%, while the number of test coverage from 20% to 75%, while the number of test cases increased from ~75 to ~2000.cases increased from ~75 to ~2000.

Many spec and implementation bugs are Many spec and implementation bugs are uncovered in the course of model creationuncovered in the course of model creation

Source: Testing Internet Explorer:Source: Testing Internet Explorer: From Underdog To Best-In-Class Browser From Underdog To Best-In-Class Browser



The AsmL The AsmL modelingmodeling toolkit toolkit

Abstract state machine Language is a Abstract state machine Language is a powerful modeling language powerful modeling language

AsmL providesAsmL provides Authoring via MS Word and XMLAuthoring via MS Word and XML Simulation and interoperability via COM and .NETSimulation and interoperability via COM and .NET Automatic test oracle Automatic test oracle Semi-automatic test case generationSemi-automatic test case generation Integration with Microsoft’s test platformIntegration with Microsoft’s test platform



AsmL: Getting Started, ‘sort’AsmL: Getting Started, ‘sort’ example example

var A as Seq of Integer = []

Swap()choose i in indices(A),

j in indices(A) where i < j and A(i) > A(j)

A(i) := A(j) A(j) := A(i)

sort()step until fixpoint

swap()

A = [2,3,1]

A = [1,2,3]

A = [2,1,3]

Non-determinism

A = [1,3,2]

Parallelism



AsmL: Predefined Data TypesAsmL: Predefined Data TypesSetsSets menu = menu = {{""hamham","","cheesecheese", "", "bltblt""}}

SequencesSequencesrandomrandom == [[3,3, 6,6, 7,7, 1,1, 55]] ////asas SeqSeq ofof Integer Integer

MapsMaps directory directory as Map of as Map of String String to to Integer = Integer = {{“police” “police” ->-> 112 , “info” 112 , “info” ->-> 118 118}}

TuplesTuplesnameAndAgenameAndAge = = ((“John”,39“John”,39)) ////((StringString,,Integer)Integer)



The cool part: Asml’s high-level-nessThe cool part: Asml’s high-level-nesscomprehensioncomprehension to pick 1 of the oldest filters in a to pick 1 of the oldest filters in a

Most Recently Used (MRU) cache.Most Recently Used (MRU) cache.

class class CacheFilterCacheFiltervar var lastAccessedlastAccessed as as DateTimeDateTime

var var MruSetMruSet as Set of as Set of CacheFilterCacheFilter = {} = {}

letlet victim = victim = anyany i i | i | i inin MruSet MruSet wherewhere ((forallforall j j inin MruSet MruSet holdsholds

// // j access time newer or equal ij access time newer or equal i(Compare((Compare(j.lastAccessed,j.lastAccessed,

i.lastAccessedi.lastAccessed) >= 0)) >= 0)))



XML & XPathXML & XPath<DocumentRoot><DocumentRoot>

<ParentElement Attrib1=1 attrib2=2 ><ParentElement Attrib1=1 attrib2=2 >

<childElementA/><childElementA/>

<childElementB/><childElementB/>

</ParentElement></ParentElement>

</DocumentRoot></DocumentRoot>

Child Axis

Attribute Axis

XPath:/DocumentRoot/ParentElement/childElementB/DocumentRoot/ParentElement/@attrib2



XPath Model – axis nodesXPath Model – axis nodes

We interpret an axis as producing either We interpret an axis as producing either 1) all of the children of a given node or 1) all of the children of a given node or 2) all of the attribute children of a given node.2) all of the attribute children of a given node.

EvaluateAxis (a as Axis, c as Context) as Seq of Node let x = ContextNode(c) match a Child : return NodeChildren(x) Axis.Attribute : return NodeAttributes(x) _ : error "EvaluateAxis: internal error"

a=Child Context=ParentNode

Returns: childElementA childElementB



XPath model – node testXPath model – node testAn axis path (for example, "child::para") has a two-step evaluation.An axis path (for example, "child::para") has a two-step evaluation.

The axis ("child") is used to generate a list of matching nodes,The axis ("child") is used to generate a list of matching nodes,

which is subsequently filtered to include only those nodes that pass the which is subsequently filtered to include only those nodes that pass the given node testgiven node test ("para"). ("para").

ap as AxisPath: let nodes = EvaluateAxis(ap.axis, c) return

[nodes(i - 1) | i in [1..Length(nodes)] where PassesNodeTest(ap.test,

Context(nodes, i))]



Using models for testUsing models for test

Model

Implementation

Test OracleTest Cases

Are run by Providesactual results for

PassNo pass

Provides expected results forGenerates

UserInfo



Tools & TechniquesTools & TechniquesTool&LanguageTool&Language

Test TechniqueTest Technique

TMTTMTFSMFSM

ASMLASML

ASMASM

State WalkState WalkTest Case GenTest Case Gen

QueriesQueriese.g. all Nodese.g. all Nodes

all Edgesall Edges

Test PropertiesTest Properties => FSM => FSM => state walk => state walk

SimulationSimulationTest OracleTest Oracle

Dynamic Dynamic TraversalTraversal Model ManagerModel Manager

Model Coverage Model Coverage via random Test via random Test Case GenCase Gen

currently currently manual – manual –

random pathsrandom paths

ASML test case ASML test case generation with generation with feedbackfeedback



AsmL-based test harnessAsmL-based test harness

AsmLmodel

Test harness

Implementationunder test

Side-by-side runsSide-by-side runs

Discrepancies Discrepancies automatically automatically flaggedflagged

Anyclient



AsmL test case generation - 1AsmL test case generation - 1

Solves two problemsSolves two problems

How to find “interesting” parameter How to find “interesting” parameter values and combinationsvalues and combinations

How to find “interesting” call sequencesHow to find “interesting” call sequences

ApproachApproach

Use information derived from the model Use information derived from the model to optimize the number of test casesto optimize the number of test cases



AsmL test case generation - 2AsmL test case generation - 2

Divide state space according to state predicatesDivide state space according to state predicates

System “covers” the state predicatesSystem “covers” the state predicates Driven by input generator (parameter selection) orDriven by input generator (parameter selection) or Driven by enabledness (call sequences)Driven by enabledness (call sequences)

Tester provides inputs, gives interesting state Tester provides inputs, gives interesting state predicates, directs the searchpredicates, directs the search



AsmL Model M

The dual role of ASM modelsThe dual role of ASM models

Test cases

Guidedgeneration

(ParamSelectwith

Feedback)

Reduce

FSMtool

IUTCall next action

Get state

AsmL Model M

Testing harness

Is Svalid according

to M?

Fail: witness

Pass Sview

view

view

FSM



AsmL test case generation - 3AsmL test case generation - 3Generate only as many test cases as needed to have full Generate only as many test cases as needed to have full

coverage of the specificationcoverage of the specification

Generator ModelInput

Coverage criteriaCoverage criteria Branch: Every “branch” of the spec must be takenBranch: Every “branch” of the spec must be taken Modified condition: independent atomic conditions Modified condition: independent atomic conditions Off-by-One: off-by-one errorsOff-by-One: off-by-one errors HyperHyperstate coverage: Every state coverage: Every abstractabstract state must be state must be

visitedvisited

Feedback



Model Branch Coverage

undecided, 5

Generator covered, 104

dead, 36

manually covered, 9

XPath Model coverage XPath Model coverage via random Param Test Generationvia random Param Test Generation

Model Statistics:Model Statistics:

1,003,638 tests generated 1,003,638 tests generated (115Mb file)(115Mb file)

Only 37 tests are needed Only 37 tests are needed to cover all 113 to cover all 113 (reachable 104+9) (reachable 104+9) branches!branches!



AsmL Test (asmlt)AsmL Test (asmlt)

asmlT can be used to:asmlT can be used to:

1.1. Find interesting sequences of method callsFind interesting sequences of method calls finds the shortest route in a generated FSM that uses every finds the shortest route in a generated FSM that uses every

edge to traverse the statesedge to traverse the states

2.2. Find interesting parameters for each method call Find interesting parameters for each method call

Access Driven Filtering Access Driven Filtering (ADF)(ADF) you write a predicate (boolean function) to decide if a given you write a predicate (boolean function) to decide if a given

parameter is interesting parameter is interesting

3.3. Perform a conformance testPerform a conformance testCompare Model with ImplementationCompare Model with Implementation



ReferencesReferencesModel-Based Test Generation ToolsModel-Based Test Generation Tools, Alan Hartman, , Alan Hartman, AGEDIS CONSORTIUM AGEDIS CONSORTIUM http://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdfhttp://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdf

Finite State Model-Based Testing on a ShoestringFinite State Model-Based Testing on a Shoestring, Harry Robinson, Harry Robinsonhttp://http://www.model-based-testing.orgwww.model-based-testing.org/shoestring.htm /shoestring.htm

Using a model-based test generator to test for standard conformanceUsing a model-based test generator to test for standard conformance E. Farchi, et al IBM SYSTEMS JOURNAL, VOL 41, NO 1, 2002E. Farchi, et al IBM SYSTEMS JOURNAL, VOL 41, NO 1, 2002http://researchweb.watson.ibm.com/journal/sj/411/farchi.pdfhttp://researchweb.watson.ibm.com/journal/sj/411/farchi.pdf

Model Based TestingModel Based Testing, , L. Apfelbaum, et al, Software Quality Week L. Apfelbaum, et al, Software Quality Week Conference in May, 1997.Conference in May, 1997.http://http://www.model-based-testing.orgwww.model-based-testing.org/sqw97.pdf /sqw97.pdf

Modeling for Testers: A Picture’s Worth 1000 WordsModeling for Testers: A Picture’s Worth 1000 Words,,Elisabeth Hendrickson, Quality Tree Software, Inc.Elisabeth Hendrickson, Quality Tree Software, Inc.http://www.raddue.com/pnsqc/Pnsqc02.pdfhttp://www.raddue.com/pnsqc/Pnsqc02.pdf

Good reference for existing tools



More ReferencesMore ReferencesAbstract State Machine Language, Microsoft ResearchAbstract State Machine Language, Microsoft Researchhttp://research.microsoft.com/fse/asml/ http://research.microsoft.com/fse/asml/

The AsmL Test Generator tool, Microsoft ResearchThe AsmL Test Generator tool, Microsoft Researchhttp://http://research.microsoft.com/fseresearch.microsoft.com/fse/AsmL/doc/AsmLTester.html /AsmL/doc/AsmLTester.html

Testing Internet Explorer:Testing Internet Explorer: From Underdog To Best-In-Class Browser From Underdog To Best-In-Class BrowserJason Taylor, Microsoft Corporation at Jason Taylor, Microsoft Corporation at StarWest 2002StarWest 2002

Intelligent Test Automation, Harry RobinsonSoftware Testing & Quality Engineering (STQE) magazine Sept/Oct 2000http://www.model-based-testing.org/robinson.pdf

Confessions of a Modeling Bigot: Parts III-IVConfessions of a Modeling Bigot: Parts III-IVMichael Corning , ASPToday Dec02, Jan03Michael Corning , ASPToday Dec02, Jan03http://www.asptoday.com/content.asp?id=2043http://www.asptoday.com/content.asp?id=2043

© 2003 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Advanced Modeling Model Based Test Generation and Abstract state machine Language Keith Stobie Test Architect XML Web Services Microsoft Corporation.

Documents

modelbased test generator

sasqag 16jan03 asml

pilot test effort

rights reserved models

testing slide

rights reserved fsm

rights reserved whats

test engineer productivity