Amost 2011 keynote

Microsoft’s Protocol Documentation Microsoft’s Protocol Documentation Program: Program:

A Success Story for Model-Based TestingA Success Story for Model-Based TestingWolfgang Grieskamp, Software Architect Windows Interoperability Engineering Team, Microsoft Corporation

Keynote given at A-MOST 2011, March 2011, Berlin, Germany.

Part 1: BackgroundPart 1: Background

Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 2

Microsoft Microsoft Open Open ProtocolsProtocols

Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation

http://www.microsoft.com/protocols

9/4/2010 3

http://www.microsoft.com/protocols/

http://www.microsoft.com/protocols/

Technical Document Testing Program Technical Document Testing Program of Windows (as of 03/09)of Windows (as of 03/09)

222 protocols/technical documents tested 22,847 pages studied and converted into

requirements 36,875 testable requirements identified and

converted into test assertions 69% tested using MBT 31% tested using traditional test automation

66,962 person days (250+ years) Hyderabad: 250 test engineers Beijing: 100 test engineers


Formalization

Technical DocumentTest Suite

Test Suite

EnglishStatements(Requirements)

AssertionsChecked against Windows(Accuracy)

• Validate consistency

• Simulate implementation process

Test-Driven Document Quality Test-Driven Document Quality AssuranceAssurance



Process: Managing a Project with 350 Test Engineers

Review after every phase•High-profile reviewer team consisting of industry experts and Microsoft senior engineers, build along the model of a program committee•Review based on process report and test suite artifacts•Verdict accept, conditional accept, reject

9/4/2010 6

Requirements TrackingRequirements Tracking


Technical Document

Requirements Spec

Model or Test Design

Test Suite

Logs

Network Captures

Statements identified in document and tracked with requirement ID

IDs referenced from model or test assertion

IDs logged at test execution time

IDs send to network as beacon packages for network monitor capturing

manual

automated

9/4/2010 7

Typical Requirements Specification


Part 2: Model-Based Test Suite Part 2: Model-Based Test Suite DevelopmentDevelopment


Model-Based Document AnalysisModel-Based Document Analysis

ModelModel

ImplementationImplementation

Expected Outputs (Test Oracle)

Expected Outputs (Test Oracle)Inputs

(Test Sequences)

Inputs(Test Sequences)

Control

DocumentDocument

Generate

Observe

Feedback

Feedback

Author

Issue

Feedback

Verdict

Feedback


Process DrilldownProcess Drilldown


Gather Requirements

Define Actions Define Adapters

Model & Explore

Generate Tests Implement Adapters

Run Tests

9/4/2010 11

Gathering RequirementsGathering Requirements


Gather Requirements


Model & Explore


Run Tests

9/4/2010 12

Developing and Implementing AdaptersDeveloping and Implementing Adapters


Gather Requirements


Model & Explore


Run Tests

9/4/2010 13

Adapter InterfaceAdapter Interface

TCTCTest Cases

(traditional or MBT)TC

TCTC

TC

TCTC

TCTC

TCTC

Adapter ImplementationAdapter Implementation

SUT(System under test)

• Abstracts SUT functionality• Contract between teams

• Test case team• Adapter team

• Abstracts SUT functionality• Contract between teams

• Test case team• Adapter team

• Pluggable• Different server setups• Different transports

• Pluggable• Different server setups• Different transports

Test Adapters


Develop Model and Generate TestsDevelop Model and Generate Tests


Gather Requirements


Model & Explore


Run Tests

Potential dependencyon data types

9/4/2010 15

Spec Explorer 2010 Technology Spec Explorer 2010 Technology BreakdownBreakdown Model programs

Guarded state update rules Rich object-oriented model state (collections, object graphs) Language agnostic (Based on .Net intermediate language

interpretation)

Trace patterns Regular style language to represent scenarios Slicing of model program by composition

Symbolic state exploration and test generation Expands parameters using combinatorial interaction testing Extracts a finite interface automaton (IA) from composed model Traverses IA to generate standalone test code –or- Runs on-the-fly tests from IA

Integrated into Visual Studio 2010


Spec Explorer 2010 Look & FeelSpec Explorer 2010 Look & Feel

Explore

Analyze

Generate

Execute

C# Model C# Model (or other .Net (or other .Net Language)Language)

Model GraphModel Graph

Test SuiteTest Suite

VSTT ResultVSTT Result


Spec Explorer Walkthrough: Chat ExampleSpec Explorer Walkthrough: Chat Example

Users can Enter the session Exit the session List all session users Broadcast a message

Received by all session users

18

Chat Room Server

User1

SUT

User2 User3

What happens if two users broadcast a message at virtually the same time?

Chat RequirementsChat Requirements

19

R1: User MUST receive response for logon request

R2: User MUST receive response for logoff request

R3: User MUST receive response for list request

R4: List response MUST contain the list of logged-on users if successful

R5: All logged-on users MUST receive broadcast message

R6: Messages from one sender MUST be received in order

Chat tracesChat traces

Is the trace correct?

(Assume two users (user1, user2) are logged on)

20

T1Broadcast(user1,”1”)Broadcast(user2,”2”)BroadcastAck(user2,”1”)BroadcastAck(user1,”2”) BroadcastAck(user1,”1”)BroadcastAck(user2,”2”)



If each user sends one message, any receiving order is correct!

Chat tracesChat traces

Is the trace correct?

(Assume two users (user1, user2) are logged on)

21

T4Broadcast(user1,”1a”)Broadcast(user1,”1b”)BroadcastAck(user2,”1a”)BroadcastAck(user1,”1a”) BroadcastAck(user1,”1b”)BroadcastAck(user2,”1b”)

T5Broadcast(user1,”1a”)Broadcast(user1,”1b”)BroadcastAck(user1,”1a”)BroadcastAck(user2,”1b”) BroadcastAck(user1,”1b”)BroadcastAck(user2,”1a”)

Local order consistency: messages sent by one user must be received in order

22

23

Spec Explorer’s Conformance NotionSpec Explorer’s Conformance Notion

Alternating simulation: SUT must “simulate” all stimuli of model Model must “simulate” all responses of SUT (responses are buffered)

24

Broadcast(1,”1”)


Model SUT



BroadcastAck(1,”1”)






Spec Explorer’s Approach to State ExplosionSpec Explorer’s Approach to State Explosion

Slicing techniques:

Parameter selection (Constraints, Pairwise, etc.) State filtering Trace patterns Requirement coverage

Slicing requires human intervention Smart slicing is an art

Spec Explorer tailored for slicing design25

Where a Trace can end: Accepting State Where a Trace can end: Accepting State ConditionCondition

Is the following a valid word?Micr

Does the following trace represents a useful test?Broadcast(1,”1”); Broadcast(1,”1”)

An Accepting state condition characterizes those states in which a trace can end Used to ensure that a trace does not stop at arbitrary points Used to ensure that a test leaves the system in a good state

Accepting state condition for Chat: All messages have been delivered to recipients

26

Demo: Chat ModelDemo: Chat Model

What the demo shows OO model state Using helper methods in models Modeling with events Slicing

Demo: Server Message Block Protocol Demo: Server Message Block Protocol v2v2What the demo shows:Modeling a key Microsoft productUsing C# to model state transition systemsUsing trace patterns to slice test purposes from modelsGenerating and executing tests


Part 3: EvaluationPart 3: Evaluation


Comparison MBT vs TraditionalComparison MBT vs Traditional


• In % of total effort per requirement, normalizing individual vendor performance

• Vendor 2 modeled 85% of all test suites, performing relatively much better than Vendor 1

9/4/2010 35

Remarks on ComparisonRemarks on Comparison

Numbers backed up by correlation analysis conducted by empirics expert

Engineers applying MBT were lay folks, most of them freshly hired from college

Few power modelers achieve much higher efficiency rates

Though model authoring takes only a smaller part of overall effort per requirement, modeling approach seems to determine efficiency


Factors for SuccessFactors for Success Tool support

Modeling in mainstream languages and environments

Visualization of exploration and actual generated test suite (no magic as in online testing)

Dealing with state explosion by scenario slicing

Training and Guidance One week modeling classes for every new hire Size of project generated critical mass for

community help Reviewing process added additional guidance


ConclusionsConclusions

Biggest application of behavioral model-based testing known so far

Probe big enough for relevant empiric results, proving efficiency gain of 42%

Will lead to more momentum on modeling in general inside of Microsoft

Model-based testing rocks!


Advertisement…Advertisement…

Microsoft offers Spec Explorer 2010 a Visual Studio Powertool via

Licensing allows for commercial use Academic institutions can join MSDN

Academic Alliance for free access to Visual Studio 2010


© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,

IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
