Comparing robustness of AIS-based middleware implementations

Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems

Comparing Robustness of AIS-Based Middleware Implementations

Zoltán Micskei, István MajzikBudapest University of

Technology and Economics

Francis Tam

Nokia Research CenterNokia Group

International Service Availability Symposium (ISAS) 2007

MotivationComparison: mostly performance. However:

Application

Component 1 Component N

SAF AIS

invalid input

A faulty application could crash even the HA middleware!

Robustness

„The degree to which a system operates correctly in the presence of oexceptional inputs or ostressful environmental conditions.”

[IEEE Std 610.12.1990]

Robustness testing

Functional testingoConformance, expected output includedo Valid inputs, some of the invalids

Robustness testing− Try to „break” the system− Large amount of invalid input

Goal

Test and compare robustness of HA MWo Based on common interface

Several fault type and mode → automatic test generation

Fault model: Primary sources

Custom Application

AIS implementation

Operating System

Hardware

API calls

OS calls

Fault model: Secondary sources

Custom Application

AIS implementation

Operating System

Hardware

External Components

Human Interface

API calls

OS calls

HW failures

Operators

Our testing toolsTBTS-TG

(type spec.)Workload

MBST-TG (mutation)

Operating system

Hardware

OS call wrapper

HA Middleware

Testing toolsTBTS-TG

(type spec.)Workload

MBST-TG (mutation)

Operating system

Hardware

OS call wrapper

HA Middleware

Type specific testing Goal: test the whole interface

saAmfInitialize

saAmfPmStart

saComponentNameGet

Handle invalidHandle closed

Handle invalidHandle closedMonitoring startedComponent not registered

Handle invalidHandle closedComponent not registeredPointer null

Type specific testing Goal: test the whole interface

saAmfInitialize

saAmfPmStart

saComponentNameGet

SaAmfHandleT SaAmfName

Handle invalidHandle closed

Name invalidComponent not registered

Type specific testing For each function

o Fill a template with the parameterso Invalid and valid values

Middleware specific: o state based callso Complex setup code for type valueso Running tests as SA-aware components

Testing toolsTBTS-TG

(type spec.)Workload

MBST-TG (mutation)

Operating system

Hardware

OS call wrapper

HA Middleware

Mutation-based testing Goal: test complex scenarios using

multiple functions

How?oWrite complex testoMutate existing code with injecting

typical robustness faults Sources to mutate

o SAFtesto Functional tests in openais

Testing toolsTBTS-TG

(type spec.)Workload

MBST-TG (mutation)

Operating system

Hardware

OS call wrapper

HA Middleware

OS call wrapper Goal: test environment conditions

Provide workload

Intercept system calls ando delay,o change return value.

Support in OS:o e.g. strace and LD_PRELOAD in Linux

Testing results Three middleware

o Openais version 0.80.1 and trunko Fujitsu Siemens SAFE4TRY

Test execution environmento Configuration file, restart MW, logging…

Results:o Differences in headerso Test program abortsoMiddleware crashes

Type specific

openais-0.80.1 openais-trunk SAFE4TRY

success 24568 26019 29663

segmentation fault 1100 1468 0

timeout 467 2178 2

SAFE4TRY seems to be more robust to

these kind of inputs

For 6 functions in openais the middleware itself crashed

In openais 0.69 segmentation fault was

8001 out of 13460

Mutation based

Example from the observed failures:

OS call wrapper

openais-0.80.1 openais-trunk SAFE4TRY

No failure observed 6 5 5

Application failed 0 2 1

Middleware failed 3 2 3

Observations:All are vulnerable for system call failureSome calls cause failure for all: e.g. socketSome depends on system: e.g. bind

Future work - Obtaining metrics

Large amount of output

Number of failed tests for a function → robustness faults in the function

Help:o Assigning expected error codeso Data mining tools / decision tree

Lessons learnt Simple tests can find robustness failures Different methods find different failures There are problems even with the headers Existing applications not up-to-date

o LDAP DN format, component name get Middleware differ heavily

o How-to start, stop; configuration files For complex scenarios, OS call failures

o Detailed workload, complex test setup needed Robustness improving