Top Banner
Budapest University of Technology and Economics Department of Measurement and Information Comparing Robustness of AIS-Based Middleware Implementations Zoltán Micskei, István Majzik Budapest University of Technology and Economics Francis Tam Nokia Research Center Nokia Group International Service Availability Symposium (ISAS) 2007
22

Comparing robustness of AIS-based middleware implementations

Aug 04, 2015

Download

Technology

Zoltan Micskei
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Comparing robustness of AIS-based middleware implementations

Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems

Comparing Robustness of AIS-Based Middleware Implementations

Zoltán Micskei, István MajzikBudapest University of

Technology and Economics

Francis Tam

Nokia Research CenterNokia Group

International Service Availability Symposium (ISAS) 2007

Page 2: Comparing robustness of AIS-based middleware implementations

MotivationComparison: mostly performance. However:

Application

Component 1 Component N

SAF AIS

invalid input

A faulty application could crash even the HA middleware!

Page 3: Comparing robustness of AIS-based middleware implementations

Robustness

„The degree to which a system operates correctly in the presence of oexceptional inputs or ostressful environmental conditions.”

[IEEE Std 610.12.1990]

Page 4: Comparing robustness of AIS-based middleware implementations

Robustness testing

Functional testingoConformance, expected output includedo Valid inputs, some of the invalids

Robustness testing− Try to „break” the system− Large amount of invalid input

Page 5: Comparing robustness of AIS-based middleware implementations

Goal

Test and compare robustness of HA MWo Based on common interface

Several fault type and mode → automatic test generation

Page 6: Comparing robustness of AIS-based middleware implementations

Fault model: Primary sources

Custom Application

AIS implementation

Operating System

Hardware

API calls

OS calls

Page 7: Comparing robustness of AIS-based middleware implementations

Fault model: Secondary sources

Custom Application

AIS implementation

Operating System

Hardware

External Components

Human Interface

API calls

OS calls

HW failures

Operators

Page 8: Comparing robustness of AIS-based middleware implementations

Our testing toolsTBTS-TG

(type spec.)Workload

MBST-TG (mutation)

Operating system

Hardware

OS call wrapper

HA Middleware

Page 9: Comparing robustness of AIS-based middleware implementations

Testing toolsTBTS-TG

(type spec.)Workload

MBST-TG (mutation)

Operating system

Hardware

OS call wrapper

HA Middleware

Page 10: Comparing robustness of AIS-based middleware implementations

Type specific testing Goal: test the whole interface

saAmfInitialize

saAmfPmStart

saComponentNameGet

Handle invalidHandle closed

Handle invalidHandle closedMonitoring startedComponent not registered

Handle invalidHandle closedComponent not registeredPointer null

Page 11: Comparing robustness of AIS-based middleware implementations

Type specific testing Goal: test the whole interface

saAmfInitialize

saAmfPmStart

saComponentNameGet

SaAmfHandleT SaAmfName

Handle invalidHandle closed

Name invalidComponent not registered

Page 12: Comparing robustness of AIS-based middleware implementations

Type specific testing For each function

o Fill a template with the parameterso Invalid and valid values

Middleware specific: o state based callso Complex setup code for type valueso Running tests as SA-aware components

Page 13: Comparing robustness of AIS-based middleware implementations

Testing toolsTBTS-TG

(type spec.)Workload

MBST-TG (mutation)

Operating system

Hardware

OS call wrapper

HA Middleware

Page 14: Comparing robustness of AIS-based middleware implementations

Mutation-based testing Goal: test complex scenarios using

multiple functions

How?oWrite complex testoMutate existing code with injecting

typical robustness faults Sources to mutate

o SAFtesto Functional tests in openais

Page 15: Comparing robustness of AIS-based middleware implementations

Testing toolsTBTS-TG

(type spec.)Workload

MBST-TG (mutation)

Operating system

Hardware

OS call wrapper

HA Middleware

Page 16: Comparing robustness of AIS-based middleware implementations

OS call wrapper Goal: test environment conditions

Provide workload

Intercept system calls ando delay,o change return value.

Support in OS:o e.g. strace and LD_PRELOAD in Linux

Page 17: Comparing robustness of AIS-based middleware implementations

Testing results Three middleware

o Openais version 0.80.1 and trunko Fujitsu Siemens SAFE4TRY

Test execution environmento Configuration file, restart MW, logging…

Results:o Differences in headerso Test program abortsoMiddleware crashes

Page 18: Comparing robustness of AIS-based middleware implementations

Type specific

openais-0.80.1 openais-trunk SAFE4TRY

success 24568 26019 29663

segmentation fault 1100 1468 0

timeout 467 2178 2

SAFE4TRY seems to be more robust to

these kind of inputs

For 6 functions in openais the middleware itself crashed

In openais 0.69 segmentation fault was

8001 out of 13460

Page 19: Comparing robustness of AIS-based middleware implementations

Mutation based

Example from the observed failures:

Page 20: Comparing robustness of AIS-based middleware implementations

OS call wrapper

openais-0.80.1 openais-trunk SAFE4TRY

No failure observed 6 5 5

Application failed 0 2 1

Middleware failed 3 2 3

Observations:All are vulnerable for system call failureSome calls cause failure for all: e.g. socketSome depends on system: e.g. bind

Page 21: Comparing robustness of AIS-based middleware implementations

Future work - Obtaining metrics

Large amount of output

Number of failed tests for a function → robustness faults in the function

Help:o Assigning expected error codeso Data mining tools / decision tree

Page 22: Comparing robustness of AIS-based middleware implementations

Lessons learnt Simple tests can find robustness failures Different methods find different failures There are problems even with the headers Existing applications not up-to-date

o LDAP DN format, component name get Middleware differ heavily

o How-to start, stop; configuration files For complex scenarios, OS call failures

o Detailed workload, complex test setup needed Robustness improving