Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems
Comparing Robustness of AIS-Based Middleware Implementations
Zoltán Micskei, István MajzikBudapest University of
Technology and Economics
Francis Tam
Nokia Research CenterNokia Group
International Service Availability Symposium (ISAS) 2007
MotivationComparison: mostly performance. However:
Application
Component 1 Component N
SAF AIS
invalid input
A faulty application could crash even the HA middleware!
Robustness
„The degree to which a system operates correctly in the presence of oexceptional inputs or ostressful environmental conditions.”
[IEEE Std 610.12.1990]
Robustness testing
Functional testingoConformance, expected output includedo Valid inputs, some of the invalids
Robustness testing− Try to „break” the system− Large amount of invalid input
Goal
Test and compare robustness of HA MWo Based on common interface
Several fault type and mode → automatic test generation
Fault model: Primary sources
Custom Application
AIS implementation
Operating System
Hardware
API calls
OS calls
Fault model: Secondary sources
Custom Application
AIS implementation
Operating System
Hardware
External Components
Human Interface
API calls
OS calls
HW failures
Operators
Our testing toolsTBTS-TG
(type spec.)Workload
MBST-TG (mutation)
Operating system
Hardware
OS call wrapper
HA Middleware
Testing toolsTBTS-TG
(type spec.)Workload
MBST-TG (mutation)
Operating system
Hardware
OS call wrapper
HA Middleware
Type specific testing Goal: test the whole interface
saAmfInitialize
saAmfPmStart
saComponentNameGet
Handle invalidHandle closed
Handle invalidHandle closedMonitoring startedComponent not registered
Handle invalidHandle closedComponent not registeredPointer null
Type specific testing Goal: test the whole interface
saAmfInitialize
saAmfPmStart
saComponentNameGet
SaAmfHandleT SaAmfName
Handle invalidHandle closed
Name invalidComponent not registered
Type specific testing For each function
o Fill a template with the parameterso Invalid and valid values
Middleware specific: o state based callso Complex setup code for type valueso Running tests as SA-aware components
Testing toolsTBTS-TG
(type spec.)Workload
MBST-TG (mutation)
Operating system
Hardware
OS call wrapper
HA Middleware
Mutation-based testing Goal: test complex scenarios using
multiple functions
How?oWrite complex testoMutate existing code with injecting
typical robustness faults Sources to mutate
o SAFtesto Functional tests in openais
Testing toolsTBTS-TG
(type spec.)Workload
MBST-TG (mutation)
Operating system
Hardware
OS call wrapper
HA Middleware
OS call wrapper Goal: test environment conditions
Provide workload
Intercept system calls ando delay,o change return value.
Support in OS:o e.g. strace and LD_PRELOAD in Linux
Testing results Three middleware
o Openais version 0.80.1 and trunko Fujitsu Siemens SAFE4TRY
Test execution environmento Configuration file, restart MW, logging…
Results:o Differences in headerso Test program abortsoMiddleware crashes
Type specific
openais-0.80.1 openais-trunk SAFE4TRY
success 24568 26019 29663
segmentation fault 1100 1468 0
timeout 467 2178 2
SAFE4TRY seems to be more robust to
these kind of inputs
For 6 functions in openais the middleware itself crashed
In openais 0.69 segmentation fault was
8001 out of 13460
Mutation based
Example from the observed failures:
OS call wrapper
openais-0.80.1 openais-trunk SAFE4TRY
No failure observed 6 5 5
Application failed 0 2 1
Middleware failed 3 2 3
Observations:All are vulnerable for system call failureSome calls cause failure for all: e.g. socketSome depends on system: e.g. bind
Future work - Obtaining metrics
Large amount of output
Number of failed tests for a function → robustness faults in the function
Help:o Assigning expected error codeso Data mining tools / decision tree
Lessons learnt Simple tests can find robustness failures Different methods find different failures There are problems even with the headers Existing applications not up-to-date
o LDAP DN format, component name get Middleware differ heavily
o How-to start, stop; configuration files For complex scenarios, OS call failures
o Detailed workload, complex test setup needed Robustness improving
