Budapest University of Technology and Economics Department of Measurement and Information Comparing Robustness of AIS-Based Middleware Implementations Zoltán Micskei, István Majzik Budapest University of Technology and Economics Francis Tam Nokia Research Center Nokia Group International Service Availability Symposium (ISAS) 2007
22
Embed
Comparing robustness of AIS-based middleware implementations
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems
Comparing Robustness of AIS-Based Middleware Implementations
Zoltán Micskei, István MajzikBudapest University of
Technology and Economics
Francis Tam
Nokia Research CenterNokia Group
International Service Availability Symposium (ISAS) 2007
A faulty application could crash even the HA middleware!
Robustness
„The degree to which a system operates correctly in the presence of oexceptional inputs or ostressful environmental conditions.”
[IEEE Std 610.12.1990]
Robustness testing
Functional testingoConformance, expected output includedo Valid inputs, some of the invalids
Robustness testing− Try to „break” the system− Large amount of invalid input
Goal
Test and compare robustness of HA MWo Based on common interface
Several fault type and mode → automatic test generation
Fault model: Primary sources
Custom Application
AIS implementation
Operating System
Hardware
API calls
OS calls
Fault model: Secondary sources
Custom Application
AIS implementation
Operating System
Hardware
External Components
Human Interface
API calls
OS calls
HW failures
Operators
Our testing toolsTBTS-TG
(type spec.)Workload
MBST-TG (mutation)
Operating system
Hardware
OS call wrapper
HA Middleware
Testing toolsTBTS-TG
(type spec.)Workload
MBST-TG (mutation)
Operating system
Hardware
OS call wrapper
HA Middleware
Type specific testing Goal: test the whole interface
saAmfInitialize
saAmfPmStart
saComponentNameGet
Handle invalidHandle closed
Handle invalidHandle closedMonitoring startedComponent not registered
Handle invalidHandle closedComponent not registeredPointer null
Type specific testing Goal: test the whole interface
saAmfInitialize
saAmfPmStart
saComponentNameGet
SaAmfHandleT SaAmfName
Handle invalidHandle closed
Name invalidComponent not registered
Type specific testing For each function
o Fill a template with the parameterso Invalid and valid values
Middleware specific: o state based callso Complex setup code for type valueso Running tests as SA-aware components
Testing toolsTBTS-TG
(type spec.)Workload
MBST-TG (mutation)
Operating system
Hardware
OS call wrapper
HA Middleware
Mutation-based testing Goal: test complex scenarios using
multiple functions
How?oWrite complex testoMutate existing code with injecting
typical robustness faults Sources to mutate
o SAFtesto Functional tests in openais
Testing toolsTBTS-TG
(type spec.)Workload
MBST-TG (mutation)
Operating system
Hardware
OS call wrapper
HA Middleware
OS call wrapper Goal: test environment conditions
Provide workload
Intercept system calls ando delay,o change return value.
Support in OS:o e.g. strace and LD_PRELOAD in Linux
Testing results Three middleware
o Openais version 0.80.1 and trunko Fujitsu Siemens SAFE4TRY
Test execution environmento Configuration file, restart MW, logging…
Results:o Differences in headerso Test program abortsoMiddleware crashes
Type specific
openais-0.80.1 openais-trunk SAFE4TRY
success 24568 26019 29663
segmentation fault 1100 1468 0
timeout 467 2178 2
SAFE4TRY seems to be more robust to
these kind of inputs
For 6 functions in openais the middleware itself crashed
In openais 0.69 segmentation fault was
8001 out of 13460
Mutation based
Example from the observed failures:
OS call wrapper
openais-0.80.1 openais-trunk SAFE4TRY
No failure observed 6 5 5
Application failed 0 2 1
Middleware failed 3 2 3
Observations:All are vulnerable for system call failureSome calls cause failure for all: e.g. socketSome depends on system: e.g. bind
Future work - Obtaining metrics
Large amount of output
Number of failed tests for a function → robustness faults in the function
Help:o Assigning expected error codeso Data mining tools / decision tree
Lessons learnt Simple tests can find robustness failures Different methods find different failures There are problems even with the headers Existing applications not up-to-date
o LDAP DN format, component name get Middleware differ heavily
o How-to start, stop; configuration files For complex scenarios, OS call failures
o Detailed workload, complex test setup needed Robustness improving