NORTH DAKOTA STATE UNIVERSITY DEPARTMENT OF COMPUTER SCIENCE
© NDSU SOFTWARE TESTING RESEARCH GROUP
Safe Runtime Validation of Behavioral Adaptations in
Autonomic SoftwarePresenter: Tariq M. King
ATC 2011September 2-4, Banff, Alberta, Canada
© NDSU SOFTWARE TESTING RESEARCH GROUP 2
Outline Motivation Introduction Testing Approach Component Design Prototype Related Work Conclusion
© NDSU SOFTWARE TESTING RESEARCH GROUP
Motivation Self-* features in autonomic computing
systems may incorporate dynamic software adaptation System adds, removes, replaces its own
components at runtime
Raises reliability concerns asnew faults may be introduced due to dynamic adaptation
Most AC research neglects that self-test should be an integral part of these types of systems
3
A NEGLECTED SELF-* CHARACTERISTIC
Self-Configure
Self-Optimize
Self-Heal
Self-Protect
Self-Test
FAULTS? FAILURES?
© NDSU SOFTWARE TESTING RESEARCH GROUP 4
To address this issue, we introduced the notion of Autonomic Self-Testing (King et. al 2007)
Idea: Investigate howAutonomic Managers(AMs) can be tailored for purpose of testing the system itself
Test Managers (TMs)Monitor, intercept and validate the adaptive changes requests of AMs
Managed Resource
Introduction AUTONOMIC SELF-TESTING
Monitor
Analyze Plan
ExecuteKnowledge
EffectorSensorAutonomic Manager
Monitor
Analyze Plan
ExecuteKnowledge
Autonomic ManagerTest Manager
© NDSU SOFTWARE TESTING RESEARCH GROUP 5
Initial work on AST concentrated on: Defining validation strategies for TMs
1. Replication with Validation (RV) – tests changes using separate copies of managed resources
2. Safe Adaptation with Validation (SAV) – tests changes in-place, directly on managed resources
Developing prototypes to investigate the feasibility of the RV strategy Lightweight, focused on localized validation
Introduction AUTONOMIC SELF-TESTING (CONT.)
© NDSU SOFTWARE TESTING RESEARCH GROUP 6
In this research, we extend previous work on AST by overcoming two of its limitations.
Major Contributions:
1. Addresses the need for system-wide validation in autonomic software through the description of an automated runtime integration testing approach
2. Investigates AST of a real-world application in which it is too expensive to maintain copies of managed resources for testing – Validates SAV
Introduction CONTRIBUTIONS
© NDSU SOFTWARE TESTING RESEARCH GROUP 7
System-Wide Validation of Behavioral Adaptations Using Self-Testable Autonomic
Components (STACS)
Overview of Testing Approach
© NDSU SOFTWARE TESTING RESEARCH GROUP 8
State-Based Safety Model
1. Extends Zhang (2004) “Enabling Safe Component-Based Software Adaptation” in WADS ‘04
© NDSU SOFTWARE TESTING RESEARCH GROUP 9
Self-Testable Autonomic Component (T, A, R, I, K)
Component Design STAC DEFINITION
© NDSU SOFTWARE TESTING RESEARCH GROUP 10
Abstract MAPE Generics
Managers (T, A)
Multi-Threading Safety Mechanisms
© NDSU SOFTWARE TESTING RESEARCH GROUP 11
Full Access to Source Code Derive suspend, resume, save
and restore state operations from AbstractResource class
Loose coupling of concreteobjects to facilitate stubinjection at runtime
Limited Access to Source Code Use wrapper class (carefully) Research on improving COTS testability
Resources (R)
© NDSU SOFTWARE TESTING RESEARCH GROUP 12
Test Interface (IT)
Test Case Execution Code Coverage Analysis
Performance
© NDSU SOFTWARE TESTING RESEARCH GROUP 13
Communication Virtual Machine (CVM) is a model-driven platform for realizing user-centric communication services Collaboration: FIU & Miami Children’s
Hospital
CVM separates the concerns of modeling, synthesis, coordination, and delivery of communication services into self-contained layers
Layer in which communication in communication takes place is called the Network Communication Broker (NCB).
Prototype COMMUNICATION VIRTUAL MACHINE
© NDSU SOFTWARE TESTING RESEARCH GROUP 14
Allen (2009) leverage AC and open-platform APIs for integrating communication services into NCB
Prototype COMMUNICATION VIRTUAL MACHINE
CVMNCB LAYER
OF
© NDSU SOFTWARE TESTING RESEARCH GROUP 15
Conducted two sets of experimental runs as part of the prototype evaluation: Test Quality Experiments – compared
the effectiveness of test sets in revealing faults and exercising the NCB program code
Performance Experiments – investigates ways to reduce the performance overhead of having a runtime testing process interleaved with the NCB
In both cases, a Mutation Analysis technique was used to simulate faulty change requests,
induce self-testing, and produce the results
Experiment Overview
© NDSU SOFTWARE TESTING RESEARCH GROUP 16
39 mutants were generated for the prototype by planting artificial faults into the interfaces and/or implementations of Skype and Smack 15 created manually, 24 automatically
generated
Test Support Tools: JUnit, Cobertura, MuJava, Eclipse TPTP
Experiments were performed on a windows-based, webcam-enabled Intel® Core 2 Duo 2.4GHz PCs with 2 GB RAM or greater
Setup Environment
© NDSU SOFTWARE TESTING RESEARCH GROUP 17
Fault Detection 35 out of the 39 mutants were detected,
producing a mutation score of 89.7% (Threat, CVM Dev/Test)
Code Coverage Recorded 63% statement coverage and 57%
branch coverage of the NCB implementation 100% method coverage of the communication
service interfaces (SkypeAdapter, SmackAdapter)
Results TEST QUALITY
© NDSU SOFTWARE TESTING RESEARCH GROUP 18
Results (cont.) PERFORMANCE
SKYPE TWO-WAY
TOSMACK
THREE-WAY
© NDSU SOFTWARE TESTING RESEARCH GROUP 19
System-wide SAV is feasible and beneficial for preventing runtime faults in the context of a real-world application
Testing adaptive changes in-place using TMs was challenging: high complexity, synchronization Significant performance overhead, further
research Safety mechanisms (locks) were instrumental
during development and runtime testing
Selecting and tailoring open-source testing tools presented unnecessary difficulties
Lessons Learned
© NDSU SOFTWARE TESTING RESEARCH GROUP 20
Da Costa et al. (2010) JAAF+T: Java Self-Adaptive Framework +
Test Modify MAPE structure to include a self-test
activity
Stevens et al. (2007) and Ramirez et al. (2008) Prototypes of Replication with Validation Autonomic Container & Autonomic Job
Scheduler
Zhang et al. (2009) and Zhao et al. (2005) Runtime verification of self-* systems
Related Work
© NDSU SOFTWARE TESTING RESEARCH GROUP 21
Presented a system-wide approach to runtime integration testing in autonomic software
Still many open research issues: Dynamic test generation Dynamic regression test scheduling Runtime test case maintenance Reducing overhead of runtime testing
Latest Direction: Cloud (CASCON 2011)
Conclusion
© NDSU SOFTWARE TESTING RESEARCH GROUP 22
Acknowledgements Dr. Masoud Milani, FIU
Dr. S. Masoud Sadjadi, FIU
Participants of the REU Summer 2006 and 2007 Programs on Autonomic Computing at FIU
Graduate students and research collaborators on CVM project
This work has been supported in part by the NSF under grant IIS-0552555.
© NDSU SOFTWARE TESTING RESEARCH GROUP 23
Thank You!Questions?
¿Preguntas?問題Sawwalвопросы質問domandeερωτήσεις