Mendosus A SAN-Based Fault Injection Test-Bed for Construction of Highly Available Network Services Xiaoyan Li, Richard Martin, Kiran Nagaraja, Thu D. Nguyen and Bin Zhang Dept. of Computer Science, Rutgers University http://www.panic-lab.rutgers.edu
29
Embed
Mendosus A SAN-Based Fault Injection Test-Bed for Construction of Highly Available Network Services Xiaoyan Li, Richard Martin, Kiran Nagaraja, Thu D.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
MendosusA SAN-Based Fault Injection Test-Bed for
Construction of Highly Available Network Services
Xiaoyan Li, Richard Martin, Kiran Nagaraja,
Thu D. Nguyen and Bin Zhang
Dept. of Computer Science, Rutgers University
http://www.panic-lab.rutgers.edu
Talk Outline
Motivation Design Implementation Benchmarks Case Studies Related Work Future Work
Motivation
Ubiquitous network access exponential growth in network services
Availability is one key challenge Networked systems are comprised of large numbers of
heterogeneous components Faults are not uncommon Complex interaction between components
Examples of costly failures: Ebay, Brittanica
Currently difficult to assess service availability How to analyze impact of failures? How to set up an appropriate test-bed?
Mendosus
Goal: provide infrastructure for service designers to assess the availability of network services
Overview: Provide flexible infrastructure to accurately model a
variety of different networking systems from the application’s point-of-view
Run application in real-time and inject faults to assess application’s behavior
Two key components: Real-time emulation of a variety of interconnects General fault injection infrastructure
Vision
Map available resources to emulated network
Design
Mendosus Architecture
Applications
KernelLatency
Routing
Fault Inclusion
Mendosus daemon
Central Controller
Network State
User Level
Fast & Reliable SAN
Emulator Module
Events
Design Decisions
Central controller Advantage: consistent network and fault information Disadvantage: limits scalability
Not involved in network emulation so should still scale well to targeted system sizes (thousands or tens of thousands of components)
Entire network state is maintained at each end node Advantage: performance Disadvantage: limits scalability
Only maintain state for LAN
Emulation module embedded within kernel Advantage: no modifications to application code Disadvantage: more difficult to modify and extend
Functional Components
Topology Maintenance
Fault Injection
Emulation
Topology Maintenance
Specification - simple ns-2 like topology scripts Specify available resources
Central controller manages topology Initializes original topology on each node Consistent view
Real time topology changes Specified as scripted events
Extend Mendosus to emulate other networks WAN: Build in performance dynamics model Wireless LAN - Realistic fault and performance models
Support pluggable modules within network components which add functionality and additional failures ! Intelligent Routing protocols (E.g. HSRP) Dynamic DNS, RR DNS
Summary
Test-bed for service designers to systematically analyze network and protocol design against failures
Results show that real-time emulation is feasible given capability of current SAN networks
Demonstrated the flexibility and usefulness of Mendosus through 2 case studies
Another step towards building highly available services…