Checkpoint & Restart for Distributed Components in XCAT3 Sriram Krishnan* Indiana University, San Diego Supercomputer Center & Dennis Gannon Indiana University.
Post on 28-Dec-2015
218 Views
Preview:
Transcript
Checkpoint & Restart for Distributed Components in XCAT3
Sriram Krishnan*
Indiana University, San Diego Supercomputer Center
&
Dennis Gannon
Indiana University
*srikrish@cs.indiana.edu
Long-running Distributed Applications on the Grid
The Problem:1 Launch simulation at Y2. Launch simulation at Z3. Link both simulations4. Execute both simulations5. Store results at X
Y
Z
The GridThe Grid
X
Need an effective way to orchestrate such computations
Checkpoint & Restart
MotivationBasic fault tolerance via periodic checkpointing
Rollback to saved checkpoint upon failure
Dynamic rescheduling of jobsCheckpoint and restart on another location
Checkpointing GoalsCorrectnessPortabilityMinimal checkpoint sizeScalability InteroperabilityCheckpoint Availability
Outline
Motivation Background
The XCAT3 frameworkCheckpoint & Restart
Checkpointing & Restart in XCAT3Software TechniquesAlgorithmsExperiments
Conclusions & Future work
Application Orchestration: Component Architectures
A Component Architecture consists of two parts: Components
Software objects that implement a set of required behaviors Frameworks
A runtime environment A set of services used by components
BenefitsEncapsulation, modular construction of programs (via
composition), reuse Component Architectures adopted in various domains
Business: EJB, CCM, COM/DCOMScientific Computing: CCA
Common Component Architecture
A ComponentID for identification & management purposes Ports: the public interfaces of a component
Defines the different ways we can interact with a component and the ways the component uses other services and components.
Image ProcessingComponent
setImage(Image I)
Image getImage()
adjustColor()
setFilter(Filter)calls doFFT(…)
Provides Ports - interfacesfunctions provided by component
Uses Ports - interface of aservice used by component
XCAT3: CCA Framework for the Grid
Grid Service Extensions (GSX) Toolkit used for OGSI Compatible Grid servicesStandard protocols used by Grid services: SOAP, HTTPhttp://www.extreme.indiana.edu/xgws/GSX
A Component is represented as a set of Grid services Provides ports, ComponentID’s are Grid services Uses ports are Grid service clients
Sriram Krishnan and Dennis Gannon. XCAT3: A Framework for CCA Components as OGSA Services. In HIPS 2004, 9th International Workshop on High-Level Parallel Programming Models and Supportive Environments. April 2004.
Checkpointing: Software Techniques
System-level TechniquesAutomatic transparent checkpointing for an
application at the operating system or middleware level
User-defined TechniquesNon-transparent checkpointing for an application that
relies on the programmer to identify the minimal information needed for restart
Checkpointing: Software Techniques
Transparent to the user: No expertise required
Not very portable across platforms
Larger checkpoint sizes: Typically complete process images stored
Less flexible: Application is treated as a black box
Not transparent to the user: Considerable expertise required
More portable across platforms
Smaller checkpoint sizes: Only minimal state stored
More flexible: Application information can be used
System-Level User-defined
Checkpointing: Examples
System-level TechniquesCondorLAM-MPIEnterprise Java BeansCORBA Components
User-defined TechniquesCUMULVSEnterprise Java BeansCORBA Components
Global Grid Forum: Grid Checkpoint/Recovery GroupUser-defined checkpointing
APIs for Grid servicesDo not address consistent
global checkpoints for distributed applications
A set of individual checkpoints that constitute a state that occurs in a failure-free, correct execution
Checkpointing Technique in XCAT3
User-defined & System-assistedUser is responsible for identifying local component stateFramework is responsible for:
Generating complete state of the component, viz. local component state, connection state, and environment state
Algorithms for generating global component states, and storing them into stable storage
Component writer implements the following methods: generateComponentState() loadComponentState() resumeExecution()
Distributed Checkpointing
Algorithm Overview: Coordinated blocking checkpoint algorithmBlock all port communication between componentsTake individual checkpoints, and commit them
atomicallyResume port communication between components
Novelty: Application to RPC-based component frameworkTypically, such algorithms are applied to messaging
frameworks
The Big Picture
ApplicationCoordinator
PersistentStorage
X Y
Z
Distributed Components on the Grid
Federation of Master (MS) & Individual Storage (IS) Services
MS
ISISISIS
Checkpoint Algorithm
ApplicationCoordinator
PersistentStorage
X Y
Z
MS
ISISISIS
CheckpointComponents
Checkpoint Algorithm
ApplicationCoordinator
PersistentStorage
X Y
Z
MS
ISISISIS
Block all port communication
between components
Checkpoint Algorithm
ApplicationCoordinator
PersistentStorage
X Y
Z
MS
ISISISIS
All communication between components blocked
Checkpoint Algorithm
ApplicationCoordinator
PersistentStorage
X Y
Z
MS
ISISISIS
Find best available Storage service URLs
Checkpoint Algorithm
ApplicationCoordinator
PersistentStorage
X Y
Z
MS
ISISISIS
Store checkpoints intoStorage services
Checkpoint Algorithm
ApplicationCoordinator
PersistentStorage
X Y
Z
MS
ISISISIS
Return storageID’sfor stored state
Checkpoint Algorithm
ApplicationCoordinator
PersistentStorage
X Y
Z
MS
ISISISIS
Atomically update locatorsfor individual checkpoints
Checkpoint Algorithm
ApplicationCoordinator
PersistentStorage
X Y
Z
MS
ISISISIS
Un-block communication between components
Checkpointing: Correctness
Consistency of Global CheckpointA flavor of coordinated blocking algorithms – well
accepted to be correct
Atomicity of CheckpointsLocators for the global checkpoint are updated
atomically after all components have been checkpointed
Not possible to have a scenario where a global checkpoint consists of a combination of old and new individual checkpoints
Restart Algorithm
Also implemented by the Application Coordinator Details
Destroy executing instances, if need beRestart all components (possibly on other resources)Load state of components from the Storage servicesResume execution of all control threads, after the
states of every component have been loaded from the Storage services
Test Application: Chem-Eng Simulation
Based on the simulation of copper electro-deposition on resistive substrate (NCSA-UIUC)Master-Worker model of executionVariable number of workers, and data size per worker
generateComponentState(), loadComponentState(), and resumeExecution() methods added to support checkpointing and restartRequired identification of the various execution states
of the master and worker components
Experiment Setup
Hardware setup8 node Linux cluster
2.8GHz dual processor Intel Xeon processorsRed Hat Linux 8.02GB Memory1Gbps EthernetSUN’s JDK 1.4.2_04
Federation of 1 Master & 8 Individual Storage services used
Single GSX-based Handle Resolver
Checkpointing: Master Processing
Checkpointing: Workers Processing
Future Work
FrameworkIntegration with the Web Service Resource
Framework (WSRF)
Fault ToleranceFault MonitoringReliable communication between componentsCheckpoint OptimizationsStorage Service Optimizations
ApplicationsUse of XCAT3 for LEAD (http://lead.ou.edu)
Conclusions
A framework for checkpointing & restart of distributed applications on the GridCCA-based component framework consistent with
Grid standardsUser-defined, platform-independent checkpointsAPIs for checkpointing, and algorithms for capturing
global checkpoints and for restart provided by the framework
http://www.extreme.indiana.edu/xcat/
Appendix
OGSI Compatibility
Representation for Provides portsIn traditional Grid/Web services, multiple ports of the
same portType are semantically equivalentCCA allows multiple ports of the same type
CCA ports can not be mapped to Web service ports!Hence, every Provides port is mapped as a separate
Grid serviceA single portType containing the Provides port interface
Representation for Uses portsClients of Grid services (Provides ports)Connections to Provides ports made at runtime
OGSI Compatibility
Representation for the ComponentIDAlso a Grid serviceActs as a Manager for the other Provides portsContains SDEs containing GSH/GSRs for the various
Provides ports
The Provides ports and ComponentID services, and the Uses ports communicate via shared state
AcmeFFT
component
Building Applications by Composition
Connect Uses Ports to Provides Ports.
Image ProcessingComponent
getImage()
adjustColor()
Image tool graphical interface component
Imagedatabase
component
setImage(…)
doFFT(…)
Restart Algorithm
Test Application: Chem-Eng Simulation
top related