Grid Resource Management: Challenges, Approaches, & Solutions Dr. Rajkumar Buyya Cloud Computing and Distributed Systems (CLOUDS) Lab. The University of.
Post on 13-Jan-2016
215 Views
Preview:
Transcript
Grid Resource Management: Challenges, Approaches, &
Solutions
Dr. Rajkumar BuyyaCloud Computing and Distributed Systems (CLOUDS)
Lab. The University of MelbourneMelbourne, Australiawww.cloudbus.org
2
Agenda
Grid Challenges Revisited Foundations of Resource Management
Challenges Decentralised Grid Scheduling Approach
Service Oriented-Grid Architecture Market-Oriented Grid Middleware On-Demand Assembly of UtilityGrids Summary
Scheduling Economics
Grid
Grid Economy
3
Grid Challenges: RM and Scheduling
Security
Resource Allocation & Scheduling
Data locality
Network Management
System Management
Resource Discovery
Uniform Access
Computational Economy
Application Construction
4
Open-Source Grid Middleware Projects
5
Driving Theme:Community vs. Utility Grids
Type
Feature
Community Grids Utility Grids
User QoS Best effort Contract/SLA
Service Pricing
Not considered /
free access
Usage, QoS level, Market supply and demand
Example Middleware
Globus, Condor, OMII, Unicore
Nimrod-G, Gridbus, & many inspired efforts (CatNets, Sun Grid Market, IBM..)
6
Why “integrate” Scalable Architecture, Business Models, and
Optimal Allocation
WWG
Pushes Grid computing into mainstream
computing
Gridbus
Foundations of Grid Resource Management
8
Resource Management Systems (RMSs): General Goals
Manage the Supply and Demand for Resources Allocate Resources such that:
They are allocated on fairly They are effectively utilised Most users are satisfied High priority jobs are given prominence
In a wide-area systems such as Grids: Additionally, we need to make sure that Resource
Providers are given appropriate “incentive” for their contribution and to ensure sustained resource sharing.
Therefore, Resource Management is a Challenging Task due to
their complex characteristics and goals.
9
Resources in Grid Environment: Characteristics
Autonomous Each have their own resource allocation policy no central control
Heterogeneous and substrate: Each resource can be different – SMPs, Clusters,
Linux, UNIX, Windows, Intel, etc. Resource owners have their own policies or
scheduling mechanisms. Varying Availability
Resource allocation/co-allocation challenge The amount of resource available various with time
10
Resources in Grid Environment: Characteristics
Size (large number of nodes, providers, consumers) Heterogeneity of:
resources (PCs, Workstatations, clusters, and supercomputers)
fabric management systems (single system image OS, queuing systems, etc.)
fabric management polices applications (scientific, engineering, and commerce) application requirements (CPU, I/O, memory, and/or
network intensive) demand patterns (peak, off-peak)
Geographic distribution and different time zones Differing goals (producers and consumers have different
objectives and strategies) Unsecure and Unreliable environment
11
RMS Architecture Alternatives
R1
R2
Rn
. ..
...
R
R1
Rn
..
R
R
R1
Rn
..
…….
Centralized (Single Resource)
Decentralized (Hierarchical)
(Multiple Domains)
Centralized (Multiple Resources)
(Single/Multiple Domains)
Decentralized(Self coordinated or Job Pool)
(Multiple Domains)
R
Scheduler Model Scheduler Architecture Example System
Unix, Linux,Windows OS
Multi-clusters/Grid Systems:
AppLes, SGE-E Nimrod-G,
Gridbus BrokerEnterprise Grids:
Alchemi, SETI@Home
1. Cooperative Clusters (exchange
workload)2. P2P systems (no exchange of jobs)
Cluster systems:PBS, LSF,
SGE, Condor
(Resource Broker orGrid Scheduler)
(Job Pool)
(Jobs) (Queue)
(Local Scheduler)
12
Centralized Vs. Decentralized Resource Management in Grids?
Traditional systems use centralised policy that need complete state-information and common fabric management policy or decentralised consensus-
based policy. Due to too many heterogenous parameters in the Grid it is
impossible to define/get: system-wide performance matrix and common fabric management policy that is acceptable to all.
Therefore, “decentralised” approach towards management of resource is advocated.
Question is: Market-oriented approaches found to be very effective in
managing complexities of decentralisation present in “human” economy, can they be applied to Grid Resource Management?
A Case for Economy-based Grid Resource Management
Service-Oriented Grid Architecture
14
What do Grid players want?
Grid Consumers Execute jobs for solving varying problem size and
complexity Benefit by utilizing distributed resources wisely Tradeoff timeframe and cost
Strategy: minimise expenses
Grid Providers Contribute resources for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity
Strategy: maximise return on investment
15
What do Grid players want & require?
Grid Service Consumers (GSCs): - minimize expenses, meet QoS How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I discover services and map jobs to meet my QoS needs? How do I manage Grid dynamics and get my work done? …
Grid Service Providers (GSPs):– maximise ROI How do I decide service pricing models ? How do I specify them ? How do I translate them into resource allocations ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …
They need mechanisms, tools and technologies that help them in value expression, value translation, and value enforcement.
16
Principle 1: Service Oriented Architecture (SOA)
A SOA is a contractual architecture for offering and consuming software as services.
There are four entities that make up an SOA service provider, service registry, and service consumer (also known as service requestor).
The functions or tasks that the service provider offers, along with other functional and technical information required for consumption, are defined in
the service definition or contract.
provider
registry
consumer
contract
17
Principle 2: Market-Oriented (Grid) Computing- (a) Sustained Resourced Sharing and (b)
Effective Management of Shared Resources
Grid Economy
18
Market-based Systems = Self-managed and Self-regulated systems.
Manage Complexity Supply and
Demand Enhance Utility
1
32
penalty
19
Cost Matrix @ Grid site X
Non-uniform costing Encourages the use
of local resources first
Real accounting system can control machine usage
11 33
22 11User 5User 5
Mach
ine 1
Mach
ine 1
User 1User 1
Mach
ine 5
Mach
ine 5
Resource Cost = Function (cpu, memory, disk, network, software, QoS, current demand, etc.)
Simple: price based on peaktime, offpeak, discount when less demand, ..
20
Market-based Computing Systems Need to Support:
To enable users (GSPs and GSCs) to realise economic value, market-based systems need to provide mechanisms for:
Value Expression a means to express their requirements, valuations, and
objectives Value Translation
scheduling policies to translate them to resource allocations
Value Enforcement mechanisms to enforce the selection and allocation of
differential services, and dynamic adaptation to changes in their availability at runtime
Market mechanisms, accounting and payment, Reservation of resources.
21
Grid Node N
Service-Oriented Grid Architecture
Grid Service Consumer
Pro
gra
mm
ing
En
viro
nm
ents
Grid Resource Broker
Grid Service Providers
Grid Explorer
Schedule Advisor
Trade Manager
Job ControlAgent
Deployment Agent
Trade Server
Resource Allocation
ResourceReservation
R1
Misc. services
Information Service
R2 Rm…
Pricing Algorithms
Accounting
Grid Node1
…
Core Middleware Services
…
…
HealthMonitor
Grid Market Services
JobExec
Info ?
Secure
Trading
QoS
Storage
Sign-on
Grid Bank
Ap
pli
cati
on
s
Data Catalogue
22
Market-Oriented Grid Software: A union of Gridbus and other
technologies
AIXSolarisWindows Linux
.NETGridFabricSoftware
GridApplications
Core GridMiddleware
User-LevelMiddleware
GridBank
Grid Exchange & Federation
JVM
Grid Scheduling:
Task, Parametric, and Components Programming
Gridbus Resource Broker
MPI
Condor SGE TomcatPBS
Aneka
Workflow APIs
IRIX OSF1 Mac
Libra
Globus Unicore ……Grid
MarketDirectory
PDB
CDB
Worldwide Grid
GridFabricHardware
Grid PortalsScience Commerce Engineering ……Collaboratories
……
Grid Storage Economy
Gri
d E
con
om
y
NorduGrid XGrid
ExcellGrid
Grid Workflow Engine
APIs/Tools:
23
On Demand Assembly of Services in Market-Oriented Grid Environments
ASP Catalogue
Grid Info Service
Grid Market Directory
GSP(Accounting Service)
GridbusGridBank
GSP(e.g., UofM)
PEGSP
(e.g., VPAC)
PE
GSP(e.g., IBM)
CPUorPE
Grid Service (GS)
(Globus)
Aneka
GS
GTS
Cluster Scheduler
Job
8
GridResource Broker
2
Visual Application Composer
Application CodeExplore
data1
36
45
Resu
lts9 7
Results+
Cost Info
10
11
Bill
12Data Catalogue
24
On Demand Assembly of Services in Market-Oriented Grid Environments:
Putting Them All Together
ASP Catalogue
Grid Info Service
Grid Market Directory
GSP(Accounting Service)
GridbusGridBank
GSP(e.g., UofM)
PEGSP
(e.g., VPAC)
PE
GSP(e.g., IBM)
CPUorPE
Grid Service (GS)
(Globus)
Aneka
GS
GTS
Cluster Scheduler
Job
8
GridResource Broker
2
Visual Application Composer
Application CodeExplore
data1
36
45
Resu
lts9 7
Results+
Cost Info
10
11
Bill
12Data Catalogue
25
Summary
Resource management is a complex undertaking as systems need to be adaptive, scalable, competitive,…, and driven by QoS.
A “computational economy”-based resource allocation helps in regulating the supply-and-demand for resources.
The use of economic paradigm for resource management and scheduling is essential for pushing Grids into mainstream computing.
Next: Nimrod-G Broker, Globus middleware, Aneka Cloud Middleware
top related