A Case for Economy Grid Architecture for Service Oriented Grid Computing Authors: Rajkumar Buyya, David Abramson & Jonathan Giddy Presenter: Diego Lopez.

A Case for Economy Grid A Case for Economy Grid Architecture for Service Architecture for Service

Oriented Grid ComputingOriented Grid Computing

Authors: Authors: Rajkumar Buyya, David Abramson & Jonathan GiddyRajkumar Buyya, David Abramson & Jonathan Giddy

Presenter: Diego LopezPresenter: Diego Lopez

Agnostic: Djuradj BabicAgnostic: Djuradj Babic

June 12, 2006June 12, 2006 22

OutlineOutline

IntroductionIntroduction Grid Economy and Resource Management Grid Economy and Resource Management

IssuesIssues Economy Models and Related WorkEconomy Models and Related Work GRACEGRACE Resource Trading and Scheduling Resource Trading and Scheduling

ExperimentationExperimentation Conclusion and Future WorkConclusion and Future Work

June 12, 2006June 12, 2006 Source: Buyya, R http://www.buyya.comSource: Buyya, R http://www.buyya.com 33

1. Introduction1. Introduction“ “ We expect that an economy driven We expect that an economy driven

approach to resource management and approach to resource management and scheduling will make a great impact on the scheduling will make a great impact on the

eventual success and widespread eventual success and widespread adoption of the Grid in day-to-day adoption of the Grid in day-to-day

computational activities.” computational activities.”

Dr. BuyyaProf. Abramson

Researcher Giddy

June 12, 2006June 12, 2006 44

1. Introduction1. Introduction

Grid environment is complex ($$$)Grid environment is complex ($$$) Different access cost modelsDifferent access cost models Dynamically varying loads and availability Dynamically varying loads and availability

conditionsconditions Use of economic models in the Grid to Use of economic models in the Grid to

encourage participation and wide-scale encourage participation and wide-scale adoptionadoption

Proposal of computational economy Proposal of computational economy framework that leverage existing Grid sitesframework that leverage existing Grid sites

June 12, 2006June 12, 2006 Source: Buyya, R A Case for Economy Grid ArchitecSource: Buyya, R A Case for Economy Grid Architecture for Service Oriented Grid Computing (Pg. 2)ture for Service Oriented Grid Computing (Pg. 2)

55

View of Economic GridView of Economic Grid

June 12, 2006June 12, 2006 66

2. Grid Economy and Resource 2. Grid Economy and Resource Management IssuesManagement Issues

Establish policies that promote Grid Establish policies that promote Grid resource sharingresource sharing

2 key players in Grid economy2 key players in Grid economy Resource providers (GSP)Resource providers (GSP) Resource consumers (GRB)Resource consumers (GRB)

Consumers interact with brokers to Consumers interact with brokers to express their express their budgetbudget and and deadlinedeadline requirements from the requirements from the GridGrid

June 12, 2006June 12, 2006 77

Proposal of GRACEProposal of GRACE Grid Architecture for Computational Grid Architecture for Computational

EconomyEconomy Leverage of existing infrastructures:Leverage of existing infrastructures:

Globus/LegionGlobus/Legion Condor/GCondor/G

Provide an infrastructure that allows for:Provide an infrastructure that allows for: Info/Market directory for publicizing entitiesInfo/Market directory for publicizing entities Model for determining value of resourcesModel for determining value of resources Resource pricing schemesResource pricing schemes Accounting, Billing and Payment mechanismsAccounting, Billing and Payment mechanisms

June 12, 2006June 12, 2006 88

3. Economy Models and Related 3. Economy Models and Related WorkWork

Possible economic models for resource trading Possible economic models for resource trading and pricing strategiesand pricing strategies

Commodity MarketCommodity Market Posted PricePosted Price BargainingBargaining Tendering/Contract-NetTendering/Contract-Net AuctionAuction Bid-based Proportional Resource SharingBid-based Proportional Resource Sharing Community/Coalition/Bartering Community/Coalition/Bartering

June 12, 2006June 12, 2006 99

Examples of Computational Examples of Computational Economy SystemsEconomy Systems

SYS_NAMESYS_NAME ECO_MODELECO_MODEL PLATFORMPLATFORM

Mariposa (’96)Mariposa (’96) BiddingBidding Dist.DbaseDist.Dbase

Mungi (’02)Mungi (’02) CommodityCommodity Storage serversStorage servers

Popcorn (’98)Popcorn (’98) AuctionAuction Web browserWeb browser

Mojo NationMojo Nation Credit-based or Credit-based or barteringbartering

Network Network storagestorage

Java MarketJava Market QoS based QoS based computationalcomputational

Web browser Web browser (applets)(applets)

June 12, 2006June 12, 2006 1010

4. GRACE4. GRACE

Use of well-adopted Grid technologies, Use of well-adopted Grid technologies, Globus/CondorGlobus/Condor

Development of middleware services for Development of middleware services for resource trading using different economic resource trading using different economic modelsmodels

Development of advanced user-centric Development of advanced user-centric Grid resource brokersGrid resource brokers

June 12, 2006June 12, 2006 http://ipdps.cc.gatech.edu/2000/papers/Abramson.pdhttp://ipdps.cc.gatech.edu/2000/papers/Abramson.pdff

1111

4.1 Grid Resource Broker (GRB)4.1 Grid Resource Broker (GRB)

Mediator between user and grid resourcesMediator between user and grid resources *Nimrod – parametric modeling language *Nimrod – parametric modeling language Use of Nimrod/G broker (superscheduler)Use of Nimrod/G broker (superscheduler)

Job Control AgentJob Control Agent Schedule AdvisorSchedule Advisor Grid ExplorerGrid Explorer Trade ManagerTrade Manager Deployment AgentDeployment Agent

June 12, 2006June 12, 2006 Src: http://www.buyya.com/ecogrid/Src: http://www.buyya.com/ecogrid/ 1212

Nimrod/G BrokerNimrod/G Broker

Grid Middleware

Nimrod/G ClientNimrod/G ClientNimrod/G Client

Grid Information Server(s)

Schedule Advisor

Trading Manager

Nimrod/G Engine

GridStore

Grid Explorer

GE GISTM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

Globus, Legion, Condor, etc.

G

G

CL

Globus enabled node.Legion enabled node.

LCondor enabled node.

RM & TSRM & TS

June 12, 2006June 12, 2006 1313

4.2 Economy Grid Middleware in 4.2 Economy Grid Middleware in Globus ContextGlobus Context

Trade Server (TS) – maximize the Trade Server (TS) – maximize the resource utility and profit for its ownerresource utility and profit for its owner

Pricing Policies – define prices for Pricing Policies – define prices for resources based on economic models resources based on economic models previously mentionedpreviously mentioned

Resource Accounting and charging – Resource Accounting and charging – tracking resource usage for billing and tracking resource usage for billing and auditing purposesauditing purposes

June 12, 2006June 12, 2006 1414

4.3 Grid Open Trading Protocols 4.3 Grid Open Trading Protocols and Deal Templateand Deal Template

Establish rules and format for exchanging Establish rules and format for exchanging commands between a GRACE client commands between a GRACE client (Trade Manager) and Trade Server(Trade Manager) and Trade Server

Deal Template (DT) containsDeal Template (DT) contains CPU timeCPU time Storage requirementsStorage requirements Initial offerInitial offer

This trading overhead can be reduced if This trading overhead can be reduced if prices are announced via GISprices are announced via GIS

June 12, 2006June 12, 2006 1515

4.4 4.4 PricingPricing, Accounting, and , Accounting, and Payment MechanismsPayment Mechanisms

N-ways to determine resource pricingN-ways to determine resource pricing Fixed price model (no QoS like today’s www)Fixed price model (no QoS like today’s www) Usage timing (peak, off-peak)Usage timing (peak, off-peak) Bulk purchaseBulk purchase Demand and supplyDemand and supply Loyalty of customers (i.e. frequent flyer miles)Loyalty of customers (i.e. frequent flyer miles) Calendar basedCalendar based

June 12, 2006June 12, 2006 1616

4.4 Pricing, 4.4 Pricing, AccountingAccounting, and , and Payment MechanismsPayment Mechanisms

Service items to be chargedService items to be charged CPU timeCPU time MemoryMemory Storage usedStorage used Software and Libraries accessed (ASP)Software and Libraries accessed (ASP)

Access to these services can be chargedAccess to these services can be charged IndividuallyIndividually Combination (costing matrix)Combination (costing matrix)

June 12, 2006June 12, 2006 *not incorporated into GRACE described in this pap*not incorporated into GRACE described in this paperer

1717

4.4 Pricing, Accounting, and 4.4 Pricing, Accounting, and Payment MechanismsPayment Mechanisms

Prepaid – purchase credits from GSP or Prepaid – purchase credits from GSP or Grid BankGrid Bank

Use and pay later (like electricity)Use and pay later (like electricity) Pay as you go (wireless calling cards)Pay as you go (wireless calling cards) Grants basedGrants based *Billing services handled by 3*Billing services handled by 3rdrd party: party:

NetChequeNetCheque PaypalPaypal

June 12, 2006June 12, 2006 1818

4.5 System Prototype & Demo 4.5 System Prototype & Demo ExperiencesExperiences

Prototype of the Nimrod/G resource Prototype of the Nimrod/G resource brokering demo held during HPDC 2000brokering demo held during HPDC 2000

Parameter study experiment performed Parameter study experiment performed over Grid resources located in both over Grid resources located in both Australia and the USAustralia and the US

Ability to change deadline and budget to Ability to change deadline and budget to trade-off cost vs. timeframe to illustrate trade-off cost vs. timeframe to illustrate Grid marketplace dynamicsGrid marketplace dynamics

June 12, 2006June 12, 2006 1919

5. Resource Trading and 5. Resource Trading and Scheduling ExperimentationScheduling Experimentation

Experiment to test operation of Grid Trade Experiment to test operation of Grid Trade Server across 5 systems (165 jobs)Server across 5 systems (165 jobs)

Use of Posted Price Market Model for the Use of Posted Price Market Model for the Nimrod/G brokeringNimrod/G brokering

Runs during peak time vs. off-peak timeRuns during peak time vs. off-peak time Access price expressed in Grid units per Access price expressed in Grid units per

CPU second (G$)CPU second (G$) Resource/service price provided by Resource/service price provided by

GRACE frameworkGRACE framework

June 12, 2006June 12, 2006 2020

Economy Grid ResultsEconomy Grid Results Cost-Optimization algorithm successfullyCost-Optimization algorithm successfully

Minimized artificial access cost per resourceMinimized artificial access cost per resource Completed within one-hour deadlineCompleted within one-hour deadline

Initial calibration phase ensures Initial calibration phase ensures completion within budget/time constraintscompletion within budget/time constraints

Scheduler excluded usage of resources Scheduler excluded usage of resources during peak timeduring peak time

Scheduler predictions met deadline using Scheduler predictions met deadline using least-expensive resources availableleast-expensive resources available

June 12, 2006June 12, 2006 2121

6. Conclusion and Future Work6. Conclusion and Future Work

GRACE leverages existing middleware GRACE leverages existing middleware systems (Condor/Legion/Globus)systems (Condor/Legion/Globus)

Nimrod/G can discover best resource Nimrod/G can discover best resource providers based on user’s requirementsproviders based on user’s requirements

Nimrod/G does not support dynamic prices Nimrod/G does not support dynamic prices once initial scheduling is proposedonce initial scheduling is proposed

Nimrod/G Nimrod/G PortalPortal available … available …