Top Banner
Nimrod/G and Grid Market A Case for Economy Grid Architecture for Service Oriented Global Grid Computing” Rajkumar Buyya, David Abramson, Jon Giddy Monash University, Melbourne, Australia www.buyya.com/ecogrid www.gridcomputing.com
16

Rajkumar Buyya, David Abramson, Jon Giddy

Feb 05, 2016

Download

Documents

Nimrod/G and Grid Market “ A Case for Economy Grid Architecture for Service Oriented Global Grid Computing”. Rajkumar Buyya, David Abramson, Jon Giddy. Monash University, Melbourne, Australia www.buyya.com/ecogrid. www.gridcomputing.com. 2100. 2100. 2100. 2100. 2100. 2100. 2100. 2100. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • Nimrod/G and Grid MarketA Case for Economy Grid Architecture for Service Oriented Global Grid ComputingRajkumar Buyya, David Abramson, Jon GiddyMonash University, Melbourne, Australiawww.buyya.com/ecogrid www.gridcomputing.com

  • Scalable HPC: Breaking Administrative BarriersDesktopSMPs or SuperComputersLocalClusterGlobalCluster/GridPERFORMANCEInter PlanetCluster/Grid ??IndividualGroupDepartmentCampusStateNationalGlobeInter PlanetUniverseAdministrative BarriersEnterpriseCluster/Grid?

  • Why Grids ? Large Scale Exploration needs themKiller Applications. Solving grand challenge applications using computer modeling, simulation and analysisLife SciencesCAD/CAMAerospaceMilitary ApplicationsDigital BiologyMilitary ApplicationsMilitary ApplicationsInternet & Ecommerce

  • Players in Grid Computing

  • What users want ?Users in Grid Economy & StrategyGrid ConsumersExecute jobs for solving varying problem size and complexityBenefit by selecting and aggregating resources wiselyTradeoff timeframe and costStrategy: minimise expensesGrid ProvidersContribute idle resource for executing consumer jobsBenefit by maximizing resource utilisationTradeoff local requirements & market opportunityStrategy: maximise returns on services

  • mix-and-match Object-orientedInternet-WWWProblem Solving ApproachMarket/Computational Economy

  • Grid Architecture for Computational EconomyGrid Node NGrid UserApplicationGrid Resource BrokerGrid Service ProvidersGrid ExplorerSchedule AdvisorTrade ManagerJob ControlAgentDeployment AgentTrade ServerResource AllocationResourceReservationR1Misc. servicesInformation Server(s)R2RmPricing AlgorithmsAccountingGrid Node1Grid Middleware ServicesHealthMonitorGrid Market ServicesJobExecInfo ?SecureTradingQoSStorageSign-on

  • Economy Grid = Globus + GRACEApplicationsMDSGRAMGlobus Security InterfaceHeartbeat MonitorNexusLocal ServicesLSFCondorGRDQBankPBSTCPSolarisIrixLinuxUDPHigh-level Services and ToolsDUROCglobusrunMPI-GNimrod/GMPI-IOCC++GlobusViewGrid StatusGASSGRACE-TSGARAGridFabricGridApps.GridMiddlewareGridToolsGBankGMDeCashJVMDUROCCore ServicesScienceEngineeringCommercePortalsActiveSheet

  • Nimrod/G : A Grid Resource BrokerA resource broker for managing and steering task farming (parametric sweep) applications on computational Grids based on deadline and computational economy. Key FeaturesA single window to manage & control experimentResource DiscoveryTrade for ResourcesResource Composition & SchedulingSteering & data managementIt allows to study the behaviour of some of the output variables against a range of different input scenarios.

  • A Glance at Nimrod-G BrokerGrid MiddlewareNimrod/G ClientNimrod/G ClientNimrod/G ClientGrid Information Server(s)Schedule AdvisorTrading ManagerNimrod/G EngineGridStoreGrid ExplorerGE GISTM TSRM & TSGrid DispatcherRM: Local Resource Manager, TS: Trade ServerGlobus,Legion, Condor-g,, Ninf,etc.GGCLGlobus enabled node.Legion enabled node.CLCondor enabled node.RM & TSRM & TS

  • A Nimrod/G ClientDeadlineLegion hostsGlobus HostsBezek is in both Globus and Legion Domains

  • Discover ResourcesDistribute JobsEstablish RatesMeet requirements ? Remaining Jobs, Deadline, & Budget ?Evaluate & RescheduleDiscover More ResourcesAdaptive Scheduling algorithmsCompose & Schedule

    Adaptive Scheduling Algorithms

    Execution Time

    (not beyond deadline)

    Execution Cost

    (not beyond budget)

    Time Minimisation

    Minimise

    Limited by budget

    Cost Minimisation

    Limited by deadline

    Minimise

    None Minimisation

    Limited by deadline

    Limited by budget

  • Globus+Legion+Condor/GAustraliaMonash Uni.:Linux clusterSolaris WSNimrod/GGlobus +GRACE_TSEuropeZIB/FUB: T3E/Mosix Cardiff: Sun E6500Paderborn: HPCLineLecce: Compaq SCCNR: ClusterCalabria: Cluster CERN: ClusterPozman: SGI/SP2Globus +GRACE_TSAsia/JapanTokyo I-Tech.:ETL, TuskubaLinux clusterGlobus/LegionGRACE_TSNorth AmericaANL: SGI/Sun/SP2USC-ISI: SGIUVa: Linux Cluster

    InternetInter-Continental Grid

  • Experimentation on the GridWorkload:165 jobs, each need 5 minute of cpu timeDeadline: 1 hrs. and budget: 800,000 unitsStrategy: minimise cost and meet deadlineExecution Cost with cost optimisationAU Peaktime:471205 (G$)AU Offpeak time: 427155 (G$)

    Resource Owner Grid servicesPeak time (G$)Off peak costLinux cluster (60 nodes)Monash, AustraliaGlobus/ Condor205IBM SP2 (80 nodes)ANL, Chicago, USGlobus/LL510Sun (8 nodes)ANL, Chicago, USGlobus/ Fork510SGI (96 nodes)ANL, Chicago, USGlobus/ Condor-G1515SGI (10 nodes)ISI, LA, USGlobus/ Fork1020

  • Execution @ AU Peak Time

    graph.peakoz

    00000

    01001

    01001

    01002

    12013

    22023

    33034

    43044

    44055

    74165

    85276

    105386

    105487

    105467

    84466

    53483

    55395

    24096

    25396

    15367

    06158

    05269

    06259

    074510

    083411

    08449

    07549

    07647

    07747

    08737

    06727

    06728

    05509

    04709

    03509

    036011

    037010

    017011

    008010

    007011

    007010

    00709

    00706

    00608

    00809

    02808

    02709

    03608

    03709

    04809

    05709

    06808

    06607

    08608

    09607

    09704

    08704

    07603

    09803

    07702

    07702

    06802

    07701

    06701

    08501

    08401

    07300

    06300

    05000

    05000

    03000

    02000

    00000

    Linux cluster - Monash (20)

    Sun - ANL (5)

    SP2 - ANL (5)

    SGI - ANL (15)

    SGI - ISI (10)

    Time (minutes)

    Jobs

    schedule.peakoz

    Linux cluster - Monash (20)Sun - ANL (5)SP2 - ANL (5)SGI - ANL (15)SGI - ISI (10)Total CPUsCost of Resources in Use

    974765798.905000000000

    974765834.041010010215

    974765876.6791010010215

    974765916.9462010020325

    974765961.3183120130775

    974766006.09632202309110

    974766050.522433034013160

    974766099.633543044015195

    974766140.084644055630018225

    974766193.7157741651238023305

    974766250.4348852761842028360

    974766279.94581053861842032420

    974766328.70391054872331034435

    974766380.083101054673440032405

    974766416.60910844666776528350

    974766465.887115348310190523285

    974766517.873125539510349027325

    974766569.701132409613113021255

    974766620.597142539613415025275

    974766671.698151536715478022220

    974766714.507150615817342520190

    974766769.82160526919070522215

    974766814.981170625920460022205

    974766866.9361807451020460026230

    974766910.5121908341121724526225

    974766944.954190844922493025210

    974766980.787200754922650525210

    974767023.283200764723734024195

    974767065.222210774723734025200

    974767105.095220873724676525190

    974767146.936220672725931022165

    974767192.881230672826417023175

    974767245.674240550928141519140

    974767286.562250470929550520145

    974767326.268250350930787017130

    974767368.9512603601130787020155

    974767408.3932703701031552520150

    974767453.6232801701132014519150

    974767493.4642800801032471018140

    974767540.2972900701132790018145

    974767585.2143000701033259517135

    974767627.01300070934043016125

    974767670.03631007063510501395

    974767714.383320060835266514110

    974767754.657330080935266517130

    974767803.536330280836199518130

    974767855.566340270936657018135

    974767894.681350360837116017125

    974767931.786360370937116019140

    974767968.023360480937421021150

    974768009.072370570937730021150

    974768046.985370680838036022150

    974768086.273380660738656519130

    974768128.989390860838814022150

    974768173.103400960739276022145

    974768217.748400970440188020120

    974768279.053410870440971019115

    974768316.76242076034175551695

    974768359.758430980341755520115

    974768411.92844077024252201690

    974768447.62744077024282351690

    974768491.79245068024312851690

    974768540.43346077014373751580

    974768579.42846067014389201475

    974768641.75647085014419801475

    974768681.08448084014435301370

    974768742.05949073004498551050

    974768775.3155006300451390945

    974768832.8465105000457420525

    974768877.9835105000459030525

    974768934.9095203000463455315

    974768979.1525302000468155210

    974769036.157540000047120500

    schedule.peakoz

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    Time (in min.)

    Cost of Resources in Use

    graph.peakus

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    Time (in min.)

    Resources (No. of CPUs) in Use

    schedule.peakus

    00000

    01001

    01001

    01001

    02001

    12002

    22012

    33122

    43232

    33333

    63443

    54453

    93442

    93462

    93492

    932102

    83471

    83271

    82371

    82461

    82460

    73460

    76460

    96460

    107210

    108210

    99100

    109000

    109000

    79000

    99000

    105000

    95000

    94000

    91000

    100000

    60000

    90000

    100000

    100000

    100040

    100040

    90070

    80070

    80080

    100080

    102070

    105050

    97050

    99040

    109020

    109010

    79030

    89020

    99020

    109010

    109020

    105030

    95020

    105020

    95020

    95010

    105000

    95000

    105000

    96000

    106000

    88000

    78000

    107000

    103000

    83000

    73000

    73000

    73000

    72000

    51000

    01000

    01000

    01000

    01000

    01000

    00000

    Linux cluster - Monash (5)

    Sun - ANL (10)

    SP2 - ANL (10)

    SGI - ANL (15)

    SGI - ISI (20)

    Time (minutes)

    Jobs

    Graph 2: Computational Scheduling during Australian Off-peak (or US peak) Time.

    Linux cluster - Monash (5)Sun - ANL (10)SP2 - ANL (10)SGI - ANL (15)SGI - ISI (20)Total CPUsCost of Resources in Use

    974761656.456000000000

    974761702.2031010010239

    974761754.5062010010239

    974761808.5353010010239

    974761842.2143020010358

    974761880.5584120020583

    974761917.12842201207103

    974761970.149533122011152

    974762014.434643232014182

    974762049.036733333153015207

    974762092.182763443153020247

    974762131.906854453466521276

    974762164.9268934421838522242

    974762194.8729934621838524272

    974762237.72610934921990527317

    974762274.887109321022927526312

    974762326.45211834715285023262

    974762373.71712832716231021242

    974762410.60913823716545021233

    974762446.3313824617035521228

    974762488.08114824607667520208

    974762529.88115734608280020222

    974762579.98815764608751523279

    974762607.98716964608751525289

    974762657.3491710721012021520218

    974762686.4551710821012021521237

    974762731.018189910013250519226

    974762771.0181910900014505519221

    974762831.1792010900015153519221

    974762892.11217900016260016206

    974762928.728219900016414018216

    974762967.9352210500017525015145

    974762997.035229500017679014140

    974763052.138239400018117013121

    974763110.9724910001925651064

    974763169.769251000001969951050

    974763221.4242660000203220630

    974763279.6652790000204715945

    974763331.22281000002047151050

    974763361.57281000002047151050

    974763422.0552910004020943014110

    974763458.9923010004020943014110

    974763502.175319007021260516150

    974763534.099318007021569015145

    974763568.951328008021884516160

    974763603.3893210008022047018170

    974763650.7363310207022528519193

    974763692.5433410505023499020220

    974763744.762359705023806521253

    974763785.881359904024436022276

    974763824.8513610902025581021251

    974763865.023710901026049020236

    974763907.432387903026997019251

    974763949.05388902027462019241

    974763989.276399902027462020246

    974764031.0554010901028404020236

    974764091.3074110902029181021251

    974764127.2644110503030571018190

    974764172.963429502031903016170

    974764223.7814310502032754517175

    974764262.887439502033081016170

    974764298.924449501033693515155

    974764358.764510500034331015145

    974764391.281469500034482014140

    974764450.2664710500035105015145

    974764502.831479600035259515159

    974764538.6714810600035259516164

    974764580.105498800036858016192

    974764617.047497800037345515187

    974764677.0655010700038145017183

    974764706.6915110300039430013107

    974764765.46252830003974501197

    974764824.65653730003989801092

    974764886.79954730004004751092

    974764923.6454730004083951092

    974764955.0575572000413105973

    974765005.625651000419355644

    974765055.8925701000422410119

    974765106.2515701000422410119

    974765156.5965801000424020119

    974765205.8045901000424020119

    974765249.4576001000425625119

    974765303.161610000042715500

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    Time (in min.)

    Cost of Resources in Use

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    Total CPUs

    Time (in min.)

    Resources (No. of CPUs) in Use

  • Execution @ AU Offpeak Time

    graph.peakoz

    00000

    01001

    01001

    01002

    12013

    22023

    33034

    43044

    44055

    74165

    85276

    105386

    105487

    105467

    84466

    53483

    55395

    24096

    25396

    15367

    06158

    05269

    06259

    074510

    083411

    08449

    07549

    07647

    07747

    08737

    06727

    06728

    05509

    04709

    03509

    036011

    037010

    017011

    008010

    007011

    007010

    00709

    00706

    00608

    00809

    02808

    02709

    03608

    03709

    04809

    05709

    06808

    06607

    08608

    09607

    09704

    08704

    07603

    09803

    07702

    07702

    06802

    07701

    06701

    08501

    08401

    07300

    06300

    05000

    05000

    03000

    02000

    00000

    Linux cluster - Monash (20)

    Sun - ANL (5)

    SP2 - ANL (5)

    SGI - ANL (15)

    SGI - ISI (10)

    Time (minutes)

    Jobs

    schedule.peakoz

    Linux cluster - Monash (20)Sun - ANL (5)SP2 - ANL (5)SGI - ANL (15)SGI - ISI (10)Total CPUsCost of Resources in Use

    974765798.905000000000

    974765834.041010010215

    974765876.6791010010215

    974765916.9462010020325

    974765961.3183120130775

    974766006.09632202309110

    974766050.522433034013160

    974766099.633543044015195

    974766140.084644055630018225

    974766193.7157741651238023305

    974766250.4348852761842028360

    974766279.94581053861842032420

    974766328.70391054872331034435

    974766380.083101054673440032405

    974766416.60910844666776528350

    974766465.887115348310190523285

    974766517.873125539510349027325

    974766569.701132409613113021255

    974766620.597142539613415025275

    974766671.698151536715478022220

    974766714.507150615817342520190

    974766769.82160526919070522215

    974766814.981170625920460022205

    974766866.9361807451020460026230

    974766910.5121908341121724526225

    974766944.954190844922493025210

    974766980.787200754922650525210

    974767023.283200764723734024195

    974767065.222210774723734025200

    974767105.095220873724676525190

    974767146.936220672725931022165

    974767192.881230672826417023175

    974767245.674240550928141519140

    974767286.562250470929550520145

    974767326.268250350930787017130

    974767368.9512603601130787020155

    974767408.3932703701031552520150

    974767453.6232801701132014519150

    974767493.4642800801032471018140

    974767540.2972900701132790018145

    974767585.2143000701033259517135

    974767627.01300070934043016125

    974767670.03631007063510501395

    974767714.383320060835266514110

    974767754.657330080935266517130

    974767803.536330280836199518130

    974767855.566340270936657018135

    974767894.681350360837116017125

    974767931.786360370937116019140

    974767968.023360480937421021150

    974768009.072370570937730021150

    974768046.985370680838036022150

    974768086.273380660738656519130

    974768128.989390860838814022150

    974768173.103400960739276022145

    974768217.748400970440188020120

    974768279.053410870440971019115

    974768316.76242076034175551695

    974768359.758430980341755520115

    974768411.92844077024252201690

    974768447.62744077024282351690

    974768491.79245068024312851690

    974768540.43346077014373751580

    974768579.42846067014389201475

    974768641.75647085014419801475

    974768681.08448084014435301370

    974768742.05949073004498551050

    974768775.3155006300451390945

    974768832.8465105000457420525

    974768877.9835105000459030525

    974768934.9095203000463455315

    974768979.1525302000468155210

    974769036.157540000047120500

    schedule.peakoz

    Time (in min.)

    Cost of Resources in Use

    graph.peakus

    Time (in min.)

    Resources (No. of CPUs) in Use

    schedule.peakus

    00000

    01001

    01001

    01001

    02001

    12002

    22012

    33122

    43232

    33333

    63443

    54453

    93442

    93462

    93492

    932102

    83471

    83271

    82371

    82461

    82460

    73460

    76460

    96460

    107210

    108210

    99100

    109000

    109000

    79000

    99000

    105000

    95000

    94000

    91000

    100000

    60000

    90000

    100000

    100000

    100040

    100040

    90070

    80070

    80080

    100080

    102070

    105050

    97050

    99040

    109020

    109010

    79030

    89020

    99020

    109010

    109020

    105030

    95020

    105020

    95020

    95010

    105000

    95000

    105000

    96000

    106000

    88000

    78000

    107000

    103000

    83000

    73000

    73000

    73000

    72000

    51000

    01000

    01000

    01000

    01000

    01000

    00000

    Linux cluster - Monash (5)

    Sun - ANL (10)

    SP2 - ANL (10)

    SGI - ANL (15)

    SGI - ISI (20)

    Time (minutes)

    Jobs

    Linux cluster - Monash (5)Sun - ANL (10)SP2 - ANL (10)SGI - ANL (15)SGI - ISI (20)Total CPUsCost of Resources in Use

    974761656.456000000000

    974761702.2031010010239

    974761754.5062010010239

    974761808.5353010010239

    974761842.2143020010358

    974761880.5584120020583

    974761917.12842201207103

    974761970.149533122011152

    974762014.434643232014182

    974762049.036733333153015207

    974762092.182763443153020247

    974762131.906854453466521276

    974762164.9268934421838522242

    974762194.8729934621838524272

    974762237.72610934921990527317

    974762274.887109321022927526312

    974762326.45211834715285023262

    974762373.71712832716231021242

    974762410.60913823716545021233

    974762446.3313824617035521228

    974762488.08114824607667520208

    974762529.88115734608280020222

    974762579.98815764608751523279

    974762607.98716964608751525289

    974762657.3491710721012021520218

    974762686.4551710821012021521237

    974762731.018189910013250519226

    974762771.0181910900014505519221

    974762831.1792010900015153519221

    974762892.11217900016260016206

    974762928.728219900016414018216

    974762967.9352210500017525015145

    974762997.035229500017679014140

    974763052.138239400018117013121

    974763110.9724910001925651064

    974763169.769251000001969951050

    974763221.4242660000203220630

    974763279.6652790000204715945

    974763331.22281000002047151050

    974763361.57281000002047151050

    974763422.0552910004020943014110

    974763458.9923010004020943014110

    974763502.175319007021260516150

    974763534.099318007021569015145

    974763568.951328008021884516160

    974763603.3893210008022047018170

    974763650.7363310207022528519193

    974763692.5433410505023499020220

    974763744.762359705023806521253

    974763785.881359904024436022276

    974763824.8513610902025581021251

    974763865.023710901026049020236

    974763907.432387903026997019251

    974763949.05388902027462019241

    974763989.276399902027462020246

    974764031.0554010901028404020236

    974764091.3074110902029181021251

    974764127.2644110503030571018190

    974764172.963429502031903016170

    974764223.7814310502032754517175

    974764262.887439502033081016170

    974764298.924449501033693515155

    974764358.764510500034331015145

    974764391.281469500034482014140

    974764450.2664710500035105015145

    974764502.831479600035259515159

    974764538.6714810600035259516164

    974764580.105498800036858016192

    974764617.047497800037345515187

    974764677.0655010700038145017183

    974764706.6915110300039430013107

    974764765.46252830003974501197

    974764824.65653730003989801092

    974764886.79954730004004751092

    974764923.6454730004083951092

    974764955.0575572000413105973

    974765005.625651000419355644

    974765055.8925701000422410119

    974765106.2515701000422410119

    974765156.5965801000424020119

    974765205.8045901000424020119

    974765249.4576001000425625119

    974765303.161610000042715500

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    Time (in min.)

    Cost of Resources in Use

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    0

    Total CPUs

    Time (in min.)

    Resources (No. of CPUs) in Use