Rajkumar Buyya, David Abramson, Jon Giddy

Nimrod/G and Grid MarketA Case for Economy Grid Architecture for Service Oriented Global Grid ComputingRajkumar Buyya, David Abramson, Jon GiddyMonash University, Melbourne, Australiawww.buyya.com/ecogrid www.gridcomputing.com

Scalable HPC: Breaking Administrative BarriersDesktopSMPs or SuperComputersLocalClusterGlobalCluster/GridPERFORMANCEInter PlanetCluster/Grid ??IndividualGroupDepartmentCampusStateNationalGlobeInter PlanetUniverseAdministrative BarriersEnterpriseCluster/Grid?

Why Grids ? Large Scale Exploration needs themKiller Applications. Solving grand challenge applications using computer modeling, simulation and analysisLife SciencesCAD/CAMAerospaceMilitary ApplicationsDigital BiologyMilitary ApplicationsMilitary ApplicationsInternet & Ecommerce

Players in Grid Computing

What users want ?Users in Grid Economy & StrategyGrid ConsumersExecute jobs for solving varying problem size and complexityBenefit by selecting and aggregating resources wiselyTradeoff timeframe and costStrategy: minimise expensesGrid ProvidersContribute idle resource for executing consumer jobsBenefit by maximizing resource utilisationTradeoff local requirements & market opportunityStrategy: maximise returns on services

mix-and-match Object-orientedInternet-WWWProblem Solving ApproachMarket/Computational Economy

Grid Architecture for Computational EconomyGrid Node NGrid UserApplicationGrid Resource BrokerGrid Service ProvidersGrid ExplorerSchedule AdvisorTrade ManagerJob ControlAgentDeployment AgentTrade ServerResource AllocationResourceReservationR1Misc. servicesInformation Server(s)R2RmPricing AlgorithmsAccountingGrid Node1Grid Middleware ServicesHealthMonitorGrid Market ServicesJobExecInfo ?SecureTradingQoSStorageSign-on

Economy Grid = Globus + GRACEApplicationsMDSGRAMGlobus Security InterfaceHeartbeat MonitorNexusLocal ServicesLSFCondorGRDQBankPBSTCPSolarisIrixLinuxUDPHigh-level Services and ToolsDUROCglobusrunMPI-GNimrod/GMPI-IOCC++GlobusViewGrid StatusGASSGRACE-TSGARAGridFabricGridApps.GridMiddlewareGridToolsGBankGMDeCashJVMDUROCCore ServicesScienceEngineeringCommercePortalsActiveSheet

Nimrod/G : A Grid Resource BrokerA resource broker for managing and steering task farming (parametric sweep) applications on computational Grids based on deadline and computational economy. Key FeaturesA single window to manage & control experimentResource DiscoveryTrade for ResourcesResource Composition & SchedulingSteering & data managementIt allows to study the behaviour of some of the output variables against a range of different input scenarios.

A Glance at Nimrod-G BrokerGrid MiddlewareNimrod/G ClientNimrod/G ClientNimrod/G ClientGrid Information Server(s)Schedule AdvisorTrading ManagerNimrod/G EngineGridStoreGrid ExplorerGE GISTM TSRM & TSGrid DispatcherRM: Local Resource Manager, TS: Trade ServerGlobus,Legion, Condor-g,, Ninf,etc.GGCLGlobus enabled node.Legion enabled node.CLCondor enabled node.RM & TSRM & TS

A Nimrod/G ClientDeadlineLegion hostsGlobus HostsBezek is in both Globus and Legion Domains

Discover ResourcesDistribute JobsEstablish RatesMeet requirements ? Remaining Jobs, Deadline, & Budget ?Evaluate & RescheduleDiscover More ResourcesAdaptive Scheduling algorithmsCompose & Schedule

Adaptive Scheduling Algorithms

Execution Time

(not beyond deadline)

Execution Cost

(not beyond budget)

Time Minimisation

Minimise

Limited by budget

Cost Minimisation

Limited by deadline

Minimise

None Minimisation

Limited by deadline

Limited by budget

Globus+Legion+Condor/GAustraliaMonash Uni.:Linux clusterSolaris WSNimrod/GGlobus +GRACE_TSEuropeZIB/FUB: T3E/Mosix Cardiff: Sun E6500Paderborn: HPCLineLecce: Compaq SCCNR: ClusterCalabria: Cluster CERN: ClusterPozman: SGI/SP2Globus +GRACE_TSAsia/JapanTokyo I-Tech.:ETL, TuskubaLinux clusterGlobus/LegionGRACE_TSNorth AmericaANL: SGI/Sun/SP2USC-ISI: SGIUVa: Linux Cluster

InternetInter-Continental Grid

Experimentation on the GridWorkload:165 jobs, each need 5 minute of cpu timeDeadline: 1 hrs. and budget: 800,000 unitsStrategy: minimise cost and meet deadlineExecution Cost with cost optimisationAU Peaktime:471205 (G$)AU Offpeak time: 427155 (G$)

Resource Owner Grid servicesPeak time (G$)Off peak costLinux cluster (60 nodes)Monash, AustraliaGlobus/ Condor205IBM SP2 (80 nodes)ANL, Chicago, USGlobus/LL510Sun (8 nodes)ANL, Chicago, USGlobus/ Fork510SGI (96 nodes)ANL, Chicago, USGlobus/ Condor-G1515SGI (10 nodes)ISI, LA, USGlobus/ Fork1020

Execution @ AU Peak Time

graph.peakoz

00000

01001

01001

01002

12013

22023

33034

43044

44055

74165

85276

105386

105487

105467

84466

53483

55395

24096

25396

15367

06158

05269

06259

074510

083411

08449

07549

07647

07747

08737

06727

06728

05509

04709

03509

036011

037010

017011

008010

007011

007010

00709

00706

00608

00809

02808

02709

03608

03709

04809

05709

06808

06607

08608

09607

09704

08704

07603

09803

07702

07702

06802

07701

06701

08501

08401

07300

06300

05000

05000

03000

02000

00000

Linux cluster - Monash (20)

Sun - ANL (5)

SP2 - ANL (5)

SGI - ANL (15)

SGI - ISI (10)

Time (minutes)

Jobs

schedule.peakoz

Linux cluster - Monash (20)Sun - ANL (5)SP2 - ANL (5)SGI - ANL (15)SGI - ISI (10)Total CPUsCost of Resources in Use

974765798.905000000000

974765834.041010010215

974765876.6791010010215

974765916.9462010020325

974765961.3183120130775

974766006.09632202309110

974766050.522433034013160

974766099.633543044015195

974766140.084644055630018225

974766193.7157741651238023305

974766250.4348852761842028360

974766279.94581053861842032420

974766328.70391054872331034435

974766380.083101054673440032405

974766416.60910844666776528350

974766465.887115348310190523285

974766517.873125539510349027325

974766569.701132409613113021255

974766620.597142539613415025275

974766671.698151536715478022220

974766714.507150615817342520190

974766769.82160526919070522215

974766814.981170625920460022205

974766866.9361807451020460026230

974766910.5121908341121724526225

974766944.954190844922493025210

974766980.787200754922650525210

974767023.283200764723734024195

974767065.222210774723734025200

974767105.095220873724676525190

974767146.936220672725931022165

974767192.881230672826417023175

974767245.674240550928141519140

974767286.562250470929550520145

974767326.268250350930787017130

974767368.9512603601130787020155

974767408.3932703701031552520150

974767453.6232801701132014519150

974767493.4642800801032471018140

974767540.2972900701132790018145

974767585.2143000701033259517135

974767627.01300070934043016125

974767670.03631007063510501395

974767714.383320060835266514110

974767754.657330080935266517130

974767803.536330280836199518130

974767855.566340270936657018135

974767894.681350360837116017125

974767931.786360370937116019140

974767968.023360480937421021150

974768009.072370570937730021150

974768046.985370680838036022150

974768086.273380660738656519130

974768128.989390860838814022150

974768173.103400960739276022145

974768217.748400970440188020120

974768279.053410870440971019115

974768316.76242076034175551695

974768359.758430980341755520115

974768411.92844077024252201690

974768447.62744077024282351690

974768491.79245068024312851690

974768540.43346077014373751580

974768579.42846067014389201475

974768641.75647085014419801475

974768681.08448084014435301370

974768742.05949073004498551050

974768775.3155006300451390945

974768832.8465105000457420525

974768877.9835105000459030525

974768934.9095203000463455315

974768979.1525302000468155210

974769036.157540000047120500

schedule.peakoz

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Time (in min.)

Cost of Resources in Use

graph.peakus

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Time (in min.)

Resources (No. of CPUs) in Use

schedule.peakus

00000

01001

01001

01001

02001

12002

22012

33122

43232

33333

63443

54453

93442

93462

93492

932102

83471

83271

82371

82461

82460

73460

76460

96460

107210

108210

99100

109000

109000

79000

99000

105000

95000

94000

91000

100000

60000

90000

100000

100000

100040

100040

90070

80070

80080

100080

102070

105050

97050

99040

109020

109010

79030

89020

99020

109010

109020

105030

95020

105020

95020

95010

105000

95000

105000

96000

106000

88000

78000

107000

103000

83000

73000

73000

73000

72000

51000

01000

01000

01000

01000

01000

00000

Linux cluster - Monash (5)

Sun - ANL (10)

SP2 - ANL (10)

SGI - ANL (15)

SGI - ISI (20)

Time (minutes)

Jobs

Graph 2: Computational Scheduling during Australian Off-peak (or US peak) Time.

Linux cluster - Monash (5)Sun - ANL (10)SP2 - ANL (10)SGI - ANL (15)SGI - ISI (20)Total CPUsCost of Resources in Use

974761656.456000000000

974761702.2031010010239

974761754.5062010010239

974761808.5353010010239

974761842.2143020010358

974761880.5584120020583

974761917.12842201207103

974761970.149533122011152

974762014.434643232014182

974762049.036733333153015207

974762092.182763443153020247

974762131.906854453466521276

974762164.9268934421838522242

974762194.8729934621838524272

974762237.72610934921990527317

974762274.887109321022927526312

974762326.45211834715285023262

974762373.71712832716231021242

974762410.60913823716545021233

974762446.3313824617035521228

974762488.08114824607667520208

974762529.88115734608280020222

974762579.98815764608751523279

974762607.98716964608751525289

974762657.3491710721012021520218

974762686.4551710821012021521237

974762731.018189910013250519226

974762771.0181910900014505519221

974762831.1792010900015153519221

974762892.11217900016260016206

974762928.728219900016414018216

974762967.9352210500017525015145

974762997.035229500017679014140

974763052.138239400018117013121

974763110.9724910001925651064

974763169.769251000001969951050

974763221.4242660000203220630

974763279.6652790000204715945

974763331.22281000002047151050

974763361.57281000002047151050

974763422.0552910004020943014110

974763458.9923010004020943014110

974763502.175319007021260516150

974763534.099318007021569015145

974763568.951328008021884516160

974763603.3893210008022047018170

974763650.7363310207022528519193

974763692.5433410505023499020220

974763744.762359705023806521253

974763785.881359904024436022276

974763824.8513610902025581021251

974763865.023710901026049020236

974763907.432387903026997019251

974763949.05388902027462019241

974763989.276399902027462020246

974764031.0554010901028404020236

974764091.3074110902029181021251

974764127.2644110503030571018190

974764172.963429502031903016170

974764223.7814310502032754517175

974764262.887439502033081016170

974764298.924449501033693515155

974764358.764510500034331015145

974764391.281469500034482014140

974764450.2664710500035105015145

974764502.831479600035259515159

974764538.6714810600035259516164

974764580.105498800036858016192

974764617.047497800037345515187

974764677.0655010700038145017183

974764706.6915110300039430013107

974764765.46252830003974501197

974764824.65653730003989801092

974764886.79954730004004751092

974764923.6454730004083951092

974764955.0575572000413105973

974765005.625651000419355644

974765055.8925701000422410119

974765106.2515701000422410119

974765156.5965801000424020119

974765205.8045901000424020119

974765249.4576001000425625119

974765303.161610000042715500

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Time (in min.)

Cost of Resources in Use

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Total CPUs

Time (in min.)

Resources (No. of CPUs) in Use

Execution @ AU Offpeak Time

graph.peakoz

00000

01001

01001

01002

12013

22023

33034

43044

44055

74165

85276

105386

105487

105467

84466

53483

55395

24096

25396

15367

06158

05269

06259

074510

083411

08449

07549

07647

07747

08737

06727

06728

05509

04709

03509

036011

037010

017011

008010

007011

007010

00709

00706

00608

00809

02808

02709

03608

03709

04809

05709

06808

06607

08608

09607

09704

08704

07603

09803

07702

07702

06802

07701

06701

08501

08401

07300

06300

05000

05000

03000

02000

00000

Linux cluster - Monash (20)

Sun - ANL (5)

SP2 - ANL (5)

SGI - ANL (15)

SGI - ISI (10)

Time (minutes)

Jobs

schedule.peakoz

Linux cluster - Monash (20)Sun - ANL (5)SP2 - ANL (5)SGI - ANL (15)SGI - ISI (10)Total CPUsCost of Resources in Use

974765798.905000000000

974765834.041010010215

974765876.6791010010215

974765916.9462010020325

974765961.3183120130775

974766006.09632202309110

974766050.522433034013160

974766099.633543044015195

974766140.084644055630018225

974766193.7157741651238023305

974766250.4348852761842028360

974766279.94581053861842032420

974766328.70391054872331034435

974766380.083101054673440032405

974766416.60910844666776528350

974766465.887115348310190523285

974766517.873125539510349027325

974766569.701132409613113021255

974766620.597142539613415025275

974766671.698151536715478022220

974766714.507150615817342520190

974766769.82160526919070522215

974766814.981170625920460022205

974766866.9361807451020460026230

974766910.5121908341121724526225

974766944.954190844922493025210

974766980.787200754922650525210

974767023.283200764723734024195

974767065.222210774723734025200

974767105.095220873724676525190

974767146.936220672725931022165

974767192.881230672826417023175

974767245.674240550928141519140

974767286.562250470929550520145

974767326.268250350930787017130

974767368.9512603601130787020155

974767408.3932703701031552520150

974767453.6232801701132014519150

974767493.4642800801032471018140

974767540.2972900701132790018145

974767585.2143000701033259517135

974767627.01300070934043016125

974767670.03631007063510501395

974767714.383320060835266514110

974767754.657330080935266517130

974767803.536330280836199518130

974767855.566340270936657018135

974767894.681350360837116017125

974767931.786360370937116019140

974767968.023360480937421021150

974768009.072370570937730021150

974768046.985370680838036022150

974768086.273380660738656519130

974768128.989390860838814022150

974768173.103400960739276022145

974768217.748400970440188020120

974768279.053410870440971019115

974768316.76242076034175551695

974768359.758430980341755520115

974768411.92844077024252201690

974768447.62744077024282351690

974768491.79245068024312851690

974768540.43346077014373751580

974768579.42846067014389201475

974768641.75647085014419801475

974768681.08448084014435301370

974768742.05949073004498551050

974768775.3155006300451390945

974768832.8465105000457420525

974768877.9835105000459030525

974768934.9095203000463455315

974768979.1525302000468155210

974769036.157540000047120500

schedule.peakoz

Time (in min.)

Cost of Resources in Use

graph.peakus

Time (in min.)

Resources (No. of CPUs) in Use

schedule.peakus

00000

01001

01001

01001

02001

12002

22012

33122

43232

33333

63443

54453

93442

93462

93492

932102

83471

83271

82371

82461

82460

73460

76460

96460

107210

108210

99100

109000

109000

79000

99000

105000

95000

94000

91000

100000

60000

90000

100000

100000

100040

100040

90070

80070

80080

100080

102070

105050

97050

99040

109020

109010

79030

89020

99020

109010

109020

105030

95020

105020

95020

95010

105000

95000

105000

96000

106000

88000

78000

107000

103000

83000

73000

73000

73000

72000

51000

01000

01000

01000

01000

01000

00000

Linux cluster - Monash (5)

Sun - ANL (10)

SP2 - ANL (10)

SGI - ANL (15)

SGI - ISI (20)

Time (minutes)

Jobs

Linux cluster - Monash (5)Sun - ANL (10)SP2 - ANL (10)SGI - ANL (15)SGI - ISI (20)Total CPUsCost of Resources in Use

974761656.456000000000

974761702.2031010010239

974761754.5062010010239

974761808.5353010010239

974761842.2143020010358

974761880.5584120020583

974761917.12842201207103

974761970.149533122011152

974762014.434643232014182

974762049.036733333153015207

974762092.182763443153020247

974762131.906854453466521276

974762164.9268934421838522242

974762194.8729934621838524272

974762237.72610934921990527317

974762274.887109321022927526312

974762326.45211834715285023262

974762373.71712832716231021242

974762410.60913823716545021233

974762446.3313824617035521228

974762488.08114824607667520208

974762529.88115734608280020222

974762579.98815764608751523279

974762607.98716964608751525289

974762657.3491710721012021520218

974762686.4551710821012021521237

974762731.018189910013250519226

974762771.0181910900014505519221

974762831.1792010900015153519221

974762892.11217900016260016206

974762928.728219900016414018216

974762967.9352210500017525015145

974762997.035229500017679014140

974763052.138239400018117013121

974763110.9724910001925651064

974763169.769251000001969951050

974763221.4242660000203220630

974763279.6652790000204715945

974763331.22281000002047151050

974763361.57281000002047151050

974763422.0552910004020943014110

974763458.9923010004020943014110

974763502.175319007021260516150

974763534.099318007021569015145

974763568.951328008021884516160

974763603.3893210008022047018170

974763650.7363310207022528519193

974763692.5433410505023499020220

974763744.762359705023806521253

974763785.881359904024436022276

974763824.8513610902025581021251

974763865.023710901026049020236

974763907.432387903026997019251

974763949.05388902027462019241

974763989.276399902027462020246

974764031.0554010901028404020236

974764091.3074110902029181021251

974764127.2644110503030571018190

974764172.963429502031903016170

974764223.7814310502032754517175

974764262.887439502033081016170

974764298.924449501033693515155

974764358.764510500034331015145

974764391.281469500034482014140

974764450.2664710500035105015145

974764502.831479600035259515159

974764538.6714810600035259516164

974764580.105498800036858016192

974764617.047497800037345515187

974764677.0655010700038145017183

974764706.6915110300039430013107

974764765.46252830003974501197

974764824.65653730003989801092

974764886.79954730004004751092

974764923.6454730004083951092

974764955.0575572000413105973

974765005.625651000419355644

974765055.8925701000422410119

974765106.2515701000422410119

974765156.5965801000424020119

974765205.8045901000424020119

974765249.4576001000425625119

974765303.161610000042715500

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Time (in min.)

Cost of Resources in Use

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Total CPUs

Time (in min.)

Resources (No. of CPUs) in Use