Top Banner
Computational Grids and Computational Economy: Nimrod/G Approach David Abramson David Abramson Rajkumar Buyya Rajkumar Buyya Jonathan Giddy Jonathan Giddy
18

Computational Grids and Computational Economy: Nimrod/G Approach David Abramson Rajkumar Buyya Jonathan Giddy.

Mar 26, 2015

Download

Documents

Ian Fitzpatrick
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • Slide 1

Computational Grids and Computational Economy: Nimrod/G Approach David Abramson Rajkumar Buyya Jonathan Giddy Slide 2 Parametric Execution of Applications Coarse-grained SPMD model Coarse-grained SPMD model Execute one application repeatedly for many combinations of input parameters Execute one application repeatedly for many combinations of input parameters Legacy applications: add iteration and distribution without modifying code Legacy applications: add iteration and distribution without modifying code New applications: remove iteration and distribution from design New applications: remove iteration and distribution from design Parametrised modeling experiments: Parametrised modeling experiments: Require very high levels of performance Generate Large amounts of work & concurrency Uncoupled computations Tolerate - moderately high latencies Slide 3 Job 1Job 2Job 3 Job 4Job 5Job 6 Job 7Job 8Job 9 Job 10Job 11Job 12 Job 13Job 14Job 15 Job 16Job 17Job 18 Description of Parameters Slide 4 Working with Small Clusters Nimrod (1994 - ) Nimrod (1994 - ) DSTC funded project Designed for department level clusters Proof of concept Clustor (Activetools) (1997 - ) Clustor (Activetools) (1997 - ) Commercial version of Nimrod Re-engineered Features Features Workstation orientation Access to idle workstations Random allocation policy Password security Nimrod (1994 - ) Nimrod (1994 - ) DSTC funded project Designed for department level clusters Proof of concept Clustor (Activetools) (1997 - ) Clustor (Activetools) (1997 - ) Commercial version of Nimrod Re-engineered Features Features Workstation orientation Access to idle workstations Random allocation policy Password security Slide 5 Clustor limitations Manual resource location Manual resource location static file of machine names No resource scheduling No resource scheduling first come first served No cost model No cost model all machines cost alike Single access mechanism Single access mechanism Slide 6 Towards Grid Computing. Source: www.globus.org & updated Slide 7 Nimrod/G - Nimrod over Globus/Grid Wide-Area Network Support Wide-Area Network Support redesigned architecture use of high-performance networks Scalable Scheduling Scalable Scheduling guaranteed deadline use of existing schedulers Computational Economy Computational Economy I am willing to pay $$, can you complete the job by given deadline trading, bidding, resource reservation... Wide-Area Network Support Wide-Area Network Support redesigned architecture use of high-performance networks Scalable Scheduling Scalable Scheduling guaranteed deadline use of existing schedulers Computational Economy Computational Economy I am willing to pay $$, can you complete the job by given deadline trading, bidding, resource reservation... Slide 8 Layered Architecture (Grid Components) Applications Core Services Metacomputing Directory Service GRAM Globus Security Interface Heartbeat Monitor Nexus Gloperf Local Services LSF CondorMPI NQEEasy TCP SolarisIrixAIX UDP High-level Services and Tools DUROCglobusrunMPI Nimrod/G MPI-IOCC++ GlobusViewTestbed Status GASS Source: www.globus.org Slide 9 Nimrod/G Architecture Grid Middleware Services Dispatcher Nimrod/G Client Grid Directory Services Schedule Advisor Resource Discovery Parametric Engine GUSTO Test Bed Persistent Info. Slide 10 Nimrod/G Interactions MDS server Resource location Queuing System GRAM server Resource allocation (local) Additional services used implicitly: GSI (authentication & authorization) Nexus (communication) User process File access GASS server Gatekeeper node Job Wrapper Computational node Dispatcher Root node Scheduler Prmtc.. Engine Slide 11 Scheduling Algorithm Find a set of machines (MDS search) Distribute jobs from root to machines Establish job consumption rate for each machine For each machine Can we meet deadline? If not, then return some jobs to root If yes, distribute more jobs to resource If cannot meet deadline with current resource Find additional resources Find a set of machines (MDS search) Distribute jobs from root to machines Establish job consumption rate for each machine For each machine Can we meet deadline? If not, then return some jobs to root If yes, distribute more jobs to resource If cannot meet deadline with current resource Find additional resources Slide 12 A Nimrod/G Client CostDeadline AvailableMachines Slide 13 Sample Applications of Nimrod Bioinformatics: Protein Modeling Bioinformatics: Protein Modeling Sensitivity experiments on smog formation Sensitivity experiments on smog formation Parametric study of Laser detuning Parametric study of Laser detuning Combinatorial Optimization: Simulated Annealing Combinatorial Optimization: Simulated Annealing Ecological Modeling: Control Strategies for Cattle Tick Ecological Modeling: Control Strategies for Cattle Tick Electronic CAD: Field Programmable Gate Arrays Electronic CAD: Field Programmable Gate Arrays Computer Graphics: Ray Tracing Computer Graphics: Ray Tracing High Energy Physics: Searching for Rare Events High Energy Physics: Searching for Rare Events Physics: Laser-Atom Collisions Physics: Laser-Atom Collisions VLSI Design: SPICE Simulations VLSI Design: SPICE Simulations Radiation Protection and Nuclear Safety Slide 14 Electronic CAD Slide 15 Some early results - Slide 16 Slide 17 Related Works AppLeS (UC. San Diego) AppLeS (UC. San Diego) application level scheduling & case-by-case NetSolve (UTK/ORNL) NetSolve (UTK/ORNL) API for creating farms DISCWorld (U. Adelaide) DISCWorld (U. Adelaide) remote information access Millennium (UC. Berkeley) Millennium (UC. Berkeley) remote execution environment on clusters and supports computational economy Slide 18 Conclusions Nimrod/G architecture offers a scalable model for resource management and scheduling on computational grids Nimrod/G architecture offers a scalable model for resource management and scheduling on computational grids Supports Computational Economy Supports Computational Economy The current model supporting Parametric Computing can be extended to support parallel jobs or any other computational model. The current model supporting Parametric Computing can be extended to support parallel jobs or any other computational model. Plan to use the concept of Advance Resource Reservation in order to offer the feature wherein the user can say I am willing to pay $, can you complete my job by this time Plan to use the concept of Advance Resource Reservation in order to offer the feature wherein the user can say I am willing to pay $, can you complete my job by this time Further Information: www.csse.monash.edu.au/~davida/nimrod.ht ml Further Information: www.csse.monash.edu.au/~davida/nimrod.ht ml