The Gridbus Toolkit for Building and Deploying eScience Applications on Utility GridsRajkumar Buyya Fellow of Grid ComputingGrid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org
*
OutlineIntroduction to eScience and ChallengesIntroduction to the Gridbus ProjectAn Overview of Gridbus ComponentsGrid Service BrokerArchitectureDesign and ImplementationScheduling AlgorithmsBioGrid Demo OR Performance EvaluationA Case Study in High Energy PhysicsEconomy-based Scheduling in Data GridsSummary
*
Prominent Grid Drivers: Emerging eScinece and eBusiness AppsNext generation experiments, simulations, sensors, satellites, even people and businesses are creating a flood of data. They all involve numerous experts/resources from multiple organization in synthesis, modeling, simulation, analysis, and interpretation.Life SciencesDigital BiologyFinance: Portfolio analysis~PBytes/secNewswire & data mining:Natural language engineeringAstronomyInternet & EcommerceHigh Energy PhysicsBrain Activity AnalysisQuantum Chemistry
*
E-Science ElementsDistributed computationPeers sharing ideas and collaborative interpretation of data/resultsE-ScientistRemote VisualizationData & Compute Service
*
Grids have Emerged as Scalable Cyberinfrastructure for e-Science Applications
*
Type of Services Modern Grids OfferComputational Services CPU cyclesSETI@Home, NASA IPG, TeraGrid, I-Grid, Data ServicesData replication, management, secure access--LHC Grid/NapsterApplication ServicesAccess to remote software/libraries and license managementNetSolve Information ServicesExtraction and presentation of data with meaningKnowledge ServicesThe way knowledge is acquired and manageddata mining.Utility Computing ServicesTowards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.Computional GridData GridASP GridInformation GridKnowledge GridUtility Grid
*
Grid Challenges
*
Some Grid Initiatives WorldwideAustraliaNimrod-GGridbusDISCWorldGrangeNet.APACGridARC eResearch?BrazilOurGrid, EasyGridLNCC-Grid + many othersChinaChinaGrid EducationCNGrid - applicationEuropeUK eScienceEU Grids..and many more...IndiaI-GridJapanNAGERIKorea...N*GridSingaporeNGPUSAGlobusNASA IPGAccessGridTeraGridCyberinfrastureand many more...Industry InitiativesIBM On Demand ComputingHP Adaptive ComputingSun N1Microsoft - .NETOracle 10gInfosys Business GridStorageTek Grid..and many morePublic ForumsGlobal Grid ForumAustralian Grid ForumConferences:CCGridGridP2PHPDChttp://www.gridcomputing.com1.3 billion 3 yrs1 billion 5 yrs450million 5 yrs486million 5 yrs1.3 billion (Rs)27 million2? billion 120million 5 yrs
*
The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on DemandWWGWorld Wide Grid!On Demand Utility ComputingGridbusDistributed Data
*
The Gridbus Project: http://www.gridbus.orgA multi-institutional Open Source R&D Project with focus on:Architecture, Specification, and Open Source Reference Implementation.Service-Oriented Grid, Utility Computing & Distributed Data and Computation EconomyScaling from Desktops, Clusters, Cluster Federation, Enterprise Grids to Global Grids.Grid Market Directory and Web ServicesGrid Bank: Accounting and Transaction Management Visual Tools for Creation of Distributed ApplicationsWorkflow Composition and Deployment ServicesData Grid Brokering and Grid Economy ServicesData Replication Strategies GridSim Toolkit: Enhanced to support Data Grid, Reservation, etc.Libra: Economic Cluster SchedulerCoupling of Clusters and Computational EconomyAlchemi: Harnessing .NET/Windows-based ResourcesWWG: Global Data Intensive Grid Testbed Application Enabler Projects:High-Energy Physics , Astronomy, Brain Activity Analysis Osaka U., Natural Language Processing, Portfolio Analysis Spain, BioGrid - WEHI (via APACGrid), SensorGrid (NICTA), Medical Imaging (HFI)Supported by:
*
Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for Resources
*
New challenges of Grid EconomyGrid Service Providers (GSPs)How do I decide service pricing models ?How do I specify them ?How do I translate them into resource allocations ?How do I enforce them ?How do I advertise & attract consumers ?How do I do accounting and handle payments?..Grid Service Consumers (GSCs)How do I decide expenses ? How do I express QoS requirements ?How do I trade between timeframe & cost ?How do I map jobs to resources to meet my QoS needs?..They need mechanisms and technologies for value expression, value translation, and value enforcement.
GRACE: Service Oriented Grid ArchitectureGRid Architecture for Computational Economy (GRACE)
*
GRACE: A ReferenceService-Oriented Grid Architecture for Computational EconomiesGrid Node NGrid ConsumerProgramming EnvironmentsGrid Resource BrokerGrid Service ProvidersGrid ExplorerSchedule AdvisorTrade ManagerJob ControlAgentDeployment AgentTrade ServerResource AllocationResourceReservationR1Misc. servicesInformation ServiceR2RmPricing AlgorithmsAccountingGrid Node1Grid Middleware ServicesHealth MonitorGrid Market ServicesJobExecInfo ?SecureTradingQoSStorageSign-onGrid BankApplicationsData Catalogue
*
Gridbus and Complementary Grid Technologies realizing GRACEAIXSolarisWindowsLinux.NETGridFabricSoftwareGridApplicationsCore GridMiddlewareUser-Level Middleware(Grid Tools)Grid BankGrid Exchange & FederationJVMGrid Brokers:X-Parameter Sweep Lang.Gridbus Data BrokerMPICondorSGETomcatPBSAlchemiWorkflowIRIXOSF1MacLibraGlobus UnicoreGrid Market DirectoryWorldwide GridGridFabricHardwarePortalsScienceCommerceEngineeringCollaboratoriesWorkflow EngineGrid Storage EconomyGrid EconomyNorduGridXGridExcellGridNimrod-G
GRIDSIM
Gridscape
*
Gridbus Technologies Application Construction Tools Visual Parametric Modeller (VPM)Grid Economy ServicesGrid Market DirectoryA Registry for publication of GSPs and their Services VO/VEGrid Bank A Grid Accounting ServicesGrid Trading ServicesData Grid Service BrokerQoS based Scheduling of Distributed Data Oriented Apps on global GridsGrid Workflow Management SystemGridscapeInteractive Grid Testbed Portal GeneratorG-monitorGrid Application Execution Management PortalGridSimA Grid Simulation ToolkitLibraEconomy based Cluster Scheduling
*
Alchemi: .NET-based Enterprise Grid Platform & Web Services
Internet
InternetAlchemi Worker AgentAlchemi ManagerAlchemi Users
Web ServicesSETI@Home like ModelGeneral PurposeDedicated/Non-dedicate workersRole-based Security.NET and Web ServicesC# ImplementationGridThread and Job Model ProgrammingEasy to setup and use
*
On Demand Assembly of Services: Putting Them All TogetherData Source(Instruments/distributed sources)Grid Service Provider (GSP) (e.g., CERN)PECluster Scheduler
Creation and Operation of Virtual EnterprisesGrid Market DirectoryGrid Bank
*
A Market-Oriented Grid Environment
*
Grid Market Infrastructure Grids need to provide an infrastructure that supports:(a) the creation of one or more GMP registries;(b) the contributors to register themselves as GSPs along with their resources/application services that they wish to provide; (c) GSPs to publish themselves in one or more GMPs along with service prices; and (d) Grid resource brokers to discover resources/services and their attributes (e.g., access price and usage constraints) that meet user QoS requirements.
*
Grid Bank: Grid Transactions Authorization, Accounting, & Payment InfrastructureGrid Resource Broker (GRB)GridBank Payment ModuleGrid Trade ServerGridBank Charging ModuleGridBank ServerEstablish Service Costs ApplicationsGrid AgentGrid Resource MeterGridChequeDeploy Agent and Submt Jobs Usage Agreement Resource UsageGridChequeGrid Service Provider (GSP)GridCheque + Resource Usage (GSC Account ChargeGrid Service Consumer (GSC)R1R2R3R4UserUser
Grid Applications: Composition and Deployment A Broker PerspectiveNimrod-G Broker: A Grid Broker for Computational Grids Gridbus Broker: A Grid Service Broker for Data Grids
*
Grid Applications and Parametric Computing Bioinformatics: Drug Design / Protein ModellingSensitivity experiments on smog formationNatural Language EngineeringEcological Modelling: Control Strategies for Cattle TickElectronic CAD: Field Programmable Gate ArraysComputer Graphics: Ray TracingHigh Energy Physics: Searching for Rare Events
Finance: Investment Risk AnalysisVLSI Design: SPICE SimulationsAerospace: Wing DesignNetwork SimulationAutomobile:Crash Simulation Data Mining Civil Engineering:Building Design astrophysics
*
ThesisBuild a task farming application (parameter sweep or bag of tasks) and execute it on Grid within T hours or early and cost not exceeding $M.
The Gridbus Grid Service Broker for Data Grid ApplicationsBuilds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, 1999-2001]AndExtends its notion for Data and Service Grids
*
Grid Service Broker (GSB)A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users QoS requirements (deadline, budget, & T/C optimisation) Key FeaturesA single window to manage & control experimentProgrammable Task Farming EngineResource Discovery and Resource Trading Optimal Data Source DiscoveryScheduling & PredicationsGeneric Dispatcher & Grid AgentsTransportation of data & sharing of resultsAccounting
*
Gridbus Broker at a GlanceHome Node/Portal-PBS-Condor-SGEAlchemiGlobusJob managerfork()batch()GridbusBrokerfork()batch()-PBS-Condor-AlchemiData StoreAccess TechnologyGrid FTPSRBGridbusagentData CatalogCredential RepositoryMyProxy
*
Gridbus Broker ArchitectureGrid MiddlewareGridbus ClientGridbus ClientGribus ClientGrid Info ServerSchedule AdvisorTrading ManagerGridbus Farming EngineRecord KeeperGrid ExplorerGE GIS, NWSTM TSRM & TSGrid DispatcherRM: Local Resource Manager, TS: Trade ServerGGCUGlobus enabled node.ALAlchemi enabled node.(Data Grid Scheduler)DataCatalogDataNodeUnicore enabled node.$$$App, T, $, Opt(Bag of Tasks Applications)
*
Gridbus Services for eScience applicationsApplication Development Environment:XML-based language for composition of task farming (legacy) applications as parameter sweep applications.Task farming APIs for new applications.Web APIs (e.g., Portlets) for Grid portal development.Workflow interface and Gridbus-enabled workflow engine.Resource Allocation and SchedulingDynamic discovery of optional computational and data nodes that meet user QoS requirements.Hide Low-Level Grid Middleware interfacesGlobus, Alchemi, Unicore, NorduGrid, XGrid, etc.
*
Gridbus Broker: XML file
1
main ./calc $X $Y
*
Portal-based Access to Grid Broker for Launching and Steering Applications
*
Figure 3 : Logging into the portal.Drug DesignMade Easy!
*
Excel Plugin to Access Gridbus Services
*
Adaptive Scheduling StepsDiscover ResourcesDistribute JobsEstablish RatesMeet requirements ? Remaining Jobs, Deadline, & Budget ?Evaluate & RescheduleDiscover More ResourcesCompose & Schedule
*
Deadline (D) and Budget (B) Constrained Scheduling Algorithms
*
Sample Applications of Gridbus BrokerMolecular Docking - WEHIDrug Discovery Brain Activity Analysis Osaka UniversityNeuroscience studiesNatural Language Engineering Melbourne NLPIndexing of newswire dataHigh Energy Physics School of Physics/MelbourneBelle experiment data analysisFinance - Portfolio Analysis U. Comp. Madrid/SpainInvestment risk analysisAstronomyAustralian Virtual ObservatorySpreadsheet Processing Microsoft Excel
Economy-based Data Grid SchedulingHigh Energy Physics as eScience Application Case Study
*
Case Study: High Energy PhysicsWhat is High Energy Physics? (HEP)Study of the fundamental constituents of matter and forces.High Energy Physics - using H.E. enables the probing of smaller distances/structures and study in early-universe like environ.Particle Physics - quanta of matter/forces and their propertiesThe Belle ExperimentKEK B-Factory, JapanInvestigating fundamental violation of symmetry in nature (Charge Parity) which may help explain the universal matter antimatter imbalance.Collaboration 400 people, 50 institutes100s TB data currently
*
Case Study: Event Simulation and AnalysisB0->D*+D*-Ks Simulation and Analysis Package - Belle Analysis Software Framework (BASF) Experiment in 2 parts Generation of Simulated Data and Analysis of the distributed data Only the Analysis is discussed here
*
Australian Belle Data Grid Testbed
*
Case Study: Input File for Analysis parameter jobf Gridfile lfn:/users/winton/fsimddks/fsimdata*.mdst;task main copy runme.grid2 node:runme.grid2 node:execute ./runme.grid2 $jobf $jobnameendtask Dynamic parameter defined to describe an input data file
Logical file name pointing to the location in the replica catalog that contains a mapping to where the physical files are present.100 data files (30MB each) were equally distributed among the five nodes
*
Resources Used and their Service Price
OrganizationNode detailsRoleCost (in G$/CPU-sec)CS,UniMelb belle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux Broker host, Data host, NWS server N.A. (Not used as a compute resource)Physics, UniMelbfleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux Replica Catalog host, Data host, Compute resource, NWS sensor2CS, University of Adelaidebelle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux Data host, NWS sensorN.A. (Not used as a compute resource)ANU, Canberrabelle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor4Dept of Physics, USydbelle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor4VPAC, Melbournebrecca-2.vpac.org180 node cluster (only head node used), LinuxCompute resource,NWS sensor6
*
Network Cost (in Grid $/Currency!)
*
Deploying Application ScenarioA data grid scenario with 100 jobs and each accessing remote data of ~30MBDeadline: 3hrs.Budget: G$ 60KScheduling Optimisation Scenario:Minimise TimeMinimise CostResults:
*
Time Minimization in Data Grids
*
Results : Cost Minimization in Data Grids
*
Observation
OrganizationNode detailsCost (in G$/CPU-sec)Total Jobs ExecutedTimeCostCS,UniMelb belle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource)----Physics, UniMelbfleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux 2394CS, University of Adelaidebelle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource)----ANU, Canberrabelle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux 422Dept of Physics, USydbelle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux 4722VPAC, Melbournebrecca-2.vpac.org180 node cluster (only head node used), Linux6232
*
Grid Workflow Management System and Broker ServicesDatabaseDatabaseTasksParametersDependenciesGMDReplicaCatalogGridbus BrokerGlobusWeb servicesHTTPGridFTPData transferWorkflow PlannerApplication CompositionScientific PortalWorkflow Enactment EngineWorkflow description & QoSInfo ServiceMDS
*
The GridSim ToolkitA Java based tool for Grid Scheduling Simulations Basic Discrete Event Simulation InfrastructureVirtual Machine (Java, cJVM, RMI) PCsClustersWorkstations. . .SMPsDistributed ResourcesGridSim ToolkitApplication ModelingInformationServicesResource AllocationGrid Resource Brokers or Schedulerss SimulationStatisticsResource Modeling and Simulation (with Time and Space shared schedulers)
Job Management ClustersSingle CPUReservationSMPsLoad PatternApplication ConfigurationResource ConfigurationVisual ModelerGrid ScenarioNetworkSimJavaDistributed SimJavaResource EntitiesOutputApplication, User, Grid Scenarios Input and ResultsAdd your own policy for resource allocation
*
Selected GridSim Users - 2002
*
Summary and ConclusionIntroduced requirements for an eScience application Demonstrated suitability of Grid computing as Cyberinfrastructure for eScience and eBusiness.Grids exploit synergies that result from cooperation of autonomous entities:Resource sharing, dynamic provisioning, and aggregation at global level. Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements.Delivering ICT services as computing utilities.Grids offer enormous opportunities for realizing eScience and eBusiness at global level.
*
Any Questions ?http://www.gridbus.org
*
Testbed DetailsAll nodes ran Globus 2.4.2ANU and Melbourne CS had 4 CPUs each.Sydney node was effectively 1 processor (SMP kernel was disabled)Adelaide Globus Gatekeeper was not functioning however we could get data off it.BASF was pre-installed on all the machines Gigabyte of code.
For resource providers, they provide service with different goalsFor consumers, they have different requirmentsHow to manage grid service supply-and-demand is hard task
So how can consumers find a resource depending on their requirements. And what type of resource that providers provide and how can they serve consumers within certain QoS.For example, when many consumers want to access same resource, who should be served firstHow to manage supply-demand balance is another challenge.High Energy Physics (HEP) is the study of the fundamental constituents of matter and the forces between these constituents. It is called High Energy Physics as using high energies enables us to probe smaller distances and structures within matter, and also allows us to study matter as it was in the early universe, the history of matter. It is also called Particle Physics as we deal with quanta of matter and forces and the properties associated with these.The study of HEP is broken into two main disciplines, theoretical and experimental. Theoretical HEP propose theories and models to describe matter, forces, their properties, actions, and interactions. Experimental HEP construct experiments or detectors and accelerators to investigate matter interactions and behaviour under high energy conditions.Experimental HEP can be roughly broken into 3 separate activities. The boundaries of these activities, in time and responsibility, are often indistinct. The activities are the construction of detectors which typically takes many years, the measurement or collection of data, and the analysis of this data. We will focus on the using data grids for the analysis of data within HEP.It has three layers. High-level application layer, workflow execution engine layer and Grid resources layer. And resource information is maintained by some info services. For example, GMD (Grid Market Directory) has VO resources information and their price information, MDS contains the information about Grid nodes)
High-level application submits users workflow descriptions to the workflow engine, and engine execute tasks on the grid resources through low level grid middleware such as Globus. It also use some data transfer service like gridftp to move data between resources. The workflow scheduler is the central component of the engine. it schedules tasks in the workflow based on their dependencies. It also queries info services to find a suitable resources for every tasks according to users QoS requirements.
This is the architecture of our current work. Users can use high-level applications such as scientific portal to define workflow and submit to workflow execution engine. Workflow scheduler is the central component, it query grid info services like Grid market directory, data catalog and machines information, to find suitable resources and dispatches workflow tasks on the resources through low-level grid middleware like globus. It also use grid file transfer system like grid ftp to transfer data between different resources.
