Dr. Rajkumar Buyya

Grid Computing and The Gridbus Toolkit: Creating and Managing Utility Grids for eScience and eBusiness ApplicationsDr. Rajkumar Buyya Fellow of Grid ComputingGrid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software Engineering The University of Melbourne, Australia gridbus.org/~raj/tut/gridbus.zip

*

4 Essential Utilities (in Home)

*

(5) IT services as the fifth utility (water, electricity, gas, telephone, IT) eScienceeBusinesseGovernmenteHealthMultilingualeEducation

*

GRIDS Lab @ MelbourneThe youngest and one of the largest research labs in the CSSE Dept:2 PostDocs2 Research Programmers7 RHD (6 PhD) students ~5 honours/masters projectsFundingNational and International organizationsAustralian Research CouncilMany industries (Sun, StorageTek, Microsoft, IBM)University-wide collaboration:Faculties of Science, Engineering, and MedicineMany national and international collaborations.AcademicsIndustriesSoftware:Our Grid middleware technologies are widely in academic and industrial users.Publication:My research team produces 20% of our Depts research output.

EducationR & D+ Community Services

*

Books at Glance: Co-authored/edited

*

Presentation OutlinePart 1: Introduction to Grid Computing and ApplicationsTechnology Evolution and Application DriversGrid Challenges, Approaches, and ArchitecturePart 2: Grid Economy and Service Oriented ComputingChallengesService-Oriented Grid Architecture (SOGA)Realisation of SOGAPart 3: Global Grids and Gridbus TechnologiesGrid Market Directory, GridBank, VPM, Grid Service Broker, G-Monitor Part 4: Performance Evaluation on the World-Wide GridCompute Grid Application eScience Application Belle Analysis Data GridPart 5: Closing RemarksOpen Challenges in Grid EconomyAnalogy to Electric Power GridSummary and Conclusion

*

Evolution: Humans eHumans (eHugging, eSmell, eFood!), Science eScience, Business eBusiness

*

Computing and Communication Technologies Evolution* Sputnik19601970197519801985199019952000* ARPANET* Email* Ethernet* TCP/IP* IETF* Internet Era* WWW Era* Mosaic* XML* PC Clusters*Crays*MPPs* Mainframes* HTML* W3C* P2P* Grids* XEROX PARC wormCOMPUTINGCommunication* Web Services* Minicomputers* PCs* WS Clusters* PDAs* Workstations* HTC2010* eScience* Computing Utility* eBusiness* SocialNet

*

Computing Evolving towards: Global/Grid ComputingPersonal DeviceSMPs or SuperComputersLocalClusterGlobalGridS E R V I C E S+PERFORMANCEInter PlanetGridIndividualGroupDepartmentCampusStateNationalGlobeInter PlanetUniverseAdministrative BarriersEnterpriseCluster/Grid

*

Leading to Grid (computing) Paradigm:Cyberinfrastructure for sharing resourcesInspired by Power Grid!

* A service-oriented/utility computing paradigm that enables seamless sharing of geographically distributed, autonomous resources.* This was the original aim of building Internet although it ended up in giving birth to email!

*

What is Grid ?(there are several definitions)A type of parallel and distributed system that enables the sharing, selection, & aggregation of geographically distributed autonomous resources:Computers PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc;Software e.g., ASPs renting expensive special purpose applications on demand;Catalogued data and databases e.g. transparent access to human genome database;Special devices/instruments e.g., radio telescope SETI@Home searching for life in galaxy.People/collaborators.depending on their availability, capability, cost, and user QoS requirements.Wide area

*

A Bird Eye View of World-Wide Grid EnvironmentGrid Resource BrokerResource BrokerApplicationGrid Information ServiceGrid Resource BrokerR2R3RNR1R4R5R6Grid Information Service

*

Type of Services Modern Grids OfferComputational Services CPU cyclesNASA IPG, WWG, TeraGrid, SETI@HomeData ServicesData replication, management, secure access--LHC Grid/NapsterApplication ServicesAccess to remote software/libraries and license managementNetSolve Information ServicesExtraction and presentation of data with meaningKnowledge ServicesThe way knowledge is acquired and managed using meta data & semantics.Utility Computing ServicesComputional GridData GridASP GridInformation GridKnowledge GridUtility Grid

*

Prominent Grid Drivers: Emerging e-Science and e-Business AppsNext generation experiments, simulations, sensors, satellites, even people and businesses are creating a flood of data. e-Science refers to the large scale science that will increasingly be carried out through distributed global collaborations enabled by the Internet.Life SciencesDigital BiologyFinance: Portfolio analysis~PBytes/secNewswire & data mining:Natural language engineeringAstronomyInternet & EcommerceHigh Energy PhysicsBrain Activity AnalysisQuantum Chemistry

*

1. Online Medical Instrumentation and NeuroscienceOsaka Univ. HospitalOsaka Univ.DV transferLife-electronics laboratory,AISTData AnalysisProvision of MEGProvision of expertise in the analysis of brain functionCybermedia CenterData GenerationVirtual Laboratoryfor medicine and brain scienceKnowledge sharingMEG sharing?Data Sharing

*

3. Enterprise Computing ApplicationsTraditional Model Grid Based ModelEmail serverWebserverDatabaseserverAppsserverUpgrade to a new serverto handlemore usersHorizontal integration of Email, Web, Data, and Apps serversService Virtualization Layer & Load Balancing

Global Grids and Challenges

*

E-Science / E-Business App ElementsDistributed computationPeers sharing ideas and collaborative interpretation of data/resultsE-ScientistRemote VisualizationData & Compute Service

*

Grids have Emerged as Cyberinfrastructure that scales from from enterprise to global

*

Grid-based Utility Computing model need to scale from desktops to Global levelPersonal DeviceSMPs or SuperComputersLocalClusterGlobalGridS E R V I C E S+PERFORMANCEInter PlanetGridIndividualGroupDepartmentCampusStateNationalGlobeInter PlanetUniverseAdministrative BarriersEnterpriseCluster/Grid

*

Grids need to offer a wide variety of servicesComputational Services CPU cyclesSETI@Home, NASA IPG, TeraGrid, I-Grid, Data ServicesData replication, management, secure access--LHC Grid/NapsterApplication ServicesAccess to remote software/libraries and license managementNetSolve Information ServicesExtraction and presentation of data with meaningKnowledge ServicesThe way knowledge is acquired and manageddata mining.Utility Computing ServicesTowards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.Computional GridData GridASP GridInformation GridKnowledge GridUtility Grid

*

Grid Challenges

*

Grid Operations Management Challenges dynamic resources, policies, and self interested entitiesGrid Economy TechnologiesGSP1GSPGSPGSP2GSP3GSP4GSP5

*

Some Grid Initiatives WorldwideAustraliaNimrod-GGridbusDISCWorldGrangeNet.APACGridARC eResearchBrazilOurGrid, EasyGridLNCC-Grid + many othersChinaChinaGrid EducationCNGrid - applicationEuropeUK eScienceEU Grids..and many more...IndiaI-GridJapanNAGERIKorea...N*GridSingaporeNGPUSAGlobusGridSecAccessGridTeraGridCyberinfrastureand many more...Industry InitiativesIBM On Demand ComputingHP Adaptive ComputingSun N1Microsoft - .NETOracle 10gInfosys Enterprise GridSatyam Enterprise GridStorageTek Grid..and many morePublic ForumsGlobal Grid ForumAustralian Grid ForumConferences:CCGridGridHPDCE-Sciencehttp://www.gridcomputing.com1.3 billion 3 yrs1 billion 5 yrs450million 5 yrs486million 5 yrs1.3 billion (Rs)27 million2? billion 120million 5 yrs

*

mix-and-match (service)Object-orientedInternet/partial-P2PNetwork enabled SolversEconomic-based Utility / Service-Oriented ComputingNimrod-G

*

The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on DemandWWGWorld Wide Grid!On Demand Utility ComputingGridbusDistributed Data

*

The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying e-Research Applications on Utility Grids

Gridbus

Distributed Data

http://www.gridbus.org

Gridbus is a open source Grid R&D project with focus on Grid Economy, Utility Grids and Service Oriented Computing. Gridbus Middleware components include:Alchemi: .NET-based Enterprise GridGrid Market Directory and Web ServicesGrid Bank: Accounting and Transaction Management Visual Tools for Creation of Distributed ApplicationsGrid Service Broker and SchedulingWorkflow Management EngineGridSim ToolkitLibra: SLA-based Resource Allocation

*

Presentation OutlinePart 1: Introduction to Grid Computing and ApplicationsTechnology Evolution and Application DriversGrid Challenges, Approaches, and ArchitecturePart 2: Grid Economy and Service Oriented ComputingChallengesService-Oriented Grid Architecture (SOGA)Realisation of SOGAPart 3: Global Grids and Gridbus TechnologiesGrid Market Directory, GridBank, VPM, Grid Service Broker, G-Monitor Part 4: Performance Evaluation on the World-Wide GridCompute Grid Application eScience Application Belle Analysis Data GridPart 5: Closing RemarksOpen Challenges in Grid EconomyAnalogy to Electric Power GridSummary and Conclusion

*

Gridbus considers: Incentive as a Design Parameter for Grid ComputingGrids aim at exploiting synergies that result from cooperation of autonomous distributed entities. Synergies include:Creation of Virtual Organisations/EnterprisesResource sharing Aggregation of resources on demand. For this cooperation to be sustainable, participants needs to have (economic) incentive. Therefore, incentive mechanisms should be considered as one of key design parameters of Grid computing.

*

Grid Economy: Methodology for Sustained Resourced Sharing and Managing Supply-and-Demand for Resources

*

Benefits of Computational EconomyIt provides an effective paradigm for managing self interested and self-regulating entities (resource owners and consumers)Helps in regulating supply-and-demand of resources.Services can be priced in such a way that equilibrium is maintained.User-centric / Utility drivenScalable:No need of central coordinator (during negotiation)Resources(sellers) and also Users(buyers) can make their own decisions and try to maximize utility and profit. Adaptable, It allows to offer different QoS (quality of services) to different applications depending the value users place on them.It offers incentive for resource owners for being part of the grid!It offers incentive for resource consumers for being good citizens.It improves the utilisation of resources.

*

It helps Users to Achieve their GoalsGrid ConsumersExecute jobs for solving varying problem size and complexityBenefit by selecting and aggregating resources wiselyTradeoff timeframe and costStrategy: minimise expensesGrid ProvidersContribute (idle) resource for executing consumer jobsBenefit by maximizing resource utilisationTradeoff local requirements & market opportunityStrategy: maximise return on investment

*

New challenges of Grid EconomyGrid Service Providers (GSPs)How do I decide service pricing models ?How do I specify them ?How do I translate them into resource allocations ?How do I enforce them ?How do I advertise & attract consumers ?How do I do accounting and handle payments?..Grid Service Consumers (GSCs)How do I decide expenses ? How do I express QoS requirements ?How do I trade between timeframe & cost ?How do I map jobs to resources to meet my QoS needs?..They need mechanisms and technologies for value expression, value translation, and value enforcement.

GRACE: A Reference Grid Economy Services ArchitectureGRid Architecture for Computational Economy (GRACE)

*

Market-based Computing Systems RequirementsTo enable users (GSPs and GSCs) to realise economic value, market-based systems need to provide mechanisms for:Value Expressiona means to express their requirements, valuations, and objectives Value Translationscheduling policies to translate them to resource allocations Value Enforcement mechanisms to enforce the selection and allocation of differential services, and dynamic adaptation to changes in their availability at runtime Market mechanisms, accounting and payment, Reservation of resources.

*

GRACE: A ReferenceService-Oriented Grid Architecture for Computational EconomiesGrid Node NGrid ConsumerProgramming EnvironmentsGrid Resource BrokerGrid Service ProvidersGrid ExplorerSchedule AdvisorTrade ManagerJob ControlAgentDeployment AgentTrade ServerResource AllocationResourceReservationR1Misc. servicesInformation ServiceR2RmPricing AlgorithmsAccountingGrid Node1Grid Middleware ServicesHealth MonitorGrid Market ServicesJobExecInfo ?SecureTradingQoSStorageSign-onGrid BankApplicationsData Catalogue

*

Realising Market-based Grid: Minimal New ComponentsGrid Market Directory ServicesGrid Trading Services for different economic modelsGrid Metering ServicesGrid Accounting and Payment ServicesGrid Service Broker

*

Gridbus and Complementary Grid Technologies realizing GRACEAIXSolarisWindowsLinux.NETGridFabricSoftwareGridApplicationsCore GridMiddlewareUser-Level Middleware(Grid Tools)Grid BankGrid Exchange & FederationJVMGrid Brokers:X-Parameter Sweep Lang.Gridbus Data BrokerMPICondorSGETomcatPBSAlchemiWorkflowIRIXOSF1MacLibraGlobus UnicoreGrid Market DirectoryWorldwide GridGridFabricHardwarePortalsScienceCommerceEngineeringCollaboratoriesWorkflow EngineGrid Storage EconomyGrid EconomyNorduGridXGridExcellGridNimrod-G

GRIDSIM

Gridscape

*

On Demand Assembly of Services: Interaction Between Grid ComponentsData Source(Instruments/distributed sources)Grid Service Provider (GSP) (e.g., CERN)PECluster Scheduler

*

*

On Demand Assembly of Services and Utility/Market-based Grid Computing

Data Source

(Instruments/distributed sources)

Grid Service Provider (GSP) (e.g., CERN)

PE

Cluster Scheduler

*

*

On Demand Assembly of Services and Utility/Market-based Grid Computing

Data Source

(Instruments/distributed sources)

Grid Service Provider (GSP) (e.g., CERN)

PE

Cluster Scheduler

*

Alchemi: .NET-based Enterprise Grid Platform & Web Services

Internet

InternetAlchemi Worker AgentsAlchemi ManagerAlchemi Users

Web ServicesSETI@Home like ModelGeneral PurposeDedicated/Non-dedicate workersRole-based Security.NET and Web ServicesC# ImplementationGridThread and Job Model ProgrammingEasy to setup and use Widely in use!

*

Some Users of AlchemiTier Technologies, USALarge scale document processing using Alchemi framework CSIRO, AustraliaNatural Resource ModelingThe Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genes Satyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi framework The University of Sao Paulo, BrazilThe Alchemi Executor as a Windows Service stochastix GmbH, GermanyAsynchronous Excel Tasks using ManagedXLL and Alchemi .Net Grid Computing framework. Many users in Universities: See next for an example.

*

Students' project gives old computers new life-1/25/2005

*

d

*

On Demand Assembly of Services and Utility/Market-based Grid Computing

Data Source

(Instruments/distributed sources)

Grid Service Provider (GSP) (e.g., CERN)

PE

Cluster Scheduler

*

Globus Technologies UsageSecurity (GSI - Globus Security Infrastructure) - single sign-on and authentication based on RSA public key cryptography technology.You need have Grid ID, public key, and private key (assigned by trusted CA)Authorization to use: You need have your Grid ID mapped to a physical (login) account on every Grid nodes that you want to use.Authentication: User proxy (trigger by grid-proxy-init) and Grid node gatekeeper authenticate each other by exchanging messages. (If you can decrypt the message that I sent by encrypting using your public key, then you are who you are claiming to be.)Information (MDS - Metacomputing Directory Service) LDAP-server based uniform access to resource structure/state information. GIIS Grid Index Information Service (one for your Grid!/organisation)GRIS Grid Resource Information Service (one for each node).Communications (grid-ftp) - multi-method communication and QoS management.Process/Job Management (GRAM - Globus Resource Allocation Manager) - Low-level (uniform) API for various local schedulers.Remote file access (GASS - Global Access to Secondary Storage).Reservation of Resources in Advance (GARA).

*

Globus Components (in One Slide)Globus SecurityInfrastructureJob ManagerGRAM client API calls to request resource allocationand process creation.MDS client API callsto locate resourcesQuery current statusof resourceCreateRSL LibraryParseRequestAllocate &create processesProcessProcessProcessMonitor &controlSite boundaryClient-side APIsMDS: Grid Index Info ServerGatekeeperMDS: Grid Resource Info ServerLocal Resource ManagerMDS client API callsto get resource infoGRAM client API statechange callbacks

*

Presentation OutlinePart 1: Introduction to Grid Computing and ApplicationsTechnology Evolution and Application DriversGrid Challenges, Approaches, and ArchitecturePart 2: Grid Economy and Service Oriented ComputingChallengesService-Oriented Grid Architecture (SOGA)Realisation of SOGAPart 3: Global Grids and Gridbus TechnologiesGrid Market Directory, GridBank, VPM, Grid Service Broker, G-Monitor Part 4: Performance Evaluation on the World-Wide GridCompute Grid Application eScience Application Belle Analysis Data GridPart 5: Closing RemarksOpen Challenges in Grid EconomyAnalogy to Electric Power GridSummary and Conclusion

*

*

On Demand Assembly of Services and Utility/Market-based Grid Computing

Data Source

(Instruments/distributed sources)

Grid Service Provider (GSP) (e.g., CERN)

PE

Cluster Scheduler

The Grid Market DirectoryGrid Vision: To enable the creation of Virtual Enterprise (VE), Virtual Oranisation (VO), or Grid MarketPlace (GMP).

*

A Market-Oriented Grid Environment

*

Grid Market Infrastructure Grids need to provide an infrastructure that supports:(a) the creation of one or more GMP registries;(b) the contributors to register themselves as GSPs along with their resources/application services that they wish to provide; (c) GSPs to publish themselves in one or more GMPs along with service prices; and (d) Grid resource brokers to discover resources/services and their attributes (e.g., access price and usage constraints) that meet user QoS requirements.

*

GMD Architecture

*

Globus MDS Vs Gridbus GMDGlobus MDSGridbus GMD

*

GSP Registration

*

GSP Service Publication

*

GSP Service Browsing

*

GMD Query Message

Query MessageSOAP MessageRepository HandlerQuery ProcessingGMD RepositoryGMD Query WebserviceRepository HandlerQuery ProcessingHTTPServer

SOAP EngineGMD RepositoryGMD Query WebserviceQuery MessageGMD Webservice clientXML

*

GMD Use Case: SC02 HPC Challenge Demonstration

*

How can I Access GMD Software ?Download, Deploy, and Use it:Open Source Reference Implementation (Java-based) is available:http://www.gridbus.org/gmd/Or Make use of Global GMD registry hosted by the Gridbus Project.For more info, Read Technical Report:A Market-Oriented Grid Directory Service for Publication and Discovery of Grid Service Providers and their Services

*

*

On Demand Assembly of Services and Utility/Market-based Grid Computing

Data Source

(Instruments/distributed sources)

Grid Service Provider (GSP) (e.g., CERN)

PE

Cluster Scheduler

Grid BankA Grid Accounting Services Architecture

*

The Grid Bank OperationsGrid Resource Broker (GRB)GridBank Payment ModuleGrid Trade ServerGridBank Charging ModuleGridBank ServerEstablish Service Costs ApplicationsGrid AgentGrid Resource MeterGridChequeDeploy Agent and Submt Jobs Usage Agreement Resource UsageGridChequeGrid Service Provider (GSP)GridCheque + Resource Usage (GSC Account ChargeGrid Service Consumer (GSC)R1R2R3R4UserUser

*

GridBank Architecture

*

Grid Bank ComponentsGrid Bank ServerRegular account management features (open, close, delete, update, browse) are supported.GridBank DatabaseGridBank Client Access InterfacePayment ModuleCharging ModuleProtocols in XML formatResource Usage Record (confirm to GGF RUR format).

*

ApplicationsGrid Resource Broker (GRB)Grid Service Provider (GSP)Grid Trade ServerGrid Resource MeterGridBank Charging ModuleR1R2R3R4Resource Usage RecordGridBankPaymentModuleGridBank ServerUserChargeable Item 1 RateChargeable Item 2 Rate...RATESItem 1 RateItem 2 Rate...XXXXXUsage Item 1Usage Item 2...=====RURCharge for Item 1Charge for Item 2...Service Cost TotalFilter relevantresource usageinformationConvert to standardResource UsageRecordGridBank systemcomponent names are in italicsGrid Components Interaction and Utilization of Grid Bank

*

Grid Bank Usage ScenarioGSPs and GSCs open account with GridBankWhen GSC wants to consume GSP service, it informs the GSP about the account to which access cost can be charged.GSPs can confirm with GridBank whether GSC has sufficient credit or even request to put the amount on hold.GSP measures the amount of resource consumed and charges the GSC account in Grid Bank.Grid Bank transfers to tokens/credits/money from GSC to GSP account; and maintains transaction details (Resource Usage Record).Grid Bank also be used for developing Scalable Authentication Infrastructure.

*

X509v3 Digital CertificateSubject:/O=Grid/O=Globus/OU=cs.uwa.edu.au/CN=Chris McDonaldClientsResourcesResource access authorization file (grid-mapfile)/O=Grid/O=Globus/OU=cs.uwa.edu.au/CN=Alexander Barmouta alex/O=Grid/O=Globus/OU=cs.mu.oz.au/CN=Rajkumar Buyya rajkumar/O=Grid/O=Globus/OU=cs.uwa.edu.au/CN=Chris McDonald chrisX509v3 Digital CertificateSubject:/O=Grid/O=Globus/OU=cs.uwa.edu.au/CN=Alexander BarmoutaX509v3 Digital CertificateSubject:/O=Grid/O=Globus/OU=cs.mu.oz.au/CN=Rajkumar BuyyaAccess Scalability Problem

*

Resource access authorization file (grid-mapfile)/O=Grid/O=Globus/OU=cs.uwa.edu.au/CN=GridBank gridbank GridBankTemplate (local) accountsgbaccount1gbaccount2gbaccount3Resource access authorization file (grid-mapfile)/O=Grid/O=Globus/OU=cs.uwa.edu.au/CN=GridBank gridbank/O=Grid/O=Globus/OU=cs.uwa.edu.au/CN=Alexander Barmouta gbaccount1Template (local) accountsgbaccount2gbaccount3Request to access resourcePassing clients Certificate SubjectExecute jobGridBank Accounts/O=Grid/O=Globus/OU=cs.uwa.edu.au/CN=Alexander Barmouta/O=Grid/O=Globus/OU=cs.mu.oz.au/CN=Rajkumar Buyya/O=Grid/O=Globus/OU=cs.uwa.edu.au/CN=Chris McDonaldGridBanks Solution to Access Scalability Problem

*

How can I Access GridBank Software ?Download, Deploy, and Use it:Open Source Reference Implementation is available:http://www.gridbus.org/For more info, Read Technical Report:GridBank: A Grid Accounting Services Architecture (GASA) for Distributed Systems Sharing and Integration

*

*

On Demand Assembly of Services and Utility/Market-based Grid Computing

Data Source

(Instruments/distributed sources)

Grid Service Provider (GSP) (e.g., CERN)

PE

Cluster Scheduler

The Gridbus Grid Service Broker for Data Grid ApplicationsBuilds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, 1999-2001]AndExtends its notion for Data and Service Grids

*

Grid Service Broker (GSB)A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users QoS requirements (deadline, budget, & T/C optimisation) Key FeaturesA single window to manage & control experimentProgrammable Task Farming EngineResource Discovery and Resource Trading Optimal Data Source DiscoveryScheduling & PredicationsGeneric Dispatcher & Grid AgentsTransportation of data & sharing of resultsAccounting

*

Gridbus Broker ArchitectureGrid MiddlewareGridbus ClientGridbus ClientGribus ClientGrid Info ServerSchedule AdvisorTrading ManagerGridbus Farming EngineRecord KeeperGrid ExplorerGE GIS, NWSTM TSRM & TSGrid DispatcherRM: Local Resource Manager, TS: Trade ServerGGCUGlobus enabled node.ALAlchemi enabled node.(Data Grid Scheduler)DataCatalogDataNodeUnicore enabled node.$$$App, T, $, Opt(Bag of Tasks Applications)

*

Gridbus Broker and Remote Service Access EnablersData StoreAccess TechnologyGrid FTPSRBCredential RepositoryMyProxyPortlets

*

Gridbus Services for eScience applicationsApplication Development Environment:XML-based language for composition of task farming (legacy) applications as parameter sweep applications.Task Farming APIs for new applications.Web APIs (e.g., Portlets) for Grid portal development.Threads-based Programming InterfaceWorkflow interface and Gridbus-enabled workflow engine.Resource Allocation and SchedulingDynamic discovery of optional computational and data nodes that meet user QoS requirements.Hide Low-Level Grid Middleware interfacesGlobus (v2, v4), SRB, Alchemi, Unicore, and ssh-based access to local/remote resources managed by XGrid, Condor, SGE.

*

Figure 3 : Logging into the portal.Drug DesignMade Easy!Click Here for Demo

*

Excel Plugin to Access Gridbus Services

*

Adaptive Scheduling StepsDiscover ResourcesDistribute JobsEstablish RatesMeet requirements ? Remaining Jobs, Deadline, & Budget ?Evaluate & RescheduleDiscover More ResourcesCompose & Schedule

*

Deadline (D) and Budget (B) Constrained Scheduling Algorithms

*

Gridbus Project: Some Applications and Users

http://www.gridbus.org

High Energy Physics: Particle Discovery

Melbourne University

NeuroScience: Brain Activity Analysis

Natural Resource Modeling

CSIRO Land and Water, Austraila.

Large Scale document processing

Tier Technologies, USA.

Detection of patterns of transcription factors in mammalian genes

*

*

On Demand Assembly of Services and Utility/Market-based Grid Computing

Data Source

(Instruments/distributed sources)

Grid Service Provider (GSP) (e.g., CERN)

PE

Cluster Scheduler

*

Case Study: High Energy Physics and Data GridThe Belle ExperimentKEK B-Factory, JapanInvestigating fundamental violation of symmetry in nature (Charge Parity) which may help explain the universal matter antimatter imbalance.Collaboration 400 people, 50 institutes100s TB data currently

*

Australian Belle Data Grid Platform

*

Case Study: Event Simulation and AnalysisB0->D*+D*-Ks Simulation and Analysis Package - Belle Analysis Software Framework (BASF) Experiment in 2 parts Generation of Simulated Data and Analysis of the distributed data

Analyzed 100 data files (30MB each) were distributed among the five nodes

*

Resources Used and their Service Price

OrganizationNode detailsRoleCost (in G$/CPU-sec)CS,UniMelb belle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux Broker host, Data host, NWS server N.A. (Not used as a compute resource)Physics, UniMelbfleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux Replica Catalog host, Data host, Compute resource, NWS sensor2CS, University of Adelaidebelle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux Data host, NWS sensorN.A. (Not used as a compute resource)ANU, Canberrabelle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor4Dept of Physics, USydbelle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux Data host, Compute resource, NWS sensor4VPAC, Melbournebrecca-2.vpac.org180 node cluster (only head node used), LinuxCompute resource,NWS sensor6

*

Network Cost (in Grid $/Currency!)

*

Deploying Application ScenarioA data grid scenario with 100 jobs and each accessing remote data of ~30MBDeadline: 3hrs.Budget: G$ 60KScheduling Optimisation Scenario:Minimise TimeMinimise CostResults:

*

Time Minimization in Data Grids

*

Results : Cost Minimization in Data Grids

*

Observation

OrganizationNode detailsCost (in G$/CPU-sec)Total Jobs ExecutedTimeCostCS,UniMelb belle.cs.mu.oz.au 4 CPU, 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource)----Physics, UniMelbfleagle.ph.unimelb.edu.au 1 CPU, 512 MB RAM, 40 GB HD, Linux 2394CS, University of Adelaidebelle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux N.A. (Not used as a compute resource)----ANU, Canberrabelle.anu.edu.au 4 CPU, 2GB RAM, 40 GB HD, Linux 422Dept of Physics, USydbelle.physics.usyd.edu.au 4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux 4722VPAC, Melbournebrecca-2.vpac.org180 node cluster (only head node used), Linux6232

*

The GridSim ToolkitA Java based tool for Grid Scheduling Simulations Basic Discrete Event Simulation InfrastructureVirtual Machine (Java, cJVM, RMI) PCsClustersWorkstations. . .SMPsDistributed ResourcesGridSim ToolkitApplication ModelingInformationServicesResource AllocationGrid Resource Brokers or Schedulerss SimulationStatisticsResource Modeling and Simulation (with Time and Space shared schedulers)

Job Management ClustersSingle CPUReservationSMPsLoad PatternApplication ConfigurationResource ConfigurationVisual ModelerGrid ScenarioNetworkSimJavaDistributed SimJavaResource EntitiesOutputApplication, User, Grid Scenarios Input and ResultsAdd your own policy for resource allocation

*

Selected GridSim Users - 2002

Workflow SchedulingSKIP (if Time Problem)

*

Grids and WorkflowGETdatasetgalfitGETdatasetstorestoresextrpreviewstackstorestorephotIssues:NamingSecurityAuthorizationService interfaceData representation and interchangeProgramming modelsWork flowetc.

Astronomical Data Analysis (Hugh Couchman, Computing in Canadian Astronomy)

*

Grid-based workflowGrid workflow A collection of tasks that are processed on distributed resources in well-defined order. DifferencesGrid workflow could be long lastingLarge data flow need to be supported (e.g. Sloan Digital Sky Survey ~Petabytes)Resources used by Grid workflow are heterogeneous Resources are dispersed across multiple administrative domainsResource availability and utilization varies dynamically over time

*

Requirements and Challenges

RequirementsComposition tools (e.g. expressing large-scale workflow)Harnessing distributed resources and services that meet user requirementsLarge-scale data transferChallengesDynamic execution environment of Grid workflowUnknown locations of intermediate dataAcquisition of resource information

*

Workflow Management SystemDeveloped a service-oriented workflow management system driven by IBM TSpacesProvides XML based language for expressing workflowAble to deploy workflow applications on global gridsServe as an infrastructure for our future work on economy-based workflow scheduling.

*

ArchitectureDatabaseDatabaseTasksParametersDependenciesGMDReplicaCatalogGridbus BrokerGlobusWeb servicesHTTPGridFTPData transferWorkflow PlannerApplication CompositionScientific PortalWorkflow Enactment EngineWorkflow description & QoSInfo ServiceMDS

*

Workflow Scheduling SystemWorkflow Coordinator (WCO) TM generation and activationLife-time of workflow executionTask Managers (TMs) Task executionResource discovery and selectionMonitoringFailure management Communication approach between WCO and TMsCommunication ModelComplexity of task dependencies (e.g. multiple parents and multiple children)Many-to-many SolutionsEvent-driven mechanismSubscription-notification Event exchange server using tuple spaces (IBM TSpaces)Workflow CoordinatorTask ManagerResourceGroupTaskGroupMonitorTask ManagerFactoryEventServiceDecentralized Scheduling Architecture

*

Event-driven Mechanism using Tuple SpacesEvent Service (IBM TSpaces)Workflow CoordinatorTask Manager ATask Manager BTask Manager N. . . . . .statusoutputnotifynotifyGrid resourcesMonitornotify

*

A Sample WF model, Task and Datalink Definition

C:port2 F:port0 D:port2 F:port1 ..

ycalc belle.anu.edu.au /data/ycalc.sh para 5 output

optionalDirected Acyclic Graph

*

Performance Evaluation (Synthetic Application on Belle Data Grid)Workflow Task ApplicationExperimental Workflow

*

Test-bed

*

Execution Progress

Tasks

1

Drag the side handles to change the width of the text block.

Time (min.)

Task

H

G

F

E

D

C

B

A

belle.cs.mu.oz.au

belle.anu.edu.au

belle.physics.usyd.edu.au

0 2.0 4.0 6.0 8.0 1012 14

*

Comparison of Sequential and Distributed Execution Distributed Execution time on Grid TestbedSequential Execution Time

*

Presentation OutlinePart 1: Introduction to Grid Computing and ApplicationsTechnology Evolution and Application DriversGrid Challenges, Approaches, and ArchitecturePart 2: Grid Economy and Service Oriented ComputingChallengesService-Oriented Grid Architecture (SOGA)Realisation of SOGAPart 3: Global Grids and Gridbus TechnologiesGrid Market Directory, GridBank, VPM, Grid Service Broker, G-Monitor Part 4: Performance Evaluation on the World-Wide GridCompute Grid Application eScience Application Belle Analysis Data GridPart 5: Closing RemarksOpen Challenges in Grid EconomyAnalogy to Electric Power GridSummary and Conclusion

Alessandro Volta in Paris in 1801 inside French National Institute shows the battery while in the presence of Napoleon IFresco by N. Cianfanelli (1841) (Zoological Section "La Specula" of National History Museum of Florence University)

*

.and in the future, I imagine a WorldwidePower (Electrical) Grid ...What ?!?!This is a mad manOh, monDieu !

*

2005 - 1801 = 204 Years

*

(5) IT services as the fifth utility (water, electricity, gas, telephone, IT) eScienceeBusinesseGovernmenteHealthMultilingualeEducation

*

Summary and ConclusionIntroduced requirements for an eScience application Demonstrated suitability of Grid computing as Cyberinfrastructure for eScience and e-Business.Grids exploit synergies that result from cooperation of autonomous entities:Resource sharing, dynamic provisioning, and aggregation at global level. Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements.Delivering ICT services as computing utilities.Grids offer enormous opportunities for realizing eScience and eBusiness at global level.

*

Any Questions ?Web - http://www.gridbus.org

*

Thanks for your attention!We Welcome Cooperation in Research and Commercialisation!http:/www.gridbus.org | http://www.gridbus.com

For resource providers, they provide service with different goalsFor consumers, they have different requirmentsHow to manage grid service supply-and-demand is hard task

So how can consumers find a resource depending on their requirements. And what type of resource that providers provide and how can they serve consumers within certain QoS.For example, when many consumers want to access same resource, who should be served firstHow to manage supply-demand balance is another challenge.Before a provider publish a service, the provider needs to register first. A registration form is required to fill out.In the registration form, providers need to give their information. Give provider nameAddress, login name, passwordAfter registration, the provider become a GMD member.

Give service name, service type, service cost like hardware cost. How much cost per cpu-sec. How much cost per application operation.Grid node name and service program pathThe query webservice client talks to grid GMD in XML called query message.Query message is encapsulated in SOAP message and sent between GMD client and GMD.

High Energy Physics (HEP) is the study of the fundamental constituents of matter and the forces between these constituents. It is called High Energy Physics as using high energies enables us to probe smaller distances and structures within matter, and also allows us to study matter as it was in the early universe, the history of matter. It is also called Particle Physics as we deal with quanta of matter and forces and the properties associated with these.The study of HEP is broken into two main disciplines, theoretical and experimental. Theoretical HEP propose theories and models to describe matter, forces, their properties, actions, and interactions. Experimental HEP construct experiments or detectors and accelerators to investigate matter interactions and behaviour under high energy conditions.Experimental HEP can be roughly broken into 3 separate activities. The boundaries of these activities, in time and responsibility, are often indistinct. The activities are the construction of detectors which typically takes many years, the measurement or collection of data, and the analysis of this data. We will focus on the using data grids for the analysis of data within HEP.In Grid computing there are many issues that need to be addressed. Like security, authorization work-flow is one of key issues. Why grid infrastructure need to provide workflow management.

This diagram shows astronomical data analysis taken from computing in Canadian Astronomy. It is written by Hugh Couchman.

This is one example shown in this picture. This picture shows a procedure of one scientific experiment. They need to get data first and go through multiple steps, the resources for processing these tasks are distributed. So grid infrastructure needs to provide environments for the users to execute such experiment automatically. Grid workflow management system can be thought of a collection of tasks that are processed on distributed resources in well-defined order. Workflow management is not a new topic. it has been developed for over ten years especially in business systems. Many technologies can be reapplied in Grid workflow system, but there are several new things.

For example one scientific workflow may take very long time. Grid workflow management system need to support large data flow. For example one astronomical project-sloan digital sky survey, the size of data flow around petabytes.One grid workflow may involve may different resources, so resources used by grid workflow are heterogeneous.And these resources may owned by different organizations and are across over multiple administrative domains.In grid environment, many users shared same resources, so the resources utilization could vary over time.In order to realize the workflow management on Grids. There are several requirements.It needs workflow composition tools for users to define their tasks and the dependencies. Need to orchestrate distributed resources and services to meet user requirementsIt need to be able to transfer large scale data between workflow resources.

grid workflow execution environment is very dynamic, because grid resources are not dedicated to the workflow users. Sometime it may be available, sometime it may not be available. And in some cases, intermediate data needed to be saved for further use, and they are managed by some data management systems, so the physical locations can not be known before execution. To get enough resource information is a challenge, since grid resources are owned by different organizations,.We also provide xml-based language for users to define their workflow. Our workflow management system provides a service for users to orchestrate grid resources and execute their workflow applications. We use IBM TSpaces to enable workflow execution. We also provide simple xml based language for the users to express tasks and their dependencies. Current system has been able to execute workflow tasks on real grid test-bed.Current work is focused on architecture part, it will use for our future work, our future will focus on economy-based workflow scheduling. It has three layers. High-level application layer, workflow execution engine layer and Grid resources layer. And resource information is maintained by some info services. For example, GMD (Grid Market Directory) has VO resources information and their price information, MDS contains the information about Grid nodes)

High-level application submits users workflow descriptions to the workflow engine, and engine execute tasks on the grid resources through low level grid middleware such as Globus. It also use some data transfer service like gridftp to move data between resources. The workflow scheduler is the central component of the engine. it schedules tasks in the workflow based on their dependencies. It also queries info services to find a suitable resources for every tasks according to users QoS requirements.

This is the architecture of our current work. Users can use high-level applications such as scientific portal to define workflow and submit to workflow execution engine. Workflow scheduler is the central component, it query grid info services like Grid market directory, data catalog and machines information, to find suitable resources and dispatches workflow tasks on the resources through low-level grid middleware like globus. It also use grid file transfer system like grid ftp to transfer data between different resources. This is the detailed architecture of our workflow scheduling system. There are three major components Workflow Coordinator, Task Mangers and Event Service. We consider Grid workflow scheduling could be very complex. Many different types of resources may be involved in one workflow, for example, some tasks may need to interact with instruments, some tasks need to query database, and some tasks may need to handle different application components. Different tasks may need to handle different types of resources and resource selection will based on different performance models. Instead of using one scheduler to handle all tasks, we use one task manager to handle one task or one task group. Task group is basically a group of tasks which can be executed on the same resource.

There is only one workflow coordinator. It generates task managers dynamically. The workflow coordinator is also responsible for whole workflow execution. Every task manager is responsible to handle task execution, resource discovery and selection for the task and monitor the status of task execution on the remote nodes.

Workflow Coordinator and Task managers are independent, they can be run in parallel. However, their behavior depends on others. For example, Task manager B cannot execute its task until all inputs are available. Its inputs are generated by task manager A. The communication models between workflow coordinator and task managers could be very complex. One task may have multiple parents tasks, and their output may generate at different time. Also one output of one task manager may be required by multiple other task managers.

So the workflow engine needs to handle the complexity of the task dependencies. The communication model between different components is many-to-many. In our system, we use event-driven mechanism with subscription-notification approach to reduce the complexity of communication processing. We also use one of tuple spaces implementation IBM TSpace to be event exchange server to drive the workflow scheduling. The detail is shown in this diagram. The middle one is tuple spaces server. By using event-driven mechanism with subscription-notification approach, every task manager doesnt need to know the detail of others, and it only generates events according to their task processing, if other task managers and workflow coordinator interested in some events, they just register to the TSpace for this event. When the event occurs, they will be notified. For example, when output of task A is available, task manager A put a output event into TSpaces and also give related information, such as the location of the output. If other task managers for example task manger B is waiting for the output of A, it will be informed.

Another benefit of this architecture is additional components are easy to plug in. For example a monitor. It can subscribe status information to tuple spaces and it will be notified when the workflow execution status. And also they can be located on different machines. Currently we model our workflow applications based on DAG, that means there is no cycle. We also provide a simple workflow language for the user to describe their tasks and dependencies. This is the example of task definition. The user can give . The user also can specify a particular resource to run this task. But it is optional, if the users dont supply, the engine will find a suitable resource to the task at the time of task execution.

Datalink is basically to define the dependencies between the tasks. In this example, Task F is dependent on Task C. Task C output 2 will go to Task F port 0.

This is an example for task and datalink definition. In the task definition, users can give task application name, the application location for run this task, And also specify their input and output. Input and output type can be file or parameter value.

Data link basically is to specify the data flow between tasks. Like in this example the input of task F depends on output of task C.

Again, this part is optional, if the resource is not specified, workflow execution engine will discover a suitable one to run this task through a directory service.

In order to evaluate our implementation, we created a workflow application. It has 8 tasks and they have dependencies. For example, task b, c and d rely on the output of A and E depend on Task C and D. We also create synthetic programs to run these tasks. We deployed the programs into three machines. They are ibm e-server with 4 CPU and running on Linux. Globus 2.4 is installed on all machines. They are located in Melbourne, Canberra and Sydney. We run the workflow engine on a desktop with CoG 1.1 installed. Task A is running on machine located in Melbourne. Task b, c and d are executed in parallel. And they were running on the machines located on Melbourne, Sydney and Canberra respectively. Task e starts after task b and c finished. Since task G and F depend the output of D, they started later. This is the execution time of each task. We also run all tasks on one server in the test-bed sequentially. This table shows the execution time. M mean minus and s mean seconds. If we look at single task, we find the execution time of every task is bigger than in sequential execution. that is because of the overhead of sending task to remote nodes. However, if we look at the total execution time, using workflow engine can provide faster solution.