The Gridbus Middleware: Building and Managing Utility Grids for Powering e-Science and e-Business Applications Dr. Rajkumar Buyya Grid Computing and Distributed.

The Gridbus Middleware:Building and Managing Utility Grids for Powering

e-Science and e-Business Applications

Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, Australiaww.gridbus.org

Gridbus Sponsors

2

Outline

Introduction to E-Science Collaborative Science & Challenges

Introduction to Grid Computing Defining Grids, Services, Challenges, Middleware Solutions

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

3

Big Science Problems & Collaborative Research

Next generation experiments, simulations, sensors, satellites, even people and businesses are creating a flood of data. They all involve numerous experts/resources from multiple organization in synthesis, modeling, simulation, analysis, and interpretation.

Life Sciences Digital Biology

Finance: Portfolio analysis

~PBytes/sec

Newswire & data mining:Natural language engineering

Astronomy

Internet & Ecommerce

High Energy Physics Brain Activity Analysis

Quantum Chemistry

4

e-Science Environment: Supporting Collaborative Science

Distributed instruments

Distributed computation

Distributed data

Peers sharing ideas and collaborative

interpretation of data/results

2100 2100 2100 2100

2100 2100 2100 2100

Remote Visualization

Data & Compute Service

Cyberinfrastructure

E-Scientist

5

Multi-institution Collaboration Challenges

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Construction

6

Grid Computing Solution: (1) providing Cyberinfrastructure for e-Science;

(2) delivering IT services as the 5th utility

E-ScienceE-Business

E-GovernmentE-Health

E-Education…

7

Outline








8

What is Grid?[Buyya et. al]

A type of parallel and distributed system that enables the sharing, exchange, selection, & aggregation of geographically distributed “autonomous” resources:

Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc;

Software – e.g., ASPs renting expensive special purpose applications on demand;

Catalogued data and databases – e.g. transparent access to human genome database;

Special devices/instruments – e.g., radio telescope – SETI@Home searching for life in galaxy.

People/collaborators.

depending on their availability, capability, cost, and user QoS requirements.

Widearea

9

How does Grids look like?A Bird Eye View of a Global Grid

Grid Resource Broker

Resource Broker

Application

Grid Information Service


databaseR2R3

RN

R1

R4

R5

R6

Grid Information Service

10

How Are Grids Used?

High-performance computing

Collaborative data-sharing

Collaborative design

Drug discovery

Financial modeling

Data center automation

High-energy physics

Life sciences

E-Business

E-ScienceNatural language processing & Data Mining

Utility computing

11

Classes of Grid Services / Types of Grids

Computational Services – CPU cycles Pooling computing power: SETI@Home, TeraGrid,

AusGrid, ChinaGrid, IndiaGrid, UK Grid,… Data Services

Collaborative data sharing generated by instruments, sensors, persons: LHC Grid, Napster

Application Services Access to remote software/libraries and license

management—NetSolve Interaction Services

eLearning, Virtual Tables, Group Communication (Access Grid), Gaming

Knowledge Services The way knowledge is acquired, processed and

managed—data mining. Utility Computing Services

Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.

Computational Grid

Data Grid

ASP Grid

Interaction Grid

Knowledge Grid

Utility Grid

infra

stru

ctu

re

Users

12

Some Characteristics of Grids: Sources of Resource Management and

Application Scheduling ChallengesNumerousresources

Different securityrequirements

& policies

Resources areheterogeneous

Geographicallydistributed

Different resourcemanagementpolicies

Connected byheterogeneous, multi-level networks

Owned by multiple organizations &

individuals

Unreliable resources and environments

Slide by Hiro

13

Some Grid Initiatives Worldwide Australia

Nimrod-G Gridbus GrangeNet. APACGrid ARC eResearch

Brazil OurGrid, EasyGrid LNCC-Grid + many others

China ChinaGrid – Education CNGrid - application

Europe UK eScience EU Grids.. and many more...

India Garuda

Japan NAREGI

Korea...N*Grid

SingaporeNGP

USA Globus GridSec AccessGrid TeraGrid Cyberinfrasture and many more...

Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft - .NET Oracle 10g Infosys – Enterprise Grid Satyam – Business Grid StorageTek –Grid.. and many more

Public Forums Open Grid Forum Australian Grid Forum Conferences:

CCGrid Grid HPDC E-Science

http://www.gridcomputing.com

1.3 billion – 3 yrs

1 billion – 5 yrs

450million – 5 yrs


1.3 billion (Rs)

27 million

2? billion


14

Layers of Grid Architecture

Grid resourcesDesktops, servers, clusters, networks, applications,

storage, devices + resource manager + monitor

Security ServicesAuthentication, Single sign-on, secure communication

Job submission, info services, Storage access, Trading, Accounting, License

Resource management and scheduling

Grid programming environment and toolsLanguages, API, libraries, compilers, parallelization tools

Grid applicationsWeb Portals, Applications,

System level

User level

Adaptiv

e M

anagem

ent

CoreMiddleware

User-LevelMiddleware

15

Open-Source Grid Middleware Projects

16

The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand

WWG

Pushes Grid computing into mainstream

computing

Gridbus

17

The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying e-Research Applications on Utility Grids

The Gridbus Project @ GRIDS Lab, The University of Melbourne: The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying eToolkit for Creating and Deploying e--Research Applications on Utility GridsResearch Applications on Utility Grids

Gridbus

Distributed Data

http://www.gridbus.org

• Gridbus is a “open source” Grid R&D project with focus on Grid Economy, Utility Grids and Service Oriented Computing.

• Gridbus Middleware components include:– Alchemi: .NET-based Enterprise Grid

– Grid Market Directory and Web Services

– Grid Bank: Accounting and Transaction Management

– Visual Tools for Creation of Distributed Applications

– Grid Service Broker and Scheduling

– Workflow Management Engine

– GridSim Toolkit

– Libra: SLA-based Resource Allocation

18

Outline








19

What does Grid players want?

Grid Consumers Execute jobs for solving varying problem size and complexity Benefit by utilizing distributed resources wisely Tradeoff timeframe and cost

Strategy: minimise expenses

Grid Providers Contribute resources for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity

Strategy: maximise return on investment

20

What does Grid players require?

They need tools and technologies that help them in value expression, value translation, and value enforcement.

Grid Service Consumers (GSCs): How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? How do I manage Grid dynamics and get my work done? …

Grid Service Providers (GSPs) How do I decide service pricing models ? How do I specify them ? How do I translate them into resource allocations ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …

21

Solution 1: Service Oriented Architecture (SOA)

A SOA is a contractual architecture for offering and consuming software as services.

There are four entities that make up an SOA service provider, service registry, and service consumer (also known as service requestor).

The functions or tasks that the service provider offers, along with other functional and technical information required for consumption, are defined in

the service definition or contract.

provider

registry

consumer

contract

22

Solution 2: Market-Oriented Grid Computing - (a) Sustained Resourced Sharing and (b)

Effective Management of Shared Resources

Grid Economy

23

Grid Node N

Service-Oriented Grid Architecture

Grid Servuce Consumer

Pro

gra

mm

ing

En

viro

nm

ents


Grid Service Providers

Grid Explorer

Schedule Advisor

Trade Manager

Job ControlAgent

Deployment Agent

Trade Server

Resource Allocation

ResourceReservation

R1

Misc. services

Information Service

R2 Rm…

Pricing Algorithms

Accounting

Grid Node1

…

Core Middleware Services

…

…

HealthMonitor

Grid Market Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign-on

Grid Bank

Ap

pli

cati

on

s

Data Catalogue

24

Layers of Grid Architecture

Grid resourcesDesktops, servers, clusters, networks, applications,

storage, devices + resource manager + monitor

Security ServicesAuthentication, Single sign-on, secure communication

Job submission, info services, Storage access, Trading, Accounting, License

Resource management and scheduling

Grid programming environment and toolsLanguages, API, libraries, compilers, parallelization tools

Grid applicationsWeb Portals, Applications,

Adaptiv

e M

anagem

ent

Application Development and Deployment Environment

Distributed Resources Coupling Services

CoreMiddleware


System level

User level

Au

tonomic/ G

rid Econ

omy

25

Gridbus and Complementary Technologies – realizing Utility Grid

AIXSolarisWindows Linux

.NETGridFabricSoftware

GridApplications

Core GridMiddleware


GridBank

Grid Exchange & Federation

JVM

Grid Brokers:

X-Parameter Sweep Lang.

Gridbus Data Broker

MPI

Condor SGE TomcatPBS

Alchemi

Workflow

IRIX OSF1 Mac

Libra

Globus Unicore ……Grid

MarketDirectory

PDB

CDB

Worldwide Grid

GridFabricHardware

……

PortalsScience Commerce Engineering ……Collaboratories

……

Workflow Engine

Grid Storage Economy

Gri

d E

con

om

y

NorduGrid XGrid

ExcellGrid

Nimrod-G

Gridscape

26

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)

(Globus)

Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Resu

lts9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

2733

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Res

ults

9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

28

Alchemi: .NET-based Enterprise Grid Platform & Web Services

InternetInternet

InternetInternet

Alchemi Worker Agents

Alchemi Manager

Alchemi Users

Web Services

Web Services

•SETI@Home like Model•General Purpose•Dedicated/Non-dedicate workers•Role-based Security•.NET and Web Services•C# Implementation•GridThread and Job Model Programming•Easy to setup and use• Widely in use!

29

Some Users of Alchemi

Tier Technologies, USALarge scale document processing using Alchemi framework

CSIRO, AustraliaNatural Resource Modeling

The Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genes

Satyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi framework

The University of Sao Paulo, BrazilThe Alchemi Executor as a Windows Service

stochastix GmbH, GermanyServing clients in International Banking/Finance sector

Many users in Universities: See next for an example.

30

Students' project gives old computers new life - 1/25/2005

31

Outline








32

A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.

It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation)

Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting

Grid Service Broker (GSB)

33

Gridbus Broker Architecture

Grid Middleware

Gridbus Client Gridbus ClientGribus Client

Grid Info Server

Schedule Advisor

Trading Manager

Gridbus Farming Engine

RecordKeeper

Grid Explorer

GE GIS, NWSTM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

G

G

CU

Globus enabled node.A

L

Alchemi enabled node.

(Data Grid Scheduler)

DataCatalog

DataNode

Unicore enabled node.

$

$

$

App, T, $, Opt

(Bag of Tasks Applications)

34

Gridbus Broker and Remote Service Access Enablers

Alchemi

Gateway

UnicoreData Store

Access Technology

Grid FTPSRB

-PBS-Condor-SGE

Globus

Job manager

fork() batch()

Gridbusagent

Data Catalog

-PBS-Condor-SGE-XGrid

SSH

fork()

batch()

Gridbusagent

Credential RepositoryMyProxy

Home Node/Portal

GridbusBroker

fork()

batch() -PBS-Condor-SGE-Alchemi-XGrid

Por

tlets

35

Gridbus Services for eScience applications

Application Development Environment: XML-based language for composition of task farming (legacy)

applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine.

Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes

that meet user QoS requirements. Hide Low-Level Grid Middleware interfaces

Globus (v2, v4), SRB, Alchemi, Unicore, and ssh-based access to local/remote resources managed by XGrid, Condor, SGE.

36

Discover Discover ResourcesResources

Distribute JobsDistribute Jobs

Establish Establish RatesRates

Meet requirements ? Remaining Meet requirements ? Remaining Jobs, Deadline, & Budget ?Jobs, Deadline, & Budget ?

Evaluate & Evaluate & RescheduleReschedule

Discover Discover More More

ResourcesResources

Compose & Compose & ScheduleSchedule

Adaptive Scheduling Steps

37

Deadline (D) and Budget (B) Constrained Scheduling Algorithms

Algorithm Execution Time (D)

Execution Cost (B)

Compute Grid

Data Grid

Cost Opt Limited by D Minimize Yes Yes

Cost-Time Opt

Minimize if possible

Minimize Yes

Time Opt Minimize Limited by B Yes Yes

Conservative-Time Opt

Minimize Limited by B, jobs have guaranteed minimum budget

Yes

38

Gridbus Project: Some Applications and Users

Gridbus Project: Gridbus Project: Some Applications and UsersSome Applications and Users

http://www.gridbus.org

BioGrid: Molecular docking for Drug-discovery

BioGrid: Molecular docking for Drug-discovery

High Energy Physics: Particle Discovery

High Energy Physics: Particle Discovery

Melbourne University

NeuroScience: Brain Activity Analysis

NeuroScience: Brain Activity Analysis

Natural Resource ModelingNatural Resource Modeling

CSIRO Land and Water, Austraila.

Large Scale document processing

Large Scale document processing

Tier Technologies, USA.

Detection of patterns of transcription factors in mammalian genes

Detection of patterns of transcription factors in mammalian genes

39

Figure 3 : Logging into the portal.

Drug DesignMade Easy!

Click Here for Demo

40

Excel Plugin to Access Gridbus Services

Excel

ExcelGrid Add-In

ExcelGrid Runner

ExcelGridJob

ExcelGrid Middleware

Gridbus Broker

Enterprise Grid

2100

2100

2100

2100

2100

2100

2100

2100

41

Outline

Introduction to the University Melbourne, GRIDS Lab, and Opportunities

Recap of the First Lecture What are Grids, Challenges, Middleware Solutions






42

Case Study: High Energy Physics and Data Grid

The Belle Experiment KEK B-Factory, Japan Investigating fundamental violation

of symmetry in nature (Charge Parity) which may help explain “why do we have more antimatter in the universe OR imbalance of matter and antimatter in the universe?”.

Collaboration 1000 people, 50 institutes

100’s TB data currently

43

Case Study: Event Simulation and Analysis

B0->D*+D*-Ks

• Simulation and Analysis Package - Belle Analysis Software Framework (BASF)• Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data

Analyzed 100 data files (30MB each) that were distributed among the five nodes within Australian Belle DataGrid platform.

44

Australian Belle Data Grid Testbed

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results

CertificateAuthority

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper


GRIDS Lab, University of Melbourne

Dept. of Physics,University of Sydney

ANU, Canberra

Dept. of Computer Science, University of Adelaide

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Intel Pentium 2.0 Ghz, 512 MB RAM

Dept. of Physics,University of Melbourne

NWSSensor

GridFTPGRIS

GlobusGatekeeper


VPACMelbourne

45

Belle Data Grid (GSP CPU Service Price: G$/sec)

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results


NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper




ANU, Canberra


NWSSensor

GridFTPGRIS

GlobusGatekeeper



NWSSensor

GridFTPGRIS

GlobusGatekeeper


NA

G$4

G$4

Datanode

G$6VPAC

MelbourneG$2

46

Belle Data Grid (Bandwidth Price: G$/MB)

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results


NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper




ANU, Canberra


NWSSensor

GridFTPGRIS

GlobusGatekeeper



NWSSensor

GridFTPGRIS

GlobusGatekeeper


NA

G$4

G$4

Datanode

G$6VPAC

MelbourneG$2

34

31

38

31

30

3336

32

47

Deploying Application Scenario

A data grid scenario with 100 jobs and each accessing remote data of ~30MB

Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario:

Minimise Time Minimise Cost

Results:

SUMMARY OF EVALUATION RESULTS

Scheduling strategy Total Time Taken (mins.)

Compute Cost (G$)

Data Cost (G$)

Total Cost (G$)

Cost Minimization 71.07 26865 7560 34425 Time Minimization 48.5 50938 7452 58390

48

Time Minimization in Data Grids

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

Time (in mins.)

Nu

mb

er

of

job

s c

om

ple

ted

fleagle.ph.unimelb.edu.au belle.anu.edu.au belle.physics.usyd.edu.au brecca-2.vpac.org

49

Results : Cost Minimization in Data Grids

0

10

20

30

40

50

60

70

80

90

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63

Time(in mins.)

Nu

mb

er o

f jo

bs

com

ple

ted

fleagle.ph.unimelb.edu.au belle.anu.edu.au belle.physics.usyd.edu.au brecca-2.vpac.org

50

SUMMARY OF EVALUATION RESULTS

Scheduling strategy Total Time Taken (mins.)

Compute Cost (G$)

Data Cost (G$)

Total Cost (G$)

Cost Minimization 71.07 26865 7560 34425 Time Minimization 48.5 50938 7452 58390

Observation

Organization

Node details Cost (in G$/CPU-sec) Total Jobs Executed

Time Cost

CS,UniMelb belle.cs.mu.oz.au4 CPU, 2GB RAM, 40 GB HD, Linux

N.A. (Not used as a compute resource)

-- --

Physics, UniMelb fleagle.ph.unimelb.edu.au1 CPU, 512 MB RAM, 40 GB HD, Linux

2 3 94

CS, University of Adelaide

belle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux

N.A. (Not used as a compute resource)

-- --

ANU, Canberra belle.anu.edu.au4 CPU, 2GB RAM, 40 GB HD, Linux

4 2 2

Dept of Physics, USyd

belle.physics.usyd.edu.au4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux

4 72 2

VPAC, Melbourne brecca-2.vpac.org180 node cluster (only head node used), Linux

6 23 2

51

Outline








52


Grids exploit synergies that result from cooperation of autonomous entities:

Resource sharing, dynamic provisioning, and aggregation at global level Great Science and Great Business!

Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications.

SOA + Market-based Grid Management = Utility Grids Grids allow users to dynamically lease Grid services at

runtime based on their quality, cost, availability, and users QoS requirements.

Delivering ICT services as computing utilities. Grids offer enormous opportunities for realizing

e-Science and e-Business at global level. Use our Gridbus technology to realise this and make money!

53

Thanks for your attention!

We Welcome Cooperation in Research and Development!http:/www.gridbus.org

The Gridbus Middleware: Building and Managing Utility Grids for Powering e-Science and e-Business Applications Dr. Rajkumar Buyya Grid Computing and Distributed.

Documents

escience environment

powering escience

grid bank

ebusiness applications

computer science

conclusion slide

grid computing solution

rajkumar buyya grid