Grid Computing and the Gridbus Middleware: Making the Global Cyberinfrastructure for e-Science and e-Business Applications a Reality Dr. Rajkumar Buyya.

Grid Computing and the Gridbus Middleware:Making the Global Cyberinfrastructure for e-Science

and e-Business Applications a Reality

Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

www.buyya.comwww.gridbus.org

2

GRIDS Lab @ Melbourne

Youngest and one of the rapidly growing research labs in our School/University:

Founded in 2002 Houses:

Research Fellows (3) Research Programmers (3) PhD candidates (10) Honours/Masters students (5+)

Funding National and International organizations Australian Research Council Many industries (Sun, StorageTek, Microsoft,

IBM, Microsoft) University-wide collaboration:

Faculties of Science, Engineering, and Medicine

Many national and international collaborations.

Academics Industries

Software: Widely in academic and industrial users.

Publication: My research team over 20% of our Dept’s

research output.

EducationR & D

+ Community Services: e.g., IEEE TC for Scalable Computing

3

Books at a Glance: Co-authored & edited

4

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service Oriented Grids and Grid Economy SOGA, Grid Market Directory, Grid Bank, Broker

Grid Service Broker Architecture, Design and Implementation

Performance Evaluation: Experiments in Creation and Deployment of Applications on Global Grids

A Case Study in High Energy Physics Summary and Conclusion

5

4 Essential Utilities and Delivery Networks

(2) Electricity

(3) Gas

(1) WaterWater

Distribution Network

(4) TelephoneTelecom Networks

6

(5) Computing Grid: Delivery IT services as the 5th utility (Power Grid inspiration)

eScienceeBusiness

eGovernmenteHealth

MultilingualeEducation

…

7

Power Grid Inspiration: Seamlessly delivering electricity as a utility to users

8

Why Grid Computing Now?Let us look at the Evolution of ICT

9

Computing and Communication Technologies Evolution & Timeline

* Sputnik

1960 1970 1975 1980 1985 1990 1995 2000

* ARPANET

* Email* Ethernet

* TCP/IP* IETF

* Internet Era * WWW Era

* Mosaic

* XML

* PC Clusters* Crays * MPPs

* Mainframes

* HTML

* W3C

* P2P

* Grids

* XEROX PARC wormCO

MP

UTIN

GC

om

mu

nic

ati

on

* Web Services

* Minicomputers

* PCs

* WS Clusters

* PDAs* Workstations

* HTC

2010

* e-Science

* Computing Utility

* e-Business

* SocialNet

ControlCentralised Decentralised

11

2100

2100 2100 2100 2100

2100 2100 2100 2100

Personal Device SMPs or SuperComputers

LocalCluster

GlobalGrid

SERV ICES

+

PERFORMANCE

Inter PlanetGrid

•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Universe

Administrative Barriers

EnterpriseCluster/Grid

Computing is Scaling: Towards Inter-Planetary Level

12

What is Grid?(there are several academic definitions, here is ours)

A type of parallel and distributed system that enables the sharing, exchange, selection, & aggregation of geographically distributed “autonomous” resources:

Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc;

Software – e.g., ASPs renting expensive special purpose applications on demand;

Catalogued data and databases – e.g. transparent access to human genome database;

Special devices/instruments – e.g., radio telescope – SETI@Home searching for life in galaxy.

People/collaborators.

depending on their availability, capability, cost, and user QoS requirements.

Widearea

13

How does Grids look like?A Bird Eye View of a Global Grid

Grid Resource Broker

Resource Broker

Application

Grid Information Service


databaseR2R3

RN

R1

R4

R5

R6

Grid Information Service

14

Classes of Grid Services / Types of Grids

Computational Services – CPU cycles Pooling computing power: SETI@Home, TeraGrid,

AusGrid, ChinaGrid, IndiaGrid, UK Grid,… Data Services

Collaborative data sharing generated by instruments, sensors, persons: LHC Grid, Napster

Application Services Access to remote software/libraries and license

management—NetSolve Interaction Services

eLearning, Virtual Tables, Group Communication (Access Grid), Gaming

Knowledge Services The way knowledge is acquired, processed and

managed—data mining. Utility Computing Services

Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.

Computational Grid

Data Grid

ASP Grid

Interaction Grid

Knowledge Grid

Utility Grid

infra

stru

ctu

re

Users

15

How Are Grids Used?

High-performance computing

Collaborative data-sharing

Collaborative design

Drug discovery

Financial modeling

Data center automation

High-energy physics

Life sciences

E-Business

E-ScienceNatural language processing & Data Mining

Utility computing

16

1. [Grid Use in Science] Online Medical Instrumentation and Neuroscience

Osaka Univ. Hospital

Osaka Univ. DV transfer

Life-electronics laboratory,AIST

Data Analysis

•Provision of MEG•Provision of expertise in the analysis of brain function

Cybermedia Center

Data Generation

Analysis Results

Analysis Results

A

Virtual Laboratoryfor medicine and brain science

•Knowledge sharing•MEG sharing?•Data Sharing

17

2. [Grid Use in Business] Enterprise Computing Application

Traditional Model Grid-based Model

Email server

Webserver

Databaseserver

Appsserver

Upgrade to a new serverto handle

more users

Utilise IT infrastructure effectively

Service Virtualization Layer & Load Balancing

18

Agenda



Service Oriented Grids and Grid Economy SOGA, Grid Market Directory, Grid Bank, Broker




19

Some Characteristics of Grids

Numerousresources

Different securityrequirements

& policies

Resources areheterogeneous

Geographicallydistributed

Different resourcemanagementpolicies

Connected byheterogeneous, multi-level networks

Owned by multiple organizations &

individuals

Unreliable resources and environments

Slide by Hiro

20

Grid Challenges

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Construction

21

Some Grid Initiatives Worldwide

Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch

Brazil OurGrid, EasyGrid LNCC-Grid + many others

China ChinaGrid – Education CNGrid - application

Europe UK eScience EU Grids.. and many more...

India Garuda

Japan NAGERI

Korea...N*Grid

SingaporeNGP

USA Globus GridSec AccessGrid TeraGrid Cyberinfrasture and many more...

Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft - .NET Oracle 10g Infosys – Enterprise Grid Satyam – Business Grid StorageTek –Grid.. and many more

Public Forums Global Grid Forum Australian Grid Forum Conferences:

CCGrid Grid HPDC E-Science

http://www.gridcomputing.com

1.3 billion – 3 yrs

1 billion – 5 yrs

450million – 5 yrs


1.3 billion (Rs)

27 million

2? billion


22

Open-Source Grid Middleware Projects

23

The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand

WWG

Pushes Grid computing into mainstream

computing

Gridbus

24

The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying e-Research Applications on Utility Grids

The Gridbus Project @ GRIDS Lab, The University of Melbourne: The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying eToolkit for Creating and Deploying e--Research Applications on Utility GridsResearch Applications on Utility Grids

Gridbus

Distributed Data

http://www.gridbus.org

• Gridbus is a “open source” Grid R&D project with focus on Grid Economy, Utility Grids and Service Oriented Computing.

• Gridbus Middleware components include:– Alchemi: .NET-based Enterprise Grid

– Grid Market Directory and Web Services

– Grid Bank: Accounting and Transaction Management

– Visual Tools for Creation of Distributed Applications

– Grid Service Broker and Scheduling

– Workflow Management Engine

– GridSim Toolkit

– Libra: SLA-based Resource Allocation

25

Why Grid Economy for Gridbus?: (1) Sustained Resourced Sharing and (2) Effective Management of Shared Resources

Grid Economy

26

Agenda



Service Oriented Grids and Grid Economy SOGA, Grid Market Directory, Grid Bank, Broker..




27

Grid Node N

A Reference Service-Oriented Architecture for Utility Grids

Grid Consumer

Pro

gra

mm

ing

En

viro

nm

ents


Grid Service Providers

Grid Explorer

Schedule Advisor

Trade Manager

Job ControlAgent

Deployment Agent

Trade Server

Resource Allocation

ResourceReservation

R1

Misc. services

Information Service

R2 Rm…

Pricing Algorithms

Accounting

Grid Node1

…

Grid Middleware Services

…

…

HealthMonitor

Grid Market Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign-on

Grid Bank

Ap

pli

cati

on

s

Data Catalogue

28

Gridbus and Complementary Technologies – realizing Utility Grid

AIXSolarisWindows Linux

.NET GridFabricSoftware

GridApplications

Core GridMiddleware

User-LevelMiddleware(Grid Tools)

GridBank

Grid Exchange & Federation

JVM

Grid Brokers:

X-Parameter Sweep Lang.

Gridbus Data Broker

MPI

Condor SGE TomcatPBS

Alchemi

Workflow

IRIX OSF1 Mac

Libra

Globus Unicore ……Grid

MarketDirectory

PDB

CDB

Worldwide Grid

GridFabricHardware

……

PortalsScience Commerce Engineering ……Collaboratories

……

Workflow Engine

Grid Storage Economy

Gri

d E

con

om

y NorduGrid XGrid

ExcellGrid

Nimrod-G

GRIDSIM

Gridscape

29

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)

(Globus)

Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Resu

lts9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

3033


ASP Catalogue

Grid Info Service



GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2



data1

36

45

Res

ults

9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

31

Alchemi: .NET-based Enterprise Grid Platform & Web Services

InternetInternet

InternetInternet

Alchemi Worker Agents

Alchemi Manager

Alchemi Users

Web Services

Web Services

•SETI@Home like Model•General Purpose•Dedicated/Non-dedicate workers•Role-based Security•.NET and Web Services•C# Implementation•GridThread and Job Model Programming•Easy to setup and use• Widely in use!

32

Some Users of Alchemi

Tier Technologies, USALarge scale document processing using Alchemi framework

CSIRO, AustraliaNatural Resource Modeling

The Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genes

Satyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi framework

The University of Sao Paulo, BrazilThe Alchemi Executor as a Windows Service

stochastix GmbH, GermanyServing clients in International Banking/Finance sector

Many users in Universities: See next for an example.

33

Students' project gives old computers new life - 1/25/2005

34

Agenda







35

A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.

It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation)

Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting

Grid Service Broker (GSB)

36

Gridbus Broker Architecture

Grid Middleware

Gridbus Client Gridbus ClientGribus Client

Grid Info Server

Schedule Advisor

Trading Manager

Gridbus Farming Engine

RecordKeeper

Grid Explorer

GE GIS, NWSTM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

G

G

CU

Globus enabled node.A

L

Alchemi enabled node.

(Data Grid Scheduler)

DataCatalog

DataNode

Unicore enabled node.

$

$

$

App, T, $, Opt

(Bag of Tasks Applications)

38

Gridbus Services for eScience applications

Application Development Environment: XML-based language for composition of task farming (legacy)

applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine.

Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes

that meet user QoS requirements. Hide Low-Level Grid Middleware interfaces

Globus (v2, v4), SRB, Alchemi, Unicore, and ssh-based access to local/remote resources managed by XGrid, Condor, SGE.

39

Discover Discover ResourcesResources

Distribute JobsDistribute Jobs

Establish Establish RatesRates

Meet requirements ? Remaining Meet requirements ? Remaining Jobs, Deadline, & Budget ?Jobs, Deadline, & Budget ?

Evaluate & Evaluate & RescheduleReschedule

Discover Discover More More

ResourcesResources

Compose & Compose & ScheduleSchedule

Adaptive Scheduling Steps

40

Deadline (D) and Budget (B) Constrained Scheduling Algorithms

Algorithm Execution Time (D)

Execution Cost (B)

Compute Grid

Data Grid

Cost Opt Limited by D Minimize Yes Yes

Cost-Time Opt

Minimize if possible

Minimize Yes

Time Opt Minimize Limited by B Yes Yes

Conservative-Time Opt

Minimize Limited by B, jobs have guaranteed minimum budget

Yes

41

Gridbus Project: Some Applications and Users

Gridbus Project: Gridbus Project: Some Applications and UsersSome Applications and Users

http://www.gridbus.org

BioGrid: Molecular docking for Drug-discovery

BioGrid: Molecular docking for Drug-discovery

High Energy Physics: Particle Discovery

High Energy Physics: Particle Discovery

Melbourne University

NeuroScience: Brain Activity Analysis

NeuroScience: Brain Activity Analysis

Natural Resource ModelingNatural Resource Modeling

CSIRO Land and Water, Austraila.

Large Scale document processing

Large Scale document processing

Tier Technologies, USA.

Detection of patterns of transcription factors in mammalian genes

Detection of patterns of transcription factors in mammalian genes

44

Agenda







45

Case Study: High Energy Physics and Data Grid

The Belle Experiment KEK B-Factory, Japan Investigating fundamental

violation of symmetry in nature (Charge Parity) which may help explain “why do we have more antimatter in the universe?”.

Collaboration 1000 people, 50 institutes

100’s TB data currently

46

Case Study: Event Simulation and Analysis

B0->D*+D*-Ks

• Simulation and Analysis Package - Belle Analysis Software Framework (BASF)• Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data

Analyzed 100 data files (30MB each) that were distributed among the five nodes within Australian Belle DataGrid platform.

47

Australian Belle Data Grid Testbed

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results

CertificateAuthority

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper


GRIDS Lab, University of Melbourne

Dept. of Physics,University of Sydney

ANU, Canberra

Dept. of Computer Science, University of Adelaide

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Intel Pentium 2.0 Ghz, 512 MB RAM

Dept. of Physics,University of Melbourne

NWSSensor

GridFTPGRIS

GlobusGatekeeper


VPACMelbourne

48

Belle Data Grid (GSP CPU Service Price: G$/sec)

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results


NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper




ANU, Canberra


NWSSensor

GridFTPGRIS

GlobusGatekeeper



NWSSensor

GridFTPGRIS

GlobusGatekeeper


NA

G$4

G$4

Datanode

G$6VPAC

MelbourneG$2

49

Belle Data Grid (Bandwidth Price: G$/MB)

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results


NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper


NWSSensor

GridFTPGRIS

GlobusGatekeeper




ANU, Canberra


NWSSensor

GridFTPGRIS

GlobusGatekeeper



NWSSensor

GridFTPGRIS

GlobusGatekeeper


NA

G$4

G$4

Datanode

G$6VPAC

MelbourneG$2

34

31

38

31

30

3336

32

50

Deploying Application Scenario

A data grid scenario with 100 jobs and each accessing remote data of ~30MB

Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario:

Minimise Time Minimise Cost

Results:

SUMMARY OF EVALUATION RESULTS

Scheduling strategy Total Time Taken (mins.)

Compute Cost (G$)

Data Cost (G$)

Total Cost (G$)

Cost Minimization 71.07 26865 7560 34425 Time Minimization 48.5 50938 7452 58390

51

Time Minimization in Data Grids

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

Time (in mins.)

Nu

mb

er

of

job

s c

om

ple

ted

fleagle.ph.unimelb.edu.au belle.anu.edu.au belle.physics.usyd.edu.au brecca-2.vpac.org

52

Results : Cost Minimization in Data Grids

0

10

20

30

40

50

60

70

80

90

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63

Time(in mins.)

Nu

mb

er o

f jo

bs

com

ple

ted

fleagle.ph.unimelb.edu.au belle.anu.edu.au belle.physics.usyd.edu.au brecca-2.vpac.org

53

SUMMARY OF EVALUATION RESULTS

Scheduling strategy Total Time Taken (mins.)

Compute Cost (G$)

Data Cost (G$)

Total Cost (G$)

Cost Minimization 71.07 26865 7560 34425 Time Minimization 48.5 50938 7452 58390

Observation

Organization

Node details Cost (in G$/CPU-sec) Total Jobs Executed

Time Cost

CS,UniMelb belle.cs.mu.oz.au4 CPU, 2GB RAM, 40 GB HD, Linux

N.A. (Not used as a compute resource)

-- --

Physics, UniMelb fleagle.ph.unimelb.edu.au1 CPU, 512 MB RAM, 40 GB HD, Linux

2 3 94

CS, University of Adelaide

belle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux

N.A. (Not used as a compute resource)

-- --

ANU, Canberra belle.anu.edu.au4 CPU, 2GB RAM, 40 GB HD, Linux

4 2 2

Dept of Physics, USyd

belle.physics.usyd.edu.au4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux

4 72 2

VPAC, Melbourne brecca-2.vpac.org180 node cluster (only head node used), Linux

6 23 2

54

Agenda







Alessandro Volta in Paris in 1801 inside French National Institute shows the battery while in the presence of Napoleon I

Fresco by N. Cianfanelli (1841) (Zoological Section "La Specula" of National History Museum of Florence University)

56

….and in the future, I imagine a WorldwidePower (Electrical) Grid …...

What ?!?!This is a mad man…

Oh, monDieu !

57

2006 - 1801 = 205 Years(Recent R&D: Delivering Internet services via Electric cables).

58

When is we going get IT as the 5th utility?(water, electricity, gas, telephone, IT)

eScienceeBusiness

eGovernmenteHealth

MultilingualeEducation

…

59

Summary and Conclusion

Grids exploit synergies that result from cooperation of autonomous entities:

Resource sharing, dynamic provisioning, and aggregation at global level Great Science and Great Business!

Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications.

Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements.

Delivering ICT services as computing utilities. Grids offer enormous opportunities for realizing

e-Science and e-Business at global level. Use our Gridbus technology to realise this and make money!

60

Thank you & Any Questions ?

Web - http://www.gridbus.org

61

Further References

Grid Computing and the Gridbus Middleware:Making the Global Cyberinfrastructure for e-Science

and e-Business Applications a Reality

Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

www.buyya.comwww.gridbus.org

Backup Slides

67

What is Grid? (It means different things to different people)

IBM On Demand Computing

Microsoft .NET

Oracle 10g

Sun N1

HP Adaptive Enterprise

United Devices and related companies: Harvesting Unused Desktop resources

68

Grid & Related Paradigms

Utility Computing• Computing “services”• No knowledge of provider• Enabled by grid technology

Utility Computing• Computing “services”• No knowledge of provider• Enabled by grid technology

Distributed Computing• Loosely coupled• Heterogeneous• Single Administration

Distributed Computing• Loosely coupled• Heterogeneous• Single Administration

Cluster• Tightly coupled• Homogeneous• Cooperative working

Cluster• Tightly coupled• Homogeneous• Cooperative working

Grid Computing• Large scale• Cross-organizational• Geographical distribution• Distributed Management

Grid Computing• Large scale• Cross-organizational• Geographical distribution• Distributed Management

6933


ASP Catalogue

Grid Info Service



GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2



data1

36

45

Res

ults

9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

71

Grid Computing and the Gridbus Middleware: Making the Global Cyberinfrastructure for e-Science and e-Business Applications a Reality Dr. Rajkumar Buyya.

Documents

computing grid

grid bank

grid economysoga

computing utility

scalable computing

grid market directory

rajkumar buyyagrid computing

research programmers