Top Banner
Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Laboratory Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org Gridbus Sponsors
67

Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

Jan 15, 2016

Download

Documents

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

Gridbus Middleware and Utility Grids:Building Autonomic and Market-Oriented Global Grids for

Delivering IT Services as the 5th Utility

Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

www.gridbus.org

Gridbus Sponsors

Page 2: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

2

GRIDS Lab @ Melbourne

Youngest and one of the rapidly growing research labs in our School/University:

Founded in 2002 Houses:

Research Fellows/PostDocs Research Programmers PhD candidates Honours/Masters students

Funding National and International organizations Australian Research Council & DEST Many industries (Sun, StorageTek, Microsoft,

IBM, Microsoft) University-wide collaboration:

Faculties of Science, Engineering, and Medicine

Many national and international collaborations.

Academics Industries

Software: Widely in academic and industrial users.

Publication: My research team produces over 20% of our

Dept’s research output.

EducationR & D

+ Community Services: e.g., IEEE TC for Scalable Computing

Page 3: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

3

(some) Books: Co-authored & edited

Page 4: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

4

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 5: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

5

4 Essential Utilities and Delivery Networks

(1) WaterWater

Distribution Network

(4) TelephoneTelecom Networks

(3) Gas

(2) Electricity

Power Grid

Page 6: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

6

(5) Computing Grid: Delivering IT services as the 5th utility (Power Grid inspiration)

eScienceeBusiness

eGovernmenteHealth

MultilingualeEducation

Page 7: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

7

Power Grid Inspiration: Seamlessly delivering electricity as a utility to users

Page 8: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

8

Grid-like Vision

In 1969, Leonard Kleinrock, one of the chief scientists of the original ARPA project which seeded the Internet, wrote:

"As of now, computer networks are still in their infancy, but as they grow up and become sophisticated, we will probably see the spread of "computer utilities", which, like present electric and telephone utilities, will service individual homes and offices across the country“

Despite major advances in hardware and software systems over the past 35 years, we are yet to realize this vision. How far are we still from delivering computing as a utility?

Let us look into the ICT evolution and project the future.

Page 9: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

9

Why Grid Computing Now?Let us look at the Evolution of ICT

Page 10: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

10

Computing and Communication Technologies Evolution: 1960-2010!

* Sputnik

1960 1970 1975 1980 1985 1990 1995 2000

* ARPANET

* Email* Ethernet

* TCP/IP* IETF

* Internet Era * WWW Era

* Mosaic

* XML

* PC Clusters* Crays * MPPs

* Mainframes

* HTML

* W3C

* P2P

* Grids

* XEROX PARC wormCO

MP

UTIN

GC

om

mu

nic

ati

on

* Web Services

* Minicomputers

* PCs

* WS Clusters

* PDAs* Workstations

* HTC

2010

* e-Science

* Computing as Utility

* e-Business

* SocialNet

ControlCentralised Decentralised

Page 11: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

11

2100

2100 2100 2100 2100

2100 2100 2100 2100

Personal Device SMPs or SuperComputers

LocalCluster

GlobalGrid

SERV ICES

+

PERFORMANCE

Inter PlanetGrid

•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Universe

Administrative Barriers

EnterpriseCluster/Grid

Computing is Scaling: Towards Inter-Planetary Level

Page 12: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

12

What is Grid? (It means different things to different people)

IBM On Demand Computing

Microsoft .NET

Oracle 10g

Sun N1 – Sun Grid Engine

HP Adaptive Enterprise

United Devices and related companies: Harvesting Unused Desktop resources

Page 13: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

13

What is Grid?[Buyya et. al.]

A type of parallel and distributed system that enables the sharing, exchange, selection, & aggregation of geographically distributed “autonomous” resources:

Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc;

Software – e.g., ASPs renting expensive special purpose applications on demand;

Catalogued data and databases – e.g. transparent access to human genome database;

Special devices/instruments – e.g., radio telescope – SETI@Home searching for life in galaxy.

People/collaborators.

depending on their availability, capability, cost, and user QoS requirements.

Widearea

Page 14: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

14

How does Grids look like?A Bird Eye View of a Global Grid

Grid Resource Broker

Resource Broker

Application

Grid Information Service

Grid Resource Broker

databaseR2R3

RN

R1

R4

R5

R6

Grid Information Service

Page 15: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

15

Classes of Grid Services / Types of Grids

Computational Services – CPU cycles Pooling computing power: SETI@Home, TeraGrid,

AusGrid, ChinaGrid, IndiaGrid, UK Grid,… Data Services

Collaborative data sharing generated by instruments, sensors, persons: LHC Grid, Napster

Application Services Access to remote software/libraries and license

management—NetSolve Interaction Services

eLearning, Virtual Tables, Group Communication (Access Grid), Gaming

Knowledge Services The way knowledge is acquired, processed and

managed—data mining. Utility Computing Services

Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.

Computational Grid

Data Grid

ASP Grid

Interaction Grid

Knowledge Grid

Utility Grid

infra

stru

ctu

re

Users

Page 16: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

16

How Are Grids Used?

High-performance computing

Collaborative data-sharing

Collaborative design

Drug discovery

Financial modeling

Data center automation

High-energy physics

Life sciences

E-Business

E-ScienceNatural language processing & Data Mining

Utility computing

Page 17: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

17

e-Science Environment: Supporting Collaborative Science

Distributed instruments

Distributed computation

Distributed data

Peers sharing ideas and collaborative

interpretation of data/results

2100 2100 2100 2100

2100 2100 2100 2100

Remote Visualization

Data & Compute Service

Cyberinfrastructure

E-Scientist

Page 18: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

18

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 19: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

19

Grid Challenges

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Construction

Page 20: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

20

Some Grid Initiatives Worldwide

Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch

Brazil OurGrid, EasyGrid LNCC-Grid + many others

China ChinaGrid – Education CNGrid - application

Europe UK eScience EU Grids.. and many more...

India Garuda

Japan NAGERI

Korea...N*Grid

SingaporeNGP

USA Globus GridSec AccessGrid TeraGrid Cyberinfrasture and many more...

Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft - .NET Oracle 10g Infosys – Enterprise Grid Satyam – Business Grid StorageTek –Grid.. and many more

Public Forums Global Grid Forum Australian Grid Forum Conferences:

CCGrid Grid HPDC E-Science

http://www.gridcomputing.com

1.3 billion – 3 yrs

1 billion – 5 yrs

450million – 5 yrs

486million – 5 yrs

1.3 billion (Rs)

27 million

2? billion

120million – 5 yrs

Page 21: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

21

Open-Source Grid Middleware Projects

Page 22: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

22

The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand

WWG

Pushes Grid computing into mainstream

computing

Gridbus

Page 23: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

23

The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying e-Research Applications on Utility Grids

The Gridbus Project @ GRIDS Lab, The University of Melbourne: The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying eToolkit for Creating and Deploying e--Research Applications on Utility GridsResearch Applications on Utility Grids

Gridbus

Distributed Data

http://www.gridbus.org

• Gridbus is a “open source” Grid R&D project with focus on Grid Economy, Utility Grids and Service Oriented Computing.

• Gridbus Middleware components include:– Alchemi: .NET-based Enterprise Grid

– Grid Market Directory and Web Services

– Grid Bank: Accounting and Transaction Management

– Visual Tools for Creation of Distributed Applications

– Grid Service Broker and Scheduling

– Workflow Management Engine

– GridSim Toolkit

– Libra: SLA-based Resource Allocation

Page 24: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

24

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 25: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

25

What do Grid players want?

Grid Consumers Execute jobs for solving varying problem size and complexity Benefit by utilizing distributed resources wisely Tradeoff timeframe and cost

Strategy: minimise expenses

Grid Providers Contribute resources for executing consumer jobs Benefit by maximizing resource utilisation Tradeoff local requirements & market opportunity

Strategy: maximise return on investment

Page 26: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

26

What do Grid players require?

They need tools and technologies that help them in value expression, value translation, and value enforcement.

Grid Service Consumers (GSCs): How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? How do I manage Grid dynamics and get my work done? …

Grid Service Providers (GSPs) How do I decide service pricing models ? How do I specify them ? How do I translate them into resource allocations ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …

Page 27: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

27

Principle 1: Service Oriented Architecture (SOA)

A SOA is a contractual architecture for offering and consuming software as services.

There are four entities that make up an SOA service provider, service registry, and service consumer (also known as service requestor).

The functions or tasks that the service provider offers, along with other functional and technical information required for consumption, are defined in

the service definition or contract.

provider

registry

consumer

contract

Page 28: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

28

Principle 2: Market-Oriented (Grid) Computing- (a) Sustained Resourced Sharing and (b)

Effective Management of Shared Resources

Grid Economy

Page 29: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

29

Market-based Systems = Self-managed and Self-regulated systems.

Complexity present in Grid systems is similar to one present in human economies.

Page 30: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

30

Grid Node N

Service-Oriented Grid Architecture

Grid Service Consumer

Pro

gra

mm

ing

En

viro

nm

ents

Grid Resource Broker

Grid Service Providers

Grid Explorer

Schedule Advisor

Trade Manager

Job ControlAgent

Deployment Agent

Trade Server

Resource Allocation

ResourceReservation

R1

Misc. services

Information Service

R2 Rm…

Pricing Algorithms

Accounting

Grid Node1

Core Middleware Services

HealthMonitor

Grid Market Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign-on

Grid Bank

Ap

pli

cati

on

s

Data Catalogue

Page 31: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

31

Gridbus and Complementary Technologies – realizing Utility Grid

AIXSolarisWindows Linux

.NETGridFabricSoftware

GridApplications

Core GridMiddleware

User-LevelMiddleware

GridBank

Grid Exchange & Federation

JVM

Grid Brokers:

X-Parameter Sweep Lang.

Gridbus Data Broker

MPI

Condor SGE TomcatPBS

Alchemi

Workflow

IRIX OSF1 Mac

Libra

Globus Unicore ……Grid

MarketDirectory

PDB

CDB

Worldwide Grid

GridFabricHardware

……

PortalsScience Commerce Engineering ……Collaboratories

……

Workflow Engine

Grid Storage Economy

Gri

d E

con

om

y

NorduGrid XGrid

ExcellGrid

Nimrod-G

Gridscape

Page 32: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

32

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)

(Globus)

Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Resu

lts9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

Page 33: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

33

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)

(Globus)

Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Resu

lts9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

Page 34: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

34

Alchemi: .NET-based Enterprise Grid Platform & Web Services

InternetInternet

InternetInternet

Alchemi Worker Agents

Alchemi Manager

Alchemi Users

Web Services

Web Services

•SETI@Home like Model•General Purpose•Dedicated/Non-dedicate workers•Role-based Security•.NET and Web Services•C# Implementation•GridThread and Job Model Programming•Easy to setup and use• Widely in use!

Page 35: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

35

Some Users of Alchemi

Tier Technologies, USALarge scale document processing using Alchemi framework

CSIRO, AustraliaNatural Resource Modeling

The Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genes

Satyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi framework

The University of Sao Paulo, BrazilThe Alchemi Executor as a Windows Service

stochastix GmbH, GermanyServing clients in International Banking/Finance sector

Many users in Universities: See next for an example.

Page 36: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

36

                           

Students' project gives old computers new life  - 1/25/2005

Page 37: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

37

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)

(Globus)

Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Resu

lts9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

Page 38: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

38

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 39: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

39

A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.

It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation)

Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting

Grid Service Broker (GSB)

Page 40: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

40

Gridbus Broker Architecture

Grid Middleware

Gridbus Client Gridbus ClientGribus Client

Grid Info Server

Schedule Advisor

Trading Manager

Gridbus Farming Engine

RecordKeeper

Grid Explorer

GE GIS, NWSTM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

G

G

CU

Globus enabled node.A

L

Alchemi enabled node.

(Data Grid Scheduler)

DataCatalog

DataNode

Unicore enabled node.

$

$

$

App, T, $, Opt

(Bag of Tasks Applications)

Page 41: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

41

Gridbus Broker: Separating “applications” from “different” remote service access

enablers and schedulers

Alchemi

Gateway

UnicoreData Store

Access Technology

Grid FTPSRB

-PBS-Condor-SGE

Globus

Job manager

fork() batch()

Gridbusagent

Data Catalog

-PBS-Condor-SGE-XGrid

SSH

fork()

batch()

Gridbusagent

Single-sign on securityHome Node/Portal

GridbusBroker

fork()

batch() -PBS-Condor-SGE-Alchemi-XGrid

Application Development Interface

Sch

ed

ulin

gIn

terfa

ces

Alogorithm1

AlogorithmN

Plugin Actuators

Page 42: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

42

Gridbus Services for eScience applications

Application Development Environment: XML-based language for composition of task farming (legacy)

applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine.

Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes

that meet user QoS requirements. Hide Low-Level Grid Middleware interfaces

Globus (v2, v4), SRB, Alchemi, Unicore, and ssh-based access to local/remote resources managed by XGrid, Condor, SGE.

Page 43: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

43

Figure 3 : Logging into the portal.

Drug DesignMade Easy!

Click Here for Demo

Page 44: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

44

Excel Plugin to Access Gridbus Services

Excel

ExcelGrid Add-In

ExcelGrid Runner

ExcelGridJob

ExcelGrid Middleware

Gridbus Broker

Enterprise Grid

2100

2100

2100

2100

2100

2100

2100

2100

Page 45: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

45

Discover Discover ResourcesResources

Distribute JobsDistribute Jobs

Establish Establish RatesRates

Meet requirements ? Remaining Meet requirements ? Remaining Jobs, Deadline, & Budget ?Jobs, Deadline, & Budget ?

Evaluate & Evaluate & RescheduleReschedule

Discover Discover More More

ResourcesResources

Compose & Compose & ScheduleSchedule

Adaptive Scheduling Steps

Page 46: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

46

Deadline (D) and Budget (B) Constrained Scheduling Algorithms

Algorithm Execution Time (D)

Execution Cost (B)

Compute Grid

Data Grid

Cost Opt Limited by D Minimize Yes Yes

Cost-Time Opt

Minimize if possible

Minimize Yes

Time Opt Minimize Limited by B Yes Yes

Conservative-Time Opt

Minimize Limited by B, jobs have guaranteed minimum budget

Yes

Page 47: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

47

Gridbus Project: Some Applications and Users

Gridbus Project: Gridbus Project: Some Applications and UsersSome Applications and Users

http://www.gridbus.org

BioGrid: Molecular docking for Drug-discovery

BioGrid: Molecular docking for Drug-discovery

High Energy Physics: Particle Discovery

High Energy Physics: Particle Discovery

Melbourne University

NeuroScience: Brain Activity Analysis

NeuroScience: Brain Activity Analysis

Natural Resource ModelingNatural Resource Modeling

CSIRO Land and Water, Austraila.

Large Scale document processing

Large Scale document processing

Tier Technologies, USA.

Detection of patterns of transcription factors in mammalian genes

Detection of patterns of transcription factors in mammalian genes

Page 48: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

48

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 49: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

49

Case Study: High Energy Physics and Data Grid

The Belle Experiment KEK B-Factory, Japan Investigating fundamental violation

of symmetry in nature (Charge Parity) which may help explain “why do we have more antimatter in the universe OR imbalance of matter and antimatter in the universe?”.

Collaboration 1000 people, 50 institutes

100’s TB data currently

Page 50: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

50

Case Study: Event Simulation and Analysis

B0->D*+D*-Ks

• Simulation and Analysis Package - Belle Analysis Software Framework (BASF)• Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data

Analyzed 100 data files (30MB each) that were distributed among the five nodes within Australian Belle DataGrid platform.

Page 51: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

51

Australian Belle Data Grid Testbed

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results

CertificateAuthority

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

GRIDS Lab, University of Melbourne

Dept. of Physics,University of Sydney

ANU, Canberra

Dept. of Computer Science, University of Adelaide

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Intel Pentium 2.0 Ghz, 512 MB RAM

Dept. of Physics,University of Melbourne

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

VPACMelbourne

Page 52: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

52

Belle Data Grid (GSP CPU Service Price: G$/sec)

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results

CertificateAuthority

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

GRIDS Lab, University of Melbourne

Dept. of Physics,University of Sydney

ANU, Canberra

Dept. of Computer Science, University of Adelaide

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Intel Pentium 2.0 Ghz, 512 MB RAM

Dept. of Physics,University of Melbourne

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NA

G$4

G$4

Datanode

G$6VPAC

MelbourneG$2

Page 53: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

53

Belle Data Grid (Bandwidth Price: G$/MB)

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results

CertificateAuthority

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

GRIDS Lab, University of Melbourne

Dept. of Physics,University of Sydney

ANU, Canberra

Dept. of Computer Science, University of Adelaide

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Intel Pentium 2.0 Ghz, 512 MB RAM

Dept. of Physics,University of Melbourne

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NA

G$4

G$4

Datanode

G$6VPAC

MelbourneG$2

34

31

38

31

30

3336

32

Page 54: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

54

Deploying Application Scenario

A data grid scenario with 100 jobs and each accessing remote data of ~30MB

Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario:

Minimise Time Minimise Cost

Results:

SUMMARY OF EVALUATION RESULTS

Scheduling strategy Total Time Taken (mins.)

Compute Cost (G$)

Data Cost (G$)

Total Cost (G$)

Cost Minimization 71.07 26865 7560 34425 Time Minimization 48.5 50938 7452 58390

Page 55: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

55

SUMMARY OF EVALUATION RESULTS

Scheduling strategy Total Time Taken (mins.)

Compute Cost (G$)

Data Cost (G$)

Total Cost (G$)

Cost Minimization 71.07 26865 7560 34425 Time Minimization 48.5 50938 7452 58390

Observation

Organization 

Node details Cost (in G$/CPU-sec) Total Jobs Executed

Time Cost

CS,UniMelb belle.cs.mu.oz.au4 CPU, 2GB RAM, 40 GB HD, Linux

N.A. (Not used as a compute resource)

-- --

Physics, UniMelb fleagle.ph.unimelb.edu.au1 CPU, 512 MB RAM, 40 GB HD, Linux

2 3 94

CS, University of Adelaide

belle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux

N.A. (Not used as a compute resource)

-- --

ANU, Canberra belle.anu.edu.au4 CPU, 2GB RAM, 40 GB HD, Linux

4 2 2

Dept of Physics, USyd

belle.physics.usyd.edu.au4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux

4 72 2

VPAC, Melbourne brecca-2.vpac.org180 node cluster (only head node used), Linux

6 23 2

Page 56: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

56

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 57: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

Alessandro Volta in Paris in 1801 inside French National Institute shows the battery while in the presence of Napoleon I

Fresco by N. Cianfanelli (1841) (Zoological Section "La Specula" of National History Museum of Florence University)

Page 58: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

58

….and in the future, I imagine a WorldwidePower (Electrical) Grid …...

What ?!?!This is a mad man…

Oh, monDieu !

Page 59: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

59

2006 - 1801 = 205 Years(Recent R&D: Delivering Internet services via Electric cables).

Page 60: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

60

When are we going to get IT as the 5th utility?(water, electricity, gas, telephone, IT)

eScienceeBusiness

eGovernmenteHealth

MultilingualeEducation

Page 61: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

61

Summary and Conclusion

Grids exploit synergies that result from cooperation of autonomous entities:

Resource sharing, dynamic provisioning, and aggregation at global level Great Science and Great Business!

Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications.

SOA + Market-based Grid Management = Utility Grids

Grids allow users to dynamically lease Grid services at runtime based on their quality, cost, availability, and users QoS requirements.

Delivering ICT services as computing utilities. Grids offer enormous opportunities for realizing

e-Science and e-Business at global level.

Page 62: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

62

Thanks for your attention!

We Welcome Cooperation in Research and Development!http:/www.gridbus.org

Page 63: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

Gridbus Middleware:Building Autonomic and Market-Oriented Global Grids for

Delivering IT Services as the 5th Utility

Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

www.gridbus.org

Gridbus Sponsors

Page 64: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

Backup Slides

Page 65: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

65

Some Characteristics of Grids

Numerousresources

Different securityrequirements

& policies

Resources areheterogeneous

Geographicallydistributed

Different resourcemanagementpolicies

Connected byheterogeneous, multi-level networks

Owned by multiple organizations &

individuals

Unreliable resources and environments

Slide by Hiro

Page 66: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

66

What are Grid benefits?

Resource sharing across multiple administrative boundaries Effective utilisation of the (existing) resources Dynamic provisioning

Application Acceleration Scalability Reliability Virtualisation:

applications, services, resources,…

Page 67: Gridbus Middleware and Utility Grids: Building Autonomic and Market-Oriented Global Grids for Delivering IT Services as the 5 th Utility Dr. Rajkumar Buyya.

Utility Grids:Autonomic and Market-Oriented Global Grids for Powering e-Science and e-Business Applications

Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

www.gridbus.org

Gridbus Sponsors