Top Banner
Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.gridbus.org www.buyya.com www.manjrasoft.com Gridbus Sponsors M anjra soft D rR ajkum arB uyya C hief E xecutive Officer M anjrasoftP ty Ltd R oom 5.31,IC T B uilding,111,Barry Street,C arlton, M elbourne,VIC 3053,A ustralia P:+61-3-8344 1344 | F :+61-3-9348 1184 E:[email protected] http://www.manjrasoft.com M an jra soft
56

Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

Mar 26, 2015

Download

Documents

Sophia Malloy
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality

Dr. Rajkumar BuyyaGrid Computing and Distributed Systems (GRIDS) LabDept. of Computer Science and Software EngineeringThe University of Melbourne, Australiawww.gridbus.orgwww.buyya.comwww.manjrasoft.com

Gridbus Sponsors

ManjrasoftDr R ajkumar B uyya

C hief E xecutive O fficer

Manjrasoft P ty L tdR oom 5.31, IC T B uilding, 111, B arry S treet, C arlton,

Melbourne, VIC 3053, AustraliaP : +61-3-8344 1344 | F : +61-3-9348 1184

E : raj@ manjrasoft.comhttp://www.manjrasoft.com

Manjrasoft

Page 2: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

2

The GRIDS Lab @ Melbourne

Youngest and one of the rapidly growing research labs in our School/University:

Founded in 2002 Houses 20+ researchers consisting of:

Research Fellows/PostDocs Software Engineers PhD candidates Honours/Masters students

Funding National and International organizations Australian Research Council & DEST Many industries (Sun, StorageTek, Microsoft,

IBM, Microsoft) University-wide collaboration:

Faculties of Science, Engineering, and Medicine

Many national and international collaborations.

Academics Industries

Software: Widely in academic and industrial users.

Publication: My research team produces over 20% of our

Dept’s research output.

EducationR & D

+ Community Services: e.g., IEEE TC for Scalable Computing

Manjrasoft

Page 3: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

3

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 4: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

4

“Computer Utilities” Vision: Implications of the Internet

1969 – Leonard Kleinrock, ARPANET project “As of now, computer networks are still in their infancy,

but as they grow up and become sophisticated, we will probably see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country”

Computers Redefined 1984 – John Gage, Sun Microsystems

“The network is the computer” 2008 – David Patterson, U. C. Berkeley

“The data center is the computer. There are dramatic differences between of developing software for millions to use as a service versus distributing software for millions to run their PCs”

2008 – “Cloud is the computer” – Buyya!

Page 5: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

5

Computing Paradigms and Attributes: Realizing the ‘Computer Utilities’

Vision Web Data Centres Utility Computing Service Computing Grid Computing P2P Computing Market-Oriented

Computing Cloud Computing …

-Ubiquitous access

-Reliability-Scalability-Autonomic-Dynamic discovery

- Composability-QoS-SLA- …

} +

Paradigms

Attributes/Capabilities

?-Trillion $ business- Who will own it?

Page 6: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

* Since Grids have been around for sometime (early 2000), do we have a unified vision of what Grids can do?

* And did we make sufficient advances to turn vision of “computer utilities” into a

reality?

-- Let us take a look at views of -“industrial” practitioners & “academics”

Page 7: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

7

“Industrial” vision of Grid computing

IBM On Demand Computing

Microsoft .NET

Oracle 10g

Sun N1 – Sun Grid Engine

HP Adaptive Enterprise

Amazon Elastic Compute Cloud Services

Manjrasoft Aneka for building enterprise Grids and Clouds.

Page 8: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

8

Most academics view: Cyberinfrastructure for conducting

collaborative (e-)Science

Distributed instruments

Distributed computation

Distributed data

Peers sharing ideas and collaborative interpretation of data/results

2100210021002100

2100210021002100

Remote Visualization

Data & Compute Service

Cyberinfrastructure

E-Scientist

Page 9: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

9

How do Grids look like?A Bird Eye View of a Global Grid

Grid Resource Broker

Resource Broker

Application

Grid Information Service

Grid Resource Broker

databaseR2R3

RN

R1

R4

R5

R6

Grid Information Service

Page 10: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

10

How do Grids look like?A Bird Eye View of a Global Grid

Grid Resource Broker

Resource Broker

Application

Grid Information Service

Grid Resource Broker

databaseR2R3

RN

R1

R4

R5

R6

Grid Information Service

Page 11: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

11

How Are Grids Used?

High-performance computing

Collaborative data-sharingCollaborative design

Drug discovery

Financial modeling

Data center automation

High-energy physics

Life sciences

E-Business

E-ScienceNatural language processing

Utility computing

Business Intelligence(Data Mining)

Page 12: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

12

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 13: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

13

Grid Challenges

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Construction

Page 14: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

14

Some Grid Initiatives Worldwide

Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch

Brazil OurGrid, EasyGrid LNCC-Grid + many others

China ChinaGrid – Education CNGrid - application

Europe UK eScience EU Grids.. and many more...

India Garuda

Japan NAREGI

Korea...N*Grid

SingaporeNGP

USA Globus TeraGrid Cyberinfrasture AutoMate and many more...

Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft - .NET Oracle 10g Amzon – Elastic Compute Cloud Infosys – Enterprise Grid Satyam – Business Grid Manjrasoft – enterprise Clouds

and Grids and many more

Public Forums Open Grid Forum Conferences:

CCGrid Grid HPDC E-Science

http://www.gridcomputing.com

1.3 billion – 3 yrs

1 billion – 5 yrs

450million – 5 yrs

486million – 5 yrs

1.3 billion (Rs)

27 million

2? billion

120million – 5 yrs

Page 15: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

15

Open-Source Grid Middleware Projects

Slide by Hiro

Page 16: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

16

Driving Theme:Community vs. Utility Grids

Type

Feature

Community Grids Utility Grids (Now Clouds)

User QoS Best effort Contract/SLA

Service Pricing

Not considered /

free access

Usage, QoS level, Market supply and demand

Example Middleware

Globus, Condor, OMII, Unicore

Nimrod-G, Gridbus, & many inspired efforts (IBM Business Grid, Sun Grid Market)

.. Amazon EC2..

Page 17: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

17

The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand

WWG

Pushes Grid computing into mainstream

computing

Gridbus

Page 18: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

18

The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying e-* Applications on Utility Grids

The Gridbus Project @ GRIDS Lab, The University of Melbourne: The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying Toolkit for Creating and Deploying ee--** Applications on Utility GridsApplications on Utility Grids

Gridbus

Distributed Data

http://www.gridbus.org

• Gridbus is a “open source” Grid R&D project with focus on Grid Economy, Utility Grids and Service Oriented Computing.

• Gridbus Middleware components include:– Aneka: .NET-based Enterprise Grid

– Tools for Rapid Application Development

– Grid Service Broker and Scheduling

– Workflow Management Engine

– Grid Market Directory & Grid Bank:

– Libra: SLA-based Resource Allocation

– GridSim Toolkit + CloudSim

– Sensor Web

– Cloud-Oriented Technologies

Manjrasoft

Page 19: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

19

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 20: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

20

What do Grid players want & require?

Grid Service Consumers (GSCs): - minimize expenses, meet QoS How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I discover services and map jobs to meet my QoS needs? How do I manage Grid dynamics and get my work done? …

Grid Service Providers (GSPs):– maximise ROI How do I decide service pricing models ? How do I specify them ? How do I translate them into resource allocations ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …

They need mechanisms, tools and technologies that help them in value expression, value translation, and value enforcement.

Page 21: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

21

Grid Node N

Service-Oriented Grid Architecture

Grid Service Consumer

Pro

gra

mm

ing

En

viro

nm

ents

Grid Resource Broker

Grid Explorer

Schedule Advisor

Trade Manager

Job ControlAgent

Deployment Agent

Information Service

Pricing Algorithms

Grid Node1

Core Middleware Services

HealthMonitor

Grid Market Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign-on

Grid Bank

Ap

pli

cati

on

s

Data Catalogue

Grid Service Providers

Trade Server

Resource Allocation

ResourceReservation

R1

Misc. services

R2 Rm…

Accounting

Page 22: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

22

Market-Oriented Grid Software: A union of Gridbus and other

technologies

AIXSolarisWindows Linux

.NETGridFabricSoftware

GridApplications

Core GridMiddleware

User-LevelMiddleware

GridBank

Grid Exchange & Federation

JVM

Grid Scheduling:

Task, Parametric, and Components Programming

Gridbus Resource Broker

MPI

Condor SGE TomcatPBS

Aneka Cloud (WS-based access + SLA

Workflow APIs

IRIX OSF1 Mac

Libra

Globus Unicore ……Grid

MarketDirectory

PDB

Worldwide Grid

GridFabricHardware

Grid PortalsScience Commerce Engineering ……Collaboratories

……

Grid Storage Economy

Gri

d E

con

om

y

NorduGrid XGrid

ExcellGrid

Grid Workflow Engine

APIs/Tools:

CDB

Page 23: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

23

On Demand Assembly of Services in Market-Oriented Grid Environments

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., Microsoft)

PEGSP

(e.g., Amazon)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)

(Globus)

Aneka

EC2

GTS

Resource Allocation

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Resu

lts9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

Page 24: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

24

ASP Catalogue

Grid Info Service

Grid Market DirectoryASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., Microsoft)

PEGSP

(e.g., Amazon)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Aneka

EC2

GTS

Resource Allocation

GSP(e.g., Microsoft)

PEGSP

(e.g., Amazon)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Aneka

EC2

GTS

Resource Allocation

Job

8

Job

Job

Job

8

GridResource Broker

2

GridResource Broker

22

Visual Application Composer

Application CodeExplore

data1

Visual Application Composer

Application CodeExplore

data1

3366

4455

Res

ults

9

Res

ults

Res

ults

9 77

Results+

Cost Info

10

Results+

Cost Info

10

1111

Bill

12

BillBill

12Data CatalogueData Catalogue

On Demand Assembly of Services in Market-Oriented Grid Environments

Page 25: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

25

Cloud Services

Infrastructure as a Services (IaaS) CPU, Storage: Amazon.com et. al

Platform as a Services (PaaS) Google App Engine, Microsoft Azure, and

Manjrasoft Aneka Software as a Service (SaaS)

SalesForce.Com

Clouds

Enterprise/Private Clouds

Public/Internet Clouds

Page 26: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

26

Layered view of services within a Cloud stack

Infrastructure as a Service (IaaS) e.g., Amazon, Nirvanix

Software as a Service (SaaS) e.g., ..SalesForce.com

Platform as a Service (PaaS) e.g., ..Aneka

Page 27: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

Aneka

A Software Platform for Building and Managing “Enterprise” Grids and

Clouds

Manjrasoft

Page 28: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

28

Aneka: A 3rd Gen enterprise Grid Technology Cloud model

Generation Properties Technologies

First(-2001)

Application-specific platforms

distributed.net,SETI@Home

Second(2002-2006)

Single programming model, rigid architecture, no QoS

UD, XtremeWeb, Alchemi, Digipede, DataSynapse, BOINC

Third(2007-2012?)

SOA, extensible architecture, multiple programming models, multi-tenancy, enterprise QoS, SLAs, market-based resource allocation

Aneka

Page 29: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

29

ANEKA – Product Overview (Alpha)

.NET based service-oriented platform for grid / cloud computing

Development and Run Time Environment

Includes Development and Management Tools

Suitable for Development of Enterprise

Grid / Cloud Applications Grid / Cloud enabling legacy

applications Ideal for Corporate

Developers, Software, SaaS, Hosting Vendors and Application / System Integrators

ANEKA Product Architecture

Message Handler / Dispatcher

Communication Layer

Container

Security

Persistence

Allocation Manager

ThreadModel

Task Model

Dataflow Model

MPIModel

Map Reduce

OtherModels

Applications

SLANegotiation

Page 30: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

30

Aneka Deployment Models

Enterprise/Private Harness LAN connected

resources Application Development,

Testing, Execution Teaching and Learning Sensitive applications

Public Hosted by a 3rd party service

provider owning a large Data Center (1000s of servers)

Offers subscription-based services to their shared infrastructure on “pay-as go” model.to many users from different organisations.

Amazon.com, Microsoft Azure Aneka SDK + Execution Manger

Aneka

Enterprise/Private Clouds

Public Clouds

Page 31: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

31

Aneka: components

public DumbTask: ITask { … public void Execute() { …… }}

for(int i=0; i<n; i++){ … DumbTask task = new DumbTask(); app.SubmitExecution(task);}

Executor

Scheduler

Executor

Executor Executor

ClientAgent

work units

internet

internet

Aneka enterprise Cloud

ClientAgent

work units

Aneka Users

Aneka Worker ServiceAneka Manager

Programming / Deployment Model

Page 32: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

32

How does it solve the problem?

An Illustratioin

Application

Manager

Executor

Manager / Executor

GThreads/Tasks

Divide the problem in to multiple small tasks and distribute them run in parallel on multiple computers within a Cloud.

Page 33: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

33

User scenario: GoFront(unit of China Southern Railway Group)

Aneka utilizes idle desktops (30) to decrease task time

from days to hours

Time (in hrs)

Single Server

Aneka Cloud

Raw Locomotive Design Files(Using AutoDesk Maya) Using Maya

Graphical Mode Directly

Case 1: Single Server

4 cores server

Aneka Maya Renderer

Use private Aneka Cloud

GoFront Private Aneka Cloud

LAN network (Running Maya Batch Mode on

demand)

Case 2: Aneka Enterprise Cloud Manjrasoft

Application: Locomotive design CAD rendering

Page 34: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

34

Aneka: How can get it?

Available to Download: Software: www.manjrasoft.com Manual: Setting up Cloud using your LAN-network computers

Teaching material parallel and distributed computing and programming, List of possible assignments for students Possible Projects for Final year students..

Price – highly affordable = Fee you charge to 1 student (each year) and all

students/teachers in entire college/university can use it! Applications

Other Departments (Physics, Chemistry, Biology, Finance, Engineering) can use it for their applications.

Page 35: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

35

ASP Catalogue

Grid Info Service

Grid Market DirectoryASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., Microsoft)

PEGSP

(e.g., Amazon)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Aneka

EC2

GTS

Resource Allocation

GSP(e.g., Microsoft)

PEGSP

(e.g., Amazon)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Aneka

EC2

GTS

Resource Allocation

Job

8

Job

Job

Job

8

GridResource Broker

2

GridResource Broker

22

Visual Application Composer

Application CodeExplore

data1

Visual Application Composer

Application CodeExplore

data1

3366

4455

Res

ults

9

Res

ults

Res

ults

9 77

Results+

Cost Info

10

Results+

Cost Info

10

1111

Bill

12

BillBill

12Data CatalogueData Catalogue

On Demand Assembly of Services in Market-Oriented Grid Environments

Page 36: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

36

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 37: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

37

A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.

It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation)

Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting

Grid Service Broker (GSB)

Page 38: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

38

Core Middleware

Gridbus User Console/Portal/Application Interface

Grid Info Server

Schedule Advisor

Trading Manager

Gridbus Farming Engine

RecordKeeper

Grid Explorer

GE GIS, NWSTM TS

RM & TS

Grid Dispatcher

G

G

CU

Globus enabled node.

AL

DataCatalog

DataNode

Amazon EC2/S3 Cloud.

$

$

$

App, T, $, Optimization Preference

workload

Gridbus Broker

Page 39: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

39

Gridbus Broker: Separating “applications” from “different” remote service access

enablers and schedulers

Aneka

AMI

Amazon EC2Data Store

Access Technology

Grid FTPSRB

-PBS-Condor-SGE

Globus

Job manager

fork() batch()

Gridbusagent

Data Catalog

-PBS-Condor-SGE-XGrid

SSH

fork()

batch()

Gridbusagent

Single-sign on securityHome Node/Portal

GridbusBroker

fork()

batch() -PBS-Condor-SGE-Aneka-XGrid

Application Development Interface

Sch

ed

ulin

gIn

terfa

ces

Alogorithm1

AlogorithmN

Plugin Actuators

Page 40: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

40

Gridbus Services for eScience applications

Application Development Environment: XML-based language for composition of task farming (legacy)

applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine. … Grid Superscalar – in cooperation with BSC/UPC

Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes that

meet user QoS requirements. Hide Low-Level Grid Middleware interfaces

Globus (v2, v4), SRB, Aneka, Unicore, and ssh-based access to local/remote resources managed by XGrid, PBS, Condor, SGE.

Page 41: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

41

Drug DesignMade Easy!

Click Here for Demo

Page 42: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

42

s

A Sample List of Gridbus Broker UsersA Sample List of Gridbus Broker UsersA Sample List of Gridbus Broker Users

http://www.gridbus.org

Molecular docking for drug design on Australian National Grid

Molecular docking for drug design on Australian National Grid

High Energy Physics: Particle Discovery

High Energy Physics: Particle Discovery

Melbourne University

NeuroScience: Brain Activity Analysis

NeuroScience: Brain Activity Analysis

EU Data Mining GridEU Data Mining Grid

DaimlerChrysler, Technion, U. Ljubljana, U. Ulster

Kidney/Human Physiome Modelling

Kidney/Human Physiome Modelling

Melbourne Medical Faculty, Université d'Evry, France

Finance /Investment Risk Studies: Spanish Stock Market

Finance /Investment Risk Studies: Spanish Stock Market

Universidad Complutense de Madrid, Spain

Page 43: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

43

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 44: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

44

Case Study: High Energy Physics and Data Grid

The Belle Experiment KEK B-Factory, Japan Investigating fundamental violation

of symmetry in nature (Charge Parity) which may help explain “why do we have more antimatter in the universe OR imbalance of matter and antimatter in the universe?”.

Collaboration 1000 people, 50 institutes

100’s TB data currently

Page 45: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

45

Case Study: Event Simulation and Analysis

B0->D*+D*-Ks

• Simulation and Analysis Package - Belle Analysis Software Framework (BASF)• Experiment in 2 parts – Generation of Simulated Data and Analysis of the distributed data

Analyzed 100 data files (30MB each) that were distributed among the five nodes within Australian Belle DataGrid platform.

Page 46: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

46

Australian Belle Data Grid Testbed

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results

CertificateAuthority

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

GRIDS Lab, University of Melbourne

Dept. of Physics,University of Sydney

ANU, Canberra

Dept. of Computer Science, University of Adelaide

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Intel Pentium 2.0 Ghz, 512 MB RAM

Dept. of Physics,University of Melbourne

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

VPACMelbourne

Page 47: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

47

Belle Data Grid (GSP CPU Service Price: G$/sec)

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results

CertificateAuthority

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

GRIDS Lab, University of Melbourne

Dept. of Physics,University of Sydney

ANU, Canberra

Dept. of Computer Science, University of Adelaide

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Intel Pentium 2.0 Ghz, 512 MB RAM

Dept. of Physics,University of Melbourne

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NA

G$4

G$4

Datanode

G$6VPAC

MelbourneG$2

Page 48: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

48

Belle Data Grid (Bandwidth Price: G$/MB)

Grid Service Broker

Replica Catalog

AARNET

NWS NameServer

VirtualOrganization

Analysis Request

Analysis Results

CertificateAuthority

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

GRIDS Lab, University of Melbourne

Dept. of Physics,University of Sydney

ANU, Canberra

Dept. of Computer Science, University of Adelaide

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Intel Pentium 2.0 Ghz, 512 MB RAM

Dept. of Physics,University of Melbourne

NWSSensor

GridFTPGRIS

GlobusGatekeeper

Dual Intel Xeon 2.8 Ghz, 2 GB RAM

NA

G$4

G$4

Datanode

G$6VPAC

MelbourneG$2

34

31

38

31

30

3336

32

Page 49: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

49

Deploying Application Scenario

A data grid scenario with 100 jobs and each accessing remote data of ~30MB

Deadline: 3hrs. Budget: G$ 60K Scheduling Optimisation Scenario:

Minimise Time Minimise Cost

Results:

SUMMARY OF EVALUATION RESULTS

Scheduling strategy Total Time Taken (mins.)

Compute Cost (G$)

Data Cost (G$)

Total Cost (G$)

Cost Minimization 71.07 26865 7560 34425 Time Minimization 48.5 50938 7452 58390

Page 50: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

50

Time Minimization in Data Grids

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

Time (in mins.)

Nu

mb

er

of

job

s c

om

ple

ted

fleagle.ph.unimelb.edu.au belle.anu.edu.au belle.physics.usyd.edu.au brecca-2.vpac.org

Page 51: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

51

Results : Cost Minimization in Data Grids

0

10

20

30

40

50

60

70

80

90

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63

Time(in mins.)

Nu

mb

er o

f jo

bs

com

ple

ted

fleagle.ph.unimelb.edu.au belle.anu.edu.au belle.physics.usyd.edu.au brecca-2.vpac.org

Page 52: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

52

SUMMARY OF EVALUATION RESULTS

Scheduling strategy Total Time Taken (mins.)

Compute Cost (G$)

Data Cost (G$)

Total Cost (G$)

Cost Minimization 71.07 26865 7560 34425 Time Minimization 48.5 50938 7452 58390

Observation

Organization 

Node details Cost (in G$/CPU-sec) Total Jobs Executed

Time Cost

CS,UniMelb belle.cs.mu.oz.au4 CPU, 2GB RAM, 40 GB HD, Linux

N.A. (Not used as a compute resource)

-- --

Physics, UniMelb fleagle.ph.unimelb.edu.au1 CPU, 512 MB RAM, 40 GB HD, Linux

2 3 94

CS, University of Adelaide

belle.cs.adelaide.edu.au4 CPU (only 1 available) , 2GB RAM, 40 GB HD, Linux

N.A. (Not used as a compute resource)

-- --

ANU, Canberra belle.anu.edu.au4 CPU, 2GB RAM, 40 GB HD, Linux

4 2 2

Dept of Physics, USyd

belle.physics.usyd.edu.au4 CPU (only 1 available), 2GB RAM, 40 GB HD, Linux

4 72 2

VPAC, Melbourne brecca-2.vpac.org180 node cluster (only head node used), Linux

6 23 2

Page 53: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

53

Agenda

Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services

Global Grids and Challenges Security, resource management, pricing models, …

Service-Oriented Grid Architecture and Gridbus Solutions

Market-based Management, GMD, Grid Bank, Aneka Grid Service Broker

Architecture, Design and Implementation Performance Evaluation: Experiments in Creation

and Deployment of Applications on Global Grids A Case Study in High Energy Physics

Summary and Conclusion

Page 54: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

54

Summary and Conclusion

Grids exploit synergies that result from cooperation of autonomous entities:

Resource sharing, dynamic provisioning, and aggregation at global level Great Science and Great Business!

Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications.

SOA + Market-based Grid Management = Utility Grids Grids allow users to dynamically lease Grid services

at runtime based on their quality, cost, availability, and users QoS requirements.

Delivering ICT services as computing utilities. Clouds are rapidly emerging, but more work is

required Federation of Clouds, Cloud Exchange, and Application Scaling

Page 55: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

55

Convergence of Competing Paradigms/Communities Needed

Web Data Centres Utility Computing Service Computing Grid Computing P2P Computing Cloud Computing Market-Oriented

Computing …

•Ubiquitous access•Reliability•Scalability•Autonomic•Dynamic discovery•Composability•QoS•SLA•…

} +

Paradigms

Attributes/Capabilities

?-Trillion $ business- Who will own it?

Manjrasoft

Page 56: Utility-Oriented Cloud & Grid Computing: A Vision, Hype, and Reality Dr. Rajkumar Buyya Grid Computing and Distributed Systems (GRIDS) Lab Dept. of Computer.

56

Thanks for your attention!

Are there any Questions? Comments/ Suggestions

We Welcome Cooperation in R&D and Business! http:/www.gridbus.org | www.Manjrasoft.com

[email protected] | [email protected]

Manjrasoft