SensorGrid: A new Cyberinfrastructure Integrating Sensor Network and Grid Computing for e-Science Applications Dr. Rajkumar Buyya Grid Computing and Distributed.

SensorGrid: A new Cyberinfrastructure Integrating Sensor Network and

Grid Computing for e-Science Applications

Dr. Rajkumar Buyya

Grid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

www.buyya.com

WW Grid

SensorGrid: A new Cyberinfrastructure for Linking

the Physical World with the Digital World

Dr. Rajkumar Buyya

Fellow of Grid Computing

Grid Computing and Distributed Systems (GRIDS) Lab. Dept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

gridbus.org/~raj/tut/gridbus.zip

WW Grid

3

GRIDS Lab @ Melbourne

The youngest and one of the largest research labs in the CSSE Dept:

2 PostDocs 2 Research Programmers 7 RHD (6 PhD) students ~5 honours/masters projects

Funding National and International organizations Australian Research Council Many industries (Sun, StorageTek,

Microsoft, IBM) University-wide collaboration:

Faculties of Science, Engineering, and Medicine

Many national and international collaborations.

Academics Industries

Software: Our Grid middleware technologies are

widely in academic and industrial users. Publication:

My research team produces 20% of our Dept’s research output.

EducationR & D

+ Community Services

4

Books at Glance: Co-authored/edited

5

Outline

Introduction Technology Trends in Sensors and Grids Sensors and Grids Integration Proposal

Grid Computing What is it?, architecture, and technologies

NOSA (NICTA Open SensorWeb Architecture)

Summary and Conclusion

6

7

8

Internet, Web, and Grid Effect

TCP/IPHTML

MosaicXML

PHASE 1. Packet Switching Networks 2. The Internet is Born 3. The World Wide Web 4. with XML 5. The Grid

1969: 4 US Universities linked to form ARPANET TCP/IP becomes core protocolHTML hypertext system created1972: First e-mail program created Domain Name System created

IETF created (1986)CERN launch World Wide Web

1976: Robert Metcalfe develops Ethernet NCSA launch Mosaic interface

0

20

40

60

80

100

120

140

1965 1970 1975 1980 1985 1990 1995 2000 2005 2010

The 'Network Effect’The 'Network Effect’ kicks in, and the web kicks in, and the web

goes critical' goes critical'

Num

ber

of

host

s(m

illio

ns)

Bus

ine

ss $

$$ ‘‘ Web

Web

Serv

ices

Serv

ices

and G

rid

and G

rid

Eff

ect

’Eff

ect

’

Grid Information Service

SensorWeb Service

Sensor Network

databaseR2

R3

RN

R1

R4

R5

R6

Grid Resource Broker

The customer can have access to the Grid Resource Broker, and then through Web Service, requests can be sent to obtain real sensor network data.

10

Traditional Devices, Sensors, and their Networks at Glance

Traditional Devices (Computers and High End Resources) are:

Powerful Connected to Power Grid – so we don’t worry too much about it power

consumption Large Storage Space Good for archival and large-scale analysis Connected by High Bandwidth/Speed Network

Sensors: Less powerful Scarcity of power (battery operated, or even self-power generated) Less Storage No good for archival Connected by Low Bandwidth/Speed Network But they can sense/smell a phenomena in the physical world.

11

It will be nice to marry them

Both of them benefit: Grids:

Get Eye to see the world (so that it can sense and assist the us)

Sensors: Off load their processing, storage, archival, analysis,

etc. requirements to the Grid.

Sensors Grids = SensorGrid

12

Outline





13

Grid (computing) Paradigm:Cyberinfrastructure for sharing resources

•Inspired by Power Grid!

•* A service-oriented/utility computing paradigm that enables seamless

sharing of geographically distributed, autonomous resources.

•* This was the original aim of building Internet although it ended up in

giving birth to email!

14

Grids have Emerged as Scalable Cyberinfrastructure for e-Science Applications


Resource Broker

Application



databaseR2R3

RN

R1

R4

R5

R6


15

Type of Services Modern Grids Offer

Computational Services – CPU cycles SETI@Home, NASA IPG, TeraGrid, I-Grid,…

Data Services Data replication, management, secure access--

LHC Grid/Napster Application Services

Access to remote software/libraries and license management—NetSolve

Information Services Extraction and presentation of data with meaning

Knowledge Services The way knowledge is acquired and managed—

data mining. Utility Computing Services

Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.

Computional Grid

Data Grid

ASP Grid

Information Grid

Knowledge Grid

Utility Grid

16

Grid Capabilities

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Construction

17

Some Grid Initiatives Worldwide

Australia Nimrod-G Gridbus DISCWorld GrangeNet. APACGrid ARC eResearch

Brazil OurGrid, EasyGrid LNCC-Grid + many others

China ChinaGrid – Education CNGrid - application

Europe UK eScience EU Grids.. and many more...

India I-Grid

Japan NAGERI

Korea...N*Grid

SingaporeNGP

USA Globus NASA IPG AccessGrid TeraGrid Cyberinfrasture and many more...

Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft - .NET Oracle 10g Infosys – Business Grid StorageTek –Grid.. and many more

Public Forums Global Grid Forum Australian Grid Forum Conferences:

CCGrid Grid HPDC E-Science

http://www.gridcomputing.com

1.3 billion – 3 yrs

1 billion – 5 yrs

450million – 5 yrs


1.3 billion (Rs)

27 million

2? billion


18

The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand

WWG

World Wide Grid!On Demand Utility

Computing

Gridbus

Distributed Data

www.gridbus.org

19

The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying e-Research Applications on Utility Grids

The Gridbus Project @ GRIDS Lab, The University of Melbourne: The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying eToolkit for Creating and Deploying e--Research Applications on Utility GridsResearch Applications on Utility Grids

Gridbus

Distributed Data

http://www.gridbus.org

• Gridbus is a “open source” Grid R&D project with focus on Grid Economy, Utility Grids and Service Oriented Computing.

• Gridbus Middleware components include:– Alchemi: .NET-based Enterprise Grid

– Grid Market Directory and Web Services

– Grid Bank: Accounting and Transaction Management

– Visual Tools for Creation of Distributed Applications

– Grid Service Broker and Scheduling

– Workflow Management Engine

– GridSim Toolkit

– Libra: SLA-based Resource Allocation

20

Gridbus Architecture Layer

Grid Resources and Local Services

Alchemi

WorkFlow and Application Programming Interface

Globus Unicore

Applications

Gridbus Grid Service Broker

Adapter Layer

Alchemi Actuator

GlobusActuator

UnicoreActuator

Grid Trading and Banking Services

Grid Economy and Allocation…

…

Grid Resources and Local Services

Alchemi

WorkFlow and Application Programming Interface

Globus Unicore

Applications

Gridbus Grid Service Broker

Adapter Layer

Alchemi Actuator

GlobusActuator

UnicoreActuator

Grid Trading and Banking Services

Grid Economy and Allocation…

…

21

On Demand Assembly of Services: Putting Them All Together

Data Source

(Instruments/distributed sources)

Data Replicator(GDMP) ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

Data

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)

(Globus)

Alchemi

GS

GTS

Cluster Scheduler

Grid Service Provider (GSP)

(e.g., CERN)

PECluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Resu

lts9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

22

Alchemi: .NET-based Enterprise Grid Platform & Web Services

InternetInternet

InternetInternet

Alchemi Worker Agents

Alchemi Manager

Alchemi Users

Web Services

Web Services

•SETI@Home like Model•General Purpose•Dedicated/Non-dedicate workers•Role-based Security•.NET and Web Services•C# Implementation•GridThread and Job Model Programming•Easy to setup and use• Widely in use!

23

Some Users of Alchemi

Tier Technologies, USALarge scale document processing using Alchemi framework

CSIRO, AustraliaNatural Resource Modeling

The Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genes

Satyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi framework

The University of Sao Paulo, BrazilThe Alchemi Executor as a Windows Service

stochastix GmbH, GermanyAsynchronous Excel Tasks using ManagedXLL and Alchemi .Net Grid Computing framework.

Many users in Universities: See next for an example.

24

Students' project gives old computers new life - 1/25/2005

The Gridbus Grid Service Broker for Data Grid Applications

Builds on the Nimrod-G Computational Grid Broker and Computational Economy [Buyya, Abramson, Giddy, Monash University, 1999-

2001]And

Extends its notion for Data and Service Grids

26

Gridbus Broker Architecture

Grid Middleware

Gridbus Client Gridbus ClientGribus Client

Grid Info Server

Schedule Advisor

Trading Manager

Gridbus Farming Engine

RecordKeeper

Grid Explorer

GE GIS, NWSTM TS

RM & TS

Grid Dispatcher

RM: Local Resource Manager, TS: Trade Server

G

G

CU

Globus enabled node.A

L

Alchemi enabled node.

(Data Grid Scheduler)

DataCatalog

DataNode

Unicore enabled node.

$

$

$

App, T, $, Opt

(Bag of Tasks Applications)

27

Gridbus Services for eScience applications

Application Development Environment: XML-based language for composition of task farming (legacy)

applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine.

Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes

that meet user QoS requirements. Hide Low-Level Grid Middleware interfaces

Globus, Alchemi, Unicore, NorduGrid, XGrid, etc.

28

Figure 3 : Logging into the portal.

Drug DesignMade Easy!

Click Here for Demo

29

Sample Applications of Gridbus Broker

Molecular Docking - WEHI Drug Discovery

Brain Activity Analysis – Osaka University Neuroscience studies

Natural Language Engineering – Melbourne NLP Indexing of newswire data

High Energy Physics – School of Physics/Melbourne Belle experiment data analysis

Finance - Portfolio Analysis – U. Comp. Madrid/Spain Investment risk analysis

Astronomy – School of Physics@UoM and QUT (Queensland University of Technology)

Australian Virtual Observatory Spreadsheet Processing

Microsoft Excel

30

Outline





31

Sun Proposal: For Integration of Sensors and High End Computers using Network

Source: Anil Velluri, Sun

32

Our Proposal: SensorGrid - Integrated sensor network and

grid architecture

sensor and actuator networks

query

response

command

actuate

grid

actionssense

33

Sensor Web and NOSA

Goal: To create Web-based sensor networks by exploiting web-connected sensors (flood gauges, air pollution monitors, mobile heart monitors, satellite-borne earth imaging monitors).

This would make all sensors and repositories of sensor data discoverable, accessible, process-able and where applicable controllable via the WWW.

OGC (Open Geospatial Consortium) describes five important encoding and service standards for a Sensor Web Implementation (see next slide).

34

What is NOSA?

The NICTA Open SensorWeb Architecture (NOSA) project is developing a complete standards compliant platform and middleware for integration of sensor networks with emerging distributed computing platforms such as Grids.

It confirms to Web Services standard defined by W3C (World-Wide Web) and SensorML (Sensor Model Language) standard defined by OpenGeospatial Consortium.

35

What are NOSA Benefits?

This integration of sensor networks with Grid computing brings out dual benefits: (i) sensor networks can off-load heavy processing activities to the Grid and (ii) Grid-based sensor applications can provide advance services for smart-sensing by deploying scenario-specific operators at runtime.

Fundamental services are provided by lower-level layer components whereas components at the higher-level layer provide tools for creation of applications and management of life-cycle of data captured through sensor networks.

36

NICTA Open SensorWeb Architecture (NOSA)

SensorWeb Fabric or Emulation

SensorWeb Core Middleware

SensorWeb User-Level Middleware

SensorWeb Applications

Sensor1 Sensor2 SensorN….

SensorNet Field and Cluster1

iModel+Encoding:1. SensorML

2. Observation & Measurements

SensorDirectory Services

SensorData Grid Services

SensorGridProcessing Services

Sensor PlanningServices

Sensor Notification

Services

SensorCollection/

ObservationServices

Sensor Coordination

Services

Sensor Programming Framework (APIs, Visual Tools)

Water Information

Network

Barrier Reef Observation

Network

SecureAustralia Network ….

ZigBee/IEEE 802.15.4 protocols

SensorWebSimulation

or Emulation

Safe Transportation

andRoads

Tsunami Detection Network

Actuator1 Actuator2 ActuatorM….

SensorConfiguration

Services

Faulty Sensor Data Correction &

Management Services

Third Party Tools

….

Pollution MonitoringNetwork

Sensor1 SensorN….

SensorNet Field /ClusterN


Actuator1 ActuatorM….

….

Sensor1 Sensor2 SensorN….

SensorNet Field and Cluster1

iModel+Encoding:1. SensorML

2. Observation & Measurements

SensorDirectory Services

SensorData Grid Services

SensorGridProcessing Services

Sensor PlanningServices

Sensor Notification

Services

SensorCollection/

ObservationServices

Sensor Coordination

Services

Sensor Programming Framework (APIs, Visual Tools)

Water Information

Network

Barrier Reef Observation

Network

SecureAustralia Network ….


SensorWebSimulation

or Emulation

Safe Transportation

andRoads

Tsunami Detection Network

Actuator1 Actuator2 ActuatorM….

SensorConfiguration

Services

Faulty Sensor Data Correction &

Management Services

Third Party Tools

….

Pollution MonitoringNetwork

Sensor1 SensorN….

SensorNet Field /ClusterN


Actuator1 ActuatorM….

….

37

OGC-Sensor Web Enablement Standard Specifications

SensorML: XML encoding language for sensors. Used to discover, query and control Web-resident sensors.

Observations & Measurements: The general models and an XML encoding for what a sensor observes or measures (The value returned by or derived from a sensor observation -e.g. quantity, count, boolean, category, ordered category, position-).

Sensor Collection Service: A service to fetch observations from a sensor or constellation of sensors. Provides real time or archived observed values. Clients can also obtain information that describes the associated sensors and platforms.

Sensor Planning Service: A service by which a client can determine collection feasibility for a desired set of collection requests for one or more mobile sensors/platforms, or the client may submit collection requests directly to these sensors/platforms. SPS enables sensor tasking, acquisition requests, processing and simulation requests, and registration for alert notification.

Web Notification Service: A service by which a client may conduct a dialog with one or more other services. Provides a means for Sensor Planning Services to alert people, software, or other sensor systems of SPS results or alerts regarding phenomena of interest.

38

A sensor node includes a sensor and radio board.

Then the sensor node could send data message through radio to Station which is the gateway connected with the PC.

Here is the architecture of the interface connecting sensor network and the real PC.

39

Crossbow Wireless Sensor Network Kit and Testbed Setup in GRIDS Lab

Base station MIB510CA

Sensor MTS 300

40

SensorWeb Implementation

41

A SensorWeb Collection Client

42

Performance of collecting auto-sending and query data

Response Ti me for col l ecti ng Ti nyDB query data

0

200

400

600

Number of Cl i ents

Seco

nd

1 record10 records20 records

1 record 34. 2635 164. 3177510 records 76. 5 395. 27252520 records 96. 05 492. 21

1 5

43

Performance of collecting auto-sending and query data

Response Ti me for col l ecti ng real - t i me data

0

100

200

300

400

Number of Cl i ents

Seco

nd

1 record10 records20 records

1 record 2. 047 2. 097 3. 569 22. 90410 records 12. 14 16. 29 27. 032 173. 4820 records 24. 52 32. 904 54. 602 354. 86

1 5 50 500

44

Outline





45


Sensors and Grids two major elements of emerging Cyberinfrastructure that support e*Applications (e-Science, e-Business, e-Health, e-Life).

They support creation of smart office, house, and business environments.

Current developments in Sensors and Grids is heavily driven by applications and both compliment and need each other.

SensorGrid is just emerging and there are many opportunities available for creating many interesting applications in various domains.

46

References

Rajkumar Buyya and Srikumar Venugopal, A Gentle Introduction to Grid Computing and Technologies, CSI Communications, pages 9-19, Computer Society of India, Vol.29, No.1, July 2005.

Chen-Khong Tham and Rajkumar Buyya, SensorGrid: Integrating Sensor Networks and Grid Computing, CSI Communications, pages 24-29, Computer Society of India, Vol.29, No.1, July 2005.

47

Any Questions ?

Web - http://www.gridbus.org

http://www.buyya.com

SensorGrid: A new Cyberinfrastructure Integrating Sensor Network and Grid Computing for e-Science Applications Dr. Rajkumar Buyya Grid Computing and Distributed.

Documents

grid effect slide

power grid

ww grid slide

rajkumar buyya grid

grid middleware technologies

conclusion slide

grids sensors

coauthorededited slide