CrossGrid WP3 Task 3.3 Grid Monitoring Trinity College Dublin (TCD, AC14 - CR11) Brian Coghlan, Stuart Kenny CYFRONET Academic Computer Centre, Krakow.

CrossGrid CrossGrid WP3WP3

Task 3.3Task 3.3Grid Grid MonitoringMonitoring

Trinity College Dublin (TCD, AC14 - CR11)Trinity College Dublin (TCD, AC14 - CR11)

Brian CoghlanBrian Coghlan, , Stuart KennyStuart Kenny

CYFRONET Academic Computer Centre, Krakow (CYFRONET Academic Computer Centre, Krakow ( CYFRO, CYFRO, CO1)CO1)

Xxx YyyXxx Yyy

ICM, University of Warsaw (ICM, University of Warsaw (ICM, ICM, AC2 - C01)AC2 - C01)

Xxx YyyXxx Yyy

Poznan 28-JAN-2002

Poznan 28-JAN-2002 CrossGrid Task 3.3

Task 3.3Task 3.3 Presentation Presentation

Will concentrate on:Will concentrate on:GoalsGoalsTechnology to be developedTechnology to be developedTechnology requiredTechnology required


Task 3.3Task 3.3 Target Audience Target Audience

Application-level:Application-level: ProgramsProgramsToolsToolsServicesServices

Poznan 28-JAN-2002

WP3Portals(3.1)

Roaming Access(3.1)

Grid Resource Management

(3.2)

Grid Monitoring

(3.3)

Optimisation of Data Access

(3.4)

Tests and Integration

(3.5)

ApplicationsWP1

End Users

WP1, WP2, WP5TestbedWP4

Performance evaluation tools

(2.4)

WP3WP3

CrossGridCrossGridTechnicalTechnical

AnnexAnnexFig. WP3-1Fig. WP3-1


WP2WP2

Application

source code

Grid Monitoring

(3.3)

Performance

measurement

MPI verification

(2.2)

Benchmarks

(2.3)

Automatic

analysis

Analytical

model

Visualization

Applications (WP1)executing on

Grid testbed (WP4)

Performance analysis (2.4)

CrossGridCrossGridTechnicalTechnical

AnnexAnnexFig. WP2-1Fig. WP2-1


Task 3.3Task 3.3

We MUST get it RIGHTWe MUST get it RIGHT

Very central functionVery central function


Task 3.3 MilestonesTask 3.3 MilestonesPM 1-3PM 1-3 Definition of requirements Definition of requirements

PM3 : Deliverable D3.1 [ALL]PM3 : Deliverable D3.1 [ALL]

PM 4-6PM 4-6 Designing of architecture, interfaces and security issue Designing of architecture, interfaces and security issue

PM6 : Deliverable D3.2 (report) [ALL]PM6 : Deliverable D3.2 (report) [ALL]

PM 6PM 6 First testbed set-up on selected sites First testbed set-up on selected sites

PM6 : Deliverable D3.2 [ALL]PM6 : Deliverable D3.2 [ALL]

PM 6-12PM 6-12 Implementation of 1 Implementation of 1stst prototype (running on local grid) prototype (running on local grid)

PM12 : Deliverables D3.3 (prototype and report)PM12 : Deliverables D3.3 (prototype and report)

PM 13-24PM 13-24 Second development phase Second development phase

PM13 (beginning) full testbed prototypePM13 (beginning) full testbed prototype

PM 13-15PM 13-15 Testing, feedback collecting Testing, feedback collecting

PM13 : Extraction of raw performance data locally for WP2.4PM13 : Extraction of raw performance data locally for WP2.4

PM 16-18PM 16-18 Refinement, implementation of WP3.3 Refinement, implementation of WP3.3

PM18 : Deliverable D3.5 Extraction of raw performance data on Grid for WP2.4PM18 : Deliverable D3.5 Extraction of raw performance data on Grid for WP2.4

PM 18-24PM 18-24 Refinement, implementation of WP3.3. Refinement, implementation of WP3.3.

??????

PM 16-24 PM 16-24 Refinement, implementation of other tasks within WP3Refinement, implementation of other tasks within WP3

PM24 : Deliverable D3.6 (prototype and report)PM24 : Deliverable D3.6 (prototype and report)

PM 25-33PM 25-33 3 3rdrd development phase development phase

PM33 (end) : Milestone M3.5 Final version of the programming environmentPM33 (end) : Milestone M3.5 Final version of the programming environment

… … and so onand so on


Task 3.3Task 3.3 Philosophy Philosophy

‘‘A good Engineer is a person A good Engineer is a person who makes a design that works who makes a design that works

with as few original ideas as with as few original ideas as possible’possible’

Freeman DysonFreeman Dyson[via Yannis Perros][via Yannis Perros]

So, wSo, where possible, reuse ideashere possible, reuse ideas from from::GlobusGlobusGGFGGFDataGridDataGridElsewhereElsewhere


Task 3.3Task 3.3 Technology to be Developed Technology to be Developed

OMISOMISInterfaceInterface

RGMARGMA OMISOMISInterfaceInterface

LocalLocalMonitorMonitor

PerformancePerformanceData StorageData Storage

RGMARGMAProducerProducer

ServiceServiceManagerManager

RGMARGMAConsumerConsumer

PerformancePerformanceToolsTools

ApplicationApplication ApplicationApplicationMonitorMonitor

Non-Non-invasiveinvasive

MonitoringMonitoring

Invasive MonitoringInvasive Monitoring

Non-invasive MonitoringNon-invasive Monitoring

fromfromTechnicalTechnical

AnnexAnnex


Invasive MonitoringCopyright

S.M.Fisher / RAL


Non-invasive MonitoringNon-invasive Monitoring


Task 3.3 Task 3.3 Technology Required: ChoicesTechnology Required: Choices

HomebrewHomebrew

Globus Sockets – current technologyGlobus Sockets – current technology

Globus:Globus:

MDS - current technologyMDS - current technology

GGF:GGF:

GMA - guidelines onlyGMA - guidelines only

DataGrid:DataGrid:

R-GMA - evolving technologyR-GMA - evolving technology

Elsewhere:Elsewhere:

Jiro - new technologyJiro - new technology


Homebrew: Globus SocketsHomebrew: Globus Sockets

Supported by Globus toolsetSupported by Globus toolset

Includes GSI securityIncludes GSI security

http://www.globus.org/http://www.globus.org/

Quick solutionQuick solution

Does not give access to grid information systemDoes not give access to grid information system


Globus: Meta-Directory Service (MDS)Globus: Meta-Directory Service (MDS)

Hierarchical SchemaHierarchical Schema

Queries constrained to schemaQueries constrained to schema

Can have parallel schemasCan have parallel schemas

Based on Lightweight Directory Access Protocol (LDAP)Based on Lightweight Directory Access Protocol (LDAP)

Good for static dataGood for static data

Less good for dynamic dataLess good for dynamic data

Now in 2Now in 2ndnd generation (MDS2 released OCT-2001) generation (MDS2 released OCT-2001)

Now uses LDAP v3Now uses LDAP v3


MDS: DataGridMDS: DataGrid

DataGrid WP3 has delivered:Coordination of GRIS/GIIS work

OpenLDAP caching backend (FTREE)

GRIS/GIIS infrastructure maintain as necessary

Little effort anticipated

FTREE maintain as necessary

Little effort anticipated

Copyright

S.M.Fisher / RAL


MDS: MDS: LDAPLDAP

LDAP reflects hierarchical database technologyLDAP reflects hierarchical database technology

LDAP is LDAP is efficientefficient if you know the query in advance if you know the query in advance::

you can build your database to you can build your database to respondrespond very rapidly very rapidly

For other queFor other queriesries, it could be , it could be veryvery inefficient:inefficient:

LDAP query language cannot give results based on computation LDAP query language cannot give results based on computation on two different objects in the structure on two different objects in the structure

Relational databaseRelational database technology technology proposedproposed by Codd by Codd ((30yrs ago30yrs ago) as) as solution to inflexibility of hierarchical solution to inflexibility of hierarchical && network data bases network data bases


GGF: Grid Monitoring Architecture (GMA)GGF: Grid Monitoring Architecture (GMA)

Open schemaOpen schema

Distributed architectureDistributed architecture

Allows for dynamic dataAllows for dynamic data

GGF is a ForumGGF is a Forum

GMA is a set of guidelinesGMA is a set of guidelines

GGF will not ‘approve’ anythingGGF will not ‘approve’ anything

Several prototype implementationsSeveral prototype implementations


GMAGMA: Architecture: Architecture

Consumers can themselves be ProducersConsumers can themselves be Producers

ConsumerConsumer

ProducerProducer

RegistryRegistry

registerregister

lookuplookup


GMA: DataGMA: Data MModelodel / Protocols / Protocols

GGF GGF dodo NOTNOT specifyspecify any data model any data model

GGF GGF dodo NOTNOT specif specifyy protocol protocolss between consumer between consumerss && producer producerss

An implementer MUST:An implementer MUST:

FirstFirst choose a suitable data model choose a suitable data model

TThenhen select suitable protocols select suitable protocols

The chosen data model The chosen data model MUSTMUST have the power to support have the power to support ALLALL required required queries queries


DataGrid: Relational GMA (R-GMA)DataGrid: Relational GMA (R-GMA)

Trying to Trying to do “better” than MDSdo “better” than MDS

An implementation of GMAAn implementation of GMA

Relational schemaRelational schema

Queries less constrained than MDSQueries less constrained than MDS

Designed to handle dynamic dataDesigned to handle dynamic data

11stst prototype released in NOV-2001 prototype released in NOV-2001

Based on servlet technologyBased on servlet technology

HTTP queries / XML result-setsHTTP queries / XML result-sets

Can be set up to access MDS dataCan be set up to access MDS data


R-GMA: Data ModelR-GMA: Data Model

DataGridDataGrid have chosen a have chosen a RELATIONALRELATIONAL data modeldata model

Not general distributed RDBMS system, but a way to use relational Not general distributed RDBMS system, but a way to use relational model in model in a a distributed environment distributed environment where ACID (Atomicity, where ACID (Atomicity, Consistency, Isolation and Durability) properties not Consistency, Isolation and Durability) properties not importantimportantProducersProducers announce:announce: SQL “CREATE TABLE”SQL “CREATE TABLE”

publish:publish: SQL “INSERT”SQL “INSERT”

ConsumersConsumers collect:collect: SQL “SELECT”SQL “SELECT”

Viewed asViewed as one huge logical data base one huge logical data base,, partitioned according to partitioned according to certain criteriacertain criteria (specified by WHERE clause as a predicate) (specified by WHERE clause as a predicate)


R-GMA: Data QualityR-GMA: Data Quality

Difficult to knowDifficult to know when to get rid of archived data when to get rid of archived data

Information may no longer be Information may no longer be ‘‘up to dateup to date’’

BUTBUT

ForFor historical historical analysisanalysis this is this is not a criteria for pruningnot a criteria for pruning

Only the Only the coconsumernsumer of data of data can devise a suitable strategycan devise a suitable strategy

Only theyOnly they know know why why theythey areare collecting the data collecting the data

ProducerProducer of data is no judge of its continued worth of data is no judge of its continued worth

Time-to-Live (Time-to-Live (TTLTTL)) of of questionablequestionable value value


R-GMA: Philosophy

Information vs. Monitoring

From the user’s point of view there there is little or no difference between “plain” information and monitoring information.• Arguments about rates of change [slowly=Information whereas

rapidly=Monitoring] are unconvincing.

• Any measurement or fact represented as a tuple.

• Add a time stamp to the tuple and the information can be stored for later analysis as monitoring information.

• At most the difference is 1 field - the time stamp.

Time is the common element

DatagridDatagrid use use R-GMAR-GMA not only for monitoring but also as the basis not only for monitoring but also as the basis of an information systemof an information system

Copyright

S.M.Fisher / RAL


R-GMA: Implementation

Consumer Servlet

RegistryAPI

Consumer Servlet

RegistryAPI

Consumer Servlet

RegistryAPI

Consumer ServletConsumer Servletand Mediatorand Mediator

RegistryAPI

Sensor CodeSensor Code

ProducerAPI

Application Code Application Code

ConsumerAPI

ProducerServlet ProducerServlet

RegistryAPI

Registry ServletRegistry Servlet

SchemaAPI

Schema ServletSchema Servlet

Event Dictionary

DBProducer

DBProducerDBProducerServletServlet

Application CodeApplication Code

ArchiverAPI

Archiver ServletArchiver Servlet

ConsumerAPIConsumer

APIConsumerAPIConsumer

API

Copyright

S.M.Fisher / RAL

lookuplookup

subscribesubscribe


R-GMA: Progress

API in both JAVA and C++

Basic sensors (i.e. cpuload) for producing information

Integration with MDS:R-GMA sensor queries MDS GRIS and republishes information

Some basic displays to consume data

Demo code to get people started

Released into Testbed1 of EU-DataGrid

Copyright

S.M.Fisher / RAL


JiroJiroTMTM Technology Technology

New technologyNew technology

http://www.jiro.com/

For distributed resource managementFor distributed resource management

Part of ‘Federated Management Architecture’Part of ‘Federated Management Architecture’

ICM will discussICM will discuss


Task 3.3: SubtasksTask 3.3: Subtasks

Task 3.3 (CYFRO / TCD / ICM)Task 3.3 (CYFRO / TCD / ICM)

Requirements SpecificationRequirements Specification

Task 3.3.1 (CYFRO - 38 MM)Task 3.3.1 (CYFRO - 38 MM)

Invasive Monitoring:Invasive Monitoring:‘‘Autonomous monitoring system for on-line and automaticAutonomous monitoring system for on-line and automatic

performance analysisperformance analysis ‘‘

Task 3.3.2 (TCD – 32 MM)Task 3.3.2 (TCD – 32 MM)

Non-invasive Monitoring:Non-invasive Monitoring:‘‘SQL-query-based tool support and interfaces to GridSQL-query-based tool support and interfaces to Grid

application programming environmentapplication programming environment’’

Task 3.3.3 (ICM - 28 MM)Task 3.3.3 (ICM - 28 MM)

Jiro:Jiro:‘‘Grid Jiro-based services for Grid infrastructure monitoGrid Jiro-based services for Grid infrastructure monitoring’ring’


Task 3.3: To Be DiscussedTask 3.3: To Be Discussed

Source Code PortabilitySource Code Portability

C / Java / Perl / Python ?C / Java / Perl / Python ?

API definitionAPI definition

For Task 3.3.1For Task 3.3.1

Open Source LicensingOpen Source Licensing

GPL ?GPL ?


WP3 Task 3.3WP3 Task 3.3

THE ENDTHE END

CrossGrid WP3 Task 3.3 Grid Monitoring Trinity College Dublin (TCD, AC14 - CR11) Brian Coghlan, Stuart Kenny CYFRONET Academic Computer Centre, Krakow.

Documents

deliverable d3

implementation of wp3

deliverables d3

data model

datagriddatagrid wp3

grid information systemglobus

report allpm

ldap query language