This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CrossGrid CrossGrid WP3WP3
Task 3.3Task 3.3Grid Grid MonitoringMonitoring
Trinity College Dublin (TCD, AC14 - CR11)Trinity College Dublin (TCD, AC14 - CR11)Brian Brian CoghlanCoghlan, , Stuart KennyStuart Kenny
CYFRONET Academic Computer Centre, Krakow (CYFRONET Academic Computer Centre, Krakow ( CYFRO, CYFRO, CO1)CO1)BartBartoszosz BalisBalis,, Slawomir Zielinski Slawomir Zielinski
ICM, University of Warsaw (ICM, University of Warsaw (ICM, ICM, AC2 - C01)AC2 - C01)KrKrzzysztof Nowinskiysztof Nowinski,, Piotr Bala Piotr Bala
Krakow MAR-2002 CrossGrid Task 3.3
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
Task 3.3Task 3.3 Participants Participants
Trinity College Dublin (TCD, AC14 - CR11)Trinity College Dublin (TCD, AC14 - CR11)Brian Brian CoghlanCoghlan coghlancoghlan@@cscs..tcdtcd..ieie
Stuart KennyStuart Kenny stuartstuart..kennykenny@@cscs..tcdtcd..ieie
CYFRONET Academic Computer Centre, Krakow (CYFRONET Academic Computer Centre, Krakow (CYFRO, CYFRO, CO1)CO1)BartBartoszosz BalisBalis balisbalis@@aghagh..eduedu..plpl
Krzysztof ZielinskiKrzysztof Zielinski kzkz@@icsics..aghagh..eduedu..plpl
ICM, University of Warsaw (ICM, University of Warsaw (ICM, ICM, AC2 - C01)AC2 - C01)Krzysztof NowinskiKrzysztof Nowinski k.k.nowinskinowinski@@icmicm..eduedu..plpl
Piotr BalaPiotr Bala balabala@@icmicm..eduedu..plpl
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
Task 3.3Task 3.3 Presentation Presentation
TO DO:TO DO:1. Subtasks specification and division1. Subtasks specification and division
2. Time schedule, timing between partners2. Time schedule, timing between partnersat least for the first milestoneat least for the first milestone
3. Definition of interfaces – too early3. Definition of interfaces – too early
4. Referencing, dependency on other WP/task4. Referencing, dependency on other WP/taskpropositions to other tasks & propositions to other tasks & WPsWPs
Task 3.3.1 (CYFRO - ?? MM, ICM – 28MM)Task 3.3.1 (CYFRO - ?? MM, ICM – 28MM)Invasive Monitoring:Invasive Monitoring:
‘‘Autonomous monitoring system for on-line and automaticAutonomous monitoring system for on-line and automatic performance analysis performance analysis ‘‘
‘‘JiroJiro -based services for Grid infrastructure-based services for Grid infrastructure monito monito ringring ’’
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
Task 3.3 Task 3.3 Time schedule, timing between partnersTime schedule, timing between partners
PM 1-3 Definition of requirementsPM 1-3 Definition of requirementsPM3 : Deliverable D3.1 [ALL]PM3 : Deliverable D3.1 [ALL]
PM 4-6 Designing of architecture, interfaces and security issuePM 4-6 Designing of architecture, interfaces and security issuePM6 : Deliverable D3.2 (report) [ALL]PM6 : Deliverable D3.2 (report) [ALL]
PM 6 FirstPM 6 First testbed testbed set-up on selected sites set-up on selected sitesPM6 : Deliverable D3.2 [ALL]PM6 : Deliverable D3.2 [ALL]
PM 6-12 Implementation of 1PM 6-12 Implementation of 1stst prototype (running on local grid) prototype (running on local grid)PM12 : Deliverables D3.3 (prototype and report)PM12 : Deliverables D3.3 (prototype and report)
Timing between partners : Timing between partners : to be decidedto be decided
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
Task 3.3 Task 3.3 Definition of interfacesDefinition of interfaces
GlobusGlobus::GlobusGlobus Sockets – mature technology Sockets – mature technologyMDS - mature technology but MDS - mature technology but obsobs ooletelete by end-2002 by end-2002
DataGridDataGrid ::RGMA -RGMA - rreleasedeleased into into DataGrid DataGrid Testbed1Testbed1 DataGrid DataGrid are keen to assist Task 3.3 are keen to assist Task 3.3
Conclusion:Conclusion:Use Use GlobusGlobus Sockets + RGMA where useful Sockets + RGMA where usefulOnly use MDS for first Only use MDS for first testbedtestbed
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
GlobusGlobus Sockets Sockets
Supported by Supported by Globus toolsetGlobus toolsetIncludes GSI securityIncludes GSI securityQuick high-performance solutionQuick high-performance solutionDoes not give access to grid information systemDoes not give access to grid information systemWill give trouble with firewallsWill give trouble with firewalls
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
RGMARGMA
PhilosophyPhilosophy
•• Any measurement or fact represented as a Any measurement or fact represented as a tupletuple•• AAdddd time stamp and time stamp and it becomesit becomes monitoring information monitoring information
•• At At most the difference is 1 fieldmost the difference is 1 field - - the time stamp the time stamp
Time is the binding elementTime is the binding element
DatagridDatagrid use use R-GMAR-GMA not only for monitoring but also as the not only for monitoring but also as thebasis of an information systembasis of an information system
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
R-GMA: Data ModelR-GMA: Data Model
DataGridDataGrid have chosen a have chosen a RELATIONALRELATIONAL data modeldata model
Not general distributed RDBMS system, but a way to useNot general distributed RDBMS system, but a way to userelational model in relational model in a a distributed environment distributed environment where ACIDwhere ACID(Atomicity, Consistency, Isolation and Durability) properties(Atomicity, Consistency, Isolation and Durability) propertiesare are notnot considered considered importantimportantProducersProducers announce:announce: SQL “CREATE TABLE”SQL “CREATE TABLE”
Viewed asViewed as one huge logical data base one huge logical data base,, partitioned according partitioned accordingto certain criteriato certain criteria (specified by WHERE clause as a (specified by WHERE clause as apredicate)predicate)
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
CrossGridCrossGrid WP2 Info Flows WP2 Info Flows
CrossGridCrossGridTechnicalTechnical
AnnexAnnexFig. WP2-1Fig. WP2-1
MPIverification
(2.2)
Benchmarks
(2.3)
Applications (WP1)executing on
Grid testbed (WP4)
Performance analysis (2.4)
Automaticanalysis
Performance
measurement
Analyticalmodel
Visualization
Application
Source
Code
GridMonitoring
(3.3)
Not now needed
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
CrossGridCrossGrid WP3 Info Flows WP3 Info Flows
CrossGridCrossGridTechnicalTechnical
AnnexAnnexFig. WP3-1Fig. WP3-1
WP3WP3Portals(3.1)
Roaming Access(3.1)
Grid ResourceManagement
(3.2)
Optimisation ofData Access
(3.4)
Tests andIntegration
(3.5)
ApplicationsWP1
End Users
WP1, WP2, WP5TestbedWP4
Performanceevaluation tools
(2.4)
GridMonitoring
(3.3)
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
WP3Grid ResourceManagement
(3.2)
*ULG0RQLWRULQJ
�����
Optimisation ofData Access
(3.4)
Performanceevaluation tools
(2.4)
inputinput
inputinput
inputinput
controlcontrolresultresult
resultresult
resultresult
WP2
CrossGridCrossGrid Task 3. Task 3.33: External Info Flows: External Info Flows
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
CrossGridCrossGrid Task 3.3: Internal Info Flows Task 3.3: Internal Info Flows
Trace infoTrace info
WP3 JiroServices
(3.3.3-CYFRO)
OMISService Manager
+ Perf Tools(3.3.1-CYFRO)
Result infoResult info
Non-invasiveMonitoring(3.3.2-TCD)
inputinput
OMISApplication Monitor
+ Local Monitor(3.3.1-CYFRO)
JiroJiro info info
OMIS infoOMIS infoApplications
Instruments
Infrastructure
inputinput
ctrlctrlsktskt
PerformanceInformation
Post-processing(3.3.1-ICM)
inputinput
dBdB
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
CrossGridCrossGrid Task 3.3: Use Cases Task 3.3: Use Cases
1.1. Don’t know exactly what info is of interest:Don’t know exactly what info is of interest:Constantly monitor - when we know, look at the accumulated infoConstantly monitor - when we know, look at the accumulated info
R-GMA supports this approachR-GMA supports this approach
3.3. Know what info is of interest, in long term:Know what info is of interest, in long term:Constantly monitor & accumulate only what is actually neededConstantly monitor & accumulate only what is actually needed
R-GMA supports by this approachR-GMA supports by this approach
2.2. Know what info is of interest, in short term:Know what info is of interest, in short term:Immediately evaluate only what is actually neededImmediately evaluate only what is actually needed
OCM is inspired by this approachOCM is inspired by this approach
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
WP3 JiroServices
(3.3.3-CYFRO)
R-GMA(DataGrid)
OMISService Manager
+ Perf Tools(3.3.1-CYFRO)
resultresult
R-GMA infoR-GMA info(includes(includes
JiroJiro, MDS,, MDS,Trace, OMISTrace, OMIS
& Result)& Result)
inputinput
inputinput
inputinput
Non-invasiveMonitoring(3.3.2-TCD)
inputinput
MDS(Globus)
OMISApplication Monitor
+ Local Monitor(3.3.1-CYFRO)
MDS infoMDS info
JiroJiro info info
OMIS infoOMIS infoApplications
Instruments
Infrastructure
Static infoStatic info
inputinput
ctrlctrlsktskt
PerformanceInformation
Post-processing(3.3.1-ICM)
inputinput
resultresultdBdB
CrossGridCrossGrid Task 3.3: Can it use R-GMA ? Task 3.3: Can it use R-GMA ?
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
Task 3.3.1 (CYFRONET, Krakow, Poland,Task 3.3.1 (CYFRONET, Krakow, Poland, ICM, University of Warsaw, Poland) ICM, University of Warsaw, Poland)
Invasive Monitoring:Invasive Monitoring:‘‘Autonomous monitoring system for on-line and automaticAutonomous monitoring system for on-line and automatic performance analysis performance analysis ‘‘
Derived from OMIS / OCM researchDerived from OMIS / OCM researchhttp://wwwhttp://wwwbode.in.tum.de/~omis/bode.in.tum.de/~omis/For on-line monitoringFor on-line monitoring
And And APART researchAPART researchhttp://www.http://www.fzfz--juelichjuelich.de/apart/.de/apart/For performance analysis of parallel programsFor performance analysis of parallel programs
•• Exploitation of time patterns in grid performanceExploitation of time patterns in grid performance : :•• Use ofUse of AI AI methodsmethods (agent (agent type simulation and neural type simulation and neural//geneticgenetic
algorithmsalgorithms))
•• VisualVisual analysis of raw analysis of raw data data•• Automated application of statistical Automated application of statistical analysisanalysis
•• Analysis of performance and resource usage patternsAnalysis of performance and resource usage patternsforfor typical applications typical applications•• VisualVisual analysis analysis•• Database of application activity Database of application activity patternspatterns
•• PerspectivePerspective::•• Optimization of predicted grid usageOptimization of predicted grid usage//performanceperformance
Task 3.3.2 (Trinity College Dublin, Ireland)Task 3.3.2 (Trinity College Dublin, Ireland)Non-invasive Monitoring:Non-invasive Monitoring:
‘‘SQL-query-based tool support and interfaces to GridSQL-query-based tool support and interfaces to Grid application programming environment application programming environment ’’
‘‘JiroJiro -based services for Grid infrastructure -based services for Grid infrastructure monitomonito ringring ’’
New technologyNew technologyhttp://www.http://www.jirojiro.com/.com/
For distributed resource managementFor distributed resource managementPart of ‘Federated Management Architecture’Part of ‘Federated Management Architecture’
1.1. Sensor data MUST be pushed to Producer Sensor data MUST be pushed to Producer servletservlete.g.e.g. for debugging, queries may focus on small sectionfor debugging, queries may focus on small section
of of logfileslogfiles, yet complete multi-GB , yet complete multi-GB logfiles logfiles must bemust bemove to Producer move to Producer servletservlet
2.2. Each query MUSTEach query MUST instantiate instantiate new objects & connections new objects & connectionse.g. when debugging, have constant interaction withe.g. when debugging, have constant interaction with
same producer, yet each query requires freshsame producer, yet each query requires freshinstantiation of Consumer instantiation of Consumer servlet servlet + http connections+ http connections
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
Problem 1:Problem 1:
dBdB
RGMAProducer
API
HiddenCanonical Producer
MySQLQueryEngine
JDBC
logfiles
SensorSensorCodeCode
databasefiles
DBProducer code
implicitinterface
queries result-set
queries result-set
CanonicalProducer
API
RGMACanonical Producer
logfiles
SensorSensorCodeCode
CanonicalQueryEnginedata
seeks
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
Problem 2:Problem 2:Synchronous Synchronous ControlControl/Data/Data
via via Globus Globus sockets socketsOMISOMIS
InterfaceInterface
Application ApplicationMonitor
LocalLocalMonitorMonitor
PerformancePerformanceData StorageData Storage
RGMAConsumer
API
dBdB
RGMAProducer
API
ProducerServlet
ConsumerServlet(s)
ArchiverServlet
ConsumerServlet
AsynchronousInformation via R-GMA ?
OMISOMISInterfaceInterface
ServiceServiceManagerManager
PerformancePerformanceToolsTools
Maybe ?Maybe ? Query RGMA to find Local Monitor Query RGMA to find Local Monitor THEN set up Control/Data connection THEN set up Control/Data connection
Krakow MAR-2002 CrossGrid Task 3.3
Copyright 2002 Brian Coghlan & Bartosz Balis
Problems with R-GMAProblems with R-GMA
1.1. Sensor data MUST be pushed to Producer Sensor data MUST be pushed to Producer servletservletDataGrid DataGrid will fix thiswill fix this 4-MAR-20024-MAR-2002
2.2. Each query MUSTEach query MUST instantiate instantiate new objects & connections new objects & connectionsDataGrid DataGrid will do will do halfway-househalfway-house solution, then investigate. solution, then investigate.
Specifically – data analysis tools situated as informationSpecifically – data analysis tools situated as informationproducers/servers and consumers pulling automatically dataproducers/servers and consumers pulling automatically datafrom higher order producersfrom higher order producers