1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 1 Marian Bubak Marian Bubak Institute of Computer Science and ACC Institute of Computer Science and ACC CYFRONET CYFRONET AGH AGH Cracow Cracow , Polan , Polan d d [email protected][email protected]and and CrossGrid, K-WfGrid, and (future) CrossGrid, K-WfGrid, and (future) GridSpace GridSpace Collaborations Collaborations From Interactive Applications From Interactive Applications and Knowledge Based Workflows and Knowledge Based Workflows to Transparent Semantic Grid to Transparent Semantic Grid (from X# and ~# to # ) (from X# and ~# to # )
52
Embed
1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 1 Marian Bubak Institute of Computer Science and ACC CYFRONET AGH Cracow, Poland AGH Cracow,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Large scale numerical simulationsLarge scale numerical simulations Computationally demanding data analysisComputationally demanding data analysis Distributed computing and storageDistributed computing and storage Remote access to experimental Remote access to experimental
equipmentsequipments A need for integration heterogeneous A need for integration heterogeneous
environments into one applicationenvironments into one application Collaborative problem solvingCollaborative problem solving Virtual organisationsVirtual organisations
Evolution in Distributed ComputingEvolution in Distributed Computing
Distributed systems operate in heterogenous Distributed systems operate in heterogenous environmentsenvironments
• Large scale resource sharingLarge scale resource sharing• InteroperabilityInteroperability• Communication via protocol stacksCommunication via protocol stacks
Service oriented architecturesService oriented architectures
Open standard integrationOpen standard integration
Virtualisation of resourcesVirtualisation of resources
Complexity of computing systems close to the Complexity of computing systems close to the limits of human capabilitylimits of human capability
Main Main CrossGrid CrossGrid ObjectivesObjectives
New category of GridNew category of Grid--enabled applicationsenabled applications• computcompute-e- and data and data--intensiveintensive• distributeddistributed• nearnear--realreal--time response (a person in a loop)time response (a person in a loop)• layered layered
New programming toolsNew programming tools
Grid more user friendly, secure and efficientGrid more user friendly, secure and efficient
Interoperability with other GridsInteroperability with other Grids
Implementation of standardsImplementation of standards
Air Air ppollution ollution fforecastorecastinging
Wave Wave modelingmodeling
Data miningData mining
Weather forecastWeather forecastinging is is thethe common link and provides common link and provides input data input data forfor all other all other activitiesactivities
Main Features:Main Features: Single sign-on / authorisationSingle sign-on / authorisation Platform independentPlatform independent Batch jobsBatch jobs MPI jobsMPI jobs Running interactive applications Running interactive applications
using java plugins or VNCusing java plugins or VNC Monitoring grid applicationsMonitoring grid applications Flexible Application frameworkFlexible Application framework User profile managementUser profile management Easy application add onEasy application add on Local and grid file management Local and grid file management
Well-defined set of web-services Well-defined set of web-services An interface for accessing HPC An interface for accessing HPC
systems and services (based on systems and services (based on various technologies) in a various technologies) in a common and standardised waycommon and standardised way
Interconnection between variousInterconnection between various grid middleware and applicationsgrid middleware and applications
Additional features:Additional features:• Virtual Directory supportVirtual Directory support• Plug-in for various grid Plug-in for various grid
1.1. Interactive workflow - uInteractive workflow - user ser interactivityinteractivity onon the Grid the GridA A user can continuously interact with user can continuously interact with a a Grid client without Grid client without waiting for twaiting for termination ermination of the jobsof the jobs submittedsubmitted
2.2. Online output control - uOnline output control - user ser oone-ne-wway ay iinteractivity nteractivity A A user can see the output of the application running user can see the output of the application running inin the Grid the Grid testbed on testbed on an an MD client synchronously with the applicationMD client synchronously with the application
3.3. Runtime steering - uRuntime steering - user ser ttwo-wo-wway ay iinteractivitynteractivityA useA use can steer the running application, either providing some can steer the running application, either providing some input data online as requested by the application input data online as requested by the application (also (also asynchronouslyasynchronously)), or suspending the process, changing some , or suspending the process, changing some input data and resuming itinput data and resuming it
Interactive Job Submission Interactive Job Submission The user submits an The user submits an
application job through the application job through the CG portal or Migrating CG portal or Migrating Desktop and the Roaming Desktop and the Roaming Access Server which Access Server which supports individual user supports individual user environments,environments,
The job is handled by the The job is handled by the Scheduler, which selects Scheduler, which selects the appropriate computing the appropriate computing resources,resources,
DataGrid software DataGrid software components are used for components are used for low-level Grid operations low-level Grid operations (submission for (submission for processing and delivery of processing and delivery of results),results),
The system bases on The system bases on Globus Toolkit v2,Globus Toolkit v2,
Other CrossGrid tools and Other CrossGrid tools and services can be used in services can be used in conjunction with running conjunction with running jobs, as requirements jobs, as requirements dictatedictate
Automatic job management for parallel Automatic job management for parallel applications:applications:
Search and selection of available resources, job Search and selection of available resources, job conditioning, job launching, job monitoring, job retry conditioning, job launching, job monitoring, job retry (in case of failures) and results retrieval.(in case of failures) and results retrieval.
OCM-G – FeaturesOCM-G – Features On-line On-line operationoperation Support for Support for multi-site grid applicationsmulti-site grid applications Low perturbationLow perturbation
• Techniques for data rate Techniques for data rate reduction reduction • LLightweight and fast socket-based communicationightweight and fast socket-based communication
FlexibleFlexible, , sservices-drivenervices-driven designdesign• No fixed metrics but a set of flexible services to construct metrics with No fixed metrics but a set of flexible services to construct metrics with
desired semanticsdesired semantics• Enables custom metrics in G-PMEnables custom metrics in G-PM
ExtendibleExtendible• AAdditional servicesdditional services can easily be added can easily be added• LLoaded dynamically at run-timeoaded dynamically at run-time
Autonomous and standardizedAutonomous and standardized • SStandard interface tandard interface • MMinimizeinimizedd effort of porting OMIS-based tools across platforms effort of porting OMIS-based tools across platforms • Enabled interoperability of multiple tools monitoring a single application.Enabled interoperability of multiple tools monitoring a single application.
UDAL - UDAL - Unified Data Access LayerUnified Data Access Layer
DatabaseStorage
NAS
Replica ManagerEDG Reptor
Applications Portal
HSM
Componentsare kinds of
plug-ins
Components are grouped by types but they differ from each other by specialization. The rule based system takes the decision which particulartype component is the best in a context.
Applications
Applications
Unified Data Access Layer based on Component-Expert Architecture
Component TypeTID1
Component TypeTID1 Component Type
TID1
Component TypeTID1
Component TypeTID2 Component Type
TID1
Component TypeTID3 Component Type
TID1
Component TypeTID1
Component TypeTID4
Unified Data Access Cost Estimation Unified Data Access via GridFTP Protocol
Grid HLAGrid HLA Management System Management System HLA management servicesHLA management services
• HLA-speaking Service for HLA-speaking Service for managing federatemanaging federatess
• RTIExec Service for RTIExec Service for managing RTIExec managing RTIExec (coordination process in (coordination process in RTI)RTI)
• Broker Broker for setting up a for setting up a federationfederation and and makmakinging migration decisionsmigration decisions
Broker decision servicesBroker decision services• Registry Registry for for storingstoring
location of location of HLA-speaking HLA-speaking serviceservicess
• Infrastructure Infrastructure Monitoring/BenchmarksMonitoring/Benchmarks for checking environment for checking environment of HLA serviceof HLA service
Migration support servicesMigration support services• Application Monitoring Application Monitoring
for monitoring for monitoring performanceperformance
FeaturesFeatures Brings interactive applications to the GridBrings interactive applications to the Grid Enables easy access toEnables easy access to the Gridthe Grid viavia WWeb eb SServices ervices Extends and enhances Extends and enhances DataGrid, GridLab, and EuroGridDataGrid, GridLab, and EuroGrid DDeveloped according to Geveloped according to GGGF and software engineering standardsF and software engineering standards
StatusStatus SStable version available table version available sincesince March 2004 as open source March 2004 as open source LLicensing: CrossGrid license based on EDG, GPLicensing: CrossGrid license based on EDG, GPL CrossGrid Tutorial available for potential usersCrossGrid Tutorial available for potential users
Fraunhofer FIRST, Berlin, GermanyFraunhofer FIRST, Berlin, Germany Institute of Computer Science, Institute of Computer Science,
University of Innsbruck,University of Innsbruck, Innsbruck, AustriaInnsbruck, Austria Institute of Informatics of the Slovak Institute of Informatics of the Slovak
Academy of Sciences, Bratislava, Academy of Sciences, Bratislava, SlovakiaSlovakia
• Enabling automatic Enabling automatic construction and reuse of construction and reuse of workflows with knowledge workflows with knowledge gathered during operationgathered during operation
• Involving monitoring and Involving monitoring and knowledge acquisition knowledge acquisition services in order to provide services in order to provide added value for end usersadded value for end users
Monitoring and Performance AnalysisMonitoring and Performance Analysis
• Monitoring and instrumentation service (MIS)Monitoring and instrumentation service (MIS)• Performance analysis service (PAS)Performance analysis service (PAS)• Data representations and service interfaces Data representations and service interfaces
Monitoring and Performance AnalysisMonitoring and Performance Analysis Monitoring and Instrumentation ServiceMonitoring and Instrumentation Service
• Instrument code regions, and activitiesInstrument code regions, and activities• Monitor infrastructure, code regions, activity execution statusMonitor infrastructure, code regions, activity execution status
Performance Analysis ServicePerformance Analysis Service• Define performance metrics for workflows Define performance metrics for workflows • Analyze monitoring data and relate the data to the workflowAnalyze monitoring data and relate the data to the workflow• Define performance properties and search performance Define performance properties and search performance
bottlenecks of workflowsbottlenecks of workflows
Data Presentations and Service InterfacesData Presentations and Service Interfaces• XML schemas for describing CPU usage,TCP bandwidth, generic XML schemas for describing CPU usage,TCP bandwidth, generic
events, profiling data, workflow activity execution status, etc. events, profiling data, workflow activity execution status, etc. • WIRL, PDQS, common service operations and specific service WIRL, PDQS, common service operations and specific service
operationsoperations
WP InterdependenciesWP Interdependencies• Scheduler and Grid Workflow Execution Service (GWES) - WP2Scheduler and Grid Workflow Execution Service (GWES) - WP2• Grid Organizational Memory - WP4Grid Organizational Memory - WP4• Knowledge Assimilation Agent - WP5 Knowledge Assimilation Agent - WP5
Other flood samples time Other flood samples time perdiodperdiod
Average monitoring Average monitoring parametersparameters
Service class (e.g. FFT Service class (e.g. FFT solver)-Service inputssolver)-Service inputs;; Service outputsService outputs; ; Service Service preconditionspreconditions, effects, effects
Workflow patternsWorkflow patterns
Workflow historyWorkflow history
Domain specific Domain specific requirements – no parallelrequirements – no parallel
Levels of KnowledgeLevels of Knowledge Knowledge is separated into three levels Knowledge is separated into three levels
• Generic – definitions of concepts, taxonomyGeneric – definitions of concepts, taxonomy• Domain specific – definition of domain specific topicsDomain specific – definition of domain specific topics• Data – individuals from a concrete applicationData – individuals from a concrete application
Knowledge is gathered in different registries Knowledge is gathered in different registries
Municipality of Genova Slovak Water Research Institute, Bratislava Slovak Hydrometeorological Institute, Bratislava Slovak Watermanagement Enterprise, Banska Stiavnica
• Various Grid middleware platforms for uniform access to Various Grid middleware platforms for uniform access to resources – difficult, complex to programresources – difficult, complex to program
• RRecent initiatives ecent initiatives onon Grid programming do not address the Grid programming do not address the dynamic nature of the Griddynamic nature of the Grid
Important features of the GridImportant features of the Grid• Grid is dynamicGrid is dynamic• Resource users do not instantiate resources on their ownResource users do not instantiate resources on their own• RResources esources are in differentare in different administrative domains administrative domains
Therefore, a Grid application has to beTherefore, a Grid application has to be• More loosely coupled (combined of autonomous elements)More loosely coupled (combined of autonomous elements)• Flexible to overcome (and benefit from) Grid dynamic natureFlexible to overcome (and benefit from) Grid dynamic nature• Adaptable to cross boundaries of various administration Adaptable to cross boundaries of various administration
FeaturesFeatures• Abstract, semantically rich layer between a user and the Abstract, semantically rich layer between a user and the
middlewaremiddleware• Set of tools for Grid application developer to make the Grid Set of tools for Grid application developer to make the Grid
programming easierprogramming easier• Strong support for developing flexible and adaptable Strong support for developing flexible and adaptable
applicationsapplications
ComponentsComponents• Grid programming languageGrid programming language• Interpreter with dynamic Interpreter with dynamic ad-hocad-hoc binding capabilities binding capabilities• Runtime environment based on tuple space ideaRuntime environment based on tuple space idea• Evolving language library to share and reuse applicationsEvolving language library to share and reuse applications
Separates the developer from Separates the developer from ever-changing Grid resource ever-changing Grid resource layerlayer
Seamlessly introduces Seamlessly introduces dynamism into newly created dynamism into newly created applicationsapplications
Provides unified access to Provides unified access to resources by means of resources by means of semantically described semantically described abstractionsabstractions
Supports evolving and well Supports evolving and well organized library of organized library of applications used up-to-dateapplications used up-to-date
Allows easy reuse of already Allows easy reuse of already built built applicationsapplications
GridSpace abstraction of data
abstraction of computations abstraction of components
From Resource Abundance to Programmable GridFrom Resource Abundance to Programmable Grid
Similar semantic description of all the resourcesSimilar semantic description of all the resources• Using common notions: dependency, requirement, Using common notions: dependency, requirement,
capabilitycapability• Based on growing Semantic Web/Grid achievementsBased on growing Semantic Web/Grid achievements• With natural ability to be extended by multiple usersWith natural ability to be extended by multiple users
AdvantagesAdvantages• Helps build new applicationsHelps build new applications• Common language for various tools and platformsCommon language for various tools and platforms• Does not enforce unification of underlying technologyDoes not enforce unification of underlying technology
Global Grid Environment
Computingpower
Data
Software
Sensors,devices
Networktransfer
DatastoragespaceCommon
SemanticDescription
Layer
The environment provides everything for The environment provides everything for anan applicationapplication: v: vast space, multiplication ast space, multiplication of resources, multitude of access of resources, multitude of access standards and protocolsstandards and protocols
IIssuessuess
• Plenty of resources to build Plenty of resources to build sophisticated applicationsophisticated applications froms from
• Each new application requires huge Each new application requires huge effort to overcome integration effort to overcome integration problemsproblems
Transparent Transparent Global GridGlobal Grid for Everyday Use for Everyday Use
The applications The applications should beshould be buil builtt/rebuil/rebuiltt almost at runtime almost at runtime Abstraction and transparencyAbstraction and transparency
• Hierarchy of resources enables customisable abstraction Hierarchy of resources enables customisable abstraction levellevel
• No more No more jjob submission black boxesob submission black boxes – control over running – control over running application will be enabled application will be enabled
• AA level of transparency helps users understand level of transparency helps users understand the the GridGrid The path The path to our main objective: plugto our main objective: pluggingging problems into the Grid problems into the Grid