Experiences of the Grid… Gavin McCance University of Glasgow NeSC Meeting, 24 October 2001.

Experiences of the Grid…

Gavin McCanceUniversity of GlasgowNeSC Meeting, 24 October 2001

NeSC 24 October 2001

Gavin McCance, University of Glasgow 2/38

Background

Experimental Particle Physics background Analysing the structure of matter

…Fortran (19)77 !

Working in ‘Grid’-like areas since January this year



GridPP 20+ institutes…



…GridPP

£17M 3-year projectWorking in collaboration with EU DataGrid project

Middleware productionIntegration of middleware technologies into HEP experimentsValidation of Grid Software



…GridPP

Initial GridPP testbed underway

A personal snapshot of activities on the grid… Middleware activities we’re

involved in Some examples

Technologies we’re using Issues with integration of ‘Grid’

with particle physics experiments



Middleware

What is middleware…???

Application programs – local gridopen()

Data access specifics – HPSS, CastorJob submission specifics – PBS, LSF

Specific security procedures

Grid middleware Layered API’s.

Transparent security. Transparent data access.

Intelligent use ofdistributed resources.



Middleware Activities

GridPP ~mirrors EU DataGrid:Workload Management What jobs go where?

Data Management (*) Where’s the (best) data?

Information Services (*) What’s the state of everything?



…Middleware Activities

Fabric Management Interfaces to underlying systems

Mass Storage Management How to get the data to/from the fabric e.g.

Implementing ‘file-save()’ APIs for different mass storage systems

Security Crops up everywhere … transparent to

applications



Data Management

Data ReplicationTransparent and Secure Data AccessMeta Data StorageQuery Optimisation



Example problem: Data Replication

Problems if data exist only in one place Multiple accesses to the same data

overload network! Petabytes! Funding constraints! e.g. CERN can’t store

all of the data required

Make Replica! But need to keep track of all the files and their various replica! Need replica catalogue!



…Catalogues

Examples solutions:Have a globally unique Logical File Name (LFN) mapping to multiple physical instances of the file (PFNs).

Replica selection required Choose the ‘best’ / ‘nearest’ / ‘fastest’

Cost modelling… how time expensive to transfer files X’ from A to B

File-1

File-1

File-1

File-1

Paris

Glasgow

Chicago

LFN



…Data Replication

Grid Data Mirroring Package C, C++, JAVA, command-line APIs

Replication issues: File transfer…

Synchronisation / consistency models Basic middleware doesn’t enforce any

policy

Scalable architectures



…GDMP

File transfer uses GridFTP Existing IETF-approved (?RFC?) ftp additions + the standard grid security (GSI)

Registers new files in replica catalogue E.g. interfaced to the existing Globus

Replica Catalogue

Basic replica manager functionality to maintain consistency of replica sets



…Implementation issues

Structure not imposed by the middleware software itself… But … must think about scalable

implementations

E.g. a RC may exist on each storage element responsible for its own filesCERN Root RC

CERN RCUK RCINFN RC

Queries will propagate down until replica information is found…



…Longer term problems

Query / Replica Optimisation Grid can make / delete replica

Eg. Many people in Glasgow & Edinburgh access the ATLAS Higgs dataset ‘A1’…

Autonomously make new replica in / near Scotland based on historical information

Grid might re-cluster data

A2

A3

A1 B1 B3

B2

GlasgowParis

A2

A3

A1

B1

B3

B2

GlasgowParis



…longer termreal Grid...real Grid...

…simulated Grid provides testing arena for more

adventurous ideas!

…simulated Grid provides testing arena for more

adventurous ideas!

MONARC simulation tool



…Integration of middleware

Many iterations of requirements and use-cases with end-users… meetings…Middleware solutions must be scalable and useable by a variety of end users HEP, Biological, Earth sciences, Astro

Always looking for common elements E.g. replica / meta-data catalogues… data

transport… security…



…examples of common interfaces:generic meta-data catalogue tools

SQL Database Service: Problem: many relational databases,

diverse security, diverse wire protocols …Solution: Build on existing wire protocols: XML

transported over HTTP(S) Grid standard security framework (GSI)



..examples

Leverage open-source technology JAVA servlet based (Apache Tomcat

engine) JDBC drivers Utilises Oracle’s XSQL servlet (open

source)

Security over HTTPS with Grid-standard GSI mechanism



…examples

Allows any HTTP compliant system e.g. Web-browsers / standard C++ HTTP libraries to access any relational database…

= SQL Database Service (Spitfire)

OraclePostgreSQL

+

PKI Security

+

Standard communicationprotocols

(XML over HTTPS)



Global Grid Forum

Global Grid Forum meetings GGF1: Amsterdam meeting in April

2001

Helps define aspects common to all Grid-like projects. E.g. architectures, ‘grid’ protocols

As example… Grid Monitoring Architecture (GMA)



Information Services - GMA

One Implementation of the GMA Globus MDS, currently based on (Open)LDAP

Hierarchical directory like structure Very fast for information retrieval if you

already know the query designed into structure.

Bad for complex or ranged queries



..complementary implementation

Producer

Producer API

Register,re-register,publish

Relational Database

Consumer

Querying API

Query

ProducerServlet

RegistryServlet

SchemaServlet

subscribestream

Implementation of GMARelational queries in SQL format



…relational GMA

Information is transferred in generic SQL format…‘Producers’ of information register themselves…‘Consumers’ construct (possibly complex) SQL query and are streamed query results directly from Producers.



…implementation

Again, uses JAVA servlets Tomcat servlet engine

Again, communication with servlet is over standard HTTP.

All the internal parts communicate via HTTP and XML modular design, easily replaceable…



Useful Tools…

JAVA… nicely platform independentUML Universal(?) Modelling Language Architecture and API’s ‘should be’

defined in this…!

CASE tools Together Control Centre



…useful tools

Globus toolkit Both the original and its java

implementation (CoG)

My experience of CoG so far is generally good…! Easy GSI authentication, Globus file

transfer, Globus job submission, MDS interface



Testbeds

For GridPP, primary testbeds are the HEP experiment ones

CERN LHC (EU DataGrid WP8) US experiments, e.g. Fermilab, SLAC

First software release now!! Integration team ‘show-and-tell’ at CERN

end of this month…



...testbed work

Grid software packaged for release to experiments!

Primarily packaged using RPM

For end of October release, supported platforms are:Linux (and Solaris on a best effort basis)



..Globus installation

Generally found the Globus software installation OK! Successfully deployed on a number of batch

systems in UK Experience fed back into eScience Centres

Difficulties were setting up and recognising each countries’ Certificate Authorities (CAs) Tricky legal implications to resolve!



Testbed work so far…

UK Certificate Authority set-up… Many institutes already on testbed

Grid Status and Network monitoring demonstrator available soon

Networking status information provided by GridPP and DataGrid networking groups!



…testbed work so far

Successful tests within ATLAS (and others) of some middleware products E.g. Large file transfers between UK,

Italy, US and CERN

Further tests planned with new release!



…experimental integration

Work to do… Taking the kit and trying to integrate it into

the experiments’ software frameworks

ATLAS/LHCb software

framework (GAUDI)

Grid middleware

GANGA framework

Make Grid Services transparently available to

ATLAS and LHCbprograms



Grid validation

Preliminary tests of basic middleware has been successful

Now we have opportunity to see how it performs and scales with real datasets and real experimental users



Overall experience

Middleware development is fun! Several good products have already been

delivered

Re-using industry standard components and protocols where they exist LDAP, SQL, HTTP(S), XML, SOAP PKI security Open Source…!



…overall

Middleware being built using a variety of languages… JAVA, C++, C, Python

APIs should be available for all JAVA, C++, C and command line… web access(?)



…overall



…finally

Preliminary grid software architectures have been definedBasic middleware has been deliveredLarge scale validation underway NOW!!

A excellent base to build on! Plenty still to do!

Experiences of the Grid… Gavin McCance University of Glasgow NeSC Meeting, 24 October 2001.

Documents

best data

data tofrom

transparent data access

year slide

data replication problems

grid middleware activities

data overload network

integration of grid