EGEE is a project funded by the European Union under contract IST-2003-508833 R-GMA status and plans Abdeslem DJAOUI / RAL GRIDPP10 meeting at CERN, 3.

EGEE is a project funded by the European Union under contract IST-2003-508833

R-GMA status and plans

Abdeslem DJAOUI / RAL

GRIDPP10 meeting at CERN, 3 June 2004

www.eu-egee.org

GRIDPP10 Meeting at CERN, 3 June 2004 - 2

Contents

• Who we are

• Reminder of what is R-GMA

• What we have done so far

• What we plan for the first release of the EGEE middleware

GRIDPP10 Meeting at CERN, 3 June 2004 - 3

JRA1-UK in EGEE

GRIDPP10 Meeting at CERN, 3 June 2004 - 4

Role of JRA1-UK in EGEE

• To meet all information and monitoring needs Expect to base much of our re-engineering work on R-GMA R-GMA used in two modes:

• as standalone • Inter-work with other information systems and displays

• JRA1-UK are not responsible for the contents of the (Events) schema but will coordinate as necessary

GRIDPP10 Meeting at CERN, 3 June 2004 - 5

Team Organization

• Mostly based in PPD at RAL

• Developers have mostly computer science/Maths/Physics background

• We have at least one weekly group meeting

• Using dotProject Allows individual to record their own progress Allows all members to see the big picture Collects the information for the EGEE time sheets

• We can be e-mailed at: [email protected]

• Web page at: http://hepunx.rl.ac.uk/egee/jra1-uk/

GRIDPP10 Meeting at CERN, 3 June 2004 - 6

People and Roles

• Based at RAL Robin Middleton (0.5)

• Cluster Manager

Steve Fisher• Technical Manager,

Documentation and Quality

Abdeslem Djaoui• Architect, deputy technical

manager and developer

Antony Wilson• Deputy technical manager and

developer

Linda Cornwall (0.5)• Documentation and Quality

Rob Byrom• Developer

Martin Craig• Developer

Steve Hicks• Developer

John Walk• Developer

A.N.Other• Integrator, sysadmin and developer

• Based at QMUL Roney Cordenonsi

• Developer

New (and started)

New (and started)

Short listing

GRIDPP10 Meeting at CERN, 3 June 2004 - 7

Unified approach to information and monitoring

• Provides information on both: The Grid itself (resources and services)

• Mainly for the other middleware packages• The user may query it to monitor the status of the Grid

Grid applications• For users

• Applied to both information and monitoring All tuples have time stamp

• Basis for other capabilities in the Grid Problem determination Resource usage Logging of information Application monitoring

GRIDPP10 Meeting at CERN, 3 June 2004 - 8

Mediator service

Registry service

R-GMA

• A Relational implementation of GMA (Grid Monitoring Architecture from GGF)

• Powerful data model and query language

• All data modelled as tables• SQL can express most queries in

one expression

• Creates impression that you have one RDBMS per VO

Information Producer

Information Consumer

Producer service

Schema serviceConsumer service

R-GMA services

GRIDPP10 Meeting at CERN, 3 June 2004 - 9

R-GMA Relational Data Model

• Producers announce: SQL “CREATE TABLE”• publish: SQL “INSERT”• Consumers collect: SQL “SELECT”

• Some Producers (and the Registry and Schema) make use of RDBMS as appropriate – but what is central is the relational model.

• Some producers make use of an in-memory data structure (for performance)

• Not a general distributed RDBMS system, but a way to use the relational model in a distributed environment where global consistency is not important

GRIDPP10 Meeting at CERN, 3 June 2004 - 10

LCG Installation of R-GMA

• Initially R-GMA will be added to LCG-2 to allow job accounting and job monitoring as requested by some experiments, it will *not* be used as an information system

• Following successful deployment at a number of sites in the UK and elsewhere, we have produced RPMs and instructions for adding R-GMA to LCG-2

• Current status in LCG-2 RPM’s delivered 21 May. These include fixes requested by LCG Testing by Piera Bettini (LCG) resumes 1 June. R-GMA is expected to be approved as an LCG-2 add-on by 15 June

GRIDPP10 Meeting at CERN, 3 June 2004 - 11

JRA1: Milestones and Deliverables for the first year

MonthDeliverables

&Milestones

Item Lead Partner

M03 MJRA1.1 Tools for middleware engineering and integration deployed CERN

M03 DJRA1.1 (Document) Architecture and Planning (Release 1) CERN

M03 MJRA1.2 Software cluster development and testing infrastructure available CERN

M05 MJRA1.3 Integration and testing infrastructure in place including test plans (Release 1)

CERN

M05 DJRA1.2 (Document) Design of grid services (Release 1) CERN

M09M09 MJRA1.4MJRA1.4 Software for the Release Candidate 1Software for the Release Candidate 1 CERNCERN

M10 MJRA1.5 Integrated Release Candidate 1 enters testing and validation period (Release 1)

CERN

M12 DJRA1.3 (Software) Software and associated documentation (Release 1) CERN

GRIDPP10 Meeting at CERN, 3 June 2004 - 12

The challenge

• We need to re-engineer existing code, and introduce new functionality while at the same time providing continuously improved frequent releases and all without breaking backwards compatibility.

GRIDPP10 Meeting at CERN, 3 June 2004 - 13

Achievements so far

• New API implemented (partially) as wrapper round the old one for the EGEE MW prototype We value feedback on the new API

• Registry replication completed and tested on our (UK) cluster testbed

• The new API includes authorisation Not complete yet No implementation yet

• Web service prototypes written for all components• Extensive UML diagrams for new design

Once design agreed we will refactor code towards it Many potential simplifications have been identified Outline plan agreed for Schema replication

GRIDPP10 Meeting at CERN, 3 June 2004 - 14

Deliverables up to end of August

• May 31 Detailed draft specification of our software On time (John Walk)

• June 30 code will be moved to the new repository, suitably restructured for

the new SCM and build system.

• July 31 Finalise Java wrapper API Complete design of new services

• August 31 The C,C++ and Python APIs will finalised

GRIDPP10 Meeting at CERN, 3 June 2004 - 15

Up to the end of December

• Sept 30 A Web services implementation release will be produced

• Oct 31 Multi VO support will be released

• Nov 30 The web services themselves will be ported to Windows We will make it very easy to build and distribute the system in

source form such that it can be readily built and installed on new platforms.

Replication of the schema and registry will be complete

• Dec 20 A complete XP port will be ready Some more improvements to the mediator and to the SQL parser

GRIDPP10 Meeting at CERN, 3 June 2004 - 16

Over the year

• During the year: Authorization code will be developed but it is not yet clear how

much of it will be ready for RC1. We will also start to look at interfacing to displays and other

information and monitoring systems by working with other clusters as appropriate understand

requirements for the R-GMA role in Accounting, Job provenance and Logging & bookkeeping.

GRIDPP10 Meeting at CERN, 3 June 2004 - 17

And finally

• Note that after August 31st there will be no external changes to the software so we will not be under pressure to release code before it is ready (improved quality).

• Delivering a working system is our top priority, so we may release less improvements than planned by Dec 20th.