JRA1 Activity Feedback Frédéric Hemmer EGEE Middleware Manager and the JRA1 team

JRA1 Activity Feedback

Frédéric Hemmer

EGEE Middleware Managerand the JRA1 team

EGEE is a project funded by the European Union under contract IST-2003-508833

www.eu-egee.org

EGEE 1st Conference, 22-Apr-2004

EGEE 1st Conference, Cork, April 22, 2004 - 2

Outline

• Status

• What has changed in Cork

• Important steps until next conference in November 2004 Or rather end of 2004 – MJRA1.4: Release Candidate 1

• Summary

EGEE 1st Conference, Cork, April 22, 2004 - 3

STATUS

EGEE 1st Conference, Cork, April 22, 2004 - 4

Software Clusters

• Tools, Testing & Integration (CERN) clusters• Development clusters:

UK CERN IT/CZ Nordic

• Clusters have a reasonable sized (distributed) development testbed Taken over from EDG Nordic cluster to be finalized Milestone MJRA1.2 (PM3) almost reached now

• Link with Integration & Tools clusters established• Clusters up and running!

• Nordic (security) cluster JRA3

EGEE 1st Conference, Cork, April 22, 2004 - 5

Design Team

• Formed in December 2003• Current members:

UK: Steve Fisher IT/CZ: Francesco Prelz Nordic: David Groep VDT: Miron Livny CERN: Predrag Buncic, Peter Kunszt,

Frederic Hemmer, Erwin Laure

• Started service design based on component breakdown defined by the LCG ARDA RTAG

• Leverage experiences and existing components from AliEn, VDT, and EDG.

• A working document• Overall design & API’s• https://edms.cern.ch/document/458972

• Basis for architecture (DJRA1.1) and design (DJRA1.2) document

EGEE 1st Conference, Cork, April 22, 2004 - 6

Towards a prototype

• Focus on key services; exploit existing components• Initially an ad-hoc installation at Cern and Wisconsin• Aim to have first instance ready by end of April

Open only to a small user community Expect frequent changes (also API changes) based on user

feedback and integration of further services• Enter a rapid feedback cycle

Continue with the design of remaining services Enrich/harden existing services based on early user/operations

feedback

• Access service: AliEn shell, APIs

• Information & Monitoring: R-GMA

• CE: AliEn CE, Globus gatekeeper,

CondorG

• Security: VOMS, myProxy

• Workload mgmt: AliEn task queue;

• SE: SRM (Castor), GridFTP, GFAL, aoid

• File Transfer Service: AliEn FTD

• File and Replica Catalog: AliEn File Catalog, RLS

Initial prototype componentsfor April’04

To be extended/ changed (e.g. WMS)

This is not a release!

It’s purely an ad-hoc installation

EGEE 1st Conference, Cork, April 22, 2004 - 7

Integration

• A master Software Configuration Plan is being finalized now

• It contains basic principles and rules about the various areas of SCM and Integration (version control, release management, build systems, bug tracking, etc)

• Compliant with internationally agreed standards (ISO 10007-2003 E, IEEE SCM Guidelines series)

• Most EGEE stakeholders have already been involved in the process to make sure everybody is aware of, contributes to and uses the plan

• An EGEE JRA1 Developer's Guide will follow shortly in collaboration with JRA2 (Quality Assurance) based on the SCM Plan

• It is of paramount importance to deliver the plan and guide as early as possible in the project lifetime

EGEE 1st Conference, Cork, April 22, 2004 - 8

SCM Contents

• SCM and Integration Implementation

• Configuration and Version Control

• Build Systems

• Release Process

• Other Configuration and Change Control Procedures

• Solid steps towards MJRA1.3 (PM5)

EGEE 1st Conference, Cork, April 22, 2004 - 9

Testing

• The 3 initial testing sites are CERN, NIKHEF and RAL More sites can join the testing activity at a later stage ! Must fulfil site requirements

• Testing activities will be driven by the test plan document

• Test plan being developed based on user requirements documents: Application requirements from NA4: HEPCAL I&II, AWG documents, Bio-

informatics requirements documents from EDG Deployment requirements being discussed with SA1 ARDA working document for core Grid services Security: work with JRA3 to design and plan security testing

• The test plan is a living document: it will evolve to remain consistent with the evolution of the software

• Coordination with NA4 testing and external groups (e.g. Globus) established

• Solid steps towards MJRA1.3 (PM5)

EGEE 1st Conference, Cork, April 22, 2004 - 10

Resources indicators

Partner FTE1 MM Assigned2 To Hire2 FTE from TA2 DeviationCERN 32.2 386.4 31.2 1.0 32 97%RAL (CCLRC) 9.0 108 8 1.0 8 89%INFN 16.5 198 14.5 2.0 16 88%DATAMAT 5.3 63 4 1.3 6 76%CESNET 4.4 52.8 4.4 0.0 4 100%CNRS 2.0 24 1 0.0 2 50%

Total effort 69.4 832.2 63.1 5.3 68 91%

Resource Plan indicatorHired or assignated up now 63.1Total FTE from TA 68.0Recruitment indicator 92.8%Still to hire 4.9

1: People planned & assigned to work2: Head count

Middleware Resources Summary

EGEE 1st Conference, Cork, April 22, 2004 - 11

Execution Plan

• Work Breakdown Structure defined

• Document needs to be completed

EGEE 1st Conference, Cork, April 22, 2004 - 12

Achievements in Cork

EGEE 1st Conference, Cork, April 22, 2004 - 13

Clusters Responsibilities

Nordic

CE

RN

CE

RN

UK

UK

IT/C

Z

IT/CZ

EGEE 1st Conference, Cork, April 22, 2004 - 14

Planning

• Evolution of the prototype Envisaged status at end of 2004:

• Key services need to fulfill all requirements (application, operation, quality, security, …) and form a deployable release

• Remaining services available as prototype Need to develop a roadmap

• Incremental changes to prototype (where possible)• Early user feedback through ARDA and early deployment on SA1 pre-

production service• Detailed release plan being planned

Converge prototype work with integration & testing activities• Need to get rolling now!• First components will start using SCM in May

EGEE 1st Conference, Cork, April 22, 2004 - 15

Guiding Principles

• Lightweight (existing) services Easily and quickly deployable

• Interoperability Allow for multiple implementations

• Resilience and Fault Tolerance

• Co-existence with deployed infrastructure Run as an application

• Service oriented approach Follow WSRF standardization No mature WSRF implementations exist to date, hence: start with

plain WS – WSRF compliance is not an immediate goal Review situation end 2004

EGEE 1st Conference, Cork, April 22, 2004 - 16

JRA1/SA1 - Process description

• No official delivery of requirements from SA1 to JRA1 stated in the TA

• The definition, discussion and agreement of the requirements has already started, done through dedicated meetings

• This is an ongoing process: Not all the requirements defined yet Set of requirements agreed, need basic

agreement to start working! But can be reviewed at any time there is a valid reason for it

EGEE 1st Conference, Cork, April 22, 2004 - 17

JRA1/SA1 - Requirements

1. Middleware delivery to SA12. Release management 3. Deployment scenarios 4. Middleware configuration

JRA1 will provide a standard set of configuration files and documentation with examples that SA1 can use to design tools. Format to be agreed between SA1-JRA1

It is the responsibility of SA1 to provide configuration tools to the sites

5. Enforcement of the procedures 6. Platforms to support

• Primary platform: Red Hat Enterprise 3.0, gcc 3.2.3 and icc8 compilers (both 32 and 64-bits) .

• Secondary platform: Windows (XP/2003), vc++ 7.1 compiler (both 32 and 64-bits)

7. Versions for compilers, libraries, third party software 8. Programming languages 9. Packaging and software distribution 10. Others

• Sites must be allowed to organize the network as they wish, internal or external connectivity, NAT, firewall, etc, all must be possible, no special constraints. WNs must not require Outgoing IP connectivity; Not inbound connectivity either.

EGEE 1st Conference, Cork, April 22, 2004 - 18

Related documents

• Requirement table stored at:

https://edms.cern.ch/document/456865

• Meeting minutes stored at:

https://edms.cern.ch/document/451069

EGEE 1st Conference, Cork, April 22, 2004 - 19

JRA1/JRA3

• A lot of progress has been achieved here Security Group formed, JRA1 members identified First meeting scheduled on May 5-6, 2004

• GAP analysis planned by then VOMS Administration support clarified

• Handled by JRA3 Issue: VOMS effort reporting

EGEE 1st Conference, Cork, April 22, 2004 - 20

JRA1/JRA4

• SCM plan presented and discussed

• More discussions on which components of JRA4 will be required in the overall architecture/design need to take place

EGEE 1st Conference, Cork, April 22, 2004 - 21

American Involvement in JRA1

• UWisc Miron Livny part of the design Team Condor Team actively involved in reengineering resource access

• In collaboration with Italian Cluster

• ISI Identification of potential contributions started (e.g. RLS)

• Focused discussions being planned

• Argonne Collaboration on Testing started Support for key Globus Components enhancements being

discussed

EGEE 1st Conference, Cork, April 22, 2004 - 22

March Issues related to other activities

• Security JRA3 Development must be part of JRA1 structure Security is not an orthogonal activity

• Same for JRA4• Interaction with SA1

Support Requirements and acceptance criteria gathering Definition of responsibilities

• Testing• Support

Packaging/Configuration/Distribution Mechanisms• American involvement still being clarified

• Issue: CNRS Involvement• JRA2 documents (standards, templates) need to be available early

And relevant to the complexity of the activity

EGEE 1st Conference, Cork, April 22, 2004 - 23

Important steps until next conference(rather: end of 2004 – MJRA1.4: 1st release candidate)

EGEE 1st Conference, Cork, April 22, 2004 - 24

Milestones and Deliverables for 2004

MonthDeliverables

&Milestones

ItemLead Partner

Status

M03 MJRA1.1 Tools for middleware engineering and integration deployed

CERN Most tools identified (SCM)Issue: CNRS person tbi

M03 DJRA1.1 (Document) Architecture and Planning (Release 1)

CERN Design document as starting point; writing organized for May

M03 MJRA1.2 Software cluster development and testing infrastructure available

CERN Done for CERN/UK/IT/CZNordic to be finalized

M05 MJRA1.3 Integration and testing infrastructure in place including test plans (Release 1)

CERN Sites identified; teams established;

M05 DJRA1.2 (Document) Design of grid services (Release 1)

CERN Design document as starting point

M09 MJRA1.4 Software for the Release Candidate 1 CERN Work on prototype started; will evolve into 1st release candidate

M08 – Amsterdam conference: Tech preview of release candidate 1 available

EGEE 1st Conference, Cork, April 22, 2004 - 25

JRA1 and other activities

• NA4 HEP: ARDA project started; ensures close relations between HEP and middleware Bio: activities with similar spirit needed – focused meeting tentatively being planned

for May

• SA1 Revision of requirements (platforms)

• JRA2 QAG started Monthly meeting established

• JRA3 Necessary structures established Focused Meeting in May

• JRA4 Architectural components required need to be clarified

• Other projects Potential drain of resources for dissemination activities

EGEE 1st Conference, Cork, April 22, 2004 - 26

Summary

• Activity progressing and on track (despite tight schedules)

• Technology Risk Will WS allow for all upcoming requirements? Divergence to standards

• Very productive conference Lively discussions in most sessions Unfortunately no time for a JRA1/NA4 session

• No amendments to TA foreseen

• Thanks to organizers

EGEE 1st Conference, Cork, April 22, 2004 - 27

Information Systems …

http://www.mylostbag.com/