Technical Overview

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Technical Overview

Balázs Kónya (Lund University)Technical Director

1st EMI Periodic ReviewBrussels, 22 June 2011

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Outline

EMI is about software, the mission of the project is to deliver software for DCIs

•The process: how the development is managed– Technical Management roles– Requirements– Development cycle

•The outcome: EMI software portfolio– EMI on the DCI landscape– EMI products and their functionality– High-level technical roadmap– Year 2 top priorities

22/06/2011 1st EMI Periodic Review 2

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Project Management

PTB in EMI


NA1, JRA1

JRA1

JRA1SA1

SA1

SA2

Software & Services

Requirements

Defines

ImplementsCertifies

Release Candidate

Process definition

Process monitoring

NA2NA2

NA1, NA2

Collaborations, exploitation, sustainability

Training

Dissemination & Out-Reach

In-Reach

NA1

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611Managing EMI development


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Technical Areas and PTs


ARC CE, UNICORE Services, gLite MPI , gLite Compute, etc

dCache, CERN Data, DGAS, StoRM ,etc

ARC Container, UNICORE Security, Cesnet Security, Argus, VOMS, etc

ARC Infosys, APEL client, DGAS Client, gLite Infosys, EMI Registry, etc

Product Teams

Dedicate teams of experts

Fully responsible for development, maintenance and

unit/system testing

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611Requirement management

• Sources of requirements– EGI-TCB: formal requests communicated via EGI Tracker – WLCG: informal requests communicated verbally or via email– Users: direct communication with PTs

• Handling of requirements– After an initial filtering requests are recorded in the EMI Req. Tracker

• Large ”background noise”: duplicates, out-of-scope, bugreports, etc...

– PTB assesses, categorizes and prioritizes requirements– Endorsed requests are translated into objectives, then to development

tasks• Adjusts and/or creates new objectives

– Continous process, though requirements may not have an immediate effect on workplan

• Special attention to synchronize with EGI/UMD cycles

• Everything is recorded, monitored and tracked:– https://savannah.cern.ch/task/?group=emi-req


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

From requirements to released products


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI on the DCI landscape


EMI

DNA1.4 - EMI Roadmap and DCI Collaborations

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611EMI software portfolio (1/2)

• Originates from ARC, dCache, gLite, UNICORE– not a complete union of the four

• Initial stack was defined by DNA1.3.1– consisted of 98 components– ”component table” was the first attempt to define ”what EMI is”

• Early consolidation during first year– iterative process via EMI 0 and EMI 1 release preparations

• Logical restructuring• Phase out (5 components)

• Current software portfolio (DNA1.3.2)– 58 products of different maturity levels– 54 products are released as part of Kebnekaise

• Became 51 after further merges

– 3 products under certification and 4 in early development– available on SL5/64


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611EMI software portfolio (2/2)

• Consists of services, clients and APIs/libraries• Integrated products that work together• EMI products contribute to the ”dynamic services” DCI

layer and offer the following functionalities:– Compute: job execution, parallel job, job scheduling – Data: file access, file transfer, metadata catalogue– Infrastructure: information publication, accounting

sensors, information discovery, monitoring probes, messaging backbone

– Security: authentication, authorization, attribute authority, attribute authority, credential management


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Evolution of the software stack

• Need to manage conflicting requirements:– Product and interface stability – Need for consolidation via portfolio cleanup and adoption of

common interfaces and libraries– Need to implement new features

• Effect on the software stack:– Established code base with little space for ”easy consolidation”

• emphasis is on hardening, cleanup

– Challenging Product phase out• transition path is necessary

– New products for consolidation• common libraries & services• for a transitional period will coexist with pre-EMI solution

– Feature requests to be implemented on top of existing products• avoid development from scratch, yet another set of prototypes


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

High-level Technical Roadmap


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Year 1 achievements


Find details in JRA1 presentation

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Year 2 top technical priorities (1/2)

• Compute:– EMI Execution Service: implementation of the agreed common

job management methods– GLUE2 support in compute clients

• Data:– Client-side GLUE2 support implementation– EMI Data Access Library design and implementation– Storage Element and catalogue synchronization

• Security:– Simplified management of security credentials (AAI)– EMI Authentication Library implementation– EMI delegation agreement


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Year 2 top technical priorities (2/2)

• Infrastructure:– EMI service registry implementation– Cloud strategy– Delivery of service monitoring via NAGIOS

• All: – Consolidation plans– Additional platforms

• Debian family and SL6

– Usability improvements • Command line parameters• Error messages


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Thank you

22/06/2011 161st EMI Periodic Review

EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Technical Documents

• Release planning, management:– https://twiki.cern.ch/twiki/pub/EMI/EMI1Schedule/kebnekaise-v.9.pdf– https://twiki.cern.ch/twiki/pub/EMI/EMI1Updates/

EMI_Release_Schedule_v0.4.pdf

• Release Tracker:– https://savannah.cern.ch/task/?group=emi-releases

• RfC (development task on a PT level): – http://bugzilla.nordugrid.org/show_bug.cgi?id=2394 (glue support for ARC CE)– https://savannah.cern.ch/bugs/?77525 (ARGUS)– https://savannah.cern.ch/bugs/?77527 (ARGUS)

• EMI Execution Service inteface: • http://cdsweb.cern.ch/record/1359908/files/EMI-ES-Specification_v1.0.odt

• Storage Accounting record:– http://cdsweb.cern.ch/record/1352472/files/StAR-EMI-tech-doc-final.doc


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Cloud position (DRAFT!)

• Presented at the EGI User Virtualization Workshop– https://www.egi.eu/indico/

conferenceDisplay.py?confId=415– https://www.egi.eu/indico/getFile.py/

access?contribId=9&resId=0&materialId=slides&confId=415


EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Amsterdam slides:What EMI can contribute with

• EMI develops/operates in the "service level”, not in the VM level• some of the EMI services can be useful on the VM-level as well:

– BDII as the "info service", storing info about VMMs, so on– a generic service registry which will enable the "hooking up" of all the

appliances– ARGUS, VOMS as authorization systems

• EMI will provide/contribute/build/test grid appliances and “landscape deployments” of EMI services– Landcape deployment example: a complete CE

• EMI will utilize VMs under/behind EMI services– e.g. Computing Elements with VM-based nodes

• EMI partners have been the driving force behind some of the “cloud-relevant” standards:– Extensions, profiles for GLUE, accounting

• Finally, EMI can offer lessons learnt in Grid – Our experience can help not to repeat the same mistakes

WP - Presenter Name - EMI First EC Review 22 June 2011 19

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Amsterdam slides: What is NOT in the scope of EMI

• Virtual Machine Management layer • Application catalogues• VM Image catalogues• Accounting systems (server side)

WP - Presenter Name - EMI First EC Review 22 June 2011 20

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611Understanding the near future...

12/05/2011 EGI User Virtualization Workshop 21