INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org An introduction to EGEE Guy Warner NeSC Edinburgh
Jan 11, 2016
INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
An introduction to EGEE
Guy WarnerNeSC Edinburgh
An introduction to EGEE, Towards e-Research, St. Andrews 2
Enabling Grids for E-sciencE
INFSO-RI-508833
Acknowledgements
This presentation includes slides and information from many colleagues in EGEE, especially from the 1st project review in February 2005, including :– Fabrizio Gagliardi (1st Review) – Bob Jones (UK AHM 2004 talk)– Ian Bird– Frédéric Hemmer– Roberto Barbera
An introduction to EGEE, Towards e-Research, St. Andrews 3
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE
• Goals• Organisation• Activities and status• FAQ
An introduction to EGEE, Towards e-Research, St. Andrews 4
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE
• Goals• Organisation• Activities and status• FAQ
An introduction to EGEE, Towards e-Research, St. Andrews 5
Enabling Grids for E-sciencE
INFSO-RI-508833
EGEE – towards e-infrastructure
EGEE is building a large-scale production grid service to:
• Underpin research, technology and public service
• Link with and build on national, regional and international initiatives
• Foster international cooperation both in the creation and the use of the e-infrastructure
Network infrastructure& Resource
centres
Op
era
tio
ns
, S
up
po
rt a
nd
tr
ain
ing
Collaboration
Pan-European Grid
An introduction to EGEE, Towards e-Research, St. Andrews 6
Enabling Grids for E-sciencE
INFSO-RI-508833
Background
• By 2003:– Grid technology shown to be viable– Large amount of functional middleware – …thanks to:
FP5 : DataGrid, DataTAG, CrossGrid, etc… USA: VDT, Globus, Condor, etc. … and others
• Next step - major production infrastructure– EGEE was proposed to the EU in 2003
• 2 year project began in April 2004, with a 4-year vision.
An introduction to EGEE, Towards e-Research, St. Andrews 9
Enabling Grids for E-sciencE
INFSO-RI-508833
In the first 2 years EGEE will
• Establish production quality sustained Grid services – 3000 users from at least 5 disciplines– integrate 50 sites into a common
infrastructure– offer 5 Petabytes (1015) storage
• Demonstrate a viable general process to bring other scientific communities on board
• Propose a second phase in mid 2005 to take over EGEE in early 2006
Pilot Added
An introduction to EGEE, Towards e-Research, St. Andrews 10
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE
• Goals• Organisation• Activities and status• FAQ
An introduction to EGEE, Towards e-Research, St. Andrews 11
Enabling Grids for E-sciencE
INFSO-RI-508833
EGEE Organisation
• 70 leading institutions in 27 countries, federated in regional Grids
• ~32 M Euros EU funding for first 2 years starting April 2004 (matching funds from partners)
• Leveraging national and regional gridactivities
• Promoting partnershipoutside EU
An introduction to EGEE, Towards e-Research, St. Andrews 12
Enabling Grids for E-sciencE
INFSO-RI-508833
Activities Definition
• Network Activities– NA1: Project Management– NA2: Dissemination and Outreach– NA3: User Training and Induction– NA4: Application Identification and Support– NA5: Policy and International Cooperation
• Service Activities– SA1: Grid Support, Operation and Management– SA2: Network Resource Provision
• Joint Research Activities– JRA1: Middleware Reengineering + Integration– JRA2: Quality Assurance– JRA3: Security– JRA4: Network Services Development
Emphasis in EGEE is on operating a productiongrid and supporting the end-users
An introduction to EGEE, Towards e-Research, St. Andrews 13
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE • Goals• Organisation• Activities and status
JR activities: Middleware Re-engineering Operations (Human) Networking
• FAQ
An introduction to EGEE, Towards e-Research, St. Andrews 14
Enabling Grids for E-sciencE
INFSO-RI-508833
LCG and EGEE
• EGEE committed to “hit the ground running” in the proposal
• Current service (“LCG-2”) based on work done in LCG
• EGEE profits from the resources - no funded computing/data resources in EGEE
• LCG obtains additional production and operation efforts
LCG : Large Hadron Collider Compute Grid
An introduction to EGEE, Towards e-Research, St. Andrews 15
Enabling Grids for E-sciencE
INFSO-RI-508833
Current production grid: LCG-2
Computing cluster Network resources Data storage
Operating system Local schedulerFile system
User access SecurityData transferInformation schema
Workload management Data managementApp monitoring system
User interfaces Applications
Hardware
System software
“Basic” services
“Collective” services
Application level services
HPSS, CASTOR…HPSS, CASTOR…
RedHat LinuxRedHat Linux NFS, …NFS, … PBS, Condor, LSF,…PBS, Condor, LSF,…
VDT (Condor, Globus, GLUE)VDT (Condor, Globus, GLUE)
EU DataGridEU DataGrid
Information system
An introduction to EGEE, Towards e-Research, St. Andrews 16
Enabling Grids for E-sciencE
INFSO-RI-508833
Current production m’ware: LCG-2
ReplicaReplicaCatalogueCatalogue
Logging &Logging &Book-keepingBook-keeping
ResourceResourceBrokerBroker
StorageStorageElementElement
ComputingComputingElementElement
Information Information ServiceService
Job Status
DataSets info
Author.&Authen.
Job S
ub
mit
Even
t
Job
Qu
ery
Job
Stat
us
Input “sandbox”
Input “sandbox” + Broker Info
Output “sandbox”
Output “sandbox”
Pu
blis
h
SE & CE info
““User User interface”interface”
An introduction to EGEE, Towards e-Research, St. Andrews 17
Enabling Grids for E-sciencE
INFSO-RI-508833
• Co-existence with deployed infrastructure– Co-existence with LCG-2 and OSG
(US) are essential for the EGEE Grid services
• Site autonomy– Reduce dependence on ‘global,
central’ services• Open source license
gLite: Guiding Principles
• Service oriented approach– Allow for multiple interoperable
implementations• Lightweight (existing) services
– Easily and quickly deployable– Use existing services where
possible Condor, EDG, Globus, LCG, …
• Portable– Being built on Scientific Linux and
Windows• Security
– Sites and Applications• Performance/Scalability &
Resilience/Fault Tolerance– Comparable to deployed
infrastructure
EDGVDT . . .
LCG . . .AliEn
An introduction to EGEE, Towards e-Research, St. Andrews 18
Enabling Grids for E-sciencE
INFSO-RI-508833
gLite Services for Release 1
Grid AccessService
API
Access Services
JobProvenance
Job Management Services
ComputingElement
WorkloadManagement
PackageManager
MetadataCatalog
Data Services
StorageElement
DataManagement
File & ReplicaCatalog
Authorization
Security Services
AuthenticationAuditing
Information &Monitoring
Information & Monitoring Services
Application
Monitoring
Site Proxy
Accounting
JRA3 UK
CERN IT/CZ
Focus on key services
An introduction to EGEE, Towards e-Research, St. Andrews 19
Enabling Grids for E-sciencE
INFSO-RI-508833
GENIUS Portals for GILDA
Running LCG2 Middlewarehttps://grid-demo.ct.infn.it
Running new gLite Middlewarehttps://glite-demo.ct.infn.it
Portal simplifies use of a Grid (and hides
changes in middleware from users)
For new application communities and for training
Grid Demonstrator (open access) – two versions
An introduction to EGEE, Towards e-Research, St. Andrews 24
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE
• Goals• Organisation• Activities and status
JR activities Operations (Human) Networking
• FAQ
An introduction to EGEE, Towards e-Research, St. Andrews 25
Enabling Grids for E-sciencE
INFSO-RI-508833
Operations - Introduction
• Strategy has been to – simplify as far as possible what is deployed, and to make that
robust and useable. – In parallel construct the essential infrastructure needed to
operate and maintain a grid infrastructure in a sustainable way.
• Current service based on work done in LCG – culminating in the current service (“LCG-2”)– Now at the point where in parallel we need to deploy and
understand gLite – whilst maintaining a reliable production service.
An introduction to EGEE, Towards e-Research, St. Andrews 26
Enabling Grids for E-sciencE
INFSO-RI-508833
Country providing resourcesCountry anticipating joining
In LCG-2: 132 sites, 30 countries 16,644 cpu ~4 PB storage
Includes non-EGEE sites:• 9 countries• 18 sites
Computing Resources: April 2005
An introduction to EGEE, Towards e-Research, St. Andrews 27
Enabling Grids for E-sciencE
INFSO-RI-508833
SA1 – Operations Structure
• Operations Management Centre (OMC):– At CERN – coordination etc
• Core Infrastructure Centres (CIC)– Manage daily grid operations –
oversight, troubleshooting– Run essential infrastructure services– Provide 2nd level support to ROCs– UK/I, Fr, It, CERN, + Russia (M12)– Taipei also run a CIC
• Regional Operations Centres (ROC)– Act as front-line support for user and
operations issues– Provide local knowledge and
adaptations– One in each region – many distributed
• User Support Centre (GGUS)– In FZK – manage PTS – provide single
point of contact (service desk)– Not foreseen as such in TA, but need is
clear
An introduction to EGEE, Towards e-Research, St. Andrews 29
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE
• Goals• Organisation• Activities and status
JR activities Operations (Human) Networking
• FAQ
An introduction to EGEE, Towards e-Research, St. Andrews 30
Enabling Grids for E-sciencE
INFSO-RI-508833
(Human) Networking Activities
• Dissemination and Outreach: 5% of EGEE budget– Dissemination – to actively promote and raise awareness of the EGEE
project http://www.eu-egee.org – Outreach – to identify and contact potential new user communities
• Training and Induction: 4% of EGEE budget– 22 partners, led by NeSC– Induction – to introduce and orient - users and members– Training – to create, collate, make available and deliver material and
courses
• Application Identification and Support– Process for selecting new application areas– Supports selected VO’s in porting applications
• International cooperation
An introduction to EGEE, Towards e-Research, St. Andrews 31
Enabling Grids for E-sciencE
INFSO-RI-508833
Planned courses and repository
•The EGEE training material is being made available to the whole community.
•EGEE repository allows flexible searches of the available material... Take a look!!!
•Courses and material via…
http://egee.nesc.ac.uk
An introduction to EGEE, Towards e-Research, St. Andrews 32
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE • Goals• Organisation• Activities and status
Middleware Re-engineering Operations (Human) Networking
• Supporting application communities
• FAQ
An introduction to EGEE, Towards e-Research, St. Andrews 33
Enabling Grids for E-sciencE
INFSO-RI-508833
Application communities and EGEE
• LCG and Bio-informatics from day 1• New application communities are selected
by the EGEE Generic Applications Advisory Panel
– See: EGEE web site (NA4 activity) and also http://agenda.cern.ch/age?a042351
• Selected are:– Computational chemistry
– Earth sciences
– Earth observation
– Astrophysics
• Also working with DILIGENT:– Virtual digital data libraries
• GILDA grid for new applications and testing before migrate to production grid
An introduction to EGEE, Towards e-Research, St. Andrews 34
Enabling Grids for E-sciencE
INFSO-RI-508833
EGEE pilot application: BioMedical
• BioMedical– Bioinformatics (gene/proteome databases
distributions)– Interactive application (human supervision
or simulation)– Security/privacy constraints
Heterogeneous data formats - Frequent data updates - Complex data sets - Long term archiving
• BioMed applications deployed – GATE - Geant4 Application for Tomographic
Emission– GPS@ - genomic web portal – CDSS - Clinical Decision Support System
http://egee-na4.ct.infn.it/biomed/applications.html
An introduction to EGEE, Towards e-Research, St. Andrews 35
Enabling Grids for E-sciencE
INFSO-RI-508833
Geophysics Applications
Seismic processing Generic Platform:
- Based on Geocluster, an industrial application – to be a starter of the core member VO.
- Include several standard tools for signal processing, simulation and inversion.
- Opened: any user can write new algorithms in new modules (shared or not)
- Free for academic research
-Controlled by license keys (opportunity to explore license issue at a grid level)
- initial partners F, CH, UK, Russia, Norway
An introduction to EGEE, Towards e-Research, St. Andrews 36
Enabling Grids for E-sciencE
INFSO-RI-508833
The MAGIC telescope
• Largest Imaging Air Cherenkov Telescope (17 m mirror dish)
• Located on Canary Island La Palma (@ 2200 m asl)
• Lowest energy threshold ever obtained with a Cherenkov telescope
Aim: detect –ray sources in the unexplored energy range: 30 (10)-> 300 GeV
An introduction to EGEE, Towards e-Research, St. Andrews 37
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE • Goals• Organisation• Activities and status
Middleware Re-engineering Operations (Human) Networking
• Policy and international relations
• FAQ
An introduction to EGEE, Towards e-Research, St. Andrews 38
Enabling Grids for E-sciencE
INFSO-RI-508833
Policy and international relations
• Further releases of the eIRG white paper on grid policy– Closer relations with the European Strategy Forum on Research
Infrastructures (ESFRI)
• Participation in EU concertation meetings– Contribute to organisation– Leadership of key working groups (e.g. security via JRA3)
• EU synergy roadmap revision– With SEE-GRID and DEISA
• Continue work with OASIS, GGF etc
An introduction to EGEE, Towards e-Research, St. Andrews 39
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE • Goals• Organisation• Activities and status
Middleware Re-engineering Operations (Human) Networking
• Supporting application communities
• FAQ
An introduction to EGEE, Towards e-Research, St. Andrews 40
Enabling Grids for E-sciencE
INFSO-RI-508833
Links to industry?
• EGEE Industry Forum– raise awareness of the project in
industry to encourage industrial participation in the project
– foster direct contact of the project partners with industry
– ensure that the project can benefit from practical experience of industrial applications
• For more info: http://public.eu-egee.org/industry/
An introduction to EGEE, Towards e-Research, St. Andrews 41
Enabling Grids for E-sciencE
INFSO-RI-508833
Open Source Software License
• The existing EGEE grid middleware (LCG-2) is distributed under an Open Source License developed by EU DataGrid project
– Derived from modified BSD - no restriction on usage (academic or commercial) beyond acknowledgement
– Approved by Open Source Initiative (OSI)
• Same approach for new middleware (gLite)
– New license agreed by partners is derived from the EDG license and takes into account feedback from the World Intellectual Property Office (WIPO)
An introduction to EGEE, Towards e-Research, St. Andrews 42
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
EGEE: Enabling Grids for E-sciencE • Goals• Organisation• Activities and status
Middleware Re-engineering Operations (Human) Networking
• Supporting application communities
• FAQ• Summary
An introduction to EGEE, Towards e-Research, St. Andrews 43
Enabling Grids for E-sciencE
INFSO-RI-508833
EGEE Summary
• EGEE runs the largest and probably the only multi-disciplinary production grid infrastructure
• A process is in place for migrating new applications to the EGEE infrastructure
• A training programme is established
• gLite “next generation” middleware released
• Plans for a follow-on phase are being developed
An introduction to EGEE, Towards e-Research, St. Andrews 44
Enabling Grids for E-sciencE
INFSO-RI-508833
EGEE www.eu-egee.orggLite www.glite.org
LCG lcg.web.cern.ch/LCG/
The Grid Cafe www.gridcafe.org•More EU sites:
•http://www.cordis.lu/ist/grids/fp6_grid_projects.htm•http://www.gridstart.org/concertation_mtg.shtml•“e-Infrastructures Reflection Group http://www.e-irg.org
•NeSC www.nesc.ac.uk
Further Information
INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
Digital libraries
An introduction to EGEE, Towards e-Research, St. Andrews 46
Enabling Grids for E-sciencE
INFSO-RI-508833
• Grids and digital libraries• DILIGENT and DELOS• DILIGENT overview
An introduction to EGEE, Towards e-Research, St. Andrews 47
Enabling Grids for E-sciencE
INFSO-RI-508833
Curation, discovery, re-use of knowledge
e-Research
The expanding horizons of grids
e-Science
An introduction to EGEE, Towards e-Research, St. Andrews 48
Enabling Grids for E-sciencE
INFSO-RI-508833
Grids: will support more than e-Research!
• Virtual Digital Libraries needed for research as well as learning
• Note also: Centrality of curation, preservation– Under-recognised by
many researchers– Hence the Digital
Curation Centre•
E-l
earn
ing
•D
igit
al l
ibra
ries
•E
-res
earc
h
• e-Infrastructure
• AAA ServicesDiagram from a slide by the UK’s JISC
An introduction to EGEE, Towards e-Research, St. Andrews 49
Enabling Grids for E-sciencE
INFSO-RI-508833
Building on e-Infrastructure in 4-D
• Across geographical distance – networks– Allow remote resources to be accessed– SuperJANET, UKLight, GEANT, …
• Across admin domains – grids– Allow resources in a VO to be shared: virtual computing
• Across time – data (knowledge) curation– Provides for future research and education– Digital Curation Centre (http://www.dcc.ac.uk/)
• Across disciplines – semantics– How interfaces to services can be understood via a shared
ontology, so services can be discovered and used outside their originating community
An introduction to EGEE, Towards e-Research, St. Andrews 50
Enabling Grids for E-sciencE
INFSO-RI-508833
Current “Grid-aware” EU projects for Digital libraries
• DELOS– Network of excellence exploring technologies for future digital
libraries “Future digital libraries should enable any citizen to access human knowledge any time and anywhere, in a friendly, multi-modal, efficient, and effective way”
– http://www.delos.info/
• DILIGENT– a DIgital Library Infrastructure on Grid-ENabled Technology that
“will allow members of dynamic virtual research organizations to create on-demand transient digital libraries based on shared computing, storage, multimedia, multi-type content and application resources”
– http://www.diligentproject.org/
An introduction to EGEE, Towards e-Research, St. Andrews 51
Enabling Grids for E-sciencE
INFSO-RI-508833
DILIGENT
• A testbed DIgital Library Infrastructure on Grid ENabled Technology
Start date September 1, 2004
Duration 36 months
Effort 1024 person months
Cost 8,9M EUR
European Community Contribution 6,3M EUR
Technical and Scientific Coordinator Consiglio Nazionale delle Ricerche-I.S.T.I. (Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo"), of Pisa, Italy
Number of partners: 14
Donatella Castelli, Scientific Co-ordinator for DILIGENT 3rd EGEE Conference - Athens, 18th April 2005
52
DLCreation service
Service C
Service B
Service A
Service D
Service E
DILIGENT DL infrastructure
simulation
Speech recognition
Feature extraction
3D processing
ConsumersConsumers ProducersProducers
An introduction to EGEE, Towards e-Research, St. Andrews 53
Enabling Grids for E-sciencE
INFSO-RI-508833
DILIGENT
• Two complementary real-life application scenarios will serve to demonstrate and validate the test-bed: one from the culture heritage domain and the other from the environmental e-Science domain
• Next activities:– Set up of a DILIGENT development infrastructure
Using gLite with additional higher level services
– Development of a first simple prototype of the Diligent infrastructure for supporting DLs of “Live documents”
Donatella Castelli, Scientific Co-ordinator for DILIGENT 3rd EGEE Conference - Athens, 18th April 2005
54
Live DocumentsToday is 18 April, 2005 and the user wants to update the report s/he wrote in January with the most recent products
International Report on Mediterranean Sea Chlorophyll Distribution during year 2003
1. Scientific and Societal Concerns Any scheme to monitor the ocean biota and their environment must strive to address the major scientific and societal concerns of the day pertaining to marine life. This section summarises some major concerns that emerged during discussions at the meeting. Many other concerns could have been included, but space precludes a complete listing of concerns. 1.1. Biodiversity and Conservation Marine biodiversity is not easy to assess and is generally poorly known. There are many complicating factors, including a three-dimensional, fluid, mobile environment, its vastness, and its challenging depths. Away from shore, primary producers and primary grazers are usually small, drifting forms that undergo spatial variability and seasonal changes. The larger invertebrate grazers have a range of life history stages, often with planktonic and benthic phases. Many large animals are migratory. Ocean habitats can be linked by the dispersal of planktonic larvae, and in this way, the systems can be interconnected even at a distance.
Jan – Apr 2003 Finally, the higher-order diversity of life is much greater in the oceans than in terrestrial systems—there are 13 unique phyla in the oceans and only one on land. Marine biodiversity is essentially the evolutionary history of life. In general, long-term environmental stability seems to increase biodiversity and, conversely, global climate change can be expected to decrease it.
International Report on Mediterranean Sea Chlorophyll Distribution during year 2003
1. Scientific and Societal Concerns Any scheme to monitor the ocean biota and their environment must strive to address the major scientific and societal concerns of the day pertaining to marine life. This section summarises some major concerns that emerged during discussions at the meeting. Many other concerns could have been included, but space precludes a complete listing of concerns. 1.1. Biodiversity and Conservation Marine biodiversity is not easy to assess and is generally poorly known. There are many complicating factors, including a three-dimensional, fluid, mobile environment, its vastness, and its challenging depths. Away from shore, primary producers and primary grazers are usually small, drifting forms that undergo spatial variability and seasonal changes. The larger invertebrate grazers have a range of life history stages, often with planktonic and benthic phases. Many large animals are migratory. Ocean habitats can be linked by the dispersal of planktonic larvae, and in this way, the systems can be interconnected even at a distance.
Jan - Apr 2003
Finally, the higher-order diversity of life is much greater in the oceans than in terrestrial systems—there are 13 unique phyla in the oceans and only one on land. Marine biodiversity is essentially the evolutionary history of life. In general, long-term environmental stability seems to increase biodiversity and, conversely, global climate change can be expected to decrease it.
Check update model/ parameters defined in the report template.Submit request to get the most recent chlorophyll product
The new product must replace the old one and the caption must become coherent with the new image
Donatella Castelli, Scientific Co-ordinator for DILIGENT 3rd EGEE Conference - Athens, 18th April 2005
55
Contacts www.diligentproject.org
Donatella Castelli (CNR-ISTI, scientific co-ordinator) [email protected]
Jessica Michael (ERCIM, administrative co-ordinator) [email protected]