EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities David Wallom University of Oxford Chair, EGI Federated Cloud
Nov 29, 2014
www.egi.eu EGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.eu EGI-InSPIRE RI-261323
Using a Widely Distributed Federated Cloud System to Support Multiple
Disparate User Communities
David Wallom University of Oxford
Chair, EGI Federated Cloud
www.egi.eu EGI-InSPIRE RI-261323
Overview
• Why • What • Who
www.egi.eu EGI-InSPIRE RI-261323
Growth of Providers
Grid Limited academic resource providers
Cloud Diversity and scale of new resource providers
www.egi.eu EGI-InSPIRE RI-261323
Growth of Userbase
20k researchers 2M researchers!
Few related use cases
Many diverse use cases
www.egi.eu EGI-InSPIRE RI-261323
Hardware Hardware
Hardware Hardware
Hardware
Cloud Management Cloud Management
Cloud Management Cloud Management
Cloud Management
User Communities
User Communities
User Communities
Federated interfaces
Federated services
• Open Standards
• Technology independant
• Provider agnosticism
• Production integration and reuse
Principles of Federation
www.egi.eu EGI-InSPIRE RI-261323
EGI Cloud Infrastructure
6
EGI Core Platform
Federated AAI
Service Registry
Monitoring Accounting
EGI Cloud Infrastructure Platform
Instance Mgmt
Information Discovery
Storage Management
Cloud Management Stacks (OpenStack, OpenNebula, Synnefo, …)
Help and Support
Security Co-ordination
Training and Outreach
EGI C
olla
bora
tion
Tool
s
EGI A
pplic
atio
n D
B Im
age
Repo
sito
ry
EGI C
loud
Ser
vice
Mar
ketp
lace
Sustainable Business Models
User Community
GSI GLUE2
OCCI CDMI
SAM UR
OVF
Cloudinit
www.egi.eu EGI-InSPIRE RI-261323
Compatibility for technology providers
Cloud Mgmt. Fram. Fed. AAI Monitoring Accounting Img. Mgmt. OCCI CDMI
OpenStack Yes Yes* Yes Yes Yes Yes
OpenNebula Yes Yes* Yes Yes Yes Yes
Synnefo Yes Yes Yes - Yes -
Cloudstack - - - - Yes -
Emotive Yes Yes* - N/A N/A Yes
SlipStream Yes N/A N/A Yes Yes Yes*
Perun Yes Yes N/A N/A N/A N/A
7
Example documentation for a CMF:- https://wiki.egi.eu/wiki/Fedcloud-tf:ResourceProviders:OpenStack
www.egi.eu EGI-InSPIRE RI-261323
Geographical dispersion • 12 countries provide 19 certified
resources – Czech Republic, Germany, Greece,
Hungary, Italy, Macedonia, Poland, Slovakia, Spain, Sweden, Turkey, United Kingdom
• 2 countries currently integrating – France, Finland, Portugal, Croatia
• 5 countries interested – Bulgaria, Israel*, The Netherlands,
Switzerland
• Worldwide interest – South Africa* (SAGrid) – South Korea* (KISTI) – United States* (NIST, NSF Centres)
* Not shown on map
www.egi.eu EGI-InSPIRE RI-261323
Utilisation
9
217352 VMs launched across 25 clouds
www.egi.eu EGI-InSPIRE RI-261323
Federated Cloud Services
Federated IaaS and STaaS Cloud
EYR TC monthly meeting 10
Tier 1: Reliable
Infrastructure Cloud
Tier 4: Zero ICT
Infrastructures
Tier 3: Platform as a Service
Tier 2: General-purpose platform services
PaaS
PaaS
DB
aaS
Had
oop
aaS
VRE
Secure storage
Key
Mgm
t
Encr
ypti
on
ACL
mgm
t
Virtual eLaboratory
17 Feb 2014
www.egi.eu EGI-InSPIRE RI-261323
AppDB – Virtual Appliance catalogue
11
www.egi.eu EGI-InSPIRE RI-261323
EGI FedCloud Communities 9/2014
• Ecology – BioVeL: Biodiversity Virtual e-Laboratory
• Structural biology – WeNMR: a worldwide e-Infrastructure for NMR and structural biology
• Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb)
• Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for
volcano and earthquakes monitoring (Collaboration with Helix Nebula)
• Software Engineering – SCI-BUS: simulated environments for portal testing
• Software Engineering – DIRAC: deploying ready-to-use distributed computing systems
• Software Engineering – Catania Science Gateway Framework
• Musicology – Peachnote: dynamic analysis of musical scores
• Earth Observation – ENVRI: Common Operations of Environmental Research
infrastructures (collaboration with EISCAT3D)
• Geology – VERCE: Virtual Earthquake and seismology Research
• Ecology – LifeWatch: E-Science European Infrastructure for Biodiversity and Ecosystem
Research
• High Energy Physics – CERN ATLAS: ATLAS processing cluster via HelixNebula
More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users
12
www.egi.eu EGI-InSPIRE RI-261323
New EGI FedCloud Communities since launch
• Education – Cranfield University distributed systems course
• Cultural Heritage – DCH-RP management of preservation services in the cloud
• Hydrological Modelling – Running Hydrological models to support real time analysis
• Bioinformatics – ELIXIR execution of the Ensamble application in the Federated Cloud
environment
• Systems implementations – deployment of FTK developed tools and services and data
preservation
• Internet of Things – Smart Grid systems investigation
• Software Development – deployment of research PaaS
• RNA Sequencing – deployment of analysis engines in the cloud
• Physiological Modelling – Calibration, scenario mapping and development
More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users
13
www.egi.eu EGI-InSPIRE RI-261323
WeNMR e-Infrastructure (VirtualCing)
EGI Federated Cloud Infrastructure
…
NMR Protein DB
ToPoS
Virtual CING Virtual
CING Virtual Cing
Virtual CING Virtual
CING Virtual Cing
Virtual CING Virtual
CING Virtual Cing
OCCI client
Virtual Common Interface for NMR structure generation (VirtualCing), developed in WeNMR, has been deployed via the EGI Federated Cloud
14
www.egi.eu EGI-InSPIRE RI-261323
ESA HN Flagship on the EGI Federated Cloud
The European Space Agency deployed part of the ESA Helix Nebula Flagship into the EGI Federated Cloud:
1. Use SlipStream Infrastructure Broker to deploy a custom computing cluster on multiple Federated Cloud sites (CESNET, GRNET)
2. Connect the computing cluster to the ESA G-POD system and run SAR processing for volcano and earthquake monitoring
15
Automatic Cluster Deployment
EGI Federated Cloud
Computing Cluster
Cluster Head
Working Nodes
OCCI
Globus
Job submission
BLUE BOX
www.egi.eu EGI-InSPIRE RI-261323 16
EISCAT 3D PoC
EISCAT_3D is a three-dimensional imaging radar to be located in the northernmost parts of Europe, to study Earth atmosphere and its coupling to space.
The EISCAT 3D Proof of Concept is a PaaS built on top of the EGI Federated Cloud for Data Access and Dissemination, based on the ENVRI tools, the EGI Federated Cloud Infrastructure and Storage services and the current EISCAT data.
www.egi.eu EGI-InSPIRE RI-261323
Peachnote
10/9/2014 17
Peachnote is a music score search engine and analysis platform.
Hundreds of thousands of music scores are being digitized by libraries all over the world. In contrast to books, they generally remain inaccessible for content-based retrieval and algorithmic analysis. There is no analogue to Google Books for music scores, and no large corpora exists that can empower advanced analysis on music scores. Peachnote want to help change that providing visitors and researchers access to a massive amount of symbolic music data.
EGI Federated Cloud
OMR Worker
PDF Splitter
OMR Feeder
OMR Worker
…
www.egi.eu EGI-InSPIRE RI-261323
OpenModeller on the Biovel Portal
10/9/2014 18
The Ecological Niche Modeling (ENM) Workflow takes as input a file containing species occurrence points to create a model with the openModeller Web Service.
• The EUBrazilOpenBio ENM service is exposed through an extended openModeller Web Service interface
• Multi-staging and multi-parametric oM experiments are implemented through COMPSs that dynamically creates the virtual resources to execute the operations.
• An OCCI connector is used for the VMs management while data management supports CDMI endpoints.
ENM Service (OMWS2)
VENUS-C Cloud Middleware
COMPSs Workflow Orchestrator OCCI CDMI
EGI Federated Cloud
Service available at https://portal.biovel.eu/
www.egi.eu EGI-InSPIRE RI-261323 10/9/2014 20
Running GNU Octave and R on the CHAIN-REDS Science Gateway
20
CHAIN-REDS Science Gateway -CHAIN-REDS RI-306819
www.egi.eu EGI-InSPIRE RI-261323
Strengthening the underpinning platform
Technology Evolution • Broader support for open standards in Cloud
management frameworks – Utilisation of rOCCI for interfaces to commercial cloud
frameworks • New feature additions to foundational tools
depending on requests – Accounting, monitoring, service discovery, Application
DB • Innovative deployments of opensource PaaS • Large improvements on documentation for all 3
stakeholder groups
22
Valu
e ad
ded
serv
ices
for U
ser
Com
mun
ities
www.egi.eu EGI-InSPIRE RI-261323
Conclusions The EGI Federated Cloud, a federation of institutional private Clouds, offering Cloud
Services to researchers in Europe and worldwide • A single cloud system able to
– Scale to user needs – Integrate multiple different providers to give resilience – Prevent vendor lock-in through open standards – Enable resource provision targeted towards the research community
• Paving the way for a global federated cloud marketplace
– Established best practice – Illustrating leadership – Open standards, open technology – Open membership, open processes – Open competition
Current capability – ~6,000 cores, 300 TB storage Promised through EGI-Inspire – 18,000 cores, 6000 TB storage 2020 Target – 1M cores, 1EB storage
23
Alison Packer, Álvaro López García, Alvaro Simon, Binh Minh Nguyen, Björn Hagemeier, Boris Parak, Boro Jakimovski, Cal
Loomis, Christos Loverdos, Daniele Cesini, Daniele Lezzi, David Blundell, Diego Scardaci, Elisabetta Ronchieri, Emir Imamagic, Enol Fernandez, Feyza Eryol, Florian Feldhaus, Gergely Sipos,
Ivan Diaz, Jan Meizner, John Gordon, Kostas Koumantaros, Malgorzata Krakowian, Marios Chatziangelou, Marco Verlato,
Marica Antonacci , Mattieu Puel, Matteo Turilli, Michel Jouvin, Michel Drescher, Miroslav Ruda, Nuno L. Ferreira, Owen Synge,
Paul Miller, Peter Solagna, Piotr Kasprzak , Roberto Barbera, Ruben Valles, Sándor Ács, Salvatore Pinto, Silvio Spardi,
Soonwook Hwang, Steven Newhouse, Stuart Pullinger, Thijs Metsch, Tomasz Szepieniec, Viet Tran, Zeeshan Ali Shah and all
the other members of the EGI federated Cloud Task Force, activity, workpackage
www.egi.eu EGI-InSPIRE RI-261323
Thank you!
Questions?
25
Task Force resources
Mailing List: [email protected]
Wiki site: http://go.egi.eu/tf-fedclouds
GitHub: https://github.com/EGI-FCTF
Indico site: https://www.egi.eu/indico/categoryDisplay.py?categId=112