ISC Cloud13 Sill - Crossing organizational boundaries in cloud computing

The Conference on the Use of Cloud for High Performance Computing (HPC)

Taking the Best from Industry & Academics: Cross-SDO Projects to Accelerate HPC Cloud Innovation

Alan Sill, Texas Tech University;VP of Standards, Open Grid Forum

Alan Sill, TTUSeptember 24, 2013

About the Open Grid Forum:

Open Grid Forum (OGF) is a leading global standards organization cooperating with many partners in the areas of cloud, grid and related forms of advanced distributed computing.The OGF community and its partners pursue these topics through an open process for development, creation and promotion of relevant specifications and use cases. The central feature of this work is open forum with open processes to champion architectural blueprints related to cloud and grid computing. The resulting specifications and standards enable pervasive adoption of advanced distributed computing techniques for business and research worldwide.

2


History and Background

• OGF began in 2001 as an organization to promote the advancement of distributed computing worldwide.

• Grid Forum --> Global Grid Forum --> GGF + Enterprise Grid Alliance --> formation of OGF in 2005.

• Mandate is to take on all forms of distributed computing and to work to promote cooperation, information exchange, best practices in use and standardization.

• OGF best known for a series of important computing, security and network standards that form the basis for major science and business-based distributed computing (BES, GridFTP, DRMAA, JSDL, RNS, GLUE, UR, etc.).

• Have also been working on cloud and Big Data standards (OCCI, WS-Agreement, DFDL, etc.) for several years.

• Cooperative work agreements with other SDOs in place.

3


• DRMAA: Distributed Resource Management Application APIGrid Engine, Open Grid Scheduler: (open source); TORQUE and related products: Adaptive Computing; PBS Works: Altair Engineering; Gridway: DSA Research; Condor: U. of Wisconsin / Red Hat;

•OGSA® Basic Execution Service Version 1.0 and BES HPC Profile:BES++ for LSF/SGE/PBS: Platform Computing; Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering;

• JSDL: Job Submission Description Language (family of specs):BES++ for LSF/SGE/PBS and Platform LSF: Platform Computing; Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering;

•WS-Agreement (family of specifications):ElasticLM License-as-a-Service: ElasticLM; BEinGrid SLA Negotiator, LM-Architecture and Framework: (Multiple partners); BREIN SLA Management Framework: (Multiple partners); WSAG4J, Web Services Agreement for Java (framework implementation): Fraunhofer SCAI.

4

OGF Standards In Use In Industry:


OGF StandardsOGF has an extensive set of applicable standards related to advanced distributed grid and cloud computing and associated storage management and network operation:• Job Submission and Workflow Management (JSDL, BES)• Federated Identity Management (FedSec-CG)• Virtual Organizations (VOMS)• Managing the Trust Eco-System (CA operations, AuthN/AuthZ)• Network Management (NSI, NML, NMC, NM)• Secure, fast multi--party data transfer (GridFTP, SRM)• Service Agreements (WS-Agreement, WS-Agreement

Negotiation)• Data Format Description (DFDL)• Cloud Computing interfaces (OCCI)• Distributed resource management APIs (DRMAA, SAGA, etc.)• Firewall Traversal (FiTP)• (Many others under development)

5


Example: Worldwide LHC Computing Grid

6

~450,000 cpu cores~430 Pb storage

Typical data transfer rate: ~12 GByte/sec

Total worldwide grid capacity: ~2x WLCG across all grids and

VOs

LSN-MAGIC MeetingFebruary 22, 2012

The Role of Standards for Risk Reduction and Inter-operation in XSEDEAndrew Grimshaw

XSEDE: The Next Generation ofUS Supercomputing Infrastructure

OGF standards power some of the largest supercomputing infrastructures in the world!

XSEDE cloud infrastructure still being written

LSN-MAGIC MeetingFebruary 22, 2012XSEDE Services Layer:

Simple services combined in many ways

8

–Resource Namespace Service 1.1–OGSA Basic Execu8on Service–OGSA WSRF BP – metadata and no8fica8on–OGSA-‐ByteIO–GridFTP–JSDL, BES, BES HPC Profile–WS Trust Secure Token Services–WSI BSP for transport of creden8als–… (more than we have room to cover here)

Examples – (not a complete list)

Basic message: XSEDE represents a phase change in the engagement of OGF standards with US cyberinfrastructure.


9

• OCCI is an API and Protocol• Sits on the boundary of a Service Provider

and Service Consumer• No assumptions about the boundary

OCCI® by OGF

Open Cloud Computing Interface®

Many real-world implementations!

Task Force – Sept 2013

Members•70 individuals•40 institutions•13 countries

Stakeholders•23 Resource Providers•10 Technology Providers•7 User Communities•4 Liaisons

Technologies•OpenNebula.•StratusLab.•OpenStack.•Synnefo.•WNoDeS.

BSC

CNRS

LMU

OeRC

Masaryk

TUD

IFAE

Cyfronet

100%IT

CESNET

RADICAL

SRCE

DANTE

FZJ

GRNET

GWDG

STFC

SARA

KTH

INFNFCTSG

EGI.eu

Imperial

CESGACETA

IFCA

IGI

IPHC

IN2P3

SZTAKI

IISAS SixSq

Credit: David Wallom Chair EGI Federated Cloud Task Force

Federation Test bed – September 2013FedCloud Core Services

Information SystemTop-DBII

MonitoringNagios

Image MetadataMarketplace

Appliance Repository

VOVOMS server

User Interfaces

OCCI ClientsrOCCI; WNoDeS-CLI

CDMI ClientsLibcdmi-java

Resource Providers

VOMS proxy

VmcatcherVmcaster

Marketplace client

SSM

CDMI server LDAP

OpenNebulaOpenStackStratusLabWNoDeS

OCCI server

AAI

VOMS Proxy

EGI Production Infrastructure

Service Availability

SAM

AccountingAPEL

Configuration DB

GOCDB

External Services

Certification Authorities

Credit: David Wallom Chair EGI Federated Cloud Task Force


12

Collaborating Organizations:

• It is often said that there are “too many standards organizations”. This is a lot like saying there is “too much software”.• Each has its own area of specialty, its own contributor base, and its own

method of funding to develop its work products.• How does OGF relate to these organizations? (Answer: Cooperatively!)

(Some of many...)

Interoperability:

Appropriate use of standards permits software processes to interoperate with other components and

infrastructures, and thus reduces risks including the risk of unwanted vendor lock-in.

This allows developers, vendors and users to focus more on higher level capabilities and therefore less on

reinventing common aspects and features of their APIs and interface modules.

Coupling standards and software innovation is therefore crucial to economical cloud development at this stage.


14

Why Pursue Standards as a Means to Interoperability?

•Risk reduction:• Lessen risk of dead-end product design and orphan

components.• Lessen or remove risk of vendor service lock-in.•Mitigate reusability barriers for software and data

access.•Provide best-of-breed development and methods.•Mix-and-match for input & output of processing steps.•Allow innovation/competition at more interesting

layers and development of better internal features.•Facilitate interoperation with other provider software

services, components and infrastructures.


Business Benefits• Having an organized set of acceptance criteria

can improve business value to members in the following ways:• For VENDORS, ensures that their product passes test

acceptance conditions, leading to fewer customer complaints in the field.

• For PROVIDERS, ensures that products they are hosting are well-behaved, with fewer unexpected error or service failure conditions.

• For USERS, ensures that the products and services they use will be interoperable with the API, framework or standard they are using as the basis for their purchased services,

• Pre-purchase specification for products and services can be done on a rational basis.

• Productivity of programmer teams is maximized by having well-defined workflows for scenario and unit testing.

15


16

OGF Cooperative AgreementsIn Place as of Late 2013

OGF and DMTF:• OGF published the the OCCI Core, Infrastructure and HTTP

Rendering specifications as GFD.183, 184 and 185, and is working on a JSON rendering. We created a joint work register with DMTF on OCCI and CIMI in 2011 and continue to cooperate with them on implementation and testing of the CIMI specification. DMTF has also joined OGF in the Cloud Plugfests.

OGF and ISO:• OGF has an ongoing Category A liaison with ISO JTC1 SC38 on

Cloud Computing and is working with ISO on joint activities. OGF and SNIA (CDMI):

• OGF has cooperative agreement w/SNIA with respect to CDMI and has co-hosted 10 Cloud Plugfests so far (series continues).


17

OGF and ETSI:• Cooperative MoU in place since 2012. OGF contributing to

ETSI CSC effort. ETSI joined the Cloud Plugfest testing as a co-sponsor in 2012 and continues to support this series.

OGF and ITU-T:• OGF has an ongoing formal liaison with ITU-T JCA Cloud.

OGF and TM Forum:• MoU in place since 2005, renewed 2012. OGF contributed to

TM Forum cross-SDO work on on “End-to-End Management of Cloud Service Agreements”, including SLAs, ongoing.

OGF and CSA:• Cooperative agreement between OGF & CSA since formation.

OGF and IEEE:• OGF co-sponsored IEEE Cloudcom conferences in 2011

(Athens) and 2012 (Taipei); ongoing engagements.



18

OGF and XSEDE:• OGF actively supports the standards process that underlies

XSEDE infrastructure and many of its ongoing projects.OGF and EGI:

• OGF works closely with EGI and associated projects; recently co-located OGF39 + Cloud Interoperability Week at EGI TF.

OGF and OCEAN:• OCEAN joined Cloud Plugfest series in 2013 as a co-sponsor

and as one of the primary sponsors of the upcoming September 2013 Cloud Interoperability Week.

OGF and OW2:• OW2 co-sponsored Sep. 2013 Cloud Interoperability Week &

Cloud Plugfest series and supports associated cloud projects.



19

OGF and NIST:• OGF contributed several talks and input to US National

Institute of Standards and Technology Cloud Standards Roadmap in 2010-2013 timeframe; AFS co-chairs SAJACC working group.

OGF and European Commission FP7/H2020 projects:• OGF technologies underpin the approaches of many

successfully funded FP-series projects and expects to participate in supporting the Horizon 2020 call.

OGF and Cloud Plugfest series:• OGF was one of the co-founding members of the ongoing

Cloud Plugfest developer-oriented testing series and co-sponsored the recent Cloud Interoperability Week. OGF provides substantial infrastructure to the ongoing series.


Alan Sill, TTU

September 24, 2013

http://collaborate.nist.gov/twiki-cloud-computing/bin/view/CloudComputing/SAJACC

NIST SAJACC Public Process

20


European Union “Standards & Interoperability for e-Infrastructure Implementation” (SIENA) Initiative

SIENA (2010-‐2012) is a Support Ac8on funded by the European Commission under FP7 (2007-‐13) CapaciDes programme.

A coordinated effort...

...towards the delivery of a future e-‐Infrastructures Roadmap....

...aligned with the needs of European and na=onal ini=a=ves and the evolving world.

IEG – Industry Expert Group

Partners

Interna8onal Experts

REB – Roadmap Editorial Board

SLG – Special Liaison Group

Standards Development Organisa8ons (SDOs)

Distributed Compu8ng Infrastructure (DCI) Ini8a8ves

Consolidated Cloud Compu8ng Workshops

Final Report:June 2012

21


22


Cloud Plugfest Developer Series:

24

Continuing seriesco-sponsored by

OGF, DMTF, SNIA, OASIS, ETSI,

OCEAN and OW2!

9 previous events held so far!

Developer-oriented in-person standards and software testing series

OPEN TO ALL!

Easy to get involved and join in events as developers or project

researchers!

Cloud Plugfest 10 just completed!

Cloud Interoperability Week Sep. 2013!

http://cloudplugfest.org


ETSI Tools Used in Cloud Plugfests:

25

Exploration

Proof that iPhones are used in space:

Link: Voyager - The Interstellar Mission


Conclusions

• OGF actively engages with partners and participants throughout the international arena through an open forum with open processes to promote best practices and standards in advanced distributed computing.

• OGF and its associated projects occupy an important role in standards and software development with significant uptake in advanced distributed computing, including cloud, grid, networking and large-scale data processing, transfer and handling through innovative cooperation with many partners.

• TTU is leveraging these standards to support a wide variety of flexible architectures for advanced scientific and business uses through the NSF Cloud and Autonomic Computing Center and is actively seeking international collaborators to extend this work more comprehensively through the cloud.

28

ISC Cloud13 Sill - Crossing organizational boundaries in cloud computing

Technology

grid computing

ogf standards ogf

open grid forum ogf

distributed computing

forms of distributed

worldwide lhc computing

advanced distributed

open grid scheduler