OGF Introductory Overview - FAS* 2014

©2013 Open Grid Forum

The Open Grid Forum: History, Introduction and Process

Alan Sill VP of Standards, OGF

Senior Scientist, High Performance Computing Center Adjunct Professor of Physics

Texas Tech University

1

FAS* Co-located Conferences September 8-12, 2013

© 2014 Open Grid Forum FAS* 2014: SAOS, P2P, CAC, OGF London, UK Sep 9, 2014

About the Open Grid Forum:

Open Grid Forum (OGF) is a leading global standards development organization operating in the areas of cloud, grid and related forms of advanced distributed computing. The OGF community pursues these topics through an open process for development, creation and promotion of relevant specifications and use cases. OGF actively engages partners and participants throughout the international arena through an open forum with open processes to champion architectural blueprints related to cloud and grid computing. The resulting specifications and standards enable pervasive adoption of advanced distributed computing techniques for business and research worldwide.

2

© 2014 Open Grid Forum FAS* 2014: SAOS, P2P, CAC, OGF London, UK Sep 9, 2014

History and Background

• OGF began in 2001 as an organization to promote the advancement of distributed computing worldwide.

• Grid Forum --> Global Grid Forum --> GGF + Enterprise Grid Alliance --> formation of OGF in 2005.

• Mandate is to take on all forms of distributed computing and to work to promote cooperation, information exchange, best practices in use and standardization.

• OGF best known for a series of important computing, security and network standards that form the basis for major science and business-based distributed computing (BES, GridFTP, DRMAA, JSDL, RNS, GLUE, UR, etc.).

• Have also been working on cloud and Big Data standards (OCCI, WS-Agreement, DFDL, etc.) for several years.

• Cooperative work agreements with other SDOs in place.3

© 2014 Open Grid Forum FAS* 2014: SAOS, P2P, CAC, OGF London, UK Sep 9, 2014

OGF Standards

OGF has an extensive set of applicable standards related to advanced distributed grid and cloud computing and associated storage management and network operation:

- Job Submission and Workflow Management (JSDL, BES) - Federated Identity Management (FedSec-CG) - Virtual Organizations (VOMS) - Managing the Trust Eco-System (CA operations, AuthN/AuthZ) - Network Management (NSI, NML, NMC, NM) - Secure, fast multi--party data transfer (GridFTP, SRM) - Service Agreements (WS-Agreement, WS-Agreement Negotiation) - Data Format Description (DFDL) - Cloud Computing interfaces (OCCI) - Distributed resource management APIs (DRMAA, SAGA, etc.) - Firewall Traversal (FiTP) - (Many others under development)

4

© 2014 Open Grid Forum FAS* 2014: SAOS, P2P, CAC, OGF London, UK Sep 9, 2014

Starting Point: OGF Documents

5

http://ogf.org/documents

© 2014 Open Grid Forum FAS* 2014: SAOS, P2P, CAC, OGF London, UK Sep 9, 2014

OGF Document Types

• Informational: To inform the community about a useful idea or set of ideas.

• Experimental: To inform the community about a useful experiment, testbed or implementation of idea or set of ideas.

• Community Practice: To inform the community of common practice or process, with the objective to influence the community.

• Recommendations: To document a specification, analogous to an Internet Standards track document. Recommendations are initially designated as "proposed," and following further experience and review may become full recommendations.

• Further information including guidance and advice contained in GFD.152 at: http://ogf.org/documents/GFD.152.pdf

6

© 2014 Open Grid Forum FAS* 2014: SAOS, P2P, CAC, OGF London, UK Sep 9, 2014

Example: Worldwide LHC Computing Grid

7

~450,000 cpu cores ~430 Pb storage

Typical data transfer rate: ~12 GByte/sec

Total worldwide grid capacity: ~2x WLCG across all grids and

VOs

The Role of Standards for Risk Reduction and Inter-operation in XSEDE

XSEDE: The Next Generation of US National Supercomputing

Infrastructure

Cloud and grid standards now power some of the largest academic supercomputing infrastructures in the world!

Blacklight  Shared  Memory  4k  Xeon  cores

!Darter  24k  cores  !Nautilus  Visualization  Data  Analytics  !Keeneland    CPU/GPGPU

Stampede  460K  cores  w.  Xeon  Phi  >1000  users  Upgrade  in  2015

Wrangler  Data  Analytics

Trestles  IO-­‐intensive  10k  cores  160  GB  SSD/Flash  !Gordon      Data  intensive  64  TB  memory  300  TB  Flash  Mem

Open  Science  Grid  High  throughput  124  sitesBlue  Waters  

Leadership  

SuperMIC  380  nodes  –  1PF  (Ivy  bridge,  Xeon  Phi,  GPU)

Over  13  million  service  units/day  typically  delivered  as  of  2014  across  all  XSEDE  supercomputing  sites  (about  3  million  core  hours/day),  totaling  about  1.6  billion  core  hours  per  year

Yellowstone  Geosciences

US National Cyberinfrastructure

Promote an open, robust, collaborative, and innovative

ecosystem

Adopt, create and disseminat

e knowledge

Extend   the impact of cyber-

infrastructurePrepare

the current and next

generation

Provide technical

expertise and support services

Collaborate with other CI groups and

projects

FutureGrid*  

Maverick  Visualization  Data  Analytics

Comet  “Long  Tail  Science”  47k  cores/2  PF  High  throughput

ACI-­‐REF  Campus  sharing,  NSF  Cloud  (shared)

Grids

Credit: Irene Qualters, US

National Science Foundation

www.egi.euEGI-InSPIRE RI-261323 www.egi.euEGI-­‐InSPIRE  RI-­‐261323

EGI international presenceStorage Value  (yearly  increase)

Disk  (PB)    235  PB  (+69%)

Tape  (PB)    176  PB  (+32%)

Value  (yearly  increase)

CPU  cores 361,300  across  53  countries  (1.44  M  job/day)

Standards-based international collaboration

EGI Federated Cloud: A successful standards-based international federated cloud infrastructure

Credit: David Wallom

Chair EGI Federated Cloud Task Force

Members •70 individuals •40 institutions •13 countries

Stakeholders •23 Resource Providers •10 Technology Providers •7 User Communities •4 Liaisons

Technologies •OpenStack •OpenNebula •StratusLab •CloudStack (in evaluation)

•Synnefo •WNoDeS

BSC

CNRS

LMU

OeRC

Masaryk

TUD

IFAE

Cyfronet

100%IT

CESNET

RADICAL

SRCE

DANTE

FZJ

GRNET

GWDG

STFC

SARA

KTH

INFNFCTSG

EGI.eu

Imperial

CESGACETA

IFCA

IGI

IPHC

IN2P3

SZTAKI

IISAS SixSq

Standards •OCCI (control) •OVF (images) •X.509 (authN) •CDMI (storage - under development)

(Updated July 2014)

© 2014 Open Grid Forum FAS* 2014: SAOS, P2P, CAC, OGF London, UK Sep 9, 2014

Continuing series… !

Oriented towards REAL DEVELOPMENT

!Past events

co-sponsored by many open source and standards-related

organizations including

OGF, DMTF, SNIA, OASIS, ETSI, OCEAN and OW2

Developer-oriented in-person

standards and software testing

series

Easy to get involved and join

in events as developers or

project researchers

Cloud Plugfest 11 just completed!

http://cloudplugfest.org

Cloud Plugfest Developer Series:

12

More events in planning pipeline.

© 2014 Open Grid Forum FAS* 2014: SAOS, P2P, CAC, OGF London, UK Sep 9, 2014

Conclusions and Future:

• OGF actively engages many partners and participants throughout the international arena through an open forum with open processes to promote best practices and standards in advanced distributed computing.

• OGF occupies an important role in standards and software development with significant uptake in advanced distributed computing, including cloud, grid, networking and large-scale data processing, transfer and handling.

• OGF documents support a variety of flexible architectures for advanced scientific, community and business uses.

• OGF’s experience has enabled distributed computing built on these architectures to provide more flexible, efficient and utility-like global infrastructures. Join us!

13