Cross-SDO Projects to Accelerate Cloud Innovation Alan Sill, Ph.D Site Director, Cloud and Autonomic Computing Center Senior Scientist, High Performance Computing Center Adjunct Professor of Physics Texas Tech University Vice President of Standards, Open Grid Forum
55
Embed
MPLS/SDN 2013 Intercloud Standardization and Testbeds - Sill
This talk givens an overview of several multi-SDO and cross-SDO activities to promote and spur innovation in cloud computing. The focus is on API development and standardization, including testbeds, test use cases, and collaborative activities between organizations to create and carry out development and testing in this area. The focus is on work being pursued through the Cloud and Autonomic Computing Center at Texas Tech University, which is part of the US National Science Foundation's Industry/University Cooperative Research Center, and on work being done by standards organizations such as the Open Grid Forum, Distributed Management Task Force, and Telecommunications Management Forum in which the CAC@TTU is involved. A summary is also given of work to produce a new round of more detailed use cases suitable for testing by the US National Institute of Standards and Technology's Standards Acceleration to Jumpstart Adoption of Cloud Computing (SAJACC) working group, with brief mention also given to other related work going on in this area in other parts of the world. Background and other standards work is also mentioned.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Cross-SDO Projects to Accelerate Cloud Innovation
Alan Sill, Ph.D Site Director, Cloud and Autonomic Computing Center Senior Scientist, High Performance Computing Center
Adjunct Professor of Physics Texas Tech University
Vice President of Standards, Open Grid Forum
Alan Sill, TTU
November 20, 2013
Standards as a Means to Interoperability
• Risk reduction: • Lessen risk of dead-end product design and orphan
components. • Lessen or remove risk of vendor service lock-in. • Mitigate reusability barriers for software and data access.
• Provide best-of-breed development and methods. • Mix-and-match for input & output of processing steps. • Allow innovation/competition at more interesting layers
and development of better internal features. • Facilitate interoperation with other provider software
services, components and infrastructures. • Approach must be explicitly cross-SDO and cross-vendor.
!2
Alan Sill, TTU
November 20, 2013
CAC Goals and VisionThe Texas Tech site intends to provide a practical work arena for development and coordination of standards, standards-based software and reference implementations applicable to cloud and other forms of advanced distributed computing. The site will fill a need to organize, classify, develop reference implementations for and otherwise contribute to standards-based software in advanced distributed computing. The vision that underlies these goals is one of harmonious, coordinated development of software that interoperates across many boundaries of deployment and implementation, and that can be repurposed, rescaled and redeployed as needed to solve a wide variety of user, vendor and supplier problems.
!4
Alan Sill, TTU
November 20, 2013
Standards, Clouds and Interoperability: Our vision at the CAC is that appropriate use of standards as part of the innovation process permits software and hardware used in clouds to interoperate with other components and infrastructures, and thus reduces risks including the risk of unwanted vendor lock-in. !This allows developers, vendors and users to focus more on higher level capabilities and therefore less on reinventing common aspects and features of their APIs and interface modules. !Coupling standards and software innovation is therefore crucial to economical cloud development at this stage.
CPU cores 361,300 across 53 countries (1.44 M job/day)
!17
Science Virtual Organizations on the OSG
• Astrophysics
• Biochemistry
• Bioinformatics
• Earthquake Engineering
• Genetics
• Gravitational-‐wave physics
• Mathematics
• Nanotechnology
• Nuclear and particle physics
LSN-MAGIC Meeting February 22, 2012
The Role of Standards for Risk Reduction and Inter-operation in XSEDE
XSEDE: The Next Generation of US Supercomputing Infrastructure
OGF standards power some of the largest supercomputing infrastructures in the world!
LSN-MAGIC Meeting February 22, 2012XSEDE Services Layer:
Simple services combined in many ways
!20
–Resource Namespace Service 1.1 –OGSA Basic Execu@on Service –OGSA WSRF BP – metadata and no@fica@on –OGSA-‐ByteIO –GridFTP –JSDL, BES, BES HPC Profile –WS Trust Secure Token Services –WSI BSP for transport of creden@als –… (more than we have room to cover here)
Examples – (not a complete list)
XSEDE represents a phase change in the engagement of modern computing standards with US cyberinfrastructure.
Alan Sill, TTU
November 20, 2013
Standardization Benefitsp Having an organized set of acceptance criteria
can improve business value to members in the following ways: n For VENDORS, ensures that their product passes test acceptance
conditions, leading to fewer customer complaints in the field. n For PROVIDERS, ensures that products they are hosting are well-
behaved, with fewer unexpected error or service failure conditions.
n For USERS, ensures that the products and services they use will be interoperable with the API, framework or standard they are using as the basis for their purchased services,
p Pre-purchase specification for products and services can be done on a rational basis.
p Productivity of programmer teams is maximized by having well-defined workflows for scenario and unit testing.
!21
Alan Sill, TTU
November 20, 2013
About the Open Grid Forum:
Open Grid Forum (OGF) is a leading global standards organization cooperating with many partners in the areas of cloud, grid and related forms of advanced distributed computing. The OGF community and its partners pursue these topics through an open process for development, creation and promotion of relevant specifications and use cases. The central feature of this work is open forum with open processes to champion architectural blueprints related to cloud and grid computing. The resulting specifications and standards enable pervasive adoption of advanced distributed computing techniques for business and research worldwide.
!22
Alan Sill, TTU
November 20, 2013
History and Background
• OGF began in 2001 as an organization to promote the advancement of distributed computing worldwide.
• Grid Forum --> Global Grid Forum --> GGF + Enterprise Grid Alliance --> formation of OGF in 2005.
• Mandate is to take on all forms of distributed computing and to work to promote cooperation, information exchange, best practices in use and standardization.
• OGF best known for a series of important computing, security and network standards that form the basis for major science and business-based distributed computing (BES, GridFTP, DRMAA, JSDL, RNS, GLUE, UR, etc.).
• Have also been working on cloud and Big Data standards (OCCI, WS-Agreement, DFDL, etc.) for several years.
• Cooperative work agreements with other SDOs in place.
OGF and the Pursuit of Open International Standards:• OGF views its mission as integrally tied to the creation and
implementation of practical standards of use across a wide variety of boundaries. • Interoperability and utility for implementation across
multiple projects is essential • Interoperability and usability across international
boundaries on a global basis is desired • OGF’s approach to standards creation and curation
promotes development of standards that will be of use to the large-scale infrastructure projects.
• Standards are developed by participants in these projects. • For the past several years, active also in cloud computing.
Alan Sill, TTU
November 20, 2013
OGF Standards Strengths
!25
§ OGF has an extensive set of applicable standards related to federated community grid and cloud computing: § Federated Identity Management (FedSec-CG) § Managing the Trust Eco-System (CA operations, AuthZ and AuthN tools) § Virtual Organizations (VOMS) and related authorization tools § Job Submission and Workflow Management (JSDL, BES, HPC Profile) § Network Management (NSI, NML, NMC, NM) § Secure, fast multi--party data transfer (GridFTP, SRM) § Data Format Description Language (DFDL) § Service Agreements (WS-Agreement, WS-Agreement Negotiation) § Cloud Computing interfaces (OCCI family of specifications) § Distributed resource management (DRMAA, SAGA, etc.) § Firewall Traversal (FiTP); Usage Accounting (UR, GLUE) § Others under development through ISOD-RG, DCIFed-WG, etc.
§ Working to gather this information to form an organized description of OGF work - an OGF “Cloud Portfolio”.
Alan Sill, TTU
November 20, 2013
• DRMAA: Distributed Resource Management Application API Grid Engine, Open Grid Scheduler: (open source); TORQUE and related products: Adaptive Computing; PBS Works: Altair Engineering; Gridway: DSA Research; Condor: U. of Wisconsin / Red Hat;
•OGSA® Basic Execution Service Version 1.0 and BES HPC Profile: BES++ for LSF/SGE/PBS: Platform Computing; Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering;
• JSDL: Job Submission Description Language (family of specs): BES++ for LSF/SGE/PBS and Platform LSF: Platform Computing; Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering;
•WS-Agreement (family of specifications): ElasticLM License-as-a-Service: ElasticLM; BEinGrid SLA Negotiator, LM-Architecture and Framework: (Multiple partners); BREIN SLA Management Framework: (Multiple partners); WSAG4J, Web Services Agreement for Java (framework implementation): Fraunhofer SCAI.
!26
OGF Standards In Use In Industry:
Alan Sill, TTU
November 20, 2013
OGF Infrastructure Area
The OGF Infrastructure groups explore and define what is needed to interface physical and virtual resources to higher level constructs. These include networks and network devices, computers and virtual machines, storage, visualization devices, instruments, and sensor technologies. • Network Measurements Working Group (NM-WG) • Network Measurement And Control Working Group (NMC-WG) • Network Mark-Up Language Working Group (NML-WG) • Network Service Interface Working Group (NSI-WG) • Open Cloud Computing Interface Working Group (OCCI-WG) • Infrastructure Services On-Demand Provisioning Research Group
(ISOD-RG) • Firewall Virtualization For Grid Applications Working Group (FVGA-WG) • Grid High-Performance Networking Research Group (GHPN-RG)
• NIST Cloud definition (NIST SP 800-145), NIST Special Publication 500-291 version 2, NIST Cloud Computing Standards Roadmap, July 2013and Cloud Computing Reference Architecture (CCRA), v1.0 (NIST SP 500-292) http://www.nist.gov/itl/cloud/publications.cfm
• ITU-T Focus Group on Cloud: Technical Report (Part 1 to 7) http://www.itu.int/en/ITU-T/focusgroups/cloud/Documents/FG-coud-technical-report.zip
• IEEE - WGs on InterCloud issues and Cloud Profiles – IEEE ICWG/2302 WG - Intercloud WG (ICWG) Working Group http://standards.ieee.org/develop/wg/ICWG-2302_WG.html– IEEE P2301, P2302 Projects http://standards.ieee.org/develop/project/2301.html, 2302.html
• IETF Internet Drafts – Cloud Reference Framework. Internet Draft, by B. Khasnabish, J. Chu, S. Ma, Y. Meng, N. So, P.
Unbehagen, M. Morrow, M. Hasan, Y. Demchenko http://tools.ietf.org/html/draft-khasnabish-cloud-reference-framework-05.txt
– Cloud Service Broker, Internet Draft by Shao Weixiang, Hu Jie, Bhumip Khasnabish. http://tools.ietf.org/html/draft-shao-opsawg-cloud-service-broker-03.txt
!29
Alan Sill, TTU
November 20, 2013
!30
• OCCI is an API and Protocol • Sits on the boundary of a Service Provider
and Service Consumer • No assumptions about the boundary
Multiple organizations have their own APIs, and real work has been done by standards organizations to catch up and offer common standards and products that can work across these. Now is the time to do this! Recent work has brought several cloud-related standards into frameworks that can be used to implement them (for example, CloudStack, OpenStack, Open Nebula, and several others) and are mature enough to apply:
n Open Cloud Computing Interface (OCCI) from OGF n Cloud Data Management Interface (CDMI) from SNIA n Data Format Description Language (DFDL) from OGF n Open Virtualization Format (OVF) from DMTF and ISO n Cloud Application Management Protocol (CAMP) from OASIS n Cloud Infrastructure Management Interface (CIMI) from DMTF n WS-Agreement and WS-Agreement Negotiation from OGF
Alan Sill, TTU
November 20, 2013
Standards Organizations Active in Cloud Computing:
• It is often said that there are “too many standards organizations”. This is a lot like saying there is “too much software”.
• Each has its own area of specialty, its own contributor base, and its own method of funding to develop its work products.
• How best to work with these organizations? (Our answer: Cooperatively!)
OGF and ETSI: Cooperative MoU in place; contributing to ETSI CSC effort. Co-host Cloud Plugfest series (cooperative with SNIA).
OGF and TM Forum: Memorandum of Understanding in place; ongoing cross-SDO document on End-to-End Management of Cloud Service Agreements, including SLAs, in progress.
OGF and CSA: Cooperative agreement between OGF and CSA in place.
OGF and IEEE: OGF co-sponsored IEEE CloudCom 2011 (Athens) and Cloudcom 2012 (Taipei); open to other engagements.
OGF Cooperative Agreements In Place as of Feb. 2013
OGF and SIENA, NIST, GICTF, Canada Cloud, etc.:
Contribute actively to to ongoing global standards roadmapping efforts.
OGF and DMTF: Joint work register on OCCI and CIMI with DMTF Cloud Management WG.
OGF and ISO; OGF and ITU-T: OGF has a formal liaisons with ISO/IEC JTC1 SC38 on Cloud Computing and with ITU-T JCA Cloud.
OGF and SNIA (CDMI): Cooperative agreement w/SNIA Cloud on CDMI and has held 7 jointly hosted Cloud Standards Plugfests so far (cooperative now with ETSI).
DMTFCloud Management Standards
Slides prepared by
Winston Bumpus – VMware
Who is DMTF
• Established in 1993 to enable more effective management of millions of IT systems worldwide by bringing the IT industry together to collaborate on the development, validation and promotion of systems management standards.
• The group spans the industry with 160 member companies and organizations, and more than 4,000 active participants crossing 43 countries.
• Strong Alliance Partnership with GICTF and other Organizations
• The DMTF board of directors is led by 17 innovative, industry-leading technology companies. They include; Broadcom Corporation; CA Technologies.; Cisco; Citrix Systems, Inc.; Fujitsu; HP; Hitachi, Ltd.; Huawei; IBM; Intel Corporation; Microsoft Corporation; NetApp; Oracle; Software AG; SunGard Availability Services; Telecom Italia and VMware, Inc.
Open Virtualization Format (OVF)A standard packaging format for virtual machines
A distribution format for VMs Supports single VM & multiple VM configurations Optimized for distribution & simple automation Vendor and platform independent Now an ANSI and ISO standard
OVF Package (myapp.ova)
myapp.ovf
XML
web.xxx
images.iso
myapp.mf
myapp.cert
An OVF package consists of One OVF descriptor with extension .ovf zero or one OVF manifest (w/ extension .mf) zero or one OVF certificate (w/ extension .cert) zero or more disk image files zero or more additional resource files (such as ISO images)
OVF 2 (Recently Released): brings an enhanced set of networking capabilities making it applicable to a broader range of use cases that are emerging as industry enters the Cloud era.
Cloud Infrastructure Management Interface (CIMI)
Model Scope: Core IaaS functionality Deploying and managing: Machines, Volumes, Networks, Monitoring, etc. Client: VM/application developer, deployer and administrator Server: IaaS Cloud Provider Version 1.1 is released and implemented
CIMI REST/HTTP-Based ProtocolSpecification currently describes a REST/HTTP binding to the model. Other bindings are Considered. Follows REST principles and describes mapping of the HTTP protocol verbs to operations on
the model. Standard HTTP status codes are used to convey the results of the operations. Serialization formats for the message body include JSON and XML
Other features: Cloud Entry Point to manage Systems, Machines, Volumes, and Networks; Grouping of resources meant to be managed as a single unit; Metering and Monitoring support; Resource metadata; Events and event logging; Jobs; Dynamic Discovery of a provider’s resources characteristics; Entity creation using Templates
Cloud Auditing Data Federation (CADF) Working GroupScope / Objectives Develop Standards for the Federation of Cloud Auditing Data
By Specifying a Normative, Prescriptive Auditing Event Data Format along with Interface Definitions and a compatible Component and Interaction model.
The Data Model will include support for: Classification by Extensible Event Taxonomies to categorize cloud provider IT Resources, event Actions and Outcomes.
Federation of Customized Auditing Reports and Logs - event data will support federation and be composable into customizable reports and logs.
The Interface Model includes: Definition of Service Methods to Manage and Federate the Data Model’s Events, Logs and Reports
Interfaces will support audit data Submission, Import and Export, Query and Subscription.
The Component and Interaction Model will Demonstrate how the Interfaces and Data Format can be used by Cloud Providers and Consumers to Support Cloud Auditing use cases.
Future work may include Profiles that extend the core data and interface specifications to accommodate particular methods of consumption
DMTF NSMWG Focus: Virtualized and Hybrid Network Environment Management
*
Management of Virtualized Network Entities
Apps and Services that Utilize Virtualized L3 Resource/Entities
Apps and Services that Utilize Virtualized Network Entities
Physical and Virtual Network Entities
Network Entities Abstraction
Virtualized Network Entities
vNE Management API
…DNS
Fire WallEdge/Core/Border Router
Load BalancerAAA Server
…DNS
Software Entitlement Working Group
Purpose: To extend the Common Information Model to capture software entitlement and usage
metrics and to deliver the associated profile. Based on ISO/IEC 19770-2 Software Identification Tags Driven by a request for JP Morgan Chase. 1. Update CIM to support Software Entitlement and Usage Metrics – Q4 2013 2. Create CIM Software Entitlement and Usage Metrics Profile - Q1 2014
For more information www.dmtf.org/cloud www.dmtf.org/OVF
European Cloud Standards Coordination (New)Multi-organization open survey commissioned by EC, coordinated by ETSI along same lines as GICTF, NIST
!54
Draft In Progress!
Alan Sill, TTU
November 20, 2013
Summaryp Many active projects are underway to document, map and extend the
important role played by standards and software development with significant uptake in advanced distributed computing, including cloud, grid, networking and large-scale data processing, transfer and handling through innovative cooperation with many partners.
p The CAC actively engages with partners and participants throughout the international arena to understand and promote best practices and standards in cloud and advanced distributed computing.
p TTU is leveraging these standards to support a wide variety of flexible architectures for advanced scientific and business uses through the NSF Cloud and Autonomic Computing Center and is actively seeking international collaborators to extend this work more comprehensively through the cloud.