DevOps and Pragmatic distribution Process Change: …...#vmworld DevOps and Pragmatic Process Change: Top 5 Processes to Accelerate DevOps Josh Miller, VMware, Inc. Kai Holthaus, VMware,

#vmworld

DevOps and Pragmatic Process Change:Top 5 Processes

to Accelerate DevOpsJosh Miller, VMware, Inc.

Kai Holthaus, VMware, Inc.

DEV2500BU

#DEV2500BUVMworld 2018 Content: Not for publication or distribution

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.

VMworld 2018 Content: Not for publication or distribution


IntroductionsAdvisory Transformation Services

Josh MillerAdvisory Strategist

AdvisoryTransformation

Services

Kai HolthausTransformation

ArchitectVMworld 2018 Content: Not for publication or distribution

Agenda


DevOps & Why Processes Need to ChangeOverview of DevOps Principles Underpinning Process Change

Managing Changes & ConfigurationsAdapting Change & Configuration Management for DevOps

Testing and DeploymentAdapting Service Testing & Release & Deployment Management for DevOps

Monitoring, Events and IncidentsAdapting Event & Incident Management for DevOps

Creating and Utilizing KnowledgeOptimizing Knowledge Management for DevOps

Making It HappenDeveloping Strategy & Executing DevOps Transformation Initiatives



DevOps & Why Processes Need to ChangeOverview of DevOps Principles Underpinning Process Change


‹#› 6©2018 VMware, Inc.

Cloud is not a “destination”, it is a strategy

DevOps is not a “tool”, it is a wholistic wayof operating



Before It Was Called “DevOps”The transformational concept was referred to by the acronym “CAMS”

CultureRewarding Innovation

Emphasizing Learning

Breaking Down

Organizational Barriers

AutomationCreating Consistency

Increasing Productivity

Preventing Defects

MeasurementBasing Decisions on Data

Enabling Transparency

Incentivizing Continuous Improvement

SharingStressing Collaboration

Sharing Tools & Lessons

Identifying People with

Similar NeedsVMworld 2018 Content: Not for publication or distribution


The Three WaysUnderlying Principles of DevOps

The First WayThe Principles of

Flow

The Second Way The Principles of

Feedback

The Third WayThe Principles of

Continual Learning & ExperimentationVMworld 2018 Content: Not for publication or distribution


Make work visible Reduce batch sizes Reduce intervals of work Build in quality by preventing

defects from being passed to downstream work centers

Constantly optimize for global goals

Emphasize the performance of the entire system, as opposed to the performance of a specific silo of work or department

Enable fast left-to-right flow of work from Development to Operations to the Customer

Overview

Activities

The First WayThe Principles of Flow

(Business) (Customer)

Dev Ops

The First Way:Increase Flow from Dev to Ops



Amplify feedback to prevent problems from happening again

Enable faster detection & recovery

See problems as they occur and swarm them until countermeasures are in place

Maximize opportunities to learn and improve

Enable the fast and constant flow of feedback from right to left at all stages of our value stream

Create ever-safer systems of work where problems are found and fixed long before a catastrophic failure occurs

Overview

Activities

The Second WayThe Principles of Feedback


Dev Ops


The Second Way:Enable fast feedback from Ops

to Dev



Establish a dynamic, disciplined, and scientific approach to experimentation and risk-taking

Explicitly reserving time to pay down technical debt, fix defects, and refactor and improve problematic areas of our code and environments

Make blameless post-mortem reports searchable & create shared code repositories

Create a culture that fosters two things: Continual experimentation, taking risks

and learning from failure Understanding that repetition and

practice is the prerequisite to mastery Transform local discoveries into global

improvements

Overview

Activities

The Third WayThe Principles of Continual Learning & Experimentation


Dev Ops


The Second Way:Enable fast feedback from Ops

to Dev

The Third Way:High-trust culture of learning and experimentation

Continually amplifying feedback loop



VirtualInfrastructure

StorageCompute Networking

Private Cloud Public Gov. CloudHybrid

PhysicalInfrastructure

Foundational Services IaaS

Platformon

DemandCaaS PaaS FaaS

ContinuousIntegration

&Continuous

Delivery

Plan Feedback

Code Commit CI Test Artifact

CD Config Control

Management&

Operations

Security

Automation

Governance

Operations

UnifiedManagement


Instances

Virtual Machines Containers

ApplicationDevelopment

Processes

VersionControl

CodeInspection

BuildManagement

TestManagement

DigitalWorkspace

Applications

Existing Apps

Mobile Devices Laptops & Desktops

Cloud Native Apps SaaS Apps

Virtual Desktops

PlatformOperators

ApplicationDevelopers

Application Users

Po

licy

& G

ove

rnan

ceInfrastructure

OperationsProcesses

ChangeManagement

Release&

Deploy

ConfigurationManagement

IncidentManagement

ServiceCatalog

Management

CapacityManagement

RequestFulfillment

DemandManagement








Platformon



&Continuous

Delivery

Plan Feedback


CD Config Control

Management&

Operations

Security

Automation

Governance

Operations

UnifiedManagement


Instances



Processes

VersionControl

CodeInspection

BuildManagement

TestManagement

DigitalWorkspace

Applications

Existing Apps



Virtual Desktops

PlatformOperators


Application Users

Po

licy

& G

ove

rnan

ceInfrastructure

OperationsProcesses

ChangeManagement

Release&

Deploy


IncidentManagement

ServiceCatalog

Management

CapacityManagement

RequestFulfillment

DemandManagement


©2018 VMware, Inc. 14

“K”nightmare on Wall StreetThe Meltdown of Knight Capital Group

Knight Capital lost $440 million in 30 minutes due to something the firm called a 'trading glitch’

• 7/31/2012: Knight Capital had ~$365M in cash and equivalents

• 8/1/2012: Knight Capital had ~$105 million in DEBT

In reality, poor software development and testing models are likely to blame

One incorrect deployment can be the end of an entire company in the digital age

https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/


https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/


Managing Changes & ConfigurationsAdapting Change & Configuration Management for DevOps



Back to the “K”nightmareChange Management – What Was Missing?

Change Management Missing



Protect Production Environment

Optimize Risk

Enforce Proper Planning Procedures

Document

External Review – (CAB)

Change Management Today

Goal Method Practices The Surprising Truth:

• External Review = ↓Throughput

• Technical team accountable for quality = ↑Throughput

• External review:• No impact on restore

times• Negligible impact on

reducing failed changes

Source: State of DevOps Report 2014 (Puppet Labs)



Perform peer reviews

Request technical approval from management (if needed)

Identify change interdependencies

Accountable for quality

Accountable for scheduling of complex changes

Coordinate implementations

Enforce policy

Change Implementers Change Advisory Board (CAB)

Lessons from Lean and DevOps

Finally: Fearlessly cut all bureaucracy = steps in process that do not bring value!VMworld 2018 Content: Not for publication or distribution


Pre-authorized

Follows established procedure

Repeatable way of dealing with a particular kind of change

Defines specific pre-defined steps

Change Model

Standard Change

Change Models & Automation

Test

Commit

Continuous Integration

Artifact Management

Continuous Deployment

Configuration Management

Control

Code

FeedbackPlan

CI / CD Pipeline:

• Optimize Risk

• Protect Production Environment



Traditional Configuration Management

Manage (Physical) CIs

Relationships Between CIs, Attributes

Relatively Stable Environment

Focus: View of Current Environment

Copyright © AXELOS Limited 2011. All rights reserved. Material is reproduced under license from AXELOS.VMworld 2018 Content: Not for publication or distribution


Configuration Management in DevOps Operations

Configuration Information is “Codified”

Pipeline Creates Identical CIs

Volatile Environment

Focus: Detect Variances

Test

Commit


Artifact Management



Control

Code

FeedbackPlan

Codify how to create CIs

Create identical CIs at high speedbased on codified information

DetectvariancesVMworld 2018 Content: Not for publication or distribution


Extensive monitoring and logging

Detect CI variances

Telemetry also used to gauge impact of incidents

No changes to CIs in production

Instead:• Change procedure how

to create CI• Redeploy using pipeline

Configuration Management Techniques

TelemetryImmutable Infrastructure

Log Files• Volatile CIs (VMs,

containers, apps)• Things deployed

using pipeline

CMDB• Service

Architecture• Service Interfaces /

Service Levels• “Non-volatile” CIsVMworld 2018 Content: Not for publication or distribution


Testing and DeploymentAdapting Service Testing & Release & Deployment Management for DevOps



Back to the “K”nightmareWhat Was and Was Not Tested

Code Functionality

(Manual)Deployment

RegressionRemediation

Not Tested

Tested



Part of Pipeline

Fast Feedback to Developers

No deployment if testing fails

Must have identical environments!

Pipeline stops when error occurs

No code can be checked in until error is fixed

Error is “swarmed”

1. Build test case

2. Demonstrate current code fails test

3. Develop functionality

4. Demonstrate new code passes test

“Automatically” keeps your test suite current!

Testing Best Practices

Automated Testing Andon Cord Test-Driven Development



Traditional Deployment Management

Copyright © AXELOS Limited 2011. All rights reserved. Material is reproduced under license from AXELOS.




Continuous Delivery

Always Ready to Deploy


Manual Trigger Possible

Execute Pipeline

Becomes Non-Issue

Deployments the DevOps Way

Planning Deployment Review & Close

Test

Commit


Artifact Management



Control

Code

FeedbackPlan

Automated Testing

Continuous Delivery

Release Build & Test



Monitoring, Events and IncidentsAdapting Event & Incident Managementfor DevOps



Back to the “K”nightmareEvent and Incident Management – What Was Missing?

Monitoring

Pipeline

Response

Relying on external “monitoring” (volume of trades reported by NYSE)

No notification that deployment went wrong

No ability to stop trading –“pull Andon cord”VMworld 2018 Content: Not for publication or distribution


The Importance of TelemetryDevOps’ Second Way: Fail Fast, Fail Forward

Failure

Understanding

Results

Failure is inherent and inevitable in complex systems

Impossible to understand how system will behave in any given situation

Doing the same thing twice does not always lead to same result

ErrorsErrors must be detected quickly, before they can have impact

Manage WorkMust manage work so that problems in design and operations are revealed



What to Monitor

TestCommit


Artifact Management



ControlCode

FeedbackPlan

Availability of dev environments, tools

Availability of version control

Check-in issues

Errors in build

Availability, performance of build tools

Testing errors

Availability, performance of testing tools

Availability of artifact management tools

Deployment errors

Availability, performance of deployment tools

Configuration variances

Monitor for availability, performance, etc.

Monitor for warning, error conditions

Build centralized monitoring framework, implement analytics

Store historical information to be able to do error investigations

Ensure amount of data is managed (log file rotation, archiving)



Monitoring must be built into architecture

Events at the very least will trigger Incident Management process

Self-healing techniques –rely on your pipeline

Handling Incidents

Prevent incidents before they occur

Use telemetry and statistical analysis to predict error conditions

Implement automated response proceduresVMworld 2018 Content: Not for publication or distribution


Creating and Utilizing KnowledgeOptimizing Knowledge Management for DevOps



Knowledge tracked in large amounts of documentation (architectures, configurations, etc.) that are outdated upon creation

Information repositories are disparate, disjointed, not readily available, and/or not readily accessible

Leadership optimized for local goals rather than global goals a.k.a “Turf Wars”

Valuable knowledge locked in employees’ heads, withheld for political reasons, and distorted for personal benefit

Predominant culture of fear and indifference where mistakes are punished

Resources who identify problems are viewed negatively as “whistle-blowers”

Suppression or cover up of failures is regular practice

“Ignorance” ManagementObserving the current state of organizational knowledge management…

Stagnant Documentation Knowledge Hoarding Low Trust Culture



How Can DevOps help?Use the “Third Way” to encourage learning and optimize knowledge management

Learning as a Part of Daily Work

Moving from Local To Global

Dedicating to Continuous

ImprovementVMworld 2018 Content: Not for publication or distribution


Implementing the PracticesOut-learn your competition to create a competitive advantage

Just Learning CultureBlameless Post Mortems

Redefine Failure & Risk

Automate Knowledge CaptureCodify Standards in Software

Centralize Knowledge Repository

Reserve Time to ImproveEncourage All to Learn & Teach

Coach, Consult, & Share



DevOps Driven Knowledge:Creates Value. Fosters Innovation. Achieves Goals.



Making It HappenDeveloping Strategy & Executing DevOps Transformation Initiatives



Transitioning to DevOps Driven OperationsMoving from Current State IT to DevOps requires a more Wholistic Approach

CapabilitiesThe transformative

abilities, that contribute to the business as

additional value, which are enabled by the

service

CompetenciesThe skills,

methodologies, processes and tooling used to successfully

manage and deliver the service

CultureThe people,

organizational groups and roles that broker, deliver, manage and consume the offered

service



4 Point Vision for DevOpsKey Areas of Focus for Successful DevOps Transformation

Application Transformation

Services & CI/CD Infrastructure & Management

Process & Organization• Application portfolio assessment & prioritization based

on impact to critical business objectives

• Application architecture assessment & transformation to determine the requirements and prioritization of re-deploying, re-platforming, re-factoring, and/or sunsetting of application instances

• Transition of software development processes & procedures to align Agile Development best practices

• Transformation infrastructure and operations processes to align to Site Reliability Engineering (SRE) standards

• Realignment of organizational structure, roles, & responsibilities to support a collaborative DevOps operating model

• Design, build, & deploy foundational services such as Infrastructure as a Service (IaaS), Container as a Service (CaaS), & Platform as a Service (PaaS)

• Implement and integrate software delivery pipelines with tools in all elements of the various DevOps tool chain stacks to support continuous integration & continuous delivery

• Implement fully virtualized compute, storage, & network infrastructure platforms to provide a consistent infrastructure layer

• Configure and integrate a common cloud management platform to provide end-to-end automation capabilities VMworld 2018 Content: Not for publication or distribution


Developing Strategy &Transformation PathTaking a capability driven approach to strategy development

INFRASTRUCTURE PROVIDER

DEPARTMENT PARTNER

DIGITALDEPARTMENT

CYBERSECURITY INSIGHT PROTECT CONTROL

CONSUMER IDENTITY APPS DESKTOP MOBILITY

Private Cloud• IT Portal• Automated Provisioning• Resiliency• Showback

Hybrid Cloud• Cloud Bursting• Relevant Service Metrics

PaaS• Converged Application

Authoring (WaaS, DBaaS)• Change and


XaaS• Service Lifecycle Extensibility • Integrated Service Lifecycle

Third Platform• Multi-Cloud• Cloud Native Apps• Big Data Driven• Agile Focused• API Driven• Continuous Service Improvement

DevOps• Micro-service Architecture• Enhanced Network Security• Continuous Delivery

Digital Convergence• New Business Models• Continuous Customer Engagement• Digital Engagement

IaaS• Service Lifecycle Management• Service Catalog• Org and Process Transformation• Chargeback

Virtual Infrastructure• Virtualize across Compute,

Storage and Network Assets• Performance and Capacity

4

31

6

5

2

7



Year 1 Year 2 Year 3

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D

Notional 3 Year Adoption Roadmap & Implementation PlanIllustrative Plan to Iteratively Deliver Cloud Infrastructure & DevOps Service Capabilities

Infr

astr

uctu

reA

pp

licat

ions

1.1 - App. Migration

Asses Pilot Migrate

Phase 3 – Improve & Extend

1.2 - App. Migration

Asses Pilot Migrate

2.1 - App. Transformation

Analyze Re-Engineer

Re-Deploy

2.2 - App. Transformation

Analyze Re-Engineer

Re-Deploy

4.1 – CI/CD Strategy

Asses Architect Plan

4.2 – CI/CD Strategy Alignment


5.1 – CI/CD Implementation

Design Build Transition

5.2 – CI/CD Augmentation


4.3 – CI/CD Strategy Alignment


5.3– CI/CD Augmentation


6.1.1 – IaaS Basic Production Implementation

6.1 – Infrastructure as a Service (IaaS)6.1.2 – IaaS Advanced Production Implementation

7.1.1 – Hosted Containers (VIC) Pilot

7.1 – Container as a Service (CaaS)7.1.2 – Hosted Containers (VIC) Production Implementation

7.1.3– Scheduled Containers (PKS) Pilot

7.1.4 – Scheduled Containers (PKS) Production Implementation

8.1.1 – Platform on Demand PaaS Pilot

8.1 – Platform as a Service (PaaS)8.1.2 – Platform on Demand PaaS Production Implementation

8.1.3– Structured PaaS (PCF) Pilot 8.1.4 – Structured PaaS (PCF) Production Implementation

Consolidated SRE Operations – IaaS, CaaS, & PaaS Services

3.1 – Application Development Operating Model Transformation (AGILE Process & Org.)

9.1 – Infrastructure & Operations Operating Model Transformation (SRE Process & Organization)

1

2

3

4

5

6

7

8

9

SRE Operations – IaaS Services

Agile Development – Wave 1 & Wave 2 Applications

Agile Development – Wave 1 Applications

Basic DevOps Converged DevOps DevOps at Scale

Phase 2 – Implement, Optimize, & ScalePhase 1 – Evaluate, Migrate, & Innovate








Platformon



&Continuous

Delivery

Plan Feedback


CD Config Control

Management&

Operations

Security

Automation

Governance

Operations

UnifiedManagement


Instances



Processes

VersionControl

CodeInspection

BuildManagement

TestManagement

DigitalWorkspace

Applications

Existing Apps



Virtual Desktops

PlatformOperators


Application Users

Po

licy

& G

ove

rnan

ceInfrastructure

OperationsProcesses

ChangeManagement

Release&

Deploy


IncidentManagement

ServiceCatalog

Management

CapacityManagement

RequestFulfillment

DemandManagement

Multi-Cloud

DeveloperReady

Infrastructure

Cloud Native Application Architecture

SiteReliability

Engineering


‹#›©2018 VMware, Inc.

Questions?

44


PLEASE FILL OUTYOUR SURVEY.Take a survey and enter a drawingfor a VMware company store gift card.

#vmworld #DEV2500BUVMworld 2018 Content: Not for publication or distribution

THANK YOU!

#vmworld #DEV2500BUVMworld 2018 Content: Not for publication or distribution



DevOps and Pragmatic distribution Process Change: …...#vmworld DevOps and Pragmatic Process Change: Top 5 Processes to Accelerate DevOps Josh Miller, VMware, Inc. Kai Holthaus, VMware,

Documents