Top Banner
TB Fullday Tutorial 6/4/2013 8:30 AM "Continuous Delivery: Rapid and Reliable Releases with DevOps Practices" Presented by: Bob Aiello CM Best Practices Consulting Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 8882688770 9042780524 [email protected] www.sqe.com
126

Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Jan 14, 2015

Download

Technology

DevOps is an emerging set of principles, methods, and practices that empower teams and organizations to rapidly deploy systems and application updates while maintaining—and even improving—quality. By lowering barriers between development, testing, and operations, DevOps practices can add tremendous business value to software projects and systems. Bob Aiello explains how to prepare for and implement continuous delivery—in both agile and non-agile environments—employing industry standard processes and automated frameworks. Bob shares DevOps best practices starting with its role early in the application lifecycle through release and application maintenance. He introduces the emerging “Infrastructure as Code” concept that automates server and system provisioning within cloud computing environments. Learn ways to overcome technical, process, and cultural challenges with DevOps. Take back a set of practical and proven practices—for automated application build, automated packaging, and automated deployment—that will put your organization on the path to rapid and reliable releases.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

 

 

TB Full‐day Tutorial 6/4/2013 8:30 AM 

      

"Continuous Delivery: Rapid and Reliable Releases with

DevOps Practices"    

Presented by:

Bob Aiello CM Best Practices Consulting

       

Brought to you by:  

  

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Page 2: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Bob Aiello CM Best Practices Consulting

Editor-in-chief of CM Crossroads and author of Configuration Management Best Practices: Practical Methods that Work in the Real World, Bob Aiello is a consultant and software engineer specializing in software process improvement, including software configuration and release management. He has more than twenty-five years of experience as a technical manager at top New York City financial services firms, where he held company-wide responsibility for configuration management. Bob served as vice chair of the IEEE 828 Standards Working Group on CM Planning and a member of the IEEE Software and Systems Engineering Standards Committee (S2ESC) Management Board. Contact Bob at [email protected], via LinkedIn, or visit cmbestpractices.com.

 

Page 3: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Continuous Delivery (DevOps Best Practices)

1

Bob Aiello, Principal Consultant and Author of Configuration Management Best Practices : Practical Methods that Work in the Real World

http://www.linkedin.com/in/BobAiellohttp://cmbestpractices.com

CM Best Practices Consulting © 2013

Page 4: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Who am I?

• CM/DevOps Lead & Consultant for over 25 years• Editor-in-Chief at CM Crossroads• Author of CM Best Practices• IEEE Management Board • Tools and process agnostic• The guy called in the middle of the night when the release doesn’t work!

2 April 9, 2013 http://cmbestpractices.com © 2013

Page 5: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Goals of this Course• Understand Continuous Delivery• Configuration Management roots• Control Dependencies & Configuration• Continuous Integration • Build and Deployment Automation• Deployment Pipeline is an Art!

Agile Release Train3 April 9, 2013 http://cmbestpractices.com © 2013

Page 6: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

DevOps Focus

• Understand DevOps Best Practices• A Little History of DevOps• Scope of DevOps and how to get started• The People side of DevOps

Establish your own plan for DevOps!

4 April 9, 2013 http://cmbestpractices.com © 2013

Page 7: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

And Don't Forget • Delivery Ecosystem• Components & Dependencies• Test vs Verification & Validation (V&V)• Don't forget the Data• Establish IT governance and compliance

So what is Continuous Delivery? 5 April 9, 2013 http://cmbestpractices.com © 2013

Page 8: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Continuous Delivery

6

• Methodology for getting software from development to release• Focus on the Deployment Pipeline• Rapid incremental deployment• Minimize Risk• Many small deployments better than big bang

http://cmbestpractices.com © 2013 April 9, 2013

Page 9: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Continuous Integration

7

• What is Continuous Integration• Why does CI work?• Martin Fowler reminds us to test• Let's consider the ergonomics

http://cmbestpractices.com © 2013 April 9, 2013

Page 10: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Lessons from Aviation

8

• Cockpit of a plane

• Controls are easy to read• Traceability• Designed to avoid mistakes

How does this relate builds?

http://cmbestpractices.com © 2013 April 9, 2013

Page 11: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Ergonomics of the Build

9

• “Bob-proof” your build• Implicit verification and validation• Avoid the possibility of mistakes• Each step should be easy to understand• One step should not break the stream• Use dashboards and reports to communicate status

http://cmbestpractices.com © 2013 April 9, 2013

Page 12: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Knight Capital Group

10

• August 1st 2012 trading disaster• Related to NYSE systems upgrade• Resulted in a $440 million loss• Loss grew as customers left the firm• Knight Capital Group merged with GETCO holding company

DevOps doesn't cost $440 million http://cmbestpractices.com © 2013 April 9, 2013

Page 13: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Introducing the Trusted Base

11

• Ensure that you know exactly what you built• Verify that the release gets deploy• Ensure that there are no unauthorized changes

Understanding Continuous Integration

http://cmbestpractices.com © 2013 April 9, 2013

Page 14: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Features of CI

12

• Source Code Management• Building the Code• Database integration• Testing • Source code inspection• Deployment

Controlling isolationhttp://cmbestpractices.com © 2013 April 9, 2013

Page 15: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Controlled Isolation

13

• Developers work in sandboxes• Deliver (actually published) your changes• Rebase before you deliver• Private builds • Manage variants

Consider the ergonomicshttp://cmbestpractices.com © 2013 April 9, 2013

Page 16: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Ergonomics of CI/CD

14

• Small incremental changes• Early Warning• Reduce risk• Easier to triage• Easier to backout

What about life support systems?

http://cmbestpractices.com © 2013 April 9, 2013

Page 17: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

I was once asked...

15

• What if you were upgrading a life support system and your loved one was impacted• How do we ensure that there are no mistakes• Join me in Detroit in July

How do we keep CI lean?http://cmbestpractices.com © 2013 April 9, 2013

Page 18: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Keeping CI Lean

16

• Too many builds• Too much noise • Tag interesting builds

What about pilots and landings?

http://cmbestpractices.com © 2013 April 9, 2013

Page 19: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Pilots Abort Landings

17

• Recently I was on a plane when another plane was on the same runway• You want to fail early when necessary• Abort bad builds and identify the cause

What are the best practices?

http://cmbestpractices.com © 2013 April 9, 2013

Page 20: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Seven Practices, Duvall p. 39

18

• Commit code frequently• Don't commit broken code• Fix broken builds immediately• Write automated developer tests• All tests & inspections must pass• Run private builds• Avoid getting broken code

http://cmbestpractices.com © 2013 April 9, 2013

Page 21: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Chicago Board Options Exchange

19

• Planned systems upgrade• Problem with staging software• Employees reported there was a problem• CBOE did not fail over

Could DevOps have helped the CBOE?

http://cmbestpractices.com © 2013 April 9, 2013

Page 22: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

What is DevOps?

20

• New Term for...• Portmanteau• Agile Systems Administration• Agile Operations• Group of Principles

Now that we cleared that up!

http://cmbestpractices.com © 2013 April 9, 2013

Page 23: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

New Term

21

• Group of concepts• Been around for a while• Use case is compelling• Stimulating discussion• Necessary to meet demand

http://cmbestpractices.com © 2013 April 9, 2013

Page 24: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Portmanteau

22

• Combination of two words• Development• Operations

Development and Operations have very different goals

http://cmbestpractices.com © 2013 April 9, 2013

Page 25: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Conflict Between Dev & Ops

23

• Development focused on delivering new functionality• Operations is focused on providing continuous (reliable) services• Manage risk!

One time I was asked to break the rules

http://cmbestpractices.com © 2013 April 9, 2013

Page 26: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Trying to make the deadline

24

• Trading system was tested and passed• Few bugs discovered• I was asked to deliver a different version than was tested

How does DevOps help balance?

http://cmbestpractices.com © 2013 April 9, 2013

Page 27: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

DevOps is also

25

• Emerging Best Practices• Collaboration between Dev & Ops• Application and Systems Deployment• Software and Systems Development

But is DevOps Agile?

http://cmbestpractices.com © 2013 April 9, 2013

Page 28: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

What about Agile?

26

• Agile Systems Administration• Agile Operations• Waterfall needs DevOps too!

Release Antipatterns...

http://cmbestpractices.com © 2013 April 9, 2013

Page 29: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Release Antipatterns

27

Deploying software manually Deploying to a production-like environment only after development is complete Manual configuration of production environment.

So what really is DevOps?http://cmbestpractices.com © 2013 April 9, 2013

Page 30: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

DevOps is Really...

28

• Developer and Operations collaboration• Crossfunctional team• Knowledge Management• Better communication

Time to get rid of silos

http://cmbestpractices.com © 2013 April 9, 2013

Page 31: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

What is Ops?

29

• Blanket term• Systems engineers• Systems administrators• Operations staff

http://cmbestpractices.com © 2013 April 9, 2013

What's honesty got to do with all this?

Page 32: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Agile on What We Know

30

• Don't try to define requirements we do not yet understand• Last responsible moment• Requirements documents that are unusable

Deming says, “drive out fear”

http://cmbestpractices.com © 2013 April 9, 2013

Page 33: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

What is a Deployment Pipeline?

31

• Build once• Deploy the same way to every environment• Smoke is essential• Deploy to a copy of Production• Manage the pipeline

Environment managementhttp://cmbestpractices.com © 2013 April 9, 2013

Page 34: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Deployment Pipeline

32

A deployment pipeline is … an automated implementation of your application’s build, deploy, test and release process

Jez Humble and David Farley’s Continuous Delivery, p 3.

http://cmbestpractices.com © 2013 April 9, 2013

Page 35: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Aim of the Pipeline• Makes building, deploying, testing and releasing software visible to everyone involved• Improves feedback so that problems are identified, and so resolved, as early in the process as possible• Enables teams to deploy and release any version of their software to any environment at will through a fully automated process (p. 4)

33http://cmbestpractices.com © 2013 April 9, 2013

Page 36: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Antipatterns

34

• Deploying Software Manually• Deploying to Production-like environment only after Development is complete• Manual Configuration of Production Environments Continuous Deployment, p. 7 – 10

http://cmbestpractices.com © 2013 April 9, 2013

Page 37: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Agile Release Train (ART)Making each product a successful and routine event – an event that is indeed planned and eagerly anticipated yet one one that happens almost on autopilot

Dean Leffingwell’s Agile Software Requirements, p. 299

35http://cmbestpractices.com © 2013 April 9, 2013

Page 38: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

How Do We Implement?Are deployment pipelines practical?How do we figure out the details?Is it worth the time and effort?What are the benefits?What are the risks?

It's really all about knowledge...36http://cmbestpractices.com © 2013 April 9, 2013

Page 39: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Knowledge Management

37

• There are always a few experts• They are not working when system glitch• Building a Knowledge Management System• But What if some people do not want to share? (caution silos ahead)

http://cmbestpractices.com © 2013 April 9, 2013

Page 40: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Beware of Silos

38

• The SAs see file systems • The DBAs have a different view• WebSphere Admins• InfoSec helped us secure (so much nothing worked)

DevOps is about sharing knowledge!

http://cmbestpractices.com © 2013 April 9, 2013

Page 41: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Where did DevOps start?

39

• O'Reilly Velocity Conference 2008 • Web Performance and Operations• “Infrastructure as Code”

http://cmbestpractices.com © 2013 April 9, 2013

Page 42: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Need for Rapid Change

40

• 2009 Presentations on developer / operations collaboration at large shops along with safe rapid change of Web environments

http://cmbestpractices.com © 2013 April 9, 2013

Page 43: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

DevOps Days

41

• Patrick Debois – DevOpsDays in 2009• Tools (actually toolchains) have brought together “the three layers of what you need for agile movement (principles, process and practices)”

http://cmbestpractices.com © 2013 April 9, 2013

Page 44: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Let's Get Into Some Details

42

• How do we implement?• How do we make pragmatic choices?• How do we do this in the real world?

My experience taking down NYSE

http://cmbestpractices.com © 2013 April 9, 2013

Page 45: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

I Was Once Accused

43

• Promoting the wrong shell scripts• Taking down the NYSE• Stopping the World Economy

Principles of Software Delivery

http://cmbestpractices.com © 2013 April 9, 2013

Page 46: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Continuous Delivery

44

• Configuration Management focus• Version control• Dependency and configuration control

Principles of Software Delivery

http://cmbestpractices.com © 2013 April 9, 2013

Page 47: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Software Delivery Principles

45

• Create repeatable, reliable process• Automate as much as possible• If it hurts, do it more often!• Build quality in from the beginning (Deming)

any more?http://cmbestpractices.com © 2013 April 9, 2013

Page 48: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

More Principles

46

• Done means released• Everyone is responsible for the delivery process• Continuous improvement• Version control is key

So what is CM?http://cmbestpractices.com © 2013 April 9, 2013

Page 49: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Configuration Management

47

• Configuration Identification• Status Accounting• Change Control• Configuration Audit

Tracking and Controlling Changes to Configuration Items

http://cmbestpractices.com © 2013 April 9, 2013

Page 50: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Configuration Identification

48

• Provides a specific and unique identity to each configuration item (e.g. binary, config file, documentation)

• Selecting the configuration items for a system and recording their functional and physical characteristics (Sevocab)

http://cmbestpractices.com © 2013 April 9, 2013

Page 51: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Status Accounting

49

• Tracking the status of a configuration item throughout its lifecycle.

• Recording and reporting of information needed to manage a configuration effectively (Sevocab)

http://cmbestpractices.com © 2013 April 9, 2013

Page 52: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Change Control

50

• Establishing checkpoints including gatekeeping (e.g. Production, QA, UAT) and configuration control.

• Identifying, documenting, approving or rejecting, and controlling changes to the project baselines (Sevocab)

http://cmbestpractices.com © 2013 April 9, 2013

Page 53: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Configuration Audit

51

• Inspect and identify the exact version of any configuration item (physical & functional)

• Independent examination of the configuration status to compare with the physical configuration (Sevocab)

http://cmbestpractices.com © 2013 April 9, 2013

Page 54: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Functional description of CM

52

• Easier to understand in the context of a lifecycle • Consisting of six core CM functions• Closely matches the job descriptions of the people doing the work• Can be tailored to your needs

So what are the six functions?http://cmbestpractices.com © 2013 April 9, 2013

Page 55: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

CM Functions

53

• Source Code Management• Build Engineering• Environment Configuration• Change Control• Release Engineering• Deployment

Let's start with a brief overviewhttp://cmbestpractices.com © 2013 April 9, 2013

Page 56: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

My buddy from Harvard

54

• Builds are too complex to automate• Some folks do not want to see automation as being possible• You may have to shadow or ask to drive• Document the procedures and then

Script your build...http://cmbestpractices.com © 2013 April 9, 2013

Page 57: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Build Principles

55

• Create a script for each stage of the build process• Use the right technology to deploy (find out what others are doing)• Use your operating systems native tools

http://cmbestpractices.com © 2013 April 9, 2013

Page 58: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

More Build Principles

56

• Idempotent – reliable and no side effects• Evolve your deployment system incrementally• Start with “attended automation”

http://cmbestpractices.com © 2013 April 9, 2013

Page 59: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Some other tips

57

• Relative paths (watch your paths)• Eliminate manual steps• Traceability from binary to source• Test targets should not fail the build

What do I do with binaries?

http://cmbestpractices.com © 2013 April 9, 2013

Page 60: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Managing Binaries

58

• Binaries can be rebuilt • Based upon baselines• Verifiable (I hope)• Don't belong in the VCS with source• Definitive Media Libraries• Release Repos

Managing Variants in the Codehttp://cmbestpractices.com © 2013 April 9, 2013

Page 61: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Version Control Features

59

• Provides history and security• Model the architecture• Reduce complexity• Model the process

More on streams

http://cmbestpractices.com © 2013 April 9, 2013

Page 62: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Source Code Management

60

• Control of every configuration item (e.g. source code, config, binaries, compile and runtime dependencies). • Much more than just checkin and checkout (version control)• Provides sanity to the development process (reduces cognitive complexity)

http://cmbestpractices.com © 2013 April 9, 2013

Page 63: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Terminology

61

• Configuration items (CIs) include binaries, source code, config files and even documents• ISO 1007 notes end user function• Bob says, “anything where getting the wrong version would be bad”

http://cmbestpractices.com © 2013 April 9, 2013

Page 64: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

What is Control?

62

• In CM, control is managing the evolution of a CI throughout its lifecycle• Change Control• Configuration Control

Is control really the right word?

http://cmbestpractices.com © 2013 April 9, 2013

Page 65: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Principles

63

• Code is locked down and can never be lost• Code is baselined marking specific milestones• Managing variants using branches• Code changed on a branch can be merged

http://cmbestpractices.com © 2013 April 9, 2013

Page 66: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

More Principles

64

• Processes are repeatable Agile and Lean• Traceability and tracking of all changes• Improves productivity and quality

http://cmbestpractices.com © 2013 April 9, 2013

Page 67: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Best Practices

65

• How do we establish source code management that adheres to these principles?• Better question is how does CM add value and help facilitate the development effort?

http://cmbestpractices.com © 2013 April 9, 2013

Page 68: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Streams

66

• Provides a clear usage paradigm• Model components and architecture• Control flow of changesets• Snapshots create baseline of code

http://cmbestpractices.com © 2013 April 9, 2013

Page 69: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Streams

67

Ability to load a particular snapshot Strong security authorization and entitlements Complete history and traceability

How about task based development?

http://cmbestpractices.com © 2013 April 9, 2013

Page 70: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Defect & Task Tracking

68

• Track changesets to workitem • Traceability to who made the change• Makes release notes a breeze to create• Ties back to requirements and test cases• Allows for ALM and workflow automation

http://cmbestpractices.com © 2013 April 9, 2013

Page 71: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

InfoSec Scans Code

69

• Source Code Inspection• Are coding practices creating risk?• Are passwords being hardcoded• Scan for complexity• Code quality

Managing globally distributed teams

http://cmbestpractices.com © 2013 April 9, 2013

Page 72: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Globally Distributed team

70

• Managing work for a globally distributed team• Effective communication• Better coordination• Traceability • Visibility

http://cmbestpractices.com © 2013 April 9, 2013

Page 73: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Build on Commit

71

• Nightly builds often enough• Build on demand• Pre-flight (private) builds• Build framework

Tame the complexity and communicate via dashboards

http://cmbestpractices.com © 2013 April 9, 2013

Page 74: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Deploy to Environment

72

• Run automated tests• Monitor the environment• Build the Ops Knowledgebase• Building our deployment framework

Infrastructure as code

http://cmbestpractices.com © 2013 April 9, 2013

Page 75: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Infrastructure as Code

73

• Provisioning Servers• Fundamental in the Cloud• What about private clouds?• Managing the OS

Puppet and chef

http://cmbestpractices.com © 2013 April 9, 2013

Page 76: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Puppet/Chef

74

• Automate provisioning, patching and configuration of operating system and application components• Systems integration framework• Scalable and extensible• Used in other deployment frameworkswww.puppetlabs.com www.opscode.com

http://cmbestpractices.com © 2013 April 9, 2013

Page 77: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

CIS Benchmark

75

• Center for Internet Security (CIS)• Consists of hundreds of recommended configurations• Code is included to verify the configuration

This is all about taming complexity

http://cmbestpractices.com © 2013 April 9, 2013

Page 78: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Taming Complexity

76

• Understand the technology• Automate everything• Do it more often• Move upstream• Build a framework

By the time we get to Production...

http://cmbestpractices.com © 2013 April 9, 2013

Page 79: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Build Once

77

• Build once – deploy everywhere• Ensure bits are identical• Build based upon baseline • Embed immutable version IDs• Configuration audit

Automated deployments

http://cmbestpractices.com © 2013 April 9, 2013

Page 80: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Deployment Frameworks

78

• Starts with scripting• Many dependencies• Taming complexities• Test each step

Traceable, Repeatable Process

http://cmbestpractices.com © 2013 April 9, 2013

Page 81: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Deploy the Same Every Environment

79

• Write a deployment framework• Practice the deploy• Well oiled machine• Repeatable and traceable

DevOps Focus

http://cmbestpractices.com © 2013 April 9, 2013

Page 82: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

DevOps

80

• Moving automation upstream• Communicating with stakeholders• Building knowledge• Infrastructure as code

Smoke testing is required

http://cmbestpractices.com © 2013 April 9, 2013

Page 83: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

SmokeTest

81

• Test the deploy itself• Put in the first trade• Verify what changed• Work with QA & Testing

Environments need to be similar to Production

http://cmbestpractices.com © 2013 April 9, 2013

Page 84: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Deploy into Copy of Production

82

• You need a dress rehearsal• Need to verify automation works• Need to know the deploy will work• Manage risks and unknowns

Deploys need to be verifiable

http://cmbestpractices.com © 2013 April 9, 2013

Page 85: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Changes Through the Pipeline

83

• Every commit triggers• Build and deploy automation• Testing the release

What are the recommended practices?

http://cmbestpractices.com © 2013 April 9, 2013

Page 86: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Pipeline Practices

84

• Only build binaries once• Deploy the same way to every environment• Smoke test Changes should propagate instantly continuously

The process itself must be testablehttp://cmbestpractices.com © 2013 April 9, 2013

Page 87: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Verification and Validation

85

• Does it meets requirements?• Are the requirements correct?• Deming – build in quality• Each step is testable

When problems occur...

http://cmbestpractices.com © 2013 April 9, 2013

Page 88: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Stop the Line

86

• Need to detect defects early• Stop the process immediately• Easier to diagnose• Easier to fix

Kanban for the Deployment Pipeline

http://cmbestpractices.com © 2013 April 9, 2013

Page 89: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Kanban

87

• Push• Pull• Implement through workflow automation

Delivery Environment as an Ecosystem

http://cmbestpractices.com © 2013 April 9, 2013

Page 90: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Delivery Ecosystem

88

• Understanding Operations• Managing Infrastructure• Server Provisioning & Configuration• Managing Middleware• Virtualization & Cloud

DevOps should focus on Ops

http://cmbestpractices.com © 2013 April 9, 2013

Page 91: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Operations

89

• Key stakeholder• Often outgunned and kept in the dark• Building a knowledgebase• Automating detection and response

How can Ops get ahead of the curve?

http://cmbestpractices.com © 2013 April 9, 2013

Page 92: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Operations in DevOps

90

• Infrastructure as Code• Provisioning servers• Monitoring the environment• Monitoring events

InfoSec is also key

http://cmbestpractices.com © 2013 April 9, 2013

Page 93: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

InfoSec

91

• Key stakeholder• Often misinformed• Policies don't secure systems• Many incidents show this is a problem area

Securing the trusted base

http://cmbestpractices.com © 2013 April 9, 2013

Page 94: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Securing the Trusted Base

92

• Builds are baselined• Version IDs are embedded• Configuration audit• Non-repudiation

Security is quality

http://cmbestpractices.com © 2013 April 9, 2013

Page 95: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Build Security In

93

• Security should be considered from the beginning• Security and quality are tightly coupled• Provision servers using standards• Control & Detect unauthorized changes

Manage components & dependencies

http://cmbestpractices.com © 2013 April 9, 2013

Page 96: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Managing Components

94

• Code should be designed into components• Reduces complexity• Interfaces are essential• Part of environment management

Managing Big Builds

http://cmbestpractices.com © 2013 April 9, 2013

Page 97: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Managing Big Builds

95

• Big builds may require multiple pipelines• Treat the team as internal products• Handle this as COTs

Configuration can be complex

http://cmbestpractices.com © 2013 April 9, 2013

Page 98: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Managing Configuration

96

• Many ways to handle this• Configuration files (httpd.conf)• Properties files (.properties)• XML as configuration (server.xml)• Default as production (so you don't forget!)

Managing Dependencieshttp://cmbestpractices.com © 2013 April 9, 2013

Page 99: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Managing Dependencies

97

• Maven and Ivy help identify dependencies• Need to be able to identify versions• Monitor and detect issues• Often controlled through data

But you have to test

http://cmbestpractices.com © 2013 April 9, 2013

Page 100: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Testing Topology

98

• Unit Testing• Functional• Regression• Integration• User Acceptance

What about non functional testing?

http://cmbestpractices.com © 2013 April 9, 2013

Page 101: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Non-functional Testing

99

• Capacity • Performance• Scalability

Shoemakers children...

http://cmbestpractices.com © 2013 April 9, 2013

Page 102: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Testing the Pipeline

100

• You need to test the automation including build, package and deployment • Fail early!• Trust but verify

Don't forget the data

http://cmbestpractices.com © 2013 April 9, 2013

Page 103: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Internal Audit Requirements

101

• Managing baselines• Traceability• Change control• Seperation of controls

Regulatory Requirements

http://cmbestpractices.com © 2013 April 9, 2013

Page 104: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Conducting an Assessment

102

• What is going well• What can be improved?

Assess to industry standards and frameworks

http://cmbestpractices.com © 2013 April 9, 2013

Page 105: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Regulatory

103

• Section 404 of the Sarbanes-Oxley Act of 2002• SSAE-16 (formerly SAS-70)• Finra• Office of the Currency

Standards and Frameworks

http://cmbestpractices.com © 2013 April 9, 2013

Page 106: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Industry Standards

104

• IEEE 828• EIA 649-B• ISO 12207 or 15288• ISO 9001

Frameworks also provide guidance

http://cmbestpractices.com © 2013 April 9, 2013

Page 107: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Frameworks

105

• Cobit for Sox Compliance• ITIL for IT Service Management• CMMI (less common in financial services)

http://cmbestpractices.com © 2013 April 9, 2013

Page 108: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Globally Distributed team

106

• Managing work for a globally distributed team• Effective communication• Better coordination• Traceability • Visibility

http://cmbestpractices.com © 2013 April 9, 2013

Page 109: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

The CD/CI/CM Process

107

• Should be Lean• Processes need to be reviewed• Tailor down or tailor up• More collaboration and consensus building• Use standards and frameworks

April 9, 2013 http://cmbestpractices.com © 2013

Page 110: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Assessment

108

• First step is to assess current practices - “As-Is”• Compare to industry standards and frameworks• Determine “To-Be” • Create a plan for improving your CM processes

April 9, 2013 http://cmbestpractices.com © 2013

Page 111: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

IT Governance & Compliance

109

• IT Governance needs to be in alignment with corporate governance• Financial reports needs to be accurate• Separation of controls• Security measures to prevent unauthorized access• Audit in place for intrusion detection

http://cmbestpractices.com © 2013 April 9, 2013

Page 112: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Sox Compliance

110

• Section 404 of the Sarbanes Oxley Act of 2002• Using ISACA Cobit 4.1 • 34 high level IT controls• PCI compliance• SAS-70

http://cmbestpractices.com © 2013 April 9, 2013

Page 113: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

ISO 9001

111

• Establishes the quality management system• ISO 90003 is the software standard in the 9000 family of standards • Uses ISO 12207 (or 15288) to specify lifecycle processes• ISO 10007 for CM• IEEE 828, EIA 649-A, Mil Std coming!

http://cmbestpractices.com © 2013 April 9, 2013

Page 114: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Which Standards?

112

• IEEE 828 – CM Planning• EIA 649-A – Non compliance• ISO 90003 to support QMS• Full lifecycle ISO 12207

Tailor !

http://cmbestpractices.com © 2013 April 9, 2013

Page 115: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Moving Upstream

113

• Dev to CM to QA to Ops• Cross functional focus• Speed up development• Build a great deployment architecture• Give it to Devs as a service!

http://cmbestpractices.com © 2013 April 9, 2013

Page 116: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Frameworks

114

• ITIL v3 including CMDBs, federated CMDBs, CMS, DML…• Cobit for SOX• CMMI ->>>> Agile

http://cmbestpractices.com © 2013 April 9, 2013

Page 117: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

How Do We Improve

115

• CSI is well - continuous• Inclusive • Transparent• Learning from mistakes

Retrospectives are essential

http://cmbestpractices.com © 2013 April 9, 2013

Page 118: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Retrospective

116

• After action review• Need open and honest evaluation• Opportunity to improve the process• Drives the entire release process

http://cmbestpractices.com © 2013 April 9, 2013

Page 119: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Plan for Improvement

117

• Improve training and use case for source code management• Improvement build automation• Setup or improve continuous integration• Automate package and deployment• Create procedures for configuration audit

April 9, 2013 http://cmbestpractices.com © 2013

Page 120: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

CM/Devops

118

• Flexible technical background• Good knowledge of development• Knowledge of QA/Ops• Strong automation skills• Some systems administration• Ability to work across silos

http://cmbestpractices.com © 2013 April 9, 2013

Page 121: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Toolsmith/Devops

119

• Strong technical background• Strong scripting skills• Diving deep into the tools including troubleshooting• Understands toolchains and finds flexible solutions• Process orientation – focus on traceability

http://cmbestpractices.com © 2013 April 9, 2013

Page 122: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Goals of this Course• Understand Continuous Delivery• Configuration Management roots• Control Dependencies & Configuration• Continuous Integration • Build and Deployment Automation• Deployment Pipeline is an Art!

Agile Release Train120 April 9, 2013 http://cmbestpractices.com © 2013

Page 123: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

DevOps Focus

• Understand DevOps Best Practices• A Little History of DevOps• Scope of DevOps and how to get started• The People side of DevOps

Establish your own plan for DevOps!

121 April 9, 2013 http://cmbestpractices.com © 2013

Page 124: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

And Don't Forget • Delivery Ecosystem• Components & Dependencies• Test vs Verification & Validation (V&V)• Don't forget the Data• Establish IT governance and compliance

So what is Continuous Delivery? 122 April 9, 2013 http://cmbestpractices.com © 2013

Page 125: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Continuous Delivery

123

• Methodology for getting software from development to release• Focus on the Deployment Pipeline• Rapid incremental deployment• Minimize Risk• Many small deployments better than big bang

http://cmbestpractices.com © 2013 April 9, 2013

Page 126: Continuous Delivery: Rapid and Reliable Releases with DevOps Practices

Continuous Delivery (DevOps Best Practices)

124

Bob Aiello, Principal Consultant and Author of Configuration Management Best Practices : Practical Methods that Work in the Real World

http://www.linkedin.com/in/BobAiellohttp://cmbestpractices.com

CM Best Practices Consulting © 2013