Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
What is “DevOps” Anyway?
Kelly GoetschDirector, Product ManagementCloud Application Foundation2015
A Pragmatic Introduction
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Agenda
3
1 What is DevOps? And why now?
2 How to Change Your Culture
3 How to Change Your Technology
4 Case Study: Oracle’s Internal Embrace of DevOps
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Dev and Ops Constantly Argue“Code is written...it’s your problem now”
4
Hey Ops - Here’s our code...good luck!
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Modern DevOps = Culture + Technology MovementCulture is what’s behind DevOps; technology is the enabler
5
Culture
Technology
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Core DevOps Principles
6
Cultural movement enabled by technology
Paid to add new features
Dev Ops
DevOps
Paid to keep system stable, fast and available
New goal:Add new features and keep the system stable, fast and available
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Characteristics of DevOps Movement
7
Principles have been around for decades
Agile Methodologies
Open Culture
Cloud-like Infrastructure
Heavy Automation
Open Source
DevOps
Movement began in startup community. Use of open source seen as integral but not technically necessary
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Old Culture and Expectations Don’t Work!Commonly overheard in offices
8
It’ll take us three months to build you
a development environment
I have to stay up all night to do a
build
The new switch we need is sitting on the loading dock
That code change will have to wait for our monthly
build
Uh...that’s not my problem. Go talk
to Ops
Why does your code always break my infrastructure?
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
DevOps Tenet #1: Culture
9
DiscussRespect
Avoid Blaming
“Done” Means
Released
•Dev respect for ops•Ops respect for dev•Don’t stereotype•Don’t just say “no”
•Don’t hide things•Ops should be in dev discussions•Dev should be in ops discussions• Shared runbooks/escalation plans•Ops should give devs access to
systems
•No fingerpointing!
•Dev’s responsibility ends when it’s in production• “Throwing it over
the wall” is dead
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
DevOps Tenant #2: Technology
10
Shared Version Control
Infra as Code
One Step Build/De
ploy
Don’t Fix Anything
•Use config mgmt to build environments• Scripts checked in
and managed as src
• Single system for code and build artifacts•Every time someone commits a
change it triggers a build and automated build verification tests• Ship trunk•Enable features through flags
•One button build/deploy (manual)• Scheduled builds/deploys• If verification fails, stop and alert
• If something breaks, re-deploy. Don’t fix• Fix environment
setup scripts
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Faster Time-to-Market
• Increase frequency of releases
• Increase accuracy of releases - avoid downtime
• Reduce the time it actually takes to perform a release
Cost
• Automate what was previously done manually. Reduces OPEX
• Prevent humans from making costly errors
• Reduce downtime, which saves money
11
Focus on Business Value
• Allow high value employees to focus on higher value activities
• Nobody benefits from doing low value activities, like setting up infrastructure
Business Value Is Driving DevOps
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Time
• Longer time to market (initially) due to work required to automate
• More builds / faster time-to-market for individual features
• Lower MTTR because problems are fixed by rebuilding
• Higher MTBF due to increased accuracy
Cost
• High up-front costs due to automation work
• Low ongoing costs because routine operations is automated
• Fewer employees wanting more pay - 10x developers want 10x pay
• Lower capex, higher opexas workloads shift to cloud
12
Culture
• Continues to empower developers
• Automation increases operational efficiencies
• Distracts developers from developing code
• Movement eschews “rockstars” but that’s naturally what happens
DevOps Implications To Business
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Agenda
13
1 What is DevOps? And why now?
2 How to Change Your Culture
3 How to Change Your Technology
4 Case Study: Oracle’s Internal Embrace of DevOps
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
How to Build a Successful DevOps Culture
14
DiscussRespect
Avoid Blaming
“Done” Means
Released
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Build Respect
• Developers should respect Operations
• Operations should respect Developers
• Don’t just say “no”
• Don’t stereotype!
15
Because the site breaks
unexpectedly
Because nobody tells
them anything
Because they say “no” all
the time
Ops Stereotype
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Discuss
• Don’t hide things! Open communication on both sides
• Operations should be in Developer discussions
• Developers should be in Operations discussions
• Build shared runbooks/escalation plans
• Operations should give Developers direct access to systems
16
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Avoid Blaming
• No fingerpointing!
• Development should have enough Operations culpability to share the blame in an outage
• Operations should have enough Development culpability to share the blame in an outage
17
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
"Done" Means Released
• The “throwing it over the wall to ops” model is dead
• Development needs to be involved with operations
• Operations needs to be involved with development
• Joint success/failure
18
Development
Operations
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Steps Required to Change Culture
19
Hire technical people who are skilled in DevOps Set up training for DevOps skills Bring in consultants to help get started
Put Dev + Ops people under same line-level managers Change the way you measure/reward performance Flatten org chart to reduce fiefdoms
Let Dev + Ops collaborate to find solutions Let Dev + Ops jointly pick tools. Limit corporate mandates Don’t restrict the choice of public cloud solutions
Sponsor teambuilding events Give Dev + Ops common goals to work towards When there's a failure, don't dwell on Dev or Ops failure
Build Trust
Bring In New Blood
Align Interests
Empower Change From the Bottom Up
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Change Your Organization’s StructureIncentives drive behavior
20
CIO
VP of Operations
Developer
VP of Development
Layers of Middle
Management
Layers of Middle
Management
Sys Admin
CIO
Developer
VP of Product
Layers of Middle
Management
Sys Admin
Report to same manager. Incentivized to work together to develop and release
Incentivized to innovate and release
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Actively Build Trust
21
Trust is the #1 ingredient to a successful DevOps culture
Dev + Ops + Social Activity Outside Work + Time = Trust
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Agenda
22
1 What is DevOps? And why now?
2 How to Change Your Culture
3 How to Change Your Technology
4 Case Study: Oracle’s Internal Embrace of DevOps
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
How to Use a Technology to Enable DevOps
23
Shared Version Control
Infra as Code
One Step Build/De
ploy
Don’t Fix Anything
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Infrastructure as Code
24
Manage it as you would any other source code
Base Image
Install Binaries
Configure Software
Make Software Work Together
Patch/Push Config Changes
Step 1Pick a Tool
Step 2Script your environment
Step 3Run your scripts against all hosts
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Humans doing anything manually will introduce errors
Consider Disabling SSH AccessThis could alienate ops people, however, as it shows a lack of trust
25
Possible solution: Remove each user’s shell
• All changes should be scripted and QA’d
– This prevents one-off changes (and errors)
– Emergency fixes can still be pushed as required
• Users can still SFTP to retrieve log files
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Shared Version Control
• Single system for code and build artifacts
• Every time someone commits a change, consider triggering a build + automatic verification tests
• Always ship trunk!
• Enable features through flags
Surprisingly not well adopted
26
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
One Step Build/Deploy
• Manual one button build/deploy
• Scheduled builds - every day, every week, etc
• Builds triggered by code checkins
• If post-build validation fails, report it
Set it and forget it
27
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Popular DevOps TechnologyRemember: You can't get DevOps by buying a single product
28
Ansible Jenkins / Hudson Rundeck
Git Perforce Subversion
Ansible Puppet Chef SaltStack
Gradle Jenkins / Hudson Robot
Artifactory Nexus
Docker Vagrant
Version Control
Build & Functional Testing Binary ManagementVirtualization
Continuous IntegrationContinuous Delivery
Configuration Management
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Agenda
29
1 What is DevOps? And why now?
2 How to Change Your Culture
3 How to Change Your Technology
4 Case Study: Oracle’s Internal Embrace of DevOps
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Product Development IT (PDIT)
• 602 Exadata
• 383 Exalogic
• 69,290 Sun x86 servers
• 6,854 SPARC servers
• 1,768 Sun ZFS appliances
• Thousands of other systems
30
Supporting enterprise, development IT, cloud and managed hosting at Oracle
924 Oracle Products135,000 Employees
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
How Did PDIT Improve Efficiency by > 2x in 18 Months?Culture changes enabled by technology
31
powered by
powered by
Culture(Campbell Webb, Sr. VP, Oracle)
+ +
CloudThe Same Products Oracle Sells
Infrastructure Automation
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Oracle Cloud is Provisioned Using Chef
Database Services
Java Services
Analytics Services
Mobile Services
Developer Services
Collaboration Services
CachingServices
Messaging Services
Notification Services
StorageServices
Application Store
IdentityServices
Powered By
32
Oracle Cloud is run on Chef
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Other PDIT Changes
33
Standardize Hardware Comprehensive Monitoring
Standardize Software (N, N-1) CMDB / Asset Management
IaaS (Nimbula/OpenStack) Proactive Monitoring
Symmetric Test/Prod/DR Environments Offshoring (1 x US + 2 x IDC shifts)
Nearline Backups, Compression, Dedup Maximize Depreciation
VLAN’s => VxLAN’s Defense in Depth Security Posture
Big Data Security Analytics (Hadoop) Centralized Identity Management
Elasticsearch+Logstash+Kibana Analytics Config Management (Chef/Puppet)
Orchestration (Mcollective)
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
Core DevOps Principles
34
Cultural movement enabled by technology
Paid to add new features
Dev Ops
DevOps
Paid to keep system stable, fast and available
Unified goal:Add new features and keep the system stable, fast and available
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.
DevOps Is Ultimately In the Eye of the BeholderWhat’s important is that you adhere to core principles that lead to business value
35