© 2017 Delphix. All Rights Reserved. Private and Confidential. © 2017 Delphix. All Rights Reserved. Private and Confidential. Kellyn Pot’Vin-Gorman | Technical Intelligence Manager| October, 2017 The Revolution DevOps to DataOps
© 2017 Delphix. All Rights Reserved. Private and Confidential.© 2017 Delphix. All Rights Reserved. Private and Confidential.
Kellyn Pot’Vin-Gorman | Technical Intelligence Manager| October, 2017
The Revolution
DevOps to DataOps
© 2017 Delphix. All Rights Reserved. Private and Confidential.
2
Kellyn Pot’Vin-Gorman
Technical Intelligence Manager, Delphix
• Multi-platform DBA, (Oracle, MSSQL, MySQL,
Sybase, Postgres…..)
• Oracle ACE Director, (Alumni)
• Oak Table Network
• APEX Women in Technology Award, CTA 2014
• STEM education with Raspberry Pi and Python
• President, Rocky Mtn Oracle User Group
• Liaison, Denver SQL Server User Group
• DevOps author, instructor and presenter.
• Author, blogger, (http://dbakevlar.com)
© 2017 Delphix. All Rights Reserved. Private and Confidential.
3
• https://en.wikipedia.org/wiki/DevOps
The Source
DevOps is a short for DEVelopment and IT OPerationS. The term is used to
refer to a set of practices that emphasize the collaboration and communication
of both software developers and information technology (IT) professionals while
automating the process of software delivery and infrastructure changes. It aims
at establishing a culture and environment where building, testing, and releasing
software can happen rapidly, frequently, and more reliably
© 2017 Delphix. All Rights Reserved. Private and Confidential.
The Life of a DBA
Database management, automating where able
Provide environments and assist with projects
Secure database environments
Optimize data access
Collaborate to solve business challenges
© 2017 Delphix. All Rights Reserved. Private and Confidential.
The Life of a DevOps Engineer
Automation through scripting
Orchestration between environments, including data sources
Security
Optimize projects, improving on each pass
Collaborate and Communicate
© 2017 Delphix. All Rights Reserved. Private and Confidential.
6
Tools Empower?
• DevOps requires agility
• Agility requires more automation
and tools to ease demands
• Even the history of DevOps
includes tools
• What tools are part of DevOps for
the DBA?
© 2017 Delphix. All Rights Reserved. Private and Confidential.
7
How DBAs Risk the Change to DevOps
• Database Administrators heavily influence decisions.
• We’re sometimes identified with manual processing as a way to measure our value.
• Introduction of the cloud, especially and SaaS, (Software as a Service) has been seen as
competitive to our role.
Still, we’re essential to a successful change, or many are working around us…
© 2017 Delphix. All Rights Reserved. Private and Confidential.
8
Why Should We Master Tools?
• We Should be part of the development process
from the beginning
• We perform code reviews
• We check in and release code
• We write code, too.
• We create automation scripts.
• We monitor and optimize
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Tools of the Development
• Oracle SQL Developer
• DBArtisan
• Redgate SQL Toolbelt and SQL Prompt
• SQL Automation PL/SQL Developer
© 2017 Delphix. All Rights Reserved. Private and Confidential.
10
DevOps Terms
• Build Automation
• Configuration Management
• Continuous Delivery/Rapid Deploy
• Functional Testing/Unit Testing/Test Driven
Development
• Monitoring/Alerting
• Packages/Containers/Virtualization
• Release Coordination/Orchestration
https://xebialabs.com/glossary/
© 2017 Delphix. All Rights Reserved. Private and Confidential.
11
Method to this Madness
• Identify Category/Methodology
• Identify tools
• Some tools will bridge more than one
category or methodology
• Dig into unique or interesting specifics
about the tool
© 2017 Delphix. All Rights Reserved. Private and Confidential.
12
Build Automation
• Maven
• Gradle
• Packer
• Cmake
• ANT
DBAs are a natural fit, due to their
history with scripting and automation
experience for backups, monitoring and
alerting.
Build links: Maven, Gradle, Make, Packer, MSBuild
© 2017 Delphix. All Rights Reserved. Private and Confidential.
13
MS Build
MS Build 15 came out in March 2017 and
part of Visual Studio
• No longer installed in the Global Assembly
cache.
• XML based
• Project file made up of items, properties
and tasks- and of course, targets.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Continuous Delivery
DBAs are much slower to
embrace continuous delivery.
The risk is high, with it’s “fail
first, fail fast” approach. Yet
we are its greatest asset.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
15
Jenkins for CI/CD
Jenkins is an open source automation server-
• Easiest executed via a jetty servlet
• Has an easy to access URL, (commonly host:8080)
• Automation is done via individual projects.
• Has numerous plugins to support environments
• Supports multiple scripting languages, including shell, batch commands and Ant scripts.
• Supports robust revision control and compatible with version control systems like CVS and
Subversion.
• Supports notifications like qmail and sendmail.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
16
Jenkins Groovy Script Example// prepare variables for subscript execution:
String subscript = "cdmserver-monitor-instance-status.groovy";
println "executing subscript ${subscript}"
// variables to be passed to the subscript
status = (action == "start" ? "started" : "stopped")
def s = ScriptlerConfiguration.getConfiguration().getScriptById(subscript)
File scriptSrc = new File(ScriptlerManagment.getScriptDirectory(),
s.getScriptPath());
© 2017 Delphix. All Rights Reserved. Private and Confidential.
17
Jenkins Plugin for Delphix
https://github.com/delphix/delphix-jenkins-plugin
© 2017 Delphix. All Rights Reserved. Private and Confidential.
18
Configuration Management
Concept & Development Testing & Verification
Operations & Support Deployment to Production
© 2017 Delphix. All Rights Reserved. Private and Confidential.
19
Ansible
• Learn about Playbooks
• Powerful collection of steps, (aka plays)
• Plays consist of a list of tasks
• Handlers allow for remote operations
• YAML Syntax
• Security configurations based on hosts and users
• Ability to run as secondary users, (privilege
escalation)
• Huge support community and best practices
© 2017 Delphix. All Rights Reserved. Private and Confidential.
20
Example of an Ansible Call
- name: Transfer and execute a script.
hosts: server
remote_user: test_user
sudo: yes
tasks:
- name: Transfer the script
copy: src=test.sh dest=/home/test_user
mode=0777
- name: Execute the script
command: sh /home/test_user/test.sh
© 2017 Delphix. All Rights Reserved. Private and Confidential.
21
Puppet
• Review, automate, deploy and invest in all your
software.
• Control and enforce consistency across environments
and platforms.
• Share, test and enforce changes on-premise and in the
cloud.
Similar to Puppet
• Chef
• Jenkins
© 2017 Delphix. All Rights Reserved. Private and Confidential.
22
Rapid Deploy
Software Content
Enablement ServiceNeed or
Focus Area
All Tiers of Specific
Environment
© 2017 Delphix. All Rights Reserved. Private and Confidential.
23
Release Coordination/Orchestration
A demand for release tools and orchestration was at times a direct result to DBAs hesitation to risk
production with a release. This helped automate and bypass the DBA to get code and features into
production.
• Docker Hub
• Artifactory
• Zephyr
• Octopus Deploy
Deployment Links- RapidDeploy, ElasticBox, JuJu
© 2017 Delphix. All Rights Reserved. Private and Confidential.
24
Octopus Deploy
• Categorized as Continuous Integration Tool
• Release and deploy
• Promote to subsequent environments
• Cross tier/application, etc.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Automation and Orchestration, DB Style
• Idera SQL Admin Toolset
• Redgate DML Automation
• Automic
• CA Automation
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Source Control and Repositories
• Git
• Bitbucket
• GitLab
• Subversion
• Mercurial
© 2017 Delphix. All Rights Reserved. Private and Confidential.
27
GitHub
Git is an open-source version control system that was started by
Linus Trovalds, (the creator of Linux)
• If you aren’t using it now, start. Yes, as a DBA…
• Great option to highlight your scripting work
GitHub Alternatives
• Subversion
• CVS
• Mercurial
© 2017 Delphix. All Rights Reserved. Private and Confidential.
28
Creating a GitHub Repository Project
© 2017 Delphix. All Rights Reserved. Private and Confidential.
30
Source/Version Control, but Data?
We all know about version
control with code, but how
are we, as DBAs doing
version control with data?
This is a concept that hasn’t
been discussed much and
DataOps is the new way of
bridging some of this
challenge.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
31
The Goals of All Tools are the Same
• Shorter Delivery Cycles
• Higher Success Rate on Releases
• Shorten Time to Address Issues
• Continued Improvement in
Communication
• Increased Revenue
© 2017 Delphix. All Rights Reserved. Private and Confidential.
32
Let’s Talk Data Gravity and It’s Impact
• Data is HEAVY.
• Relational databases are heavily weighted and can
be very impacting to DevOps implementations.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
33
Oh DBA, to Embrace Virtualization…
A technical approach in which users and applications do
not use physical machines, but simulated systems
running on actual, “real” hardware. Virtualization can be
used to eliminate resource usage and enable savings for
databases, network, file and application management,
along with server infrastructure.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
36
Why Database Virtualization is Different
• Many tools take considerable time to recover.
• DevOps automate an “undo” for development and
testing that includes data rewind.
• Include a self-service tool that will allow for rewind
without DBA intervention.
• Allows for data version control and DataOps, the
next step in DevOps
© 2017 Delphix. All Rights Reserved. Private and Confidential.
37
Virtualization Tools
• Delphix
• Veritas Velocity
• Oracle Thin Cloning Features
• Actifio
© 2017 Delphix. All Rights Reserved. Private and Confidential.
▶▶▶ Virtualize and Deployed▶ ▶ ▶
600GB
Virtualization 101 via Delphix
Storage Pool for Delphix
QADEV PATCH TEST
PRODUCTIONDatabase/App Tier
1 TB
1 TB
Read From Production
Each Virtual Database takes up around 5-10Gb upon creation, (dependent upon
parameters)TEST
Read AND Write
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Data Virtualization: Space Savings
Delphix Virtualization EngineValidated Sync Target
Dozen Virtual Database Copies, (VDBs)
© 2017 Delphix. All Rights Reserved. Private and Confidential.
41
The Future is Now- Containers/Data Pods
Containers offer the ability to isolate application code
and/or the whole infrastructure stack into a package able
entity to ease deployment, even from the same kernel.
This is a powerful tool for DevOps to ease deployment for
complex tiers, applications and multiple data stores.
Data Pods is the next step, where a pod is created from
virtualized environments.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
42
Docker
• Virtualization is important,
this is OS level
• Resource level isolation at
the kernel level
• Technology such as
OverlayFS, allows for
containers, which removes
overhead of other
virtualization technologies.
Docker Alternatives:
• Open Container
Initiative, (OCI)
• Kubernetes
• CoreOS
• Apache Mesos
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Why Create Data Pods
• Development is done by project or feature vs. tier or
product.
• With the introduction to the cloud, the business
requires those that can do more at a global scale.
• Ease of management and maintenance
• Provide more value to the business
© 2017 Delphix. All Rights Reserved. Private and Confidential.
45
Data Pods
Database
Unstructured
Data
Application
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Data Pods for Ease of Delivery
Create “Container”
Rsync (UNIX/Linux)
Robocopy (Windows)
Reporting
NFS
iSCSI
Development
NFS
iSCSI
Testing
NFS
iSCSI
Delphix Virtualization EngineSame Storage Size as Production
Deployed using Jenkins
© 2017 Delphix. All Rights Reserved. Private and Confidential.
47
How Does This Improve Agile?
Finance
HR
CRM
© 2017 Delphix. All Rights Reserved. Private and Confidential.
48
Timeline in a Project
Waterfall- 5 Releases
January February March April
Agile with Virtualization- 12 Releases
January February March April
Database Refresh
Development Production Release
Testing Cycle
The Goal of
Agile
Data Holdup to Development
Cycle
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Data Versioning
• Delphix Jet Stream
• Redgate SQL Data Compare
• Oracle SQL Developer
© 2017 Delphix. All Rights Reserved. Private and Confidential.
50
Branching and Bookmarking
• The ability to mark each iteration of development
with a bookmark
• Simplify to lock and deliver while testing a
consistent image via a virtual database, (VDB)
• If a something goes wrong, the ability to
“bookmark”, (and subsequent snapshot) to
deliver to development to address.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Self-Service
Release 1.2 Release 1.3 Branch 1.3
Branch 1.1
© 2017 Delphix. All Rights Reserved. Private and Confidential.
55
Scrum
https://www.scrumalliance.org/community/articles/2014/april/devops-and-agile
© 2017 Delphix. All Rights Reserved. Private and Confidential.
58
Crystal
Alistair Cockburn said, “Crystal is a family of human-powered, adaptive,
ultra light, ‘stretch-to-fit’ software development methodologies.”
• People are the most important aspect of Crystal.
• Development is self-sufficient and self-organizing, (very different from
Scrum)
• Very lightweight
• Very little documentation, overhead management or reporting.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
60
Collaboration Practices can Mutate
• Methods- Scrum, KanBan, Extreme Programming, Continuous
Delivery
• Official hybrid approaches like Scrumban and KanPlan
• Numerous other combinations created from any of the
originals.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
• All IaaS solutions provide encryption in-flight and encryption at-rest• But encryption doesn’t protect data as much as it needs to be .
• Europe already requires data masking, not just data encryption for any confindential data, (GDPR):
• http://ec.europa.eu/justice/data-protection/article-29/documentation/opinion-recommendation/files/2014/wp216_en.pdf
Confidential data
© 2017 Delphix. All Rights Reserved. Private and Confidential.
• Encryption is reversible data obfuscation, which is very different from masking data.
• Data masking is non-reversible.
• It solves the issue at the data level.
• Is authentication and authorization in non-production in compliance with security goals?
• All organizations will soon need to review if critical data in non-production environments be accessible to developers, testers and users.
Confidential data
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Encryption and Masking
• Optim
• Delphix Masking
• Dataveil
• CA Test Data Manager
• Oracle Encryption and Masking
© 2017 Delphix. All Rights Reserved. Private and Confidential.
• Masking personally-identifiable, (PII, HIPPA, PCI, etc.) information renders it useless from a security standpoint
• Resolves both the technical and personal responsibility issue.
• The data can be masked before it moves to non-production, removing unnecessary risk.
Why Masking is Part of the Answer
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Confidential data
Production
Non-Production
Exposure
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Confidential data
Production
Non-Production
Exposure
Encryption
Masking
Solution
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Auditing Solutions
• Idera SQL Secure and SQL Compliance
Manager
• Redgate SQL Source Control
• Oracle Auditing Features
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Data Causes Friction
• If those managing the data sources aren’t included and working well with those needing that data to produce features and products, friction is the result.
• DBAs are consistently working with archaic utilities that guarantee outcome, rarely speed of results.
A DBA is only as good as their last backup.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
What is Data Gravity?
Data Gravity is the ability for data to attract applications, services, etc. As with the laws
of physical gravity, data, due to its mass, will attract all else that has less mass.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
When you consider…
An estimate of a 10% increase in data accessibility will result in more than $65 million in additional net income, you can see why data
gravity is a big deal.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
How much data are we talking about?
• More data has been created in just the last two years than the previous history of humanity
• 1.7MB of new information created every second per human on the planet.
• Introduction of big data often has same development pain points.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
74
Know Your Tools
Incredible variation of tools and knowing tool classifications. Use
the right tool for the job and recognize that each tools has its
own “flavor.”
© 2017 Delphix. All Rights Reserved. Private and Confidential.
Remove the Roadblocks
• Remove the data as the roadblock.• Use virtualization and containers to simplify.• Self-service portals for developers and testers to
refresh and develop they way they do in an agile environment.
• Tools with an interface made for Agile and Development data at its focus.
© 2017 Delphix. All Rights Reserved. Private and Confidential.
76
Review- DBA Focus Areas
• Update skills, (scripting, monitoring, automating) to remove bottlenecks
• Enhance monitoring utilities that take infrastructure and development tools into
overall view
• Recognize a need to become more skilled in scripting and become
knowledgeable of DevOps tools.
• Data is EVERYWHERE! Recognize the integral part Database Administrators
play in DevOps and learn more about the next generation- DataOps!
Don’t fear the future, embrace it.
DBA
DevOpsSkills
© 2017 Delphix. All Rights Reserved. Private and Confidential.
77
The Periodic Table of DevOps Tools
https://xebialabs.com/periodic-table-of-devops-tools/
© 2017 Delphix. All Rights Reserved. Private and Confidential.
78
Read- The Phoenix Project and the DevOps Handbook!
Delphix with SQL Server- the Basics
https://docs.delphix.com/docs/delphix-administration/sql-server-environments-and-data-
sources/managing-sql-server-environments/overview-of-setting-up-sql-server-
environments
Delphix Upgrade Workflow: https://community.delphix.com/delphix/topics/tip-of-the-day-
upgrading-a-sql-server-dsource
Upgrading the Dsource after an Upgrade: https://docs.delphix.com/docs/delphix-
administration/sql-server-environments-and-data-sources/virtualizing-databases-using-
delphix-with-sql-server/managing-sql-server-dsources/additional-dsource-topics/upgrading-
a-dsource-after-a-sql-server-upgrade
Delphix in the Cloud
https://www.delphix.com/solutions/cloud-migration-virtual-data
References and Tips
© 2017 Delphix. All Rights Reserved. Private and Confidential.© 2017 Delphix. All Rights Reserved. Private and Confidential.
Kellyn Pot’Vin-Gorman
Technical Intelligence [email protected]://dbakevlar.com