DevOps - CUNA Councils · DevOps Tools. Build Server. DEV Env. QA Env. UI Automated Testing [Azure] Build Verification Test Env. DevOps Tools. Orchestration Configuration Management

DevOps

The What, Why and How

Presenters

John TomanSenior Manager, Software

[email protected]

John ClarkSenior Solutions Architect

[email protected]

How Does My Credit Union …• Improve speed to market for my integrated

software, with lower risk?• Make software deployment less expensive, with

less complexity?• Foster better collaboration between IT and my

lines of business?• Respond quicker to feedback from key

stakeholders and customers?

DevOps• Is a culture and a practice• Emphasizes collaboration and

communication– Development– Information Technology– Others

• Automates software delivery and infrastructure changes

DevOps: Software Development

DevQA

IT

DevOps

DevOps: Integrating Software

QA

IT

DevOps

Culture• Remove boundaries between

development, QA, IT• Collaborate on every change• Provide continuous feedback

Ingredients• Agile methodologies• Continuous Integration (CI)• Continuous, automated Testing

(CT)• Continuous Delivery (CD)

– Package– Release– Provision– Configure

Automation

Agile’s Role• Keep iterations short

– Small changes– Make it work, make it better

• Limit work in progress– Prioritize

• Respond quickly to feedback

Continuous Integration (CI)• Integrate early, integrate often• Compile and test code on every committed

change• Short feedback loop• Prioritize build stability• Promotes higher quality

Continuous Testing (CT)

• Execute automated tests as part of software delivery pipeline– Build Verification Testing (BVT)

• Exposes application risks quickly

• Utilizes fewer testing resources

Not Quite DevOps: Continuous Testing

DevQA

CT

Continuous Delivery (CD)• Allows for incremental upgrades into production• Builds on CI and CT• Lots of automation tooling support

– From source control to production• Not the same as Continuous Deployment

– Continuous Delivery may not result in release• DevOps is broader in scope (cultural)

Enabling Technologies• Source Repository• Build / Orchestration Server(s)• Virtual Machine or Container Infrastructure• Test Automation Tooling• Configuration Management Tooling• Deployment Tooling• Cloud Computing

DevOps @ Symitar• Continuous Integration adoption began 2010• Continuous Delivery began 2016• Early focal points:

– Selecting automation tooling– Automating internal deployment– Provisioning

• Systems• Episys (SymXchange)• Databases (SYMs)

– Restoration• Targeting internal production systems (QA, Dev)

Problems We’re Trying to Solve• Manual system configuration

– Deployment– Configuration– Restoration

• Limited self service– Helpdesk ticket required for most issues

• Load Episys service pack or release• Load AIX patch• Restore SYM database• Enabling Episys modules (SymXchange, etc.)• Reload system

Problems We’re Trying to Solve• Longer term

– Long release cycles– Speed to market– Higher efficiency

Tools used for Automation / CD

DevOpsTools

Build Server

DEV Env

QA EnvUI

Automated Testing [Azure]

Build Verification

Test Env

DevOps ToolsOrchestrationConfiguration ManagementSystem Deployment

CD / Automation Tool Selection• Orchestration

– Jenkins

• Configuration Management– Chef

• System Deployment– IBM Network Installation Manager (NIM)

CD / Automation Tools OverviewDeploy

Base image

Post Deploy Configuration

Configuration changes

Auditing

Automated Testing

Chef

Jenkins

NIM

CD / Automation Tools

Orchestration tool

• Initiate jobs• Self service interface• Remote command execution

via SSH• Role based access to GUI

interface• Public Key or password

authentication to end nodes• Server: RHEL

Configuration Management

• Automation tasks:• Application updates• Post deployment

configuration• System auditing• Initiated from Jenkins or

client• GUI Interface• AIX/Windows Client• Initiated via client or via Knife

command line tool• Server: RHEL

System Deployment

• IBM Network Installation Manager

• AIX/Episys image deployment via network

• System restore• System image repository• Initiated from Jenkins• Server: AIX LPAR

Chef Configuration Management• Codification of infrastructure• Provisioning

– Automate system deployment/restoration– Cloud management (Azure, Google Cloud Platform, etc.)

• Ongoing Management– System auditing

• Open Source Cookbooks– Benefit from development community

CD / Automation Solution

AIX 7.1Episys 2016.00

Final

Chef

Jenkins

NIM

SSH keys

Episys Modules

AIX patch

Episys SP

LPAR Orchestration

Post Deployment Configuration

ConfigurationChanges

KSH scripts

expect TCLNetworking

Transaction Test Tools

Add Users

Configuration files

Auditing

SSH keys

Passwords

Automated Testing

Security Patches

System Deployment Process Flow

Create LPAR

• Jenkins execute commands on HMC Server

• Jenkins executes commands on NIM Server to install image

• Trigger job to bootstrap chef client with initial cookbooks

Chef initial run

• Install chef-client• Setup

configurations• Configure

mounts• Enable/Disable

services• Modify packages• Install Episys

Cookbook

• Test Media• Update Episys• Update AIX

Service level• System Backup

Maintenance

• Jenkins job monitoring Utils

• Chef client daemon execution reports unstable nodes

• Chef-client audit mode

• Infrastructure monitoring

Automated Testing Process Flow

• AIX 7.1• Episys 2016.00

LPAR Orchestration

• Episys SP• AIX Patch• Episys Modules• Transaction

testing tools

Post Deployment

Configuration• Users, Network• Permissions• Mounts, Services• Passwords• SSH Keys

Configuration Changes

• KSH Scripts• expect TCL• Coded UI

Automated Testing

Continuous Delivery Solution

Build Server Baseline

2016.01

2016.00

Baseline

2016.01

2016.00BVT

BVT

BVT

Build Verification Test LPARs

2016.00

2016.01

Baseline

Dev QA

Chef

Integration Integration

BVT

Integration

QuestQuest

QuestQuest

UI Automated Testing(Azure)

BVT

BVT

BVT

BVT

Jenkins

QuestQuest

QuestQuest

Symitar Success Story: Release Testing• Before

– Manual refresh of systems between tests– Request to system admins

• Dependent on system admin schedule• Average turnaround time: 4 hours

– Limits on test cases executed due to time constraints• Now

– Self service tool for QA• Completed in 15 minutes• Removed dependency on system admin schedule

– Allowed for 5X release load test cases

Symitar Success Story: Failover Testing• Before

– Manual system setup – very error prone/labor intensive• Now

– Self-service automated system deployments– QA staff can build their own environments using GUI self

service tool• Fully scripted and automated backend• Estimate 8 hours labor saved per environment build (160 hours per

release)

Symitar Success Story: AIX Patching• Before

– All system setup and test cases done manually– AIX patch testing required 3 people / 2 weeks

• Now– Self service interface for QA

• Automation of system setup/restore• Automation of test cases

– Rapid re-run of tests following defect fix– AIX patch testing completed by 1 person

• Included re-run of tests following defect fix

Symitar Success Story: Self Service• Before

– All system deployment performed manually– Request to system admins

• Now– Self service interface– Full automation of system deployment

• Reduced turnaround time• Improved end-user experience

Scenario: Deploy an Episys Patch• Current Process

– Software patches deployed manually– Request to system admin team

• Goals– Fully automated software patch deployment– End-user self service option– No need for system admin intervention– Foundation for automated build testing

Scenario: Build a Test System• Current Process

– Test systems deployed manually– Limited build verification testing

• Critical test cases executed late in development cycle

• Goals– Fully automated test system deployment– Fully automate build verification testing– Execute critical test cases throughout development cycle

Scenario: Integrate a New Feature • Current Process

– New feature integration requires specialized staff and procedures• Slow and lengthy process• SME’s spending time on system setup, not testing

– End result: limited internal deployment and testing• Goals

– Provide self service tool• Fully automated backend

– Simple and rapid deployment– More extensive internal adoption and testing

Takeaway #1• Don’t need to be a development shop to

adopt or realize benefits of DevOps

Takeaway #2• DevOps is culture and practice

– Cultural change needs burn in time– Practice requires tooling

Takeaway #3• Automation is the key driver of

– Efficiency– Quality– Time to market

• Automate all the way to production

Takeaway #4• Start small• Tackle one item at a time• Prioritize

Takeaway #5• Utilize feedback loops

– Involve stakeholders, integrators, developers, testers, customers

We’re Done• Questions?

[email protected]@jackhenry.com