Top Banner
W11 DevOps & Testing 5/4/16 13:45 Continuous Integration Testing Techniques to Improve Chef Cookbook Quality Presented by: Glenn Buckholz Coveros, Inc. Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-268-8770 904-278-0524 [email protected] http://www.stareast.techwell.com/
27

Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

Apr 15, 2017

Download

Software

Josiah Renaudin
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

W11  DevOps  &  Testing  5/4/16  13:45  

Continuous  Integration  Testing  Techniques  to  Improve  Chef  Cookbook  

Quality  

Presented  by:  

Glenn  Buckholz  

Coveros,  Inc.  

Brought  to  you  by:    

350  Corporate  Way,  Suite  400,  Orange  Park,  FL  32073  

888-268-8770 904-278-0524 [email protected] http://www.stareast.techwell.com/

Page 2: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

Glenn  Buckholz  Coveros,  Inc.  

With  fifteen  years  of  industry  experience,  Glenn  Buckholz  leads  continuous  integration  and  deployment  automation  efforts  at  Coveros.  His  career  began  as  a  consultant  implementing  automated  test  frameworks  and  introducing  the  concept  of  change  management  to  many,  many  projects.  Glenn  then  decided  to  become  a  part  of  honest  society  and  settled  down  at  the  Public  Company  Accounting  Oversight  Board  as  their  full8 time  enterprise  change  manager.  Several  years  later,  he  joined  Coveros,  where  he  specializes  in  implementing  agile  practices  and  CI,  and  engineering  configuration  management  instead  of  simply  documenting  it.  

Page 3: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Continuous Integration Testing

Techniques to Improve Chef

Cookbook Quality

STAREAST – 04 May 2016

Glenn Buckholz

[email protected]

Page 4: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Agenda

•What did I do?

•What is problem?

•What are the needs?

•Demo

•Architecture

•Workflow

•Testing Techniques and Tools

•Metrics?

•Docker and Cloud

•Discussion

Page 5: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

What did I do?

● Used a modern CI/CD toolchain to apply

continuous integration testing techniques to

Chef code (CI):

○ Static analysis

○ Chefspec testing

○ Automated Functional Testing

● Introduces the idea of automatically making

cookbooks available when they are ready

(CD).

○ There is still a human gate between

available and in use by way of version

pinning.

Page 6: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

What is the problem?

● Infrastructure as Code is currently the wild

west with little formal testing methodology.

● Chef lacks coherent CM on its own.

○ Cookbook versions and freezing are not

enough.

● No quick way to advance a piece of

infrastructure from one state to another and

back for purposes of testing.

● No minimal standards and requirements on

cookbook functionality.

Page 7: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

What are the needs?

YOU ARE CHANGING THE TIRE AT

55MPH AND CAN’T STOP!!! ● Find flaws early.

● Ability to test pieces of an architecture

without having to have a whole system.

○ Unit testing system components.

● Simulate the effect of a deploy on a running

system.

● Dashboarding so you can trend code

quality.

Page 8: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Definitions

● Hands Off Deploy - Deployment that is

completely driven by Chef or some other

infrastructure tool.

● Static Analysis - evaluation of the code

without execution.

● Feedback loop - The events between when

a developer commits their code and results

can be observed.

● Application Code - The custom or home

grown software the current IT organization

is curating.

Page 9: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Demo

Page 10: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Architecture

Local Git Branch

Chefspec

Test Kitchen

Food Critic

Developer Workstation

Docker

Container

Binary Repository

Docker

Repository

Local Testing

Docker

Container

Upload Cookbook

Page 11: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Workflow

1. Clone master branch on the developer

machine.

2. Make changes and run static analysis.

3. Run Chefspec and test kitchen locally on a

docker container.

4. Push changes to remote branch.

5. Jenkins detects changes and checks out

branch onto a slave node.

Page 12: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Workflow

6. Jenkins merges the code with master

locally.

7. Chef reruns the static analysis.

8. Chef reruns Chefspec and test kitchen.

9. Results and metrics are recorded.

10.Successful tests see code merged into

master.

11.Cookbooks are uploaded to the chef

server.

12.After the code is executed in production,

AMI and Docker containers are

automatically updated from production.

Steps 8-9

automate

integration

testing

Page 13: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Reuse of well known patterns

● Feature Branches

○ Small bite sized changes.

● Automated Merging

○ Let the Jenkins objectively merge the

code after certain minimum standards are

achieved by the developer.

● Local tests that mirror a golden standard

○ Give the developers a cheat sheet.

● Short feedback loops.

● Auto Promotion

○ Let Jenkins keep the working cookbooks

in sync with SCM

Smaller changes are

more likely to make it

through the pipeline.

Page 14: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Testing Techniques

● Tabula Rasa

○ Start with a base image

○ Apply Security Hardening

○ Install stack fresh

○ Deploy application code

○ Run tests

● Production Approximation

○ Start with a clone from production

○ Run cookbooks

■ Only changes are executed

○ Run test

Page 15: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Testing Techniques

● Test in parallel

○ Do not block on critical resources clone

them.

○ VMs and containers are cheap, create

one for every commit you want to test.

○ Make unit and integration test cases

isolated. (Script system state for each

test case do not rely on other tests)

● Test local

○ Very short feedback loop for developers.

○ Reduces most pipeline failures to

integration errors.

Page 16: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Testing Tools

● Unit Testing

○ chefspec - extension of Rspec

● Functional Testing

○ BATS or chefspec

● Static Analysis

○ Foodcritic - lint like

● Git hooks

○ Make certain static tests run as a git hook

making them prerequisites to commit or

push.

Page 17: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Testing Tools

● Git Branching

○ Let the branch name decide if the CI

stack will run the changes.

○ You may want to use a prefix CI-

branchname

Page 18: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Testing Tools - Jenkins Master

● Jenkins master only monitors git.

○ Container on slave in the cloud for quick

tests.

○ VMs in cloud for executing tests in a

production like system.

■ Template AMI matches target machine.

○ All “work” takes place off the master to allow

testing in parallel.

● Create Jenkins jobs to capture production

templates post deployment. (AMI or docker

image)

Page 19: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Testing Tools - Developer Workstation

● Provide an easy to install package that includes:

○ Tools - Chefspec, Foodcritic, etc…

○ Docker settings.

○ Access to a docker repository.

○ Templates to create new cookbooks

● Enough memory and CPU to run the tools.

● DO NOT ALLOW COOKBOOK UPLOAD PRIVILEGES!!!!!

● DO NOT ALLOW MERGE TO MASTER PRIVILEGES!!!!!

● The ability for a developer to find mistakes locally is the

most cost effective testing technique available to an

organization. (Short Feedback Loop)

Page 20: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Transitioning to the CI Approach

● From the previous slides we have the tools to create

an method for retrofitting a CI driven approach to an

existing Chef infrastructure.

● On a separate branch create cookbooks that install

a piece of the production stack on a server or

container.

● On a feature branch develop just the chef code

needed to deploy your custom application.

● Slowly merge the two once per iteration. Merge one

item from the stack branch and merge your install

code to master.

Page 21: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Transitioning to the CI Approach

● Do not try to boil the ocean.

● Start with installing the application code.

● Move up to controlling the application stack

one layer at a time.

● This maps to Agile sprints and iterations.

● Cookbooks should be idempotent.

Master (What is in Chef Server)

Install Code

(Feature

Branch)

Install Code

(Feature

Branch)

Install Code

(Feature

Branch)

Install Code

(Feature

Branch)

Install Code

(Feature

Branch)

Infrastructure or Application Stack Cookbooks (Apache, Tomcat, etc.)

Test

Here

Test

Here

Test

Here Test

Here

Page 22: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Metrics

● Failed Merges

○ How many times did a developer cause a

potential merge conflict?

● Static analysis thresholds

○ How many times did a commit fail?

● Failed rspec tests.

● Failed functional tests.

● How long is the component down during

upgrade.

Page 23: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Docker and Cloud

● Docker

○ Can be very fast

○ Need a local Docker repository.

● Cloud

○ You can use a mirror of production

environment.

○ Need to find a way to clone VMs without

downtime or cluster components.

● If production uses docker this is a moot

point, just use docker.

Page 24: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Discussion Points

● Tool chain can be complicated.

○ If you already invested in CI you have a sunk cost

with many of the tools already.

● Two major types of transitions:

○ Introducing CI to chef.

○ Introducing Chef and CI to a project.

● Start small, big bang automation is high risk.

● Make the quality gates automated and give the

developers the answer key, don’t change the test, and

let Jenkins proctor the exam.

● Allocate time to sync production changes to testing

images.

Page 25: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

What Problems Did We Solve?

● Version management - The git cookbook version linkage

provides change traceability. This is enforced by Jenkins.

● Enforcing testing in general for Infrastructure - By ceeding

cookbook upload control to Jenkins we can have a

minimal set of quality standards for all deployments and

changes.

● Making Testing accessible to the developers - By putting a

framework at their fingertips. This makes it easier for

developers to follow the process than circumvent the

process.

● Provide testing metrics so cookbook quality can be

analyzed.

● Provided an automated testing framework to provide quick

feedback on integration errors.

● Disposable infrastructure for rapidly changing server state.

Page 26: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

What Problems Didn’t We Solve?

● Adoption

○ Change it hard, things can be done to ease

the transition but it cannot be forced. You

still have to convince developers the value

proposition of writing the tests.

● Integration errors

○ While we can find them faster, there is no

tool I know of that can predict code

interaction accurately.

Page 27: Continuous Integration Testing Techniques to Improve Chef Cookbook Quality

© Copyright 2016 Coveros, Inc. All rights reserved.

Thoughts? Questions?

Thank you for your time.