TheTest!Expert'sRolein!DevOps! · Fundamental conflict - Dev vs. Ops •Operations •Risk Minimization - Stability, SLAs, Uptime, Response Time •Scripts and procedures to install

W4 DevOps Wednesday, October 17th, 2018 10:15 AM

The Test Expert's Role in DevOps

Presented by:

Mike Sowers

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/

Mike Sowers Coveros' Executive Vice President of Training, Mike Sowers has more than twenty-‐five years of practical experience as a global leader of internationally distributed test teams across multiple industries. Mike is a senior leader, skilled in working with both large and small organizations to improve their software development, testing, and delivery approaches. He has worked with companies -‐ including Fidelity Investments, PepsiCo, FedEx, Southwest Airlines, Wells Fargo, ADP, and Lockheed -‐ to improve development approaches, increase software quality, reduce time to market, and decrease costs. With his passion for helping teams deliver software faster, better, and cheaper, Mike has mentored and coached senior software leaders, small teams, and direct contributors worldwide.

© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 1

Agility. Security. Delivered.

The Test Experts Role in DevOps

Michael D. Sowers@michaelsowers4

Coveros, Inc.


Mike Sowers is Executive VP for Coveros and has more than twenty-five years of practical experience as a global leader of internationally distributed test teams across multiple industries. Mike is a senior leader, skilled in working with both large and small organizations to improve their agile software development, testing, and DevOps delivery approaches.

He has worked with companies—including Fidelity Investments, PepsiCo, FedEx, Southwest Airlines, Wells Fargo, ADP, and Lockheed—to improve development approaches, increase software quality, reduce time to market, and decrease costs. With his passion for helping teams deliver software faster, better, and cheaper, Mike has mentored and coached senior software leaders, project teams, and direct contributors worldwide.

Michael Sowers@MichaelSowers4

Trainer


About Coveros

• Coveros helps organizations accelerate the delivery of secure, reliable software using agile methods

• Coveros Services• Agile transformations & coaching• Agile development and testing• DevOps implementations• Software security analysis

• Agile, DevOps & Security training

• Software Products• SecureCI – open source CI/CD toolchain• Selenified – open source automated testing

framework

Areas of Expertise


Agenda

• DevOps?• Why DevOps• What is DevOps

• CI Testing• Process and Tasks

• CD Testing• Process and Tasks

• Summary


DevOps is a software development method that stresses communication, collaboration and integration between software developers, testers, and operations engineering.

DevOps extends the principles of Agileinto Operations

Like Agile, DevOps can be usedoutside software development toimprove IT operations in general

DevOps is NOT a tool or a particularprocess

Definition

What is DevOps?


• Systems Thinking – Optimize the entire system

• Amplify Feedback Loops – shorter, incremental feedback

• Culture of Experimentation and Learning From Gene Kim, Co-author of The Phoenix Project

DevOps Principles – “The 3 Ways”

What is DevOps?

https://itrevolution.com/the-three-ways-principles-underpinning-devops/


• Repeatable and Reliable Software Delivery• Increased Deployment Frequency

• Reduced Lead time for Changes

• Reduced Mean time to Recover (MTTR)

• Reduced Failure Rates

• Collective Ownership • Collaboration• Continuous Communication• Unified Processes and Tooling

• Continuous Improvement “Kaizen”

DevOps Goals

What is DevOps?


DevOps Mindset

Fundamental conflict - Dev vs. Ops

• Operations• Risk Minimization - Stability, SLAs,

Uptime, Response Time

• Scripts and procedures to install production system components

• Development• Rapid introduction of change

(delivery of features)

• Scripts and procedures to install development system components


Successful DevOps Methods and Practices

What is DevOps?

Continuous Deployment

Continuous Integration

Lean Product Management

Continuous Testing

Continuous Monitoring

Comprehensive Change Management

Dev Ops


What is DevOps?

DevOps Terminology

• Build automation – Automating the compilation, linking, and packaging of software applications

• Continuous integration – Leveraging build automation capabilities to integrate and test software on a regular basis to thwart integration and quality issues earlier in the process

• Check-in builds – Builds performed whenever code is changed. Often includes unit tests and code quality checks.

• Nightly builds – Integration builds performed nightly to assure nothing changed during the day breaks existing functionality. Regression testing included. Platform TBD.


What is DevOps?

DevOps Terminology

• Continuous delivery – A series of practices to assure code can be rapidly and safely deployed to downstream environments (QA, staging, production, etc.) by delivering every change to downstream environments while ensuring functionality works as expected through automated and manual testing.

• Continuous deployment – A series of practices to assure code can be rapidly and safely deployed to production by delivering every change to a production (or production-like) environment while ensuring functionality works as expected through automated testing.


What is DevOps?

Environments

• While continuously testing and building software is important for a robust final product, it is difficult to continuously release software while creating it.

• Having multiple environments allows for continuous development and low level testing to continue, while having more static versions of the software available for testers.

• Typically, a set of 4 or 5 environments will exist• Development• Dev Integration (CI)• QA• Staging• Production


What is DevOps?

Environments (cont.)

• Development• Local development / test environments where new code is created and

locally tested prior to checking changes into a source code repository

• Development Integration / CI• Updated continuously, always with the most up to date code that

passed all static analysis, unit tests, and code coverage criteria

• QA• Should be updated at least nightly, always with the last code to pass all

continuous integration and nightly regression testing

• Staging• Mimics production• Used for any testing that must be performed on production-like

environments to be valid

• Production• Updated for new releases / patches only, always with stable fully tested

code


DevOps Delivery Pipeline

Process of taking a code change from developers and getting it deployed

into production or delivered to the customerthrough a series of testing environments

• Key DevOps metric:• How long does it take for a change in

the code to get into production?


Delivery Pipeline (example)

Requirement

Code

Check-in

Unit Level, CI,

Regression

Typical Testing Activities with appropriate quality gates

Trigger

Deploy to QA/Test

Story & Use Case

LevelDeploy to Staging

System Integration

& Non-functionals

Deploy to Pre-Prod

Compliance &

Final AcceptanceDeploy to Prod

Early Rapid Feedback

No Late Surprises

A Balancing Act


Testing in DevOps

Testing activities are done across environments where appropriate

Testing here is:

Code focused Quality focused Delivery focused

In Sprints


Quality Gates

• Acceptance criteria that determines whether an application is ready to move to the next stage in a delivery pipeline

• A “good” quality gate does the following:• Enforces best practices

• Ensures team norms and expectations are enforced

• Continuously changes with continuous team improvements

• Finds a balance between enforcing quality, but not becoming too restrictive


Quality Gates and the Definition of Done

• Agile teams define a “Definition of Done” for User Stories, Sprints, and Releases as part of Initial Planning.

• Necessary amounts of testing for units, stories, integration of stories, use cases, etc.

• Code quality and security standards

• Performance standards

• Your Definition of Done needs to be part of your quality gates!

• Where specific ‘Done’ criteria reside in your delivery pipeline depends upon your environments and delivery process


“When we compared high performers to low performers, we found that high performers are doing significantly less manual work, and so have automated:

• 33 percent more of their configuration management.• 27 percent more of their testing.• 30 percent more of their deployments.• 27 percent more of their change approval processes." State of DevOps 2017

Benefits of Automation

Automation DOES make a difference


Continuous Integration is a software development practice where software changes are continuously integrated together and tested to

make sure a software application is always stable and software developers receive rapid feedback if there is a quality problem

What is Continuous Integration?


Continuous Integration within DevOps

Continuous Integration Continuous Delivery


Automate Test as Part of Continuous Integration


Typical Continuous Integration Pipeline

Local

Development

Source code control

Jenkins

Check-in Builds

Integration

To

Continuous

Delivery


CI Step 1: Local build and test

To

Continuous

Delivery

Jenkins

Check-in Builds

IntegrationLocal

Development

Source code control

Typical activities1. Checkout code2. Modify code3. Unit test4. Merge code5. Run local tests

- unit- acceptance- security- performance

4. Check in code

Test Experts Role/Tasks:• Help define success criteria

for local• Assist with environment

design/configuration• Contribute to Unit Test

Design• Participate in Reviews• Design, Implement &

Automate Story Acceptance Test

• Assist with dependency management

• Ensure critical Non-Functionals are being tested


CI Step 1: Local build and test

To

Continuous

Delivery

Jenkins

Check-in Builds

IntegrationLocal

Development

Source code control

Typical activities1. Checkout code2. Modify code3. Unit test4. Merge code5. Run local tests

- unit- acceptance- security- performance

4. Check in code

Testing Tip: The more production-like our local environments are, the earlier we canbegin testing non-functional requirements like security and performance

Testing Tip: Have the developers integrate their code locally prior to check in and runstory acceptance tests to validate stories as they go. Test with them too.


CI Step 2: Validate code quality & integrate

To

Continuous

Delivery

Local

DevelopmentIntegration

Source code control

Jenkins

Check-in BuildsTypical activities1. Jenkins detects

code changes2. Compiles code3. Runs quality gate

tests- unit & coverage- acceptance- code analysis- security

4. Packages app ifall quality gatespass

Test Experts Role/Tasks:• Help define success

criteria for check-in• Understand Static

Analysis Results (risks)• Analyze Code Coverage• Participate in Reviews• Design, Implement &


• Validate definition of done

• Contribute to regression strategy

• Ensure critical Non-Functionals are being considered


CI Step 2: Validate code quality and integrate

To

Continuous

Delivery

Local

DevelopmentIntegration

Source code control

Jenkins

Check-in BuildsTypical activities1. Jenkins detects

code changes2. Compiles code3. Runs quality gate

tests- unit & coverage

- acceptance- code analysis- security

4. Packages app ifall quality gatespass

Testing Tip: Check-in builds should take no longer than 10 minutes as developersshould be waiting for feedback before beginning new work

Testing Tip: Security testing must be lightweight to be useful during check-ins. Don’ttry to run comprehensive scans at this step


CI Step 3: Validate code integration

To

Continuous

Delivery

Local

Development

Source code control

Jenkins

Check-in Builds

IntegrationTypical activities1. Provision envs2. Install / conf

application3. Run smoke test4. Run quality gate

tests- acceptance- lightweight

security- performance

5. Lightweightregression


criteria for environments• Analyze critical coverages

(design, requirements, function, interface, etc.)

• Define, Implement and Automate Smoke Tests

• Design, Implement & Automate Story Acceptance Test

• Validate definition of done

• Design, Implement & Automate regression tests


CI Step 3: Validate code integration

To

Continuous

Delivery

Local

Development

Source code control

Jenkins

Check-in Builds

IntegrationTypical activities1. Provision envs2. Install / conf

application3. Run smoke test4. Run quality gate

tests- acceptance- lightweight

security- performance

5. Lightweightregression

Testing Tip: All testing during CI must be automated

Testing Tip: If you are able to use cloud-based services, consider parallelizing yourCI process to enable more testing during your CI window (hourly, nightly)


CI Step 4: Package release candidate

Local

Development

Source code control

Jenkins

Check-in Builds

IntegrationIf all quality gates pass:1. package (and tag) application as a

release candidate2. Destroy integration environment3. Enter continuous delivery process

If any quality gates fail:1. Notify development team of errors2. Keep integration envs available for

debugging

To

Continuous

Delivery


criteria for quality gates

• Help define, implement and maintain team quality dashboard

• Ensure appropriate configuration management

• Contribute to root cause analysis and continuous improvement


Continuous delivery – A series of practices to assure code can be rapidly and safely deployed to downstream environments (QA,

staging, production, etc.) by delivering every change to downstream environments while ensuring functionality works as expected

through automated & manual testing.

Continuous deployment – A series of practices to assure code can be rapidly and safely deployed to production by delivering every change to a production (or production-like) environment while ensuring functionality works as expected through automated

testing.

What are Continuous Delivery & Deployment?


Continuous Integration within DevOps

Continuous Integration Continuous Delivery


Automate Test as Part of Continuous Delivery


CD Step 5: Test release candidateFrom

Continuous

Integration

Staging

ProductionQA

Jenkins

Typical activities1. Choose release candidate

to test2. Provision environments3. Install / conf application4. Run smoke test5. Run QA tests

- Acceptance / Explore- Performance- Secure code scan- Full security test

6. If quality gates pass,mark release ‘ready forstaging’


criteria for release candidate

• Help define environment and application configurations to be tested

• Track coverages • Define, Implement and

Automate Smoke Tests• Design, Implement &


• Run Non-Functional Tests• Design, Implement &

Automate regression tests


CD Step 5: Test release candidateFrom

Continuous

Integration

Staging

ProductionQA

Jenkins

Typical activities1. Choose release candidate

to test2. Provision environments3. Install / conf application4. Run smoke test5. Run QA tests

- Acceptance / Explore- Performance- Secure code scan- Full security test

6. If quality gates pass,mark release ‘ready forstaging’

Testing Tip: While there are applications wherein the CD testing process can becompletely automated, it is the exception and NOT the rule

Testing Tip: Consider integrating your QA and Staging Smoke Tests into CI ifenvs are elastic as it will identify issues earlier and save testers time


CD Step 6: Test against production envFrom

Continuous

Integration

QA Production

Jenkins

StagingTypical activities1. Compliance testing2. Full load test3. Red team /

penetration test4. Rollback procedure

test OR A/B switchover test

5. If quality gates pass, production ready release candidate


criteria for compliance• Run Full Non-Functional

Tests• Validate

Rollback/Recovery• Test Feature Switches• Run full Story Tests• Run full Regression Test• Tests all critical

configurations• Validate any

dependencies• Validate monitoring


CD Step 6: Test against production envFrom

Continuous

Integration

QA Production

Jenkins

StagingTypical activities1. Compliance testing2. Full load test3. Red team /

penetration test4. Rollback procedure

test OR A/B switchover test

5. If quality gates pass, production ready release candidate

Testing Tip: Many downstream testing activities performed in Staging becomeredundant as you gain confidence in the quality of release candidates

Testing Tip: It is critical to effectively test your rollback / recovery processes


CD Step 7: Deploy to productionFrom

Continuous

Integration

QA

Staging

Production

Jenkins

Typical activities1. Automated / push

button deployment2. Deployment of

reliability monitoring3. Deployment of

security monitoring


criteria for production deployment

• Run Production Implementation tests

• Validate data/data migrations

• Validate monitoring


Wrap-Up


Questions?

Michael D. Sowers

@MichaelSowers4

Coveros, Inc.

TheTest!Expert'sRolein!DevOps! · Fundamental conflict - Dev vs. Ops •Operations •Risk Minimization - Stability, SLAs, Uptime, Response Time •Scripts and procedures to install

Documents