Best Practices for Accelerating Continuous Testing

Seven Best Practices for

Accelerating Continuous Testing

Marc Hornbeek

Senior Solutions Architect

2 Spirent Communications

“There is no continuous delivery

without testing”

Dev Ops

3 Spirent Communications

Fast CT is required to race

through the many test steps in

the pipeline from Pre-Flight

through to Delivery

4 Spirent Communications

CT infrastructure must

be stable and accurate

because test failures at

any point in the pipeline

cause delays or roll-

backs

5 Spirent Communications

Consequences of

poor CT practices

80% failure rate when

using current tools

IDC Fortune 1000 survey,

December 2014

6 Spirent Communications

Accelerated Continuous Testing

Best Practices Principles

Comprehensive

&

Results

Test fast

&

Don’t Stop

Powerful

Relevant

7 Spirent Communications

CT best practices are across the pipeline

“Shift Left”

8 Spirent Communications

9 Spirent Communications

Focus on customer relevant defects

not quantity of defects

“Quality is everyone’s Responsibility.”

Dr. W. Edwards Deming

10 Spirent Communications

Pre-Flight Team-work Design for CT

CT End-to-End Culture

11 Spirent Communications

CT Process design

Fault tolerant

Obfuscation

Intelligent dashboards

CT-Ready Infrastructure

12 Spirent Communications

CT-Ready Tools

Restful APIs

Cache and pipeline

Virtual and physical

Program agnostic

Large scale

13 Spirent Communications

CT Tools Integration

Pre-Flight

CI Tests

Integration

Regression

System

Deployment

14 Spirent Communications

Orchestrate Test Topologies

Physical and Virtual

Utilization

measurement

Topologies

Test Automation

15 Spirent Communications

Accelerate Test Execution

Powerful servers

Test design

Pipeline

Scale horizontally

Thresholds

Dynamic test schedules

16 Spirent Communications

Fast Relevant CT Analytics

Continuous Monitoring (CM)

Relevance

Dashboards, Alerts, logs,

thresholds

Aggregation

Snapshots and trends

Telescope for diagnostics

17 Spirent Communications

Case Study

360 developers, six sites

36 MLOC+, 100+ features

86 build targets

complex interdependencies

200+ topologies

12K test cases

80+ deployment packages

(physical and virtual) 1,280,000,000

test possibilities

18 Spirent Communications

The primary contributing

factor was CT and CM !

CT Case Results

Metric

Major release

(#months)

Minor release

(#weeks)

# Features

Defects

Integrations / day

Tests / day

Automated tests

After

3

2

150

10

100

10

85%

Before

6

4

113

1260

0.5

0.3

5%

19 Spirent Communications

Best practices tool Best Practices Assessment Tool

simplifies the assessment process.

20 Spirent Communications

Summary

7 CT best practices emphasize

speed, don’t stop,

comprehensive and relevant.

Benchmark against Best

Practices Blueprint and

DevOps Assessment.

Don't fall into the “do-CT-

yourself / NIH” trap!

“There is no continuous delivery

without testing”

21 Spirent Communications

www.spirent.com/clear

1) CT Blueprint white papers

2) CT eBook

3) CT Assessment Tool

Take-aways

http://www.spirent.com/go/AMR

22 Spirent Communications

How can Spirent help?:

Test tools

CT Orchestration

Lab Management

Professional Services

23 Spirent Communications

© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name

“Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved.

Specifications subject to change without notice.

spirent.com

Thank you

Seven Best Practices for Accelerating Continuous Testing

Marc Hornbeek

Senior Solutions Architect

[email protected]

Twitter: mhexcalibur