© 2019 Juniper Networks Juniper Business Use Only...© 2019 Juniper Networks Juniper Business Use Only EVOLUTION OF NETWORK ARCHITECTURE VS. NETOPS In spite of years of focus on network

© 2019 Juniper Networks Juniper Business Use Only


CONFIDENTIALITY AND LEGAL NOTICE

This material contains information that is confidential and proprietary to Juniper Networks, Inc. Recipient may not

distribute, copy, or repeat information in the document without a signed non-disclosure agreement (NDA).

Any statements of product direction contained in this presentation sets forth Juniper Networks’ current intention and is

subject to change at any time without notice. No purchases are contingent upon Juniper Networks delivering any

feature or functionality depicted in this presentation.

Copyright 2019 Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo,

Juniper, Junos, and NXTWORK are registered trademarks of Juniper Networks, Inc. in the United States and other

countries. All other trademarks, service marks, registered marks, or registered service marks are the property of

their respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper

Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.


James Kelly, Juniper Networks

Paul Arsenault, Blackberry

NETWORK AUTOMATION

Getting from Automatable to Automated


EFFECTIVE

RELIABLE

VELOCITY, AGILITY,SCALE, REACH

EFFICIENT

DIGITAL OPS,DEVOPS,

PUBLIC CLOUD, IoT

SECURITY,SRE

EXPERIMENTATION,BIG DATA, AI

TCO,OPEN SOURCE

Hierarchy of needs


STATE OF

NETWORK AUTOMATION


NETWORK AUTOMATION

AUTOMATED NETWORKS


FROM making better networks TO making networking better

NETWORKS(noun)

NETWORKING(verb)

success

Automation is an operational paradigm


TWO SIDES TO AUTOMATED NETWORKS

DEV OPS

Automated NetOpsVendor’s Goal Customer’s Goal

EVOLVING TOOLS & ARCHITECTURE • Quality products• Autonomous (SDN)• Abstraction (intent)• Analytics• APIs and tool integrations

EVOLVING OPS & NRE• Codified workflows• Infrastructure as code (GitOps)• Automated building and testing (CI)• Automated deployments (CD)• Automated response (CR)

VENDOR CUSTOMER

Simpler and Automatable(building the tools)

Engineering Reliable Services(using the tools)


EVOLUTION OF NETWORK ARCHITECTURE VS. NETOPS

In spite of years of focus on network automation:

• Programmability, APIs and API protocols

• SDN, NFV, virtual networks

• SDKs and toolkits

• Integrations with tooling like Ansible, Puppet, Chef, Salt, etc.

Today’s average NetOps work looks like the 1980s

• Still manual rote tasks and toil

• Manual work maybe from CLI to GUI, but not yet to APIs

• Midnight maintenance windows are normal

• Change is the main cause of outage

• People are CLI jockeys / CCIE technicians more than engineers

FROMSCALE-UP

TO SCALE-OUT

SOME THINGS HAVE CHANGED … AND SOME THINGS HAVE NOT

The 80s called and…

They want their CLI back


EVOLUTION OF NETWORK ARCHITECTURE VS. NETOPS

In spite of years of focus on network automation:

• Programmability, APIs and API protocols

• SDN, NFV, virtual networks

• SDKs and toolkits

• Integrations with tooling like Ansible, Puppet, Chef, Salt, etc.

Today’s average NetOps work looks like the 1980s

• Still manual rote tasks and toil

• Manual work maybe from CLI to GUI, but not yet to APIs

• Midnight maintenance windows are normal

• Change is the main cause of outage

• People are CLI jockeys / CCIE technicians more than engineers

FROMSCALE-UP

SOME THINGS HAVE CHANGED … AND SOME THINGS HAVE NOT

TO SCALE-OUT


CONSUMPTION IS THE CONTEMPORARY CHALLENGE

of data center NetOps are still manual

82%

Mistakes of the past• Forcing engineers’ rebirth as a “developers”• Over-sensationalizing of tech has left behind

transformation of people & processes

Ironically…humans are the heroesin the journey to automated NetOps

BRIGHT SPOT:

NRENetwork reliability engineer(ing)


WHAT DOES SUCCESS

LOOK LIKE?


NRE: INSPIRATION FOR NETWORKERS

Workbook now FREE

2019

DevOps is Coined

2009

Google publishes SRE

2016

DevOps Handbook

2015

Treats Ops as a software engineering problem… More engineering rigor and detail


WHAT IS NETWORK RELIABILITY ENGINEERING

Nre

Core networking

fundamentals still matter.

It’s right in the name.

nRe

Represents a better way

of doing things.

Emphasizes the true goal

of automation

https://juniper.net/us/en/products-services/what-is/nre/

nrE

You can’t buy engineering

- you DO it. Sidesteps the

“productization” of

automation

Codify Automate Test Monitor Measure

https://juniper.net/us/en/products-services/what-is/nre/

Juniper Business Use Only

BLACKBERRY’SAUTOMATION PROGRAM

Juniper Business Use Only © 2017 BlackBerry. All Rights Reserved. 19

19

Table of Contents• Why Automate

• What Makes Automating Hard

• What We Did

• How We Did It

• How The Business Benefits

• What’s Next


Why Automate?


21

Security

▪ Cylance Acquisition – BlackBerry is now branded as a Cyber Security company


22

Security

▪ Cylance Acquisition – BlackBerry is now branded as a Cyber Security company

▪Over the last couple of years, the number of catalogued vulnerabilities has almost doubled

▪Over 70% of organizations take up to 90 days to patch

▪Once a threat is made public the clock starts ticking

▪ It takes attackers time to develop tools to exploit new vulnerabilities

▪ If you can test and deploy a fix faster than the attacker can develop an exploit you get to live

another day

Juniper Business Use Only 23© 2017 BlackBerry. All Rights Reserved.

Need to Patch Faster

▪ Traditional way of certifying new code for production deployment is

resource intensive and slow

▪ Human resources

▪ Lab resources


Balancing Act

STABLE

NETWORK


Balancing Act

STABLE

NETWORK

CYBER

THREATS


Balancing Act

STABLE

NETWORK

CYBER

THREATSAUTOMATIO

N


Balancing Act

STABLE

NETWORK

CYBER

THREATSAUTOMATIO

N

Enterprises Need to Balance the

Requirement to Patch Security

Vulnerabilities Against the Need to

Provide a Stable Network

Infrastructure

✓ Automated Testing

✓ Regression as well as New Features

✓ 24x7

✓ Infrastructure as Code

✓ Standards-based Design

✓ Vendor Agnostic

✓ Open Source Tools

✓ Automated Deployment


28

Other Benefits

▪ Get more done in less time

▪ Employees focus on higher-value work

▪ Improved employee engagement

▪ Simplified operations

▪ Better test coverage


What Makes Automating Hard?


Complicated Landscape

▪ Enterprises typically have a heterogeneous network environment

▪Multi-vendor

▪ Vendor-specific, monolithic operating systems

▪ Different feature sets with customer-specific use cases and

configurations




▪Multi-vendor



configurations

▪ Talent with the required skills is at a premium

▪ Staff that have the networking and programming skills and

experience are very hard to find




▪Multi-vendor



configurations

▪ Talent with the required skills is at a premium

▪ Staff that have the networking and programming skills and

experience are very hard to find

▪ Different vendors have different maturity levels with respect to their

automation capabilities


BlackBerry Solution


34

What We Did

▪ Built a vendor-agnostic automation framework based on open-source tools that support automated

testing, deployment, and configuration management


35

What We Did



▪ Git

▪ Repository (code, variable files, templates, playbooks, test reports)

▪ YAML and JSON describe specific instances

▪ JINJA2 templates describe configuration standards


36

What We Did



▪ Git




▪ Ansible

▪ Does the heavy-lifting of interacting with devices

▪ YAML Playbooks define the operations we want to perform via Ansible

▪ Templates and variable files with Ansible modules generate

device-specific configurations


37

What We Did



▪ Git




▪ Ansible





▪ Jenkins

▪ Orchestrates our actions using pipelines

custom-built to match our workflows

▪ Integrated with our ticketing system


38

What We Did



▪ Git




▪ Ansible





▪ Jenkins

▪ Orchestrates our actions using pipelines

custom-built to match our workflows

▪ Integrated with our ticketing system

▪ Python Scripts


Deployment

▪ To mitigate the vulnerabilities we need to update the devices to the

new certified code


Deployment

▪ To mitigate the vulnerabilities we need to update the devices to the

new certified code

▪ Built deployment tools built on top of our automation framework

▪Orchestrated by Jenkins

▪ Use Ansible where possible

▪ Custom Python where required

▪ Incorporates traffic shifting for routers and HA for switching

▪ Pre and post-checks


What It Looks Like

▪ Automation Framework using open source, vendor-agnostic

tools

▪ Regression test suite that covers our own specific uses

cases and configurations

▪ Dynamic generated and easy-to-read test reports

▪ A library of test cases

▪ Deployment framework to push code and manage

configuration

▪ Eco-system built around the automation framework


Automation Building Blocks

Git

Repository

Network

EngineerLab Jenkins

Production

Jenkins

Production

AnsibleUpgrade Tools

Production

Devices

Test Traffic

Lab Ansible

Upgrade Tools

Regression

Test Suite

Test Bed


Code Certification Testing Process

Git

Repository

Review Test

Results

Updated

Vendor Code

ProductionTest ResultsRegression

Test Suite

Test Bed

Passed?Yes

No


How We Did It


How We Did It

Created a new culture over a period of 2 years


How We Did It


▪ Identified innovators and leaders within our staff


How We Did It



▪ Empowered staff to be creative – permission to fail

▪ Encouraged and enabled peer-to-peer collaboration and support


How We Did It





▪ Provided baseline formal training and detailed knowledge transfers

for all technical staff

▪Mandated that all new platforms require automated testing and

deployment before being released to production

▪ Set corporate goals and objectives around automation


How We Did It





▪ Provided baseline formal training and detailed knowledge transfers

for all technical staff

▪Mandated that all new platforms require automated testing and

deployment before being released to production

▪ Set corporate goals and objectives around automation

▪ Set cut-off dates for manual changes and measured using KPIs that

are shared with the team

▪ Included automation capabilities as part of network equipment

vendor selection


Business Benefits


51

More Secure…Better Quality…Faster

▪ New code with security vulnerabilities patched delivered to production at a much faster rate

▪90% reduction in certification time – 10 weeks down to 1 week

▪80% reduction in deployment time for new code – 6 months down to 5 weeks

▪ Improved test coverage

▪ Better code quality from vendors


Simplified Operations

Automation Drives Standardization

▪ Defined standards and templates enforced via the automation

framework and the eco-system

▪ Test cases that are reusable by operations as sanity checks for

troubleshooting as well as pre and post-change

▪Measurable KPIs to gauge improvements

▪ Current and standards-based network operating system

deployments means more consistency and improved feature

availability

▪ Standardized and well-known environment reduces the risk of

unexpected behaviors


53

Improved Employee Engagement

▪ Employees can spend their time on higher-level and more interesting work

▪Less time writing and approving MOPs

▪ Improved service delivery

▪ Challenging work that demands new skills coupled with on-the-job training keeps employees

engaged

▪ Smaller and more agile team can achieve more in less time


What’s Next?


It’s a Journey, Not a Destination

▪ Automation framework has a lifecycle of its own

▪ Content requires frequent updating

▪ New test cases developed when corner cases are

discovered and issues are encountered in production

▪ Updates to test cases as the environment changes

▪ New platforms require automation to be written

Automation is never complete


What’s Next?

▪ Software-defined lab

▪ Use automation to orchestrate building the lab topology dynamically on-demand

▪ Leverage NFV to cost-effectively expand the capabilities of the lab

▪ Coordinated deployments using automation

▪ Use Jenkins jobs to call other Jenkins jobs to coordinate deployments across platforms

▪ This includes multi-cloud deployments

▪ Terraform deploys public cloud environments and calls our automation to do the on-prem portion

▪ Continuous Integration Continuous Delivery (CICD)

▪ Integration with Slack or an equivalent

▪ Minor changes coordinated and approved in real-time

▪ Change notifications sent via channels to which stakeholders (eg. NOC) subscribe

▪ Pre-deployment change validation and testing in lab


Thank You


5-STEP JOURNEY


Manual Ops

• NetOps at the device or system UI

• Engineers are more technicians than technologists

Continuous Processes,Continuous Pipeline

• Dev/test environment

• CI-CD-CR DevNetOps pipeline for changes to intent and code

• Fast fail feedback, micro changes, staging and canary deployments

• Toolchain and code to automate analytics for regulation / remediation

As-code, Test-driven

• Connect actions to triggers

• Rethink troubleshooting as testing

• Everything is code (even configs) to be tested

• Security integrated from the start

Ad Hoc Automated Workflows

• Automate the design of NetOps workflows

• Focus on frequent troubleshooting or read-only tasks before config management

Engineering Outcomes

• Manage, don’t maximize, reliability by higher-order metrics

• NRE outcomes with service-level objectives, indicators and agreements (SLO/SLI/SLA)

• Use error budgets, toil budgets

• Chaos engineering

• Manage dependencies, separation of concerns

AUTOMATED NETOPS 5-STEP

DESTINATIONPeople:

Network ReliabilityEngineers (NRE)

Process:

“DevNetOps”And NRE’ing

Technology:

Abstracted, automated and autonomous NetOps


THANK YOU

© 2019 Juniper Networks Juniper Business Use Only...© 2019 Juniper Networks Juniper Business Use Only EVOLUTION OF NETWORK ARCHITECTURE VS. NETOPS In spite of years of focus on network

Documents