Devops (start walking in the same direction) by ops

DEVOPSSTART WALKING IN THE SAME DIRECTION

WHO I AM?

Demis Rizzotto

System Engineer,DevOps Linux Engineer @Exxoss

Currently consultant @Lampiris

THE PROBLEM OF DOWNWARD SPIRAL

DIAMETRICALLY OPPOSED GOALS AND INCENTIVES

DEV▸ Respond to the rapidly

changing competitive landscape

OPS▸ Provide stable, reliable, and

secure service to the customer

THE CORE, CHRONIC CONFLICT WHEN ORGANIZATIONAL MEASUREMENTS AND INCENTIVES ACROSS DIFFERENT SILOS PREVENT THE

ACHIEVEMENT OF GLOBAL, ORGANIZATIONAL GOALS

FRUSTRATION

▸Unnecessary process steps

▸Unnecessary rework

▸Unnecessary features

▸Waiting for someone else to act on an action or task

▸Waiting for an approval

▸Waiting for an environment

▸Creating a ticket for a manual task

▸Reporting to management by manually updating spreadsheets

▸Manual status reporting

▸…

RESULTS

▸Fragile application are prone to failure

▸Ever increasing backlog and technical debt

▸Urgent security rework and remediation

▸More urgent projects into the queue

▸Big deployments very difficult to diagnose when fail

DEVOPS

More features quicker Stability

DEVOPS

DEV OPS

FACILITATE AND AUTOMATE DEPLOYMENT

HELP DEV TEAM BECOME AUTONOMUS

FACILITATE DIAGNOSTIC

GET DEVS INTERESTED IN « PROD STUFF »

GIVE ACCESS TO LOGS

GIVE ACCESS TO MONITORING

OPTIMIZE LOGS

DEFINE METRICS

QUALITY TESTS

MONITOR YOUR APPS

DEV IN DEVOPS

DEV IN DEVOPS

IMPROVEMENTS

▸ The ability to accurately track dependencies

▸ Services that are resilient and degrade gracefully

▸ The ability to archive data to manage the size of the production data set

▸ The ability to easily search and understand log messages across services

▸ The ability to trace requests from users through multiple services

▸ Simple, centralized runtime configuration using feature flags, …

DEV IN DEVOPS

▸ Make Dev responsible for their own services

▸ Integrate Dev into the incident management processes

▸ Have Dev cross-train Ops

▸ Embed Ops knowledge and capabilities into Dev

▸ Design for IT Operations

EMBED IT OPERATIONS INTO DEVELOPMENT

OPS IN DEVOPS

OPS IN DEVOPS

IMPROVEMENTS

▸ Self service

▸ Infrastructure as code

▸ Standardize infrastructure

▸ Make all infrastructure data visible

▸ Make application info data visible

▸ Modify the incident resolution process and blameless post-mortems

▸ Monitor the health of the deployment pipelines

OPS IN DEVOPS

▸ Single “repository of truth” containing both the code and environments

▸ Create the one-step Dev, Test and Production environment build process

▸ Extend the deployment pipeline processes into production

EXTEND DEVELOPMENT INTO IT OPERATIONS

WE ARE ALL DEVOPS

WE ARE ALL DEVOPS

▸ Stop think . Do

▸ Share responsability

▸ It’s Never Someone Else’s Problem

▸ Dedicated Designated Ops in every team

▸ Different type of devops, but devops is possible for all type and size of companies

▸ Iterate!

▸ Identify bottleneck and remediate

▸ Define small target and share results

DEVOPS IS FOR ALL

DEVOPS IS FOR ALL

DEVOPS IS FOR ALL

▸ AUTONOMUS TEAMS

▸ Trust but verify

▸ Local autonomy, but global consistency

▸ SELF SERVICE

▸ How ? API, web portal, infra as code , AWS accounts, …

▸ AUTOMATE

▸ If you are tired to do it manually. Automate it.

▸ REDUCE NON-VALUE ADDED

▸ If isn't your business it's maybe better don't do it your self

KEY OF SUCCESS : REDUCE FRUSTRATION

INFRASTRUCTURE

ON DEMAND ( IAAS )

EASIER TO REBUILD THAN TO REPAIR

INFRASTUCTURE

INFRASTRUCTURE

IMMUTABLE INFRASTRUCTURE▸ Automate the setup and deployment for every part and

every layer of your infrastructure.

▸ Never change any part of your system once it is deployed. If you need to change it, deploy a new system.

AVOID HISTORY

INFRASTRUCTURE

INFRASTRUCTURE AS CODE

Versionning and code review

Easier track changes

Bootstrap your infrastructure from scratch

Automate

CI/CD

TESTING

TESTING

CONTINUOUS TESTING

▸ Automated tests

▸ code quality

▸ Unit tests

▸ Stress tests

▸ Don’t forget to test infrastructure (load banacer, recovery etc.)

▸ Extend you build pipeline

SECURITY

SECURITY

INFORMATION SECURITY AS EVERYONE’S JOB, EVERY DAY

▸ Updates

▸ Use secrets management

▸ Integrate security into our CI/CD pipeline

▸ Check code libraries and their recommended configurations

▸ Dependency scanning

▸ Static analysis

▸ Dynamic analysis (ex: OWASP Zed Attack Proxy + Selenium)

SECURITY

SECURITY METRICS MONITORING

▸ Successful and unsuccessful user logins

▸ User password resets

▸ User email address resets

▸ Database syntax error

▸ User credit card changes

DEPLOYMENTS

DEPLOYMENTS

DEPLOYMENTS ARE ROUTINE AND PREDICTABLE

No stress . It’s a routine.

On business day when everyone is already in the office and without our customers even noticing

DEPLOYMENTS

HOW▸ Work in small batches and delivering quickly and

frequently

▸ Blue green deployment

▸ Feature toggle

▸ Canary testing

▸ Dark launch

▸ Automated deployments

MONITORING

ALERTING

▸ Mesure service health not only server health (Health pattern)

▸ Reducing Alert Fatigue

▸ Cut alerts that aren’t actionable

▸ Adjust thresholds

▸ Consolidate related alerts

▸ Give alerts relevant names & descriptions

▸ Make sure the right people are getting alerts

▸ Multiple notification rules and choose right channel

▸ Use incident management tools(Pagerduty, Victoryops,..)

MONITORING

IF IT MOVES, TRACK IT

ANOMALY DETECTIONTRENDS

APM

End user monitoring

Application Performance Management

DELIVERY PIPELINE METRICS - TEAM LEVEL DASHBOARD

LOGS

LOGS

▸ Set your log severity right

▸ Remember you will not be the only one reading these logs

▸ Track your communication with other systems

▸ Log exceptions correctly

▸ Use an ID to track your events

▸ Normalize yours logs

▸ Separate and Centralize your Log Data

LOGS

DOCUMENT, MEASURE, COMMUNICATE AND COLLABORATE

DOCUMENT

WRITING EFFECTIVE DOCUMENTATION

▸ Treat documentation like a requirement

▸ Put the Information in the Most Appropriate Place

▸ *Keep your documentation close to the code

▸ Keep documentation just simple enough, but not too simple

▸ Avoid creating large volumes of documentation of questionable detail which become obsolete shortly after they are written

MEASURE

MEASURE PROGRESS

▸ KPIs

▸ Deployment frequency

▸ Deployment speed

▸ Failure rate

▸ Time from code to prod

▸ Share goals

▸ Increase the visibility of work

COMMUNICATE

▸ Find better ways to communicante

▸ Promote day to day direct communication

▸ Coffee machine meetings

▸ Chat (Slack, HipChat, Team,…)

▸ Use video (Slack, Skype, Hangout,..)

▸ Quality over quantity

▸ Post Mortem meetings

▸ Bootcamps,…

IN DEVOPS CULTURE, COMMUNICATION AND COLLABORATION ARE KEY

COLLABORATE

TIPS FOR BETTER COLLABORATION

▸ Change the seating and rotate people through development

▸ Improve the environment for collaboration (Breakout areas, whiteboards, nice meeting rooms, a decent coffee machine)

▸ Cross domain workshops for analyze your process

▸ Block time for process improvement work (lean)

▸ Run lunch and learn / brown bag sessions

▸ Put a persistent chat room in place

Demis Rizzotto

[email protected]

@DemisRiz

THANKS YOU!QUESTIONS?