Top Banner
Abraham Marin-Perez @AbrahamMarin fromfragiletoagile.c om Keeping Your CI / CD Pipeline as Fast as It Needs to Be #MCRJava @AbrahamMarin @EqualExperts
106

Keeping your CI/CD pipeline as fast as it needs to be

Jan 15, 2017

Download

Software

Abraham Marin-Perez
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Keeping your CI/CD pipeline as fast as it needs to be

Abraham Marin-Perez@AbrahamMarin

fromfragiletoagile.com

Keeping Your CI / CD Pipeline as Fast as It Needs to Be

#MCRJava @AbrahamMarin @EqualExperts

Page 2: Keeping your CI/CD pipeline as fast as it needs to be

About Me

Page 3: Keeping your CI/CD pipeline as fast as it needs to be

About Me

Page 4: Keeping your CI/CD pipeline as fast as it needs to be

About Me

Page 5: Keeping your CI/CD pipeline as fast as it needs to be

About Me

Page 6: Keeping your CI/CD pipeline as fast as it needs to be

About Me

Page 7: Keeping your CI/CD pipeline as fast as it needs to be

About Me

Page 8: Keeping your CI/CD pipeline as fast as it needs to be

About Me

Page 9: Keeping your CI/CD pipeline as fast as it needs to be
Page 10: Keeping your CI/CD pipeline as fast as it needs to be

About This Talk

Page 11: Keeping your CI/CD pipeline as fast as it needs to be

Continuous Integration: check everything is still working after every commit

Continuous Deployment: every successful commit turns into a release

About This Talk

Page 12: Keeping your CI/CD pipeline as fast as it needs to be

About This Talk

Page 13: Keeping your CI/CD pipeline as fast as it needs to be

About This Talk

Page 14: Keeping your CI/CD pipeline as fast as it needs to be

About This Talk

Page 15: Keeping your CI/CD pipeline as fast as it needs to be

About This Talk

Page 16: Keeping your CI/CD pipeline as fast as it needs to be
Page 17: Keeping your CI/CD pipeline as fast as it needs to be

SUPER APP

# Files: 75# Tests: 800Build Time: 4 minOutput: superapp.war

Page 18: Keeping your CI/CD pipeline as fast as it needs to be

SUPER APP

# Files: 113# Tests: 1200Build Time: 6 minOutput: superapp.war

Page 19: Keeping your CI/CD pipeline as fast as it needs to be

SUPER APP

# Files: 169# Tests: 1800Build Time: 9 minOutput: superapp.war

Page 20: Keeping your CI/CD pipeline as fast as it needs to be

Slow feedback

Broken builds mask issues

Development paralysis

Impact on ability to meet our SLAs

Pay per use

Missed business opportunities

The Problems Of Size

Page 21: Keeping your CI/CD pipeline as fast as it needs to be

Live with it

Partial CD: only quick tests

Phased CD: split into components

Test Deprecation Policy

Microservices

How Organisations Manage Size

Page 22: Keeping your CI/CD pipeline as fast as it needs to be

Microservices

Page 23: Keeping your CI/CD pipeline as fast as it needs to be

Microservices

Page 24: Keeping your CI/CD pipeline as fast as it needs to be

Microservices

Page 25: Keeping your CI/CD pipeline as fast as it needs to be
Page 26: Keeping your CI/CD pipeline as fast as it needs to be
Page 27: Keeping your CI/CD pipeline as fast as it needs to be

SUPER APP

# Files: 169# Tests: 1800Build Time: 9 minOutput: superapp.war

APP BACKEN

DSUPER

APP

# Files: 115# Tests: 1200Build Time: 6 minOutput: superapp.war

# Files: 72# Tests: 800Build Time: 4 minOutput: appbackend.jar

Page 28: Keeping your CI/CD pipeline as fast as it needs to be

Microservices?

Page 29: Keeping your CI/CD pipeline as fast as it needs to be

Microservices

Page 30: Keeping your CI/CD pipeline as fast as it needs to be

Microservices

Page 31: Keeping your CI/CD pipeline as fast as it needs to be

Microservices

Page 32: Keeping your CI/CD pipeline as fast as it needs to be

Microservices

Page 33: Keeping your CI/CD pipeline as fast as it needs to be

Microservices

Page 34: Keeping your CI/CD pipeline as fast as it needs to be

Microservices

Page 35: Keeping your CI/CD pipeline as fast as it needs to be

Build Pipeline Becomes a Network

Page 36: Keeping your CI/CD pipeline as fast as it needs to be
Page 37: Keeping your CI/CD pipeline as fast as it needs to be

To Caesar what is Caesar’sBased on John Ferguson Smart’s“Real-World Strategies for Continuous Delivery with maven and Jenkins”

http://youtu.be/McTZtyb9M38

Page 38: Keeping your CI/CD pipeline as fast as it needs to be

John’s approachMaven wasn’t built for Continuous Deployment

commitcommitcommit...

0.0.1-SNAPSHOT

Release!

0.0.1

Page 39: Keeping your CI/CD pipeline as fast as it needs to be

Don’t use RELEASE pluginUse VERSIONS plugin

Set version to <version scheme>.<build number>Run mvn deployCommit pom file to repository

John’s approach

Page 40: Keeping your CI/CD pipeline as fast as it needs to be

Set version to <version scheme>.<build number>

mvn versions:set –DnewVersion=**your version**

John’s approach

Page 41: Keeping your CI/CD pipeline as fast as it needs to be

Run mvn deploy

mvn clean deploy

John’s approach

Page 42: Keeping your CI/CD pipeline as fast as it needs to be

Commit pom file to repository<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-scm-plugin</artifactId> <version>1.9.1</version> <executions> <execution> <id>commit</id> <phase>deploy</phase> <goals> <goal>checkin</goal> </goals> </execution> </executions></plugin>

John’s approach

Page 43: Keeping your CI/CD pipeline as fast as it needs to be

John’s approach

Page 44: Keeping your CI/CD pipeline as fast as it needs to be

0.0.1.1commit

BUILD!

0.0.1.2commit

BUILD!

0.0.1.3commit

BUILD!

John’s approach

Page 45: Keeping your CI/CD pipeline as fast as it needs to be

How do you scale this?

Page 46: Keeping your CI/CD pipeline as fast as it needs to be

<dependency> <groupId>com.superappfactory</groupId> <artifactId>appbackend</artifactId> <version>LATEST</version></dependency>

<dependency> <groupId>com.superappfactory</groupId> <artifactId>appbackend</artifactId> <version>??????</version></dependency>

APP BACKEN

DSUPER

APP

appbackend.jar superapp.war

Setting up dependencies

Page 47: Keeping your CI/CD pipeline as fast as it needs to be
Page 48: Keeping your CI/CD pipeline as fast as it needs to be
Page 49: Keeping your CI/CD pipeline as fast as it needs to be
Page 50: Keeping your CI/CD pipeline as fast as it needs to be

Setting up dependencies

APP BACKEN

DSUPER

APP

appbackend.jar superapp.war

Page 51: Keeping your CI/CD pipeline as fast as it needs to be

Rebuilding old versions

Page 52: Keeping your CI/CD pipeline as fast as it needs to be

Using “LATEST” makes it impossible to build old versions correctly

Rebuilding old versions

Page 53: Keeping your CI/CD pipeline as fast as it needs to be

<dependency> <groupId>com.superappfactory</groupId> <artifactId>appbackend</artifactId> <version>1.5.3.1</version></dependency>

<dependency> <groupId>com.superappfactory</groupId> <artifactId>appbackend</artifactId> <version>???????</version></dependency>

Rebuilding old versionsAPP

BACKEND

SUPER APP

appbackend.jar superapp.war

Page 54: Keeping your CI/CD pipeline as fast as it needs to be

Update versions of dependenciesmvn versions:use-latest-releases

Page 55: Keeping your CI/CD pipeline as fast as it needs to be

APP BACKEN

DSUPER

APP

APP BACKEN

DSUPER

APPDATA MODE

L

SUPER APP

DATA MODEL

GUI

APP BACKEND

Page 56: Keeping your CI/CD pipeline as fast as it needs to be
Page 57: Keeping your CI/CD pipeline as fast as it needs to be

commitbuild

deploy

commit

Problem: Infinite Trigger

Page 58: Keeping your CI/CD pipeline as fast as it needs to be
Page 59: Keeping your CI/CD pipeline as fast as it needs to be
Page 60: Keeping your CI/CD pipeline as fast as it needs to be

APP BACKEN

DSUPER

APP

commit

Problem: Unnecessary Rebuilds

Page 61: Keeping your CI/CD pipeline as fast as it needs to be
Page 62: Keeping your CI/CD pipeline as fast as it needs to be

Problem: Unnecessary Rebuilds

Page 63: Keeping your CI/CD pipeline as fast as it needs to be

Get last committer

buildAgent?

Proceed normally

Don’t run build

NO YES

touch skip_build

Problem: Unnecessary Rebuilds

Page 64: Keeping your CI/CD pipeline as fast as it needs to be

<profiles><!-- Plugins that need to be disabled when doing a no-run --> <profile> <id>do.nothing</id> <activation> <file>

<exists>skip_build</exists> </file> </activation>

<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.1</version> <configuration> <skipMain>true</skipMain> <skip>true</skip> </configuration> </plugin>

Problem: Unnecessary Rebuilds

Page 65: Keeping your CI/CD pipeline as fast as it needs to be

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-install-plugin</artifactId> <version>2.5.2</version> <configuration> <skip>true</skip> </configuration></plugin>

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-deploy-plugin</artifactId> <version>2.8.2</version> <configuration> <skip>true</skip> </configuration></plugin>

Problem: Unnecessary Rebuilds

Page 66: Keeping your CI/CD pipeline as fast as it needs to be

APP BACKEND

SUPER APP

commit

Problem: Unnecessary Rebuilds

Page 67: Keeping your CI/CD pipeline as fast as it needs to be

Problem: Unnecessary Rebuilds

Page 68: Keeping your CI/CD pipeline as fast as it needs to be

APP BACKEND

SUPER APP

commit

Problem: Necessary Rebuilds

Page 69: Keeping your CI/CD pipeline as fast as it needs to be

Get last committer

buildAgent?

Proceed normally

Don’t run build

NO YES

Problem: Necessary Rebuilds

Page 70: Keeping your CI/CD pipeline as fast as it needs to be
Page 71: Keeping your CI/CD pipeline as fast as it needs to be

Get last committer

buildAgent?

Proceed normally

NO YES

Check dependencies

Up to date?

NO Don’t run build

YES

touch skip_build

Page 72: Keeping your CI/CD pipeline as fast as it needs to be

Problem: Necessary Rebuilds

Page 73: Keeping your CI/CD pipeline as fast as it needs to be

build

deploy

commit

commit

commitpom.xml

Problem: Doomed Build

Page 74: Keeping your CI/CD pipeline as fast as it needs to be

Get last committer

buildAgent?

Proceed normally

NO YES

Check dependencies

Up to date?

NO Don’t run build

YES

touch skip_build

Page 75: Keeping your CI/CD pipeline as fast as it needs to be

Get last committer

buildAgent?

Check pom.xml

NO YES

Check dependencies

Up to date?

NO

Don’t run build

YES

touch skip_build

Up to date?

NO

Proceed normally

YES

Page 76: Keeping your CI/CD pipeline as fast as it needs to be

Problem: Doomed Build

Page 77: Keeping your CI/CD pipeline as fast as it needs to be
Page 78: Keeping your CI/CD pipeline as fast as it needs to be
Page 79: Keeping your CI/CD pipeline as fast as it needs to be

A real case scenario

Page 80: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Parent POM

Logging

Page 81: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Parent POM

Logging

28%

Page 82: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Parent POM

Logging

28%28%

28%

Page 83: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Parent POM

Logging

28%28%

28%

20%

Page 84: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Parent POM

Logging

48%

28%

28%

20%

Page 85: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Parent POM

Logging

Page 86: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Page 87: Keeping your CI/CD pipeline as fast as it needs to be

Build Time (BT): time an individual build takes to run

Change Rate (CR): percentage of commits upon an individual build with respect to the whole system

Useful Metrics

Page 88: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Page 89: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Parent POM

Logging

28%

Page 90: Keeping your CI/CD pipeline as fast as it needs to be

Impact Time (IT): total time to run a build and all the builds that will be triggered as a result

Useful Metrics

Page 91: Keeping your CI/CD pipeline as fast as it needs to be

No dependants IT(A) = BT(A)

A

Useful Metrics

Page 92: Keeping your CI/CD pipeline as fast as it needs to be

Serial execution IT(A) = BT(A) + IT(B) + IT(C)

B

A

C

Useful Metrics

Page 93: Keeping your CI/CD pipeline as fast as it needs to be

Parallel execution IT(A) = BT(A) + max(IT(B), IT(C))

B

A

C

Useful Metrics

Page 94: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Page 95: Keeping your CI/CD pipeline as fast as it needs to be

Weighted Impact Time (WIT): impact time of a build weighted according to its change rage

WIT(A) = IT(A) * CR(A)

Useful Metrics

Page 96: Keeping your CI/CD pipeline as fast as it needs to be

Average Impact Time (AIT): total time needed, on average, to execute all necessary builds after any given commit anywhere in the system

AIT = WIT(A) + WIT(B) + ... + WIT(Z)

Useful Metrics

Page 97: Keeping your CI/CD pipeline as fast as it needs to be

Sample Thresholds

Page 98: Keeping your CI/CD pipeline as fast as it needs to be

Average Impact Time

Average Impact Time is what indicates how well you have scaled your system

Sample Thresholds

Page 99: Keeping your CI/CD pipeline as fast as it needs to be

Maximum Impact Time

In a worst-case scenario, a build won’t take longer than this.

Sample Thresholds

Page 100: Keeping your CI/CD pipeline as fast as it needs to be

Maximum Impact Time for Critical Components

The same, but only for your most sensitive modules (log-in, payment gateway, etc.)

Beware of dependencies!

Sample Thresholds

Page 101: Keeping your CI/CD pipeline as fast as it needs to be

Service

Service

Service

Page 102: Keeping your CI/CD pipeline as fast as it needs to be

Manual processingtakes time...

Page 103: Keeping your CI/CD pipeline as fast as it needs to be

Most CI systems provide an API Calculations aren’t complex Multiple graphical tools available

Automating Build Analysis

Page 104: Keeping your CI/CD pipeline as fast as it needs to be

github.com/quiram/build-hotspotsBuild Hotspots

Page 105: Keeping your CI/CD pipeline as fast as it needs to be

https://commons.wikimedia.org/wiki/File:2012_Italian_GP_-_Lotus_wheel.jpg

Page 106: Keeping your CI/CD pipeline as fast as it needs to be

Thank You

@EqualExperts

equal-experts

equalexperts.com

Thank Youfromfragiletoagile.com

@AbrahamMarin#FastCI #MCRJava


Related Documents
Frequently Asked Questions Pipeline Pathways Keeping ...€¦ · Vinyl, wood, and/or chain link fences may cross the pipeline if there are no fence posts installed within five feet

Frequently Asked Questions Pipeline Pathways Keeping...

Category: Documents
Building an AppSec Pipeline: Keeping your program, and your life, sane

Building an AppSec Pipeline: Keeping your program, and your....

Category: Technology
Keeping More Women in the Leadership Pipeline€¦ · KEEPING MORE WOMEN IN THE LEADERSHIP PIPELINE; DRIVERS, FACILITATORS AND ENABLERS TO WOMEN’S ADVANCEMENT Charlene Dawn Frail

Keeping More Women in the Leadership Pipeline€¦ ·...

Category: Documents
Pfizer Pipeline - Amazon Web Services...Pfizer Pipeline Snapshot as of April 28, 2015 Pfizer Pipeline Snapshot as of February 27, 2015 Registration Recent Approvals - Ibrance (palbociclib)

Pfizer Pipeline - Amazon Web Services...Pfizer Pipeline...

Category: Documents
R&D Activities as Driver of Efficient Pipeline Integrity ... · R&D Activities as Driver of Efficient Pipeline Integrity ... Safety approach 5. Protection measures 6. Pipeline ...

R&D Activities as Driver of Efficient Pipeline Integrity...

Category: Documents
SAN DIEGO COUNTY WATER AUTHORITY PIPELINE RELINING PROGRAM Keeping the Water Flowing

SAN DIEGO COUNTY WATER AUTHORITY PIPELINE RELINING PROGRAM.....

Category: Documents
Secrets to Keeping Lean as a Fighting Machine

Secrets to Keeping Lean as a Fighting Machine

Category: Documents
PIPELINE INFRASTRUCTURE LIMITED · Details of the Company Pipeline Infrastructure Limited (Formerly known as Pipeline Infrastructure Private Limited) CIN: U60300MH2018PLC308292 Registered

PIPELINE INFRASTRUCTURE LIMITED · Details of the Company.....

Category: Documents
py-zmq-pipeline Documentation · Py-zmq-pipeline is as high level pipeline pattern implemented in Python_ with ZeroMQ_. TheZeroMQ Pipeline patternis a data and task distribution pattern

py-zmq-pipeline Documentation · Py-zmq-pipeline is as high...

Category: Documents
Burdekin River Pipeline EA Amendment · pipeline is also approved as an external water pipeline infrastructure corridor as per Schedule A of the site’s EA EPML00982113. The remaining

Burdekin River Pipeline EA Amendment · pipeline is also...

Category: Documents
Trans Mountain Pipeline ULC › application... · an application by Trans Mountain Pipeline ULC . as General Partner of Trans Mountain Pipeline L.P. (collectively “Trans Mountain”)

Trans Mountain Pipeline ULC › application... · an...

Category: Documents
Writing data analysis pipeline as ruby gem

Writing data analysis pipeline as ruby gem

Category: Presentations & Public Speaking