Speed up your build pipeline for faster feedback

Speed Up Your Build Pipeline for Faster FeedbackAshish Parkhi

IDeaS a SAS Company

[email protected]

@AshishParkhi

http://ashishparkhi.com

2

Build Pipeline – 10 Months Ago

About 60 to 90 minutes

3

Impact on life.

Image source – http://ak3.picdn.net/shutterstock/videos/5132438/preview/stock-footage-mixed-ethnicity-group-of-medical-professionals-

working-late-at-night-are-looking-at-a-computer.jpg

http://the247analyst.files.wordpress.com/2011/10/dealing-with-pressure.jpg

http://www.dimitri.co.uk/business/business-images/worker-alone-dark-office.jpg

http://cdn.sheknows.com/articles/2012/10/crying-little-girl.jpg

4

Build Pipeline - Now – under 10 Minutes

5

Focus on the

Bottlenecks

Divide and

Conquer

Key Principles to Speed Up

Your Build Pipeline

Fail Fast

6

Focus on the

Bottlenecks

Divide and

Conquer

Key Principles to Speed Up

Your Build Pipeline

Fail Fast

7

Disk IO – ExampleFocus on Bottleneck

File Operations

8

Disk IO – ExampleFocus on Bottleneck

Database operations.

Image Source - https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTPdVawndjUZbU2PDn-oKgjBPqmgDqr3PPZatZh9kxEgNi71AND

http://www.dba-oracle.com/images/large_disk_hot_files.gif

9

Disk IO – AlternativeFocus on Bottleneck

• Avoid file operations – e.g. duplicating workspace

Image Source - http://3.bp.blogspot.com/-bqTjSN7pSpg/UbqyjVojEFI/AAAAAAAADBw/PWe0kiuRHJ4/s200/no+duplicate+content.jpg

10

Disk IO – AlternativeFocus on Bottleneck

• Avoid file operations – e.g. Jar creation.

Image Source - http://i1.wp.com/blog.quoteroller.com/wp-content/uploads/2013/04/Dont-start-from-scratch.png?resize=800%2C264

11

Disk IO – AlternativeFocus on Bottleneck

• Robocopy/rsync.

Image source - http://www.asustor.com/images/admv2/022_Rsync%20Backup-The-ultimate-tool-for-remote-backup.png

12

Disk IO – AlternativeFocus on Bottleneck

• Test on smaller but apt data set.

Image source - http://4.bp.blogspot.com/_4hvqisoH9CE/TSZIs7eiSAI/AAAAAAAAA7E/vanj6bGD8XQ/s1600/big-vs-small-left.jpg

13

Disk IO - Alternative - SSDFocus on Bottleneck

• HDD (Toshiba MQ01ACF050 500GB SATA III) vs SSD

(Samsung PM851 512GB mSata)

14

Disk IO - Alternative - SSDFocus on Bottleneck

• HDD SSD

15

Disk IO - Alternative – In Memory DBFocus on Bottleneck

Memory (Heap) Engine

– had some limitations over myisam engine.

16

Disk IO - Alternative – In Memory DBFocus on Bottleneck

– was not supporting many MySQL queries so was

discarded.

17

Disk IO - Alternative – In Memory DBFocus on Bottleneck

database

– looked promising as it could support many MySQL

queries but still required couple of modifications to

our code.

18

Disk IO - Alternative – In Memory DBFocus on Bottleneck

– looked most promising as it is wire compatible with

MySQL, which means without code changes I could

just point to memsql and be done with it.

19

Disk IO - Alternative – RAM DriveFocus on Bottleneck

• SoftPerfect RAM Disk

Disk IO - Alternative – RAM DriveFocus on Bottleneck

• RAM Drive

21

Disk IO - Alternative – RAM DriveFocus on Bottleneck

• RAM Drive – did not work

22

Disk IO - Alternative – RAM DriveFocus on Bottleneck

• RAM Drive – did not work

23

CPU - ProfilingFocus on Bottleneck

24

CPU – Profiling - InsightsFocus on Bottleneck

Image source - https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcQde6NeSrbuv40CIhKtFa1OuIQXf7F7esMJKp1Ie7zmH2t29l6Z

Scanning resource bundle files from jars.

25

CPU – Profiling - InsightsFocus on Bottleneck

Image source - http://2.bp.blogspot.com/-uKMyLlB3F7o/Tqn_6yqdElI/AAAAAAAAB94/_1FMbHJFQBQ/s1600/weight-lift-cartoon.jpg

Loading Spring Application Context.

26

CPU – Profiling - InsightsFocus on Bottleneck

Image source - http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/00000/7000/000/7029/7029.strip.gif

Avoiding unnecessary activities during build e.g. sending

out email.

27

CPU – Profiling - InsightsFocus on Bottleneck

java.util.Calendar is horribly slow.

Total processing time took 20.72 minutes out of which Date

Arithmetic took 18.15 minutes which is about 87.6% of the

total processing time!

28

CPU – Profiling - InsightsFocus on Bottleneck

java.util.Calendar is horribly slow. We switched to joda

date library and deprecated java.util.Date API.

Now Date Arithmetic takes 1.30 minutes; that’s a massive

saving of 93.77%

29

CPU - ANT 1.7 Junit task options.Focus on Bottleneck

30

Focus on the

Bottlenecks

Divide and

Conquer

Key Principles to Speed Up

Your Build Pipeline

Fail Fast

31

CPU - Running Tests ConcurrentlyDivide and Conquer

• Create parallel jobs.

32

CPU - Running Tests ConcurrentlyDivide and Conquer

• Distribute tasks across multiple slaves.

Image source - https://wiki.jenkins-ci.org/download/attachments/2916393/logo.png?version=1&modificationDate=1302753947000

33

CPU - Running Tests ConcurrentlyDivide and Conquer

Image source - http://sharpreflections.com/wp-content/uploads/2012/06/multi_core_cpu.png

• Using @RunWith(ConcurrentJunitRunner.class).

– Curtesy - Mathieu Carbou

http://java.dzone.com/articles/concurrent-junit-tests

– Maven Surefire plugin has built in mechanism.

34

Focus on the

Bottlenecks

Divide and

Conquer

Key Principles to Speed Up

Your Build Pipeline

Fail Fast

35

Restructure The Build PipelineFail Fast

Image Source - http://javapapers.com/wp-content/uploads/2012/11/failfast.jpg

• We want our builds to give us fast feedback. Hence it is very important to

prioritize our build tasks based on what is most likely to fail first.

• Push unnecessary stuff to a separate build – Things like JavaDocs can be

done nightly.

• Separate out fast and slow running tests.

36

Incremental Build vs. Clean BuildFail Fast

• Local dev builds are incremental, instead of clean

builds, as it helps with faster feedback and fail fast.

37

Summary

• Focus on bottlenecks

– Avoid Disk IO - File operations, file based database operations.

– Use smaller datasets, robocopy, rsync

– Use in-memory databases, Ram Drives, SSDs.

– Perform CPU profiling, scan logs, to uncover the unknown.

– Verify build tool settings.

• Divide and Conquer

– Create smaller jobs that can run in parallel.

– Distribute jobs across multiple slaves.

– Write tests that can run in isolation and use ConcurrentJunitRunner to run them

in parallel.

• Fail Fast

– Restructure the build pipeline to uncover failures soon.

– Incremental Builds

38

Build Time Vs No Of Builds

Removed Workspace Duplication

Ant Junit Task – Fork Once

Ram Disk

Caching Resource

Caching Spring Context

Avoided Email

Joda DateTime

Deprecated Date API

Concurrent Junit

Runner

39

Impact on life

Image source - http://t3.gstatic.com/images?q=tbn:ANd9GcTCvK8pY5qcp7Gl3ZBjxN1mc1HVHdiy1sQhByKeGgUk_5eJuUk7cA

https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcQpoUXqhEpdGl1cLzn4gQsng_GyxUmOKWxYUH6GfrjN_FRUYPxw-Q

40

Resources

• Jenkins – http://jenkins-ci.org/

• CI – http://en.wikipedia.org/wiki/Continuous_integration

• Mklink – http://technet.microsoft.com/en-us/library/cc753194.aspx

• http://ant.apache.org/manual/Tasks/junit.html

• http://java.dzone.com/articles/javalangoutofmemory-permgen

• SSD – http://en.wikipedia.org/wiki/Solid-state_drive

• Hybrid disk – http://en.wikipedia.org/wiki/Hybrid_drive

• HSQL – http://hsqldb.org/

• H2 – http://www.h2database.com/html/main.html

• Memsql – http://www.memsql.com/

• MySQL is bazillion times faster than MemSQL

• Tmpfs – http://en.wikipedia.org/wiki/Tmpfs

• http://blog.laptopmag.com/faster-than-an-ssd-how-to-turn-extra-memory-into-a-ram-disk

• RAM Disk Software Benchmarked

• http://jvmmonitor.org/

• http://searchvmware.techtarget.com/tip/VMware-snapshot-size-and-other-causes-for-slow-

snapshots

• http://blogs.agilefaqs.com/2014/10/03/key-principles-for-reducing-continuous-integration-build-

time/

• http://googletesting.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html

• http://www.infoq.com/presentations/Development-at-Google

• http://crystalmark.info/software/CrystalDiskMark/index-e.html41

Thank You!

Ashish Parkhi

IDeaS a SAS Company

[email protected]

@AshishParkhi

http://ashishparkhi.com