JUC Europe 2015: Multi-Node Environment as a Jenkins Slave (Compound-Slave)

Multi-node environment as a Jenkins slave aka “compound-slave”

Denis Chernilevskiy

Head of Media Advertising Department

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

Common use-case the “prehistory”

Multi-node environment as a Jenkins slave

Common use-case

4

He’s famous The “slave” The “job”

│ Test this please!

The quest

6

Nodes Cloud Deploy Get results Test

The solution

▌  Cloud – Amazon/OpenStack/Docker/…

▌  Deploy – JClouds/DockerCloud + Chef/Salt/Ansible/…

▌  Test – PyTest/TestNG/…

▌  Results – Allure Framework/xUnit/…

7

How it works

▌  The common job concept

›  Take 1 slave

›  Deploy the system under test to the slave

›  Deploy and run tests on the same slave

›  Clean the slave

›  Return the slave to the pool

8

│ Perfectly fits with 1 slave

│ But what if…

11

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

The “real life” of QA/DevOps the problem

Multi-node environment as a Jenkins slave

│ Yandex MediaAd platform

│ Test this please!

Process requirements

▌  Auto build, deploy, test

▌  Continuous Integration

▌  “Green trunk”

15

System components

▌  Front-end UI and API

▌  Ad.Management

▌  Ad.Selection ▌  Ad.Storage

▌  SocDem prediction & machine learning

16

│ None fits into 1 slave…

The obvious “not a solution” J

▌  Use Jenkins

▌  Get a slave for the job

▌  Get some more cloud nodes (somehow)

▌  Use the slave as a launchpad (run deploy, tests remotely)

18

│ Not for us J

Why not?

▌  Laziness. The less new code, the better! ▌  Separate nodes management

▌  JClouds not reused ▌  Lack of visibility

20

│ Several slaves per job?

│ Need to refactor core…

│ Several nodes as a slave!

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

“Compound-slaves” plugin the solution

Multi-node environment as a Jenkins slave

Features

▌  Group several nodes as a compound-slave

›  Existing nodes grouping

›  Get & group from the cloud

25

Features

▌  Cloud provisioning

›  Plugins with a “Cloud” interface

›  Using labels from the cloud plugin

›  Number of each label to provision

26

Features

▌  Roles and labelling

›  Distinct nodes

›  Default labels

27

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

Here comes the DEMO! J

Features

▌  General

›  Built-in slave management

›  Built-in job-slave provisioning

▌  Visibility

›  Built-in dashboards

›  Built-in job steps configuration

29

│ Profit! J

Links

▌  wiki.jenkins-ci.org/display/JENKINS/Compound+Slaves

▌  github.com/jenkinsci/compound-slaves-plugin

31

Contacts

[email protected]

+79250351311 @dchernilevskiy

denis.chernilevskiy

Denis Chernilevskiy

Head of Yandex MediaAd Department