OpenCms Days 2015 Workflow using Docker and Jenkins

Daniel Seidel – Alkacon Software GmbH

OpenCms development

workflow using Jenkins and

Docker

28.09.2015

2

What can you expect?

Introduction to the Alkacon

development infrastructure

Presentation of the new

OpenCms Dev-Features

Development Scenario: Core 3

localhost

ci-autobuild

ci-10-0-x

ci-release

Development Scenario: Core 4

localhost

ci-autobuild

ci-10-0-x

ci-release

Why does it work good?

● Much Java Programming

● Each developer with his own task

● Always default configuration

What is different from the core scenario?

● Less Java Code, more Formatters etc.

● Developers may interact closer

● Special configurations

● Far more module import/export

● Less restarts necessary (jar update)

5

Development Scenario: Website

Is the core infrastructure good as well?

ci-autobuild

ci-10-0-x

ci-release

Development Scenario: Website 6

localhost

Why doesn’t it work good?

● Painful module import/export/check-in

● Configurations may diverge

● Merge conflicts for XML contents

● Many projects, many servers

● Problems getting from development

to production

7

Website: Better environment

localhost

website

docu

h

apro

xy

ci-docker

...

online.server.com

HTTP and SMB

HTTP and SMB

8

Website: Accomplishments (I)

localhost

online.server.com

HTTP and SMB

HTTP and SMB

website

docu

h

apro

xy

ci-docker

...

Docker intro (very short)

● Container like minimal VM

● Build from scripts

● Communication facilities

● Very portable

Docker benefits

● One (simple) server

● Automatic setup for containers

● Shared base images

● Easy switch

development production

9

Website: Accomplishments (II)

localhost

online.server.com

HTTP and SMB

HTTP and SMB

website

docu

h

ap

roxy

ci-docker

...

Flexible workflow

● Work on

● Server only

● Mount VFS

● Locally only

● Check-in from

● OpenCms

● Server/Docker

● Localhost

● Even the production server

● Live Demo

Git & Jenkins & Docker

Demo

Demo Demo Demo

デモ

Development infrastructure

● SMB module management:

● Import, export or delete modules via network share

● Git support for OpenCms

● Check in to Git from the OpenCms workplace

● XSLT transformations for configuration files

● Manipulate the configuration on start up

● Reduced export format

● Reduce manifest changes

● Avoid merge conflicts

11

We use: New OpenCms features

● New Wrapper for CmsJLanRepository

● Configure in opencms-importexport.xml

12

SMB module management (I)

<opencms>  <repositories> <repository name="OPENCMS" class="org.opencms.jlan.CmsJLanRepository> <params>  <param name="wrapper"> org.opencms.file.wrapper.CmsResourceWrapperModules </param> </params> </repository> </repositories> </opencms>

● When you mount the network share:

● A special folder “/modules” is present

● With the empty subfolder “import”

● And the subfolder “export” listing all installed modules

● With the folders you perform module operations

13

SMB module management (II)

Operation Performed via

Import module (add or replace) Drop module zip into “/modules/import/”

Export module Copy module zip from “/modules/export/”

Delete module Delete module zip in “/modules/export/”

How to determine the import site?

● New option in the module management

● Edit module: “Import site”

● Specifies the import site

● Default

● Root site for the SMB module management

● Current site in the administration view

14

SMB module management (III)

● Live Demo

Add, update, export, delete a module

Demo

Demo Demo Demo

デモ

SMB module management

How to check in a module?

16

Git support for OpenCms (I)

Can we do this automatically?

● Live Demo

From OpenCms to Git in one step

Demo

Demo Demo Demo

デモ

Git support

How does the support work?

● Workplace app + shell script + config file

● WEB-INF/git-scripts/module-checkin.sh (executable)

● WEB-INF/git-scripts/module-checkin.conf (config)

● Also: Stand-alone as shell script + config file

● Useful with SMB share’s modules feature

● One-click update of (local) repository

18

Git support for OpenCms (II)

config

executable

Read defaults

Read defaults

Export modules and

Call with params update

How to enable Git support?

● Make the file

● WEB-INF/git-scripts/module-checkin.sh

● executable.

● Place a suitable configuration under

● WEB-INF/git-scripts/module-checkin.conf

● (you may copy the module-checkin.conf.demo)

19

Git support for OpenCms (III)

Usage for website development

● Git support

● Direct check-in via the workplace app

● Backup configured modules on Docker restart via

script

● (local check-in via script)

● SMB module management

● Backup configured modules on Docker restart

● (local check-in via script)

● (one-click module updates)

20

Short summary – feature usage

The problem ● Different config for different dockers

● Different config development production

21

Config adjustment via XSLT (I)

Typically replaced items ● Site configuration (hostname)

● Email server configuration

● ...

Possible solutions ● Different config files

● Script-based config file manipulation during setup

● XSLT transformation on config load

Implementation details

● One XSLT file for all OpenCms config files

● XSLT 1.0 + special macro + parameters

● Transformations performed on start-up

How to apply transformations?

● Just place a file opencms-configuration.xslt

under WEB-INF/config/

22

Config adjustment via XSLT (II)

23

A typical XSLT file

<?xml version="1.0" encoding="UTF-8" ?> <xsl:stylesheet xmlns:xsl="..." version="1.0">  <xsl:param name="file" />  <xsl:output doctype-system="@dtd@" indent="yes" />  <xsl:template match="@* | node()"> <xsl:copy><xsl:apply-templates select="@* | node()"/></xsl:copy> </xsl:template>  <xsl:template match="/opencms/system/sites"> <sites> <workplace-server>http://days-demo</workplace-server> </sites> </xsl:template> </xsl:stylesheet>

What happens if You alter configurations

inside OpenCms?

● Question for you

Where to look for details?

● CmsConfigurationManager

● transformConfiguration(...)

Good for “Playing”

● The adjusted configuration is written in the log

at debug level

24

Just for the records

● Live Demo

How we use for website development

Demo

Demo Demo Demo

デモ

XSLT transformations

What’s the problem?

● Module manifest is very verbose

● In particular information like ● Export date

● Modification dates

● Authors

● (UUIDs)

● Information of “non-module” parent folders

● May cause merge conflicts

● Will cause many unnecessary and confusing

changes in the Git repository

26

Reduced export mode (I)

Possible solutions

● Do not write “critical” data to the manifest at all

● Replace “critical” data with macros, e.g.,

● FILE_TIME

● IMPORT_TIME

Possible problems

● Information loss

● What are the best dates?

● Time shifts?

● Renewing exported JSPs needs newer timestamp

27

Reduced export mode (II)

Current solution

● Two export modes

● “Default”: with all information, optional macros for

modification dates

● “Reduced”: with “critical” information omitted

● Mode set on a module basis

28

Reduced export mode (III)

Discuss possible problems or suggest

improvements!

● Live Demo

Reduced vs. Default

Demo

Demo Demo Demo

デモ

Comparison of export formats

● The shown setup

● Is very flexible in how you work

● Simplifies module import/export significantly

● Uses various additional software

● Jenkins

● Docker

● Git

● Eases deployment

● Takes advantage of various new OpenCms

features

30

Summary – Development setup

● SMB module management

● Good for automatization

● Eases module import / export

● Auto-install via auto-copy on build (we use gradle)

● Be aware: restarts for code changes remain

● Git support

● Wide variety of usage scenarios ● App at server or localhost

● Script only and SMB-share

● Fine-grained control on module check-in steps

● Consider: precise commit maximal automation

● At the moment: only for Linux (shell script)

31

Summary - Features

● XSLT-Transformations

● Easy config adjustment for different servers

● Transformations applied on startup

● One XSLT file for all files included by opencms.xml

● Good for updates?

● Reduced export format

● Purpose: Avoid merge conflicts

● Think about information loss

● Alternative:

timestamp macros (FILE_TIME, IMPORT_TIME)

32

Summary - Features

● Git-support:

● User-specific settings

● (cross-platform solution via jGit)

● Module export format

● Enable reduced export also for database exports

● (Separate manifest in various files)

● One per resource

● One for the module

● (Combine reduced format and timestamp macros)

33

Planned improvements

Any suggestions? Please let us know!

Daniel Seidel

Alkacon Software GmbH

http://www.alkacon.com

http://www.opencms.org

Thank you very much for your attention!