Making software development processes to work for you

Making software development processes to work for youOptimizing investments to balance between bureaucracy and total liberalism in software development

Mikko Paukkila – 9-Oct-2014

Background• Mikko Paukkila is a Principal Systems Engineer in software build,

release, testing and integration area. He has experience of multiple software development environments like Linux, Mac, Windows, Android, Qt, Symbian etc. He has also wide knowledge of agile & lean processes and related tools from small 10-20 people software&hardware development product projects to very large programs of 1000+ developers. He has been driving deployment of continuous integration processes in an international environment and has been administrating related tools like Jenkins and Gerrit. Expert of SCM tools like git. Mikko currently for Nokia Technologies for imaging technology business.

• Disclaimer: The presentation will contain a lot of opinions that are mine, not company’s.

Some words about software development processes

From need to productPurpose of software development processes&tools should be enabled smooth flow between these phases:

1. From (customer) need to requirement

2. From requirement to a software change

3. From software change to high quality product

4. Product creation/finalization process

Require

ment

managem

en

t

Pro

duct

creatio

n

Softw

are

develo

pm

ent

And not forgetting feedback loop.

Continuous Integration enables fast and extensive feedback of software changes that are in our release chain. This is done by automated tools, but it requires that developer commits software change(s)

often and react fast if there are problems in builds.

CI is done to enable errors and integration problems to be found earlier.

This eventually leads us to produce better quality software already from the beginning and enables also us to get our

products and solutions faster to the market.

Basic Continuous Integration system:

Continuous Integration

Developer practices:

1. Commit changes often

2. Follow the status (build & test results)

3. React when needed (debug, fix, merge etc.)

AutomatedtestingFast builds

Enablers of Continuous Integration

Version control system (git,…)

Build system (e.g. Jenkins)

Test automation system

Developers

+ other linked practices: Test Driven Development, …

Instant feedback and easy access to the latest environment and results

What? Why?

CI shift

Error cost curve:

CI

Continuous Integration tools

Tool selections: SCM• git is my option

• git is the most powerful SCM tool, but requires time to learn even most important tricks of it

• Use that SCM tool that developers have mostly got used it or if you some other specific requirements (like that your customers requires so)

• git submodules solution sucks – If you need hierarchy of repositories then use for example Google’s repo tool

• What to consider? Distributed development – central SCM vs. distributed one? A lot of open source alternatives vs. commercial ones

commit e83c5163316f89bfbde7d9ab23ca2e25604af290Author: Linus Torvalds <[email protected]>Date:   Thu Apr 7 15:13:13 2005 -0700

    Initial revision of "git", the information manager from hell

Tool selections: repository hosting• Host yourself or outside?

Security

Performance

Price (also internal workforce costs)

• Gerrit Open source alternative with code review solution

Qt and Google are using too, but have intenal forked versions

Access management on quite detailed level and requires some learning

git is the only supported version control tool

https://android-review.googlesource.com/ , https://codereview.qt-project.org/

• Github SAAS or Github Enterprise

Git only

• Gitorious

• Bitbucket by Atlassian

• Deveo Finnish version supporting git, Mercurial, Subversion

Continuous Integration system• If your project is not having an automated build&test system, then make sure you get it!

• Quite many open source alternatives: Buildbot Cruisecontrol Jenkins And commercial solutions too, like ElectricCommanders, Team

Foundation Server, Bamboo by Atlassian

• Unless you totally hate graphical interfaces or Java or open source software, then Jenkins is your choice.

• Jenkins has probably the biggest Continuous Integration solution user space and very wide support for multiple tools via plugins

• At least 61000 installations globally and 730+ plugins (information from May 2013, Cloudbees)

• Java based, you can develop own plugins if needed

• Master – slave architecture You can have how many (slave) build machines that you like.

Operations are run normally via ssh. Commercial additions like Jenkins Enterprise by Cloudbees

http://pages.cloudbees.com/rs/cloudbees/images/Jenkins-Enterprise-by-CloudBees.pdf

Software change flows

Code reviews! Code reviews! Code reviews!

• If you are running a project in which there is no code review practice, then think it again!

• Code review is one of the most valuable part of steps in release chain. Fast feedback One of the best ways to prevent quality to drop

• By code review (tool) you can: Get feedback from your colleagues Make better visible what changes are coming and prepare testing Run what ever prechecks that you need Visualize your changes

• What can go wrong? You code review tool is not part of chain – requires that developer need to do many additional steps

to get a change under review.

• Tool & practice recommendations http://en.wikipedia.org/wiki/Gerrit_%28software%29 http://en.wikipedia.org/wiki/Crucible_%28software%29 http://en.wikipedia.org/wiki/List_of_tools_for_code_review

Qt contribution model

Picture from: http://qt-project.org/wiki/Qt-Contribution-Guidelines

1)

2)

3)

4)

5)

6)

Another release modelReviewer(approval,

optional verification)

Developer

Newcommit Gerrit

Pre-build &

testCommit

Verification

SubmitCI build & test

Environment file for CI

built environment

Ok?

Environment file for verified

environmentOk?

Release verification(manual testing)

Ok?

Developers

Commit gets alreadyavailable for other developers

CI version for developers

Stable version for developers

Longer build & test

Testing

Development environmentlevels

Bleeding edge Stable

Build phase1. Fetch source code

2. Run checks that you can run already at this phase and which are not taking too much time Like lint, sonar, valgring etc. Commercial options: Coverity etc.

3. Compile it Not only for your main target, but to all of your product’s target environments:

Linux? Windows? iOS? Mobile Platforms?

4. Provide feedback to the review system and directly to the developer if build fails

• Package your code & binaries if those are reusable in another continuation steps

• Run additional analyzer that are taking long time separately, like performance tools

Test phase• Do you have already automated tests in place?

• Do you have already automated tests in place?

• Do you have already automated tests in place?

• There is a huge amount of test frameworks and test runners available for various use cases. Choose what fits best for your technologies and processes.

• Normally test automation is not enough, but you need also some manual testing

Places for optimization

CI

CI

Too complex and slow models

Applications

Middleware

OS

Adaptation

Chipset

CI for platform

CI for adaptation

(CI for Chipset SW)

CI

CI

CI

CI for package /package wrapper

CI

Package

CI for package /package wrapper

CI

CI for package /package wrapper

CI

CI for package /package wrapper

CI

Deliveries via packages

Package

• Gating is done for package or package wrapper depending on teams• Multiple concurrent configurations shall be supported in all CI systems• Asset Creation environment is set up as is reasonable for the program

Common CI environments

CI Build & test setup

PackageCI

PackageCI

PackageCI

PackageCI

PackageCI

PackageCI

PackageCI

PackageCI

PackageCI

PackageCI

PackageCI

PackageCI

Quality Gating level (package)

CI

Ass

et

Cre

ati

on C

I se

tups

CI

CI for sub-release verification: Package / Package wrapper

CI for pre-verification: Asset Creation / Team CI

CI for release verification

Developers

Software changes

Team

Com

mit G

ate

CI

Team CI

What problems do

you find?

Daily/weekly releases

Daily/weekly releases

Process optimizations• How long it takes from software change to get the change into a

product? Worse example: 1-2 weeks Best example: less than 15 minutes Optimizations:

Build time Review time Too many steps in the process?

• Quality Gating Too heavy? Not at all?

• Test automation Do you really have it in place?

Requirements / Errors / User Stories / Tasks / Defects / Epics / handling ’needs’• We use Atlassian Jira

Together with Agile plugin

• Various available solutions I have experince on Bugzilla, Rally, ScrumWorks, Lotus Notes, ...

• ”We have to use Scrum to be Agile?” – really? Are you solving your problems by forcing team(s) to use specific model? Agile/Scrum came with quite big hype There is no single right way to deploy agile processes You must be agile also by taking processes in use in an agile way.

Not only forcing the same way as those are defined in books. When Scrum is good? How about Kanban?

ScrumContinous Integration

DevOps

Startup way of working in big companies

Waterfall

?LeanAgile

Kanban

Other communication solutions for developers• Messaging:

Do you have a solution for developers to use for communication? Hipchat, IRC+Quassel (one to many and one to one) Lync, Skype (one to one + meetings) Important especially if working on multiple locations

• Information management Do developers have a solution to share instructions etc. TWiki, Concluence, Sharepoint, (Lotus Notes) etc.

Developer environment• Can any developer repeat your product build?

• If not, then make it happen.

• It is better to package all tool that you need for compile Put those into git reposity or package those otherwise Support by installation scripts

• Do developers have a working IDE package and don’t need to spend hours in installing & updating applications?

Technical Optimizations• Speed?

Put more money to build machines Start from changing from HDD to SDD -> normally huge improvement What steps can be run in parallel? Or reduce some step? Some technical optimizations: ccache, icecc, ElectricCloud, btrfs, network

optimizations

• Money? Are you investment on okay level? Consider using more open source tools

• Maintenance Virtualization Automatic ’installation’: puppet, netboot

• Do you have all you tools integrated to your CI system? Also requirement and error management systems

Bureaucrazy or enhancement?• Are your software development processes and tools more supporting

management?

• Or

• Are your software development processes and tools supporting software development?

• Yes, visibility is important and tools for that, but not those tools that are bringing still more steps for developer to get changes in.

Do you have the culture in place?• Continuous Integration developer practices:

1. Commit changes often

2. Follow the status (build & test results)

3. React when needed (debug, fix, merge etc.)

• Do you have code review practice in use?

• Do you have practices for test automation?

• Do you allow others to touch ’any’ components?