Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

Post on 15-Jan-2015

5003 Views

Category:

Technology

0 Downloads

Preview:

Click to see full reader

DESCRIPTION

Attend this talk and learn how Puppet Labs handles community contributions and the FOSS tools we’ve published to automate much of the process. Puppet Labs handles thousands of contributions from hundreds of contributors and we’ve integrated Github, TravisCI and Trello to manage all of it. Come see how we do it and what we’ve built! Jeff McCune Software Developer, Puppet Labs Recovering systems engineer turned software engineer, responsible for helping the Puppet community contribute their code to our open source projects.

Transcript

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 1/29

Automating Community ContributionsJeff McCune

Twitter: @0xEFFEmail: jeff@puppetlabs.com

http://puppetlabs.com

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 2/29

We'll Cover:Process

Challenging 2012 process

Improved 2013 process

Automation & Tooling

Github (patches)

Travis (tests)

Trello (work items)

Puppet Webhooks (automation)

Questions

Please ask questions during or after the talk, there will be time.

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 3/29

Merge more patches with automationDedicated people work well

Context switching is a challenge

Automation

Reduce context switchingSingle pane of glass

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 4/29

2012 ProcessPlatform Team

Bug Fixes

New Features

Security Incidents

Community Pull Requests

Two types of work

Planned - Bugs and Features

Unplanned - Pull Requests and Security

Pull requests often came last in the priority list

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 5/29

2012 Process OutcomeLarge and growing backlog

No response for months

Sprints to "catch up"

Public complaints from key community members

Stressed out team

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 6/29

2012 Results

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 7/29

2013 ProcessCommunity Team

Community Pull Requests

Goal: 2 PM next business day response

One type of work, not two

Reduced context switching

Platform Team

Bug Fixes

New Features

Security Incidents

Reduced context switching

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 8/29

2013 ProcessCommunity Team responsibilities:

Community Pull Requests

Platform Team responsibilities:

Bug Fixes

New Features

Security Incidents

One type of work:

Unplanned - Pull Requests and Security

Pull requests are consistently prioritized.

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 9/29

2013 ResultsWe're merging ~ 15 community patches per week

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 10/29

Process ConclusionDedicated People

Minimize context switching

Decreased backlog, faster response time.

Backlog down to 25 from 80

New pull requests responded to within two days instead of months.

Automating the processWith a solid process in place, we focused on automating the tedious aspects.

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 11/29

Work in Flight

Trello

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 12/29

Automation and ToolingGithub (patches)

Travis (tests)

Trello (work items)

Three separate tools, lots of switching back and forth.

Puppet Webhooks puts it all in once place.

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 13/29

Why more Tooling?Too many sources of information are tedious and error prone to review.

Issue Tracker (Should I work on this now?)

Jenkins and Travis (Does this patch break stuff?)

Github (What are people saying about this patch?)

CLA (Has the contributor signed the license agreement?)

Trello (What should we be working on?)

All of this information is important. It's also spread out on four or more different sites. Itwas often overlooked.

A solution is to move all of the information into one place.

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 14/29

All information in TrelloOur goal is to get all of the information we need to review a pull request into Trello.

New pull request creates a card.

Code comments

Contributor bio

CLA Information

select(contributor)

Completed this week

Puppet, Facter, Hiera, stdlib on one board

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 15/29

Gepetto BotGepetto Bot is a service account with a profile on Github, Trello, and Heroku. Thepuppet-webhooks Heroku app logs into these services as Gepetto Bot.

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 16/29

Consolidated InformationGepetto Bot gathers contributor bio using the Github API:

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 17/29

Board ColumnsCards move left to right. Gepetto Bot creates them in the Response Needed column.

Response Needed

Accepted

Doing

Waiting on Contributor

Waiting > 1 week

Going through CI

Finished this Iteration

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 18/29

How it worksWebhooks

Github executes a callback when events happen. The callback is a very simple HTTPPOST with a JSON body containing data about the event.

Think push notifications for the web.

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 19/29

How it worksThe Puppet Webhooks code running in Heroku receives the HTTP POST and:

1. Github posts JSON to our Heroku app2. Persist the JSON to a Delayed Job queue3. Start a Delayed Job worker process using workless gem4. Data is posted to Trello in a matter of seconds

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 20/29

Why use Delayed Job?Heroku provides:

Small PostgreSQL databases for free

750 hours of process time for free

31 x 24 = 744 hours for the web process

6 "extra" hours for free

Automatic retries with back-off

The workless gem allows us to only run DJ worker processes when there's actuallywork to be done.

Note: While this is all free of charge, a credit card is required to use the Heroku APIwith workless.

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 21/29

Automatic Process ManagementThe web process runs persistently.

bundle exec thin start ­p $PORT ­e $RACK_ENV

The worker process runs only when there's work to be done.

bundle exec rake jobs:worksilent

Workless automatically adjusts the worker dyno:

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 22/29

What is automated?New Pull Request => New Trello Card

Code Comment => Trello Comment

Collect Contributor Bio

Card due Date

Weekly Summary of completed cards

Unimplemented event handlersPull Request is closed

New code is pushed

Travis CI tests complete

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 23/29

Weekly summary emailSent to puppet-dev mailing list weekly

Automated summary of completed Trello cards

Scans card comments and extracts summary: matches

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 24/29

How the weekly summary worksGithub Gist as a persistent data store!

1. App reads a Liquid template from a raw github URL2. Collect all completed Trello cards3. Fill in the template with the card data4. Post the resulting Markdown back to the Gist

No persistent state in the app or the database. Liquid template is easily updated.

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 25/29

Summary TemplateHeader:

## Puppet Dev Community Summary

 * Finished Card Total: **{{ cards.size }}**{% for section in sections %} * {{ section[0] }}: {{ section[1].size }}{% endfor %}

Sections:

{% for section in sections %}## {{ section[0] }}

Cards:

{% for card in section[1] %}#### [{{card.title}}]({{card.url}})

{{card.message }}

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 26/29

Post back to Githubheroku run bundle exec rake jobs:summary

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 27/29

Tooling Wrap UpAutomate the Tedious Stuff

Try the Puppet Webhooks App

github.com/puppetlabs/puppet-webhooks

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 28/29

What we coveredProcess

Context Switches = (ಥ_ಥ)

Dedicated People = (◕‿◕)

Automation & Tooling

Github, Travis, Trello

Puppet Webhooks automates all three

Key Message

Merge more patches with automation

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 29/29

Thank You! Questions?

Jeff McCune0xEFF

jeff@puppetlabs.com

top related