Top Banner
8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis localhost:9090/onepage 1/29 Automating Community Contributions Jeff McCune Twitter: @0xEFF Email: [email protected] http://puppetlabs.com
29

Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

Jan 15, 2015

Download

Technology

Puppet Labs

Attend this talk and learn how Puppet Labs handles community contributions and the FOSS tools we’ve published to automate much of the process. Puppet Labs handles thousands of contributions from hundreds of contributors and we’ve integrated Github, TravisCI and Trello to manage all of it. Come see how we do it and what we’ve built!

Jeff McCune
Software Developer, Puppet Labs
Recovering systems engineer turned software engineer, responsible for helping the Puppet community contribute their code to our open source projects.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 1/29

Automating Community ContributionsJeff McCune

Twitter: @0xEFFEmail: [email protected]

http://puppetlabs.com

Page 2: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 2/29

We'll Cover:Process

Challenging 2012 process

Improved 2013 process

Automation & Tooling

Github (patches)

Travis (tests)

Trello (work items)

Puppet Webhooks (automation)

Questions

Please ask questions during or after the talk, there will be time.

Page 3: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 3/29

Merge more patches with automationDedicated people work well

Context switching is a challenge

Automation

Reduce context switchingSingle pane of glass

Page 4: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 4/29

2012 ProcessPlatform Team

Bug Fixes

New Features

Security Incidents

Community Pull Requests

Two types of work

Planned - Bugs and Features

Unplanned - Pull Requests and Security

Pull requests often came last in the priority list

Page 5: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 5/29

2012 Process OutcomeLarge and growing backlog

No response for months

Sprints to "catch up"

Public complaints from key community members

Stressed out team

Page 6: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 6/29

2012 Results

Page 7: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 7/29

2013 ProcessCommunity Team

Community Pull Requests

Goal: 2 PM next business day response

One type of work, not two

Reduced context switching

Platform Team

Bug Fixes

New Features

Security Incidents

Reduced context switching

Page 8: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 8/29

2013 ProcessCommunity Team responsibilities:

Community Pull Requests

Platform Team responsibilities:

Bug Fixes

New Features

Security Incidents

One type of work:

Unplanned - Pull Requests and Security

Pull requests are consistently prioritized.

Page 9: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 9/29

2013 ResultsWe're merging ~ 15 community patches per week

Page 10: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 10/29

Process ConclusionDedicated People

Minimize context switching

Decreased backlog, faster response time.

Backlog down to 25 from 80

New pull requests responded to within two days instead of months.

Automating the processWith a solid process in place, we focused on automating the tedious aspects.

Page 11: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 11/29

Work in Flight

Trello

Page 12: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 12/29

Automation and ToolingGithub (patches)

Travis (tests)

Trello (work items)

Three separate tools, lots of switching back and forth.

Puppet Webhooks puts it all in once place.

Page 13: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 13/29

Why more Tooling?Too many sources of information are tedious and error prone to review.

Issue Tracker (Should I work on this now?)

Jenkins and Travis (Does this patch break stuff?)

Github (What are people saying about this patch?)

CLA (Has the contributor signed the license agreement?)

Trello (What should we be working on?)

All of this information is important. It's also spread out on four or more different sites. Itwas often overlooked.

A solution is to move all of the information into one place.

Page 14: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 14/29

All information in TrelloOur goal is to get all of the information we need to review a pull request into Trello.

New pull request creates a card.

Code comments

Contributor bio

CLA Information

select(contributor)

Completed this week

Puppet, Facter, Hiera, stdlib on one board

Page 15: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 15/29

Gepetto BotGepetto Bot is a service account with a profile on Github, Trello, and Heroku. Thepuppet-webhooks Heroku app logs into these services as Gepetto Bot.

Page 16: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 16/29

Consolidated InformationGepetto Bot gathers contributor bio using the Github API:

Page 17: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 17/29

Board ColumnsCards move left to right. Gepetto Bot creates them in the Response Needed column.

Response Needed

Accepted

Doing

Waiting on Contributor

Waiting > 1 week

Going through CI

Finished this Iteration

Page 18: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 18/29

How it worksWebhooks

Github executes a callback when events happen. The callback is a very simple HTTPPOST with a JSON body containing data about the event.

Think push notifications for the web.

Page 19: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 19/29

How it worksThe Puppet Webhooks code running in Heroku receives the HTTP POST and:

1. Github posts JSON to our Heroku app2. Persist the JSON to a Delayed Job queue3. Start a Delayed Job worker process using workless gem4. Data is posted to Trello in a matter of seconds

Page 20: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 20/29

Why use Delayed Job?Heroku provides:

Small PostgreSQL databases for free

750 hours of process time for free

31 x 24 = 744 hours for the web process

6 "extra" hours for free

Automatic retries with back-off

The workless gem allows us to only run DJ worker processes when there's actuallywork to be done.

Note: While this is all free of charge, a credit card is required to use the Heroku APIwith workless.

Page 21: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 21/29

Automatic Process ManagementThe web process runs persistently.

bundle exec thin start ­p $PORT ­e $RACK_ENV

The worker process runs only when there's work to be done.

bundle exec rake jobs:worksilent

Workless automatically adjusts the worker dyno:

Page 22: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 22/29

What is automated?New Pull Request => New Trello Card

Code Comment => Trello Comment

Collect Contributor Bio

Card due Date

Weekly Summary of completed cards

Unimplemented event handlersPull Request is closed

New code is pushed

Travis CI tests complete

Page 23: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 23/29

Weekly summary emailSent to puppet-dev mailing list weekly

Automated summary of completed Trello cards

Scans card comments and extracts summary: matches

Page 24: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 24/29

How the weekly summary worksGithub Gist as a persistent data store!

1. App reads a Liquid template from a raw github URL2. Collect all completed Trello cards3. Fill in the template with the card data4. Post the resulting Markdown back to the Gist

No persistent state in the app or the database. Liquid template is easily updated.

Page 25: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 25/29

Summary TemplateHeader:

## Puppet Dev Community Summary

 * Finished Card Total: **{{ cards.size }}**{% for section in sections %} * {{ section[0] }}: {{ section[1].size }}{% endfor %}

Sections:

{% for section in sections %}## {{ section[0] }}

Cards:

{% for card in section[1] %}#### [{{card.title}}]({{card.url}})

{{card.message }}

Page 26: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 26/29

Post back to Githubheroku run bundle exec rake jobs:summary

Page 27: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 27/29

Tooling Wrap UpAutomate the Tedious Stuff

Try the Puppet Webhooks App

github.com/puppetlabs/puppet-webhooks

Page 28: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 28/29

What we coveredProcess

Context Switches = (ಥ_ಥ)

Dedicated People = (◕‿◕)

Automation & Tooling

Github, Travis, Trello

Puppet Webhooks automates all three

Key Message

Merge more patches with automation

Page 29: Automating Community Code Contributions to Puppet with Ruby, GitHub, Heroku, Trello and Travis

8/24/13 Automating Community Contributions to Puppet with Github, Heroku, Trello and Travis

localhost:9090/onepage 29/29

Thank You! Questions?

Jeff McCune0xEFF

[email protected]