se4fx

1

SoftwareEngineering for

Firefox

Nicholas Nethercote

2

Firefox is a web browser

•Hopefully you all know that Firefox is a web browser.

3

Developed by Mozilla and volunteers

•Firefox is an open-source project that’s developed by Mozilla and alarge number of volunteers.•Mozilla is a non-profit organisation with about 350 employees. Itsheadquarters is in California, it has smaller offices in Toronto,Vancouver, Paris, Tokyo, Beijing and Auckland. And a decent fractionof its employees don’t live in any of those cities, and so work fromhome. I’m one of them.

4

Over 400 million users

•Firefox has over 400 million users. It’s the second-most widely usedweb browser after Internet Explorer, and depending on who you ask, ithas somewhere between 20 and 30% of the global browser marketshare.

5

Firefox is big and complicated

•But the thing about Firefox that is most relevant to today’s talk is that itis a very big and complicated piece of software.

6

Millionsof linesof code

Totals grouped by language (dominant language first):cpp: 1949586 (57.28%)ansic: 1133826 (33.31%)asm: 93172 (2.74%)xml: 58409 (1.72%)sh: 56585 (1.66%)python: 51073 (1.50%)perl: 24498 (0.72%)java: 22640 (0.67%)objc: 4947 (0.15%)cs: 4185 (0.12%)yacc: 1763 (0.05%)pascal: 1012 (0.03%)lex: 861 (0.03%)exp: 449 (0.01%)php: 244 (0.01%)awk: 211 (0.01%)sed: 51 (0.00%)ada: 49 (0.00%)csh: 10 (0.00%)

Total Physical Source Lines of Code (SLOC) = 3,403,571Development Effort Estimate, Person-Years (Person-Months) = 1,022.26 (12,267.13) (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))Schedule Estimate, Years (Months) = 7.46 (89.47) (Basic COCOMO model, Months = 2.5 * (person-months**0.38))Estimated Average Number of Developers (Effort/Schedule) = 137.11Total Estimated Cost to Develop = $ 138,093,482 (average salary = $56,286/year, overhead = 2.40).

•To give you an idea, the code repository contains almost 3.5 millionlines of code, which takes up over 350 MB of disk space.•There’s a program called SLOCCount that you can use to analyzecodebases, and it estimates that this represents over 1,000 personyears of effort, and that this labour alone is worth $138 million dollars. Isuspect that’s probably an underestimate.

7

Thousands of contributors

•This work has been done by thousands of contributors over a period ofmore than a decade. This slide shows part of Firefox’s credits page,which includes the names of more than 1,700 people from all over theworld.

8

In 80 languages

•Furthermore, Firefox is one of the mostly widely-translated programs inthe world, with translations into 80 languages.•(Some of them are missing from this slide because my computer didn’thave the right fonts for all of them.)

9

Dozens of supported platforms

•And finally, it’s available on dozens of platforms. It’s always beenavailable for desktop computers, and a mobile version that works onAndroid and Maemo phones has recently become available.

10

This talk:processes &

infrastructure

•For all this to come together requires a lot of software engineeringprocesses and infrastructure, and that’s what I’ll be talking about today.

11

The life cycle of a change

1. File2. Code3. Test4. Review5. Land6. Merge

7. Release

•I’ll start by talking about all the steps that are taken for a single changeto make its way into Firefox.•The first step is to file a bug in Bugzilla.

12

https://bugzilla.mozilla.org

•Bugzilla is Mozilla’s issue tracking software. Basically it’s a databasewith a web front-end.•“Bugzilla” is something of a misleading name. “Issuezilla” or“Changezilla” would be a more accurate, but less catchy name. Almostall code changes have a Bugzilla entry associated with them. The onlyexception are trivial changes, eg. fixing a typo in a comment or fixing acompiler warning.•So you should be aware that in what follows, I’ll use the term “bug” tomean to a bug report in Bugzilla. Now a Bugzilla “bug” may be aboutan actual defect in the software, but it could be about a new feature, ora refactoring, or any other kind of change you can think of.

13

https://bugzilla.mozilla.org/

show_bug.cgi?id=637549

•Mozilla’s Bugzilla installation was started in 1998 and now has over600,000 bugs. Here are some of the important fields for one such bug.•The title is a short description of the code change implemented by thebug.•The “status” shows what stage the bug is at. For example: new,assigned, resolved.•The “product” field describes what product the bug relates to. Mozillahas several products other than Firefox, some of which share code withFirefox, and all of them are covered by the same Bugzilla installation.•The “component” field describes what part of the product the bugrelates to. There are enough product and component choices thatfinding the right one is not always easy.•The “assigned to” field shows who is working on the bug.•The “depends on” and ”blocks” fields show dependencies betweenbugs. For example, bug A might need to be finished before bug B canbe started.•There are various other, less important fields that I haven’t shownhere.•Finally, there are comments, which is where the initial description ofthe bug and all the subsequent discussion occurs.

14



7. Release

•Once the bug is filed, it’s time to actually do the coding.

15

Source CodeManagement

•In order to do that, you have to get a copy of the code.•Mozilla uses a distributed version control system called Mercurial tostore all its source code.•Mercurial is a powerful system and is quite similar to Git, which is abetter known system. Mozilla uses Mercurial instead of Git becausewhen the decision was made back in 2007, Git’s Windows supportwasn’t very good and this was a crucial requirement. Prior to Mercurial,Mozilla used a much older and less powerful version control systemcalled CVS.

16

Which repository?

•Because Firefox is such a big project, the code is actually storedacross a number of code repositories.•The most important repository is called mozila-central, and it’s theclosest thing there is to “the latest code”.•For some parts of the code, people work directly with mozilla-central.•For other parts of the code, there is a specific repository where work isdone, and that work is periodically merged into mozilla-central. Forexample, all work on the JavaScript engine -- such as the bug I showedearlier -- is done in the “tracemonkey” repository.

17

> hg clone ssh://hg.mozilla.org/tracemonkeyrequesting all changesadding changesetsadding manifestsadding file changesadded 64411 changesets with 300129 changes to 72895 filesupdating to branch default50326 files updated, 0 files merged, 0 files removed, 0files unresolved

> hg tipchangeset: 67780:4250c3243b93user: David Anderson <[email protected]>date: Thu Apr 14 13:45:06 2011 -0700summary: Fix clang warnings (bug 647280, r=luke).

Getting the code

•You get a copy of the code by cloning a repository, using theMercurial’s “clone” command.•Once you’ve cloned a repository, you have a perfect copy of therepository from Mozilla’s servers. You have the full history of everychange that has been made since the code was transferred from CVSto Mercurial back in 2007.•Mercurial’s “tip” command gives the details of the most recent committo the repository, and you can see from its output that over 60,000commits have been made to this repository since the code wastransferred from CVS to Mercurial back in 2007.•Even better, with a distributed version control system like Mercurial,each clone is a complete copy of the repository, including every one ofthose 60,000 changes, and you can move back to any of thoseversions at will.•If you want history of changes older than that, the old CVS repositoriesare still available on Mozilla’s servers.

18

make -f client.mk buildCreating directory buildCreating directory build/pgoCreating directory build/pgo/blueprintCreating directory build/pgo/js-inputCreating directory build/unixCreating directory build/win32Creating directory build/win32/crashinjectdllCreating directory configCreating directory config/mkdependCreating directory config/nsprCreating directory config/tests/src-simpleCreating directory probesCreating directory extensionsCreating directory browserCreating directory browser/appCreating directory browser/app/profile/extensionsCreating directory browser/baseCreating directory browser/componentsCreating directory browser/components/buildCreating directory browser/components/certerrorCreating directory browser/components/dirproviderCreating directory browser/components/feedsCreating directory browser/components/feeds/publicCreating directory browser/components/feeds/srcCreating directory browser/components/migrationCreating directory browser/components/migration/publicCreating directory browser/components/migration/srcCreating directory browser/components/placesCreating directory browser/components/places/srcCreating directory browser/components/preferencesCreating directory browser/components/privatebrowsingCreating directory browser/components/privatebrowsing/srcCreating directory browser/components/safebrowsingCreating directory browser/components/safebrowsing/srcCreating directory browser/components/searchCreating directory browser/components/sessionstoreCreating directory browser/components/sessionstore/srcCreating directory browser/components/sidebarCreating directory browser/components/sidebar/srcCreating directory browser/components/shellCreating directory browser/components/shell/publicCreating directory browser/components/shell/srcCreating directory browser/components/wintaskbarCreating directory browser/fuelCreating directory browser/fuel/publicCreating directory browser/fuel/srcCreating directory browser/installerCreating directory browser/installer/windowsCreating directory browser/localesCreating directory browser/themesCreating directory browser/themes/pinstripe/browserCreating directory browser/themes/pinstripe/communicatorCreating directory browser/themes/winstripe/browserCreating directory browser/themes/winstripe/communicatorCreating directory browser/branding/nightlyCreating directory browser/branding/nightly/contentCreating directory browser/branding/nightly/localesCreating directory browser/base/content/testCreating directory browser/components/certerror/testCreating directory browser/components/preferences/testsCreating directory browser/components/search/testCreating directory browser/components/sessionstore/testCreating directory browser/components/sessionstore/test/browserCreating directory browser/components/shell/testCreating directory browser/components/feeds/test

libtool: compile: gcc -DHAVE_CONFIG_H -I. -I/Users/njn/moz/ws1/js/src/ctypes/li

bffi -I. -I/Users/njn/moz/ws1/js/src/ctypes/libffi/include -Iinclude -I/Users/nj

n/moz/ws1/js/src/ctypes/libffi/src -Wall -g -fexceptions -O2 -MTsrc/x86/ffi64.l

o -MD -MP -MF src/x86/.deps/ffi64.Tpo -c/Users/njn/moz/ws1/js/src/ctypes/libffi

/src/x86/ffi64.c -fno-common -DPIC -o src/x86/ffi64.olibtool: compile: gcc -DHAVE_CONFIG_H -I. -

I/Users/njn/moz/ws1/js/src/ctypes/libffi -I. -I/Users/njn/moz/ws1/js/src/ctypes/libffi/include -Iinclude -

I/Users/njn/moz/ws1/js/src/ctypes/libffi/src -I. -

I/Users/njn/moz/ws1/js/src/ctypes/libffi/include -Iinclude -I/Users/njn/moz/ws1/js/src/ctypes/libffi/src -MT

src/x86/darwin64.lo -MD -MP -MF src/x86/.deps/darwin64.Tpo -c

/Users/njn/moz/ws1/js/src/ctypes/libffi/src/x86/darwin64.S -fno-common -DPIC -o src/x86/darwin64.olibtool: link: ar cru .libs/libffi_convenience.a src/debug.o

src/prep_cif.o src/types.o src/raw_api.o src/java_raw_api.o src/closures.o src/x86/ffi.o

src/x86/darwin.o src/x86/ffi64.o src/x86/darwin64.olibtool: link: ar cru .libs/libffi.a src/debug.o src/prep_cif.o

src/types.o src/raw_api.o src/java_raw_api.o src/closures.o src/x86/ffi.o

src/x86/darwin.o src/x86/ffi64.o src/x86/darwin64.o/usr/bin/ranlib: file: .libs/libffi_convenience.a(raw_api.o) has no

symbols/usr/bin/ranlib: file: .libs/libffi_convenience.a(java_raw_api.o) has no

symbols/usr/bin/ranlib: file: .libs/libffi_convenience.a(ffi.o) has no symbols/usr/bin/ranlib: file: .libs/libffi_convenience.a(darwin.o) has no

symbols/usr/bin/ranlib: file: .libs/libffi.a(raw_api.o) has no symbols/usr/bin/ranlib: file: .libs/libffi.a(java_raw_api.o) has no symbols/usr/bin/ranlib: file: .libs/libffi.a(ffi.o) has no symbols/usr/bin/ranlib: file: .libs/libffi.a(darwin.o) has no symbolslibtool: link: ranlib .libs/libffi.alibtool: link: ranlib .libs/libffi_convenience.aranlib: file: .libs/libffi_convenience.a(raw_api.o) has no symbolsranlib: file: .libs/libffi_convenience.a(java_raw_api.o) has no symbolsranlib: file: .libs/libffi_convenience.a(ffi.o) has no symbolsranlib: file: .libs/libffi_convenience.a(darwin.o) has no symbolsranlib: file: .libs/libffi.a(raw_api.o) has no symbolsranlib: file: .libs/libffi.a(java_raw_api.o) has no symbolsranlib: file: .libs/libffi.a(ffi.o) has no symbolsranlib: file: .libs/libffi.a(darwin.o) has no symbolslibtool: link: ( cd ".libs" && rm -f "libffi_convenience.la" && ln -s

"../libffi_convenience.la" "libffi_convenience.la" )libtool: link: ( cd ".libs" && rm -f "libffi.la" && ln -s "../libffi.la"

"libffi.la" )Section [Build] not found.jsoplengen.cppjsanalyze.cppjsapi.cppjsarena.cppjsarray.cppjsatom.cppjsbool.cppjsclone.cppjscntxt.cppjscompartment.cpp

make -f /Users/njn/moz/ws1/client.mk realbuildAdding client.mk options from /Users/njn/.mozconfig-optg64: MOZ_MAKE_FLAGS=--quiet -j2 MOZ_OBJDIR=$(TOPSRCDIR)/optg64cd /Users/njn/moz/ws1/optg64/Users/njn/moz/ws1/configureAdding configure options from /Users/njn/.mozconfig-optg64: --enable-application=browser --enable-tests --enable-optimize=-O2 -gcreating cache ./config.cachechecking host system type... x86_64-apple-darwin10.7.0checking target system type... x86_64-apple-darwin10.7.0checking build system type... x86_64-apple-darwin10.7.0checking for mawk... nochecking for gawk... gawkchecking for perl5... /opt/local/bin/perl5checking for gcc... gcc-4.2checking whether the C compiler (gcc-4.2 ) works... yeschecking whether the C compiler (gcc-4.2 ) is a cross-compiler... nochecking whether we are using GNU C... yeschecking whether gcc-4.2 accepts -g... yeschecking for c++... g++-4.2checking whether the C++ compiler (g++-4.2 ) works... yeschecking whether the C++ compiler (g++-4.2 ) is a cross-compiler... nochecking whether we are using GNU C++... yeschecking whether g++-4.2 accepts -g... yeschecking for ranlib... ranlibchecking for as... /usr/bin/aschecking for ar... archecking for ld... ldchecking for strip... stripchecking for windres... nochecking whether gcc-4.2 and cc understand -c and -o together... yeschecking how to run the C preprocessor... gcc-4.2 -Echecking how to run the C++ preprocessor... g++-4.2 -Echecking for a BSD compatible install... /usr/bin/install -cchecking whether ln -s works... yeschecking for minimum required perl version >= 5.006... 5.012003checking for full perl installation... yeschecking for python2.7... nochecking for python2.6... /opt/local/bin/python2.6checking for doxygen... :checking for autoconf... /opt/local/bin/autoconfchecking for unzip... /usr/bin/unzipchecking for zip... /usr/bin/zipchecking for makedepend... /usr/X11/bin/makedependchecking for xargs... /usr/bin/xargschecking for rpmbuild... :checking for pbbuild... nochecking for xcodebuild... /usr/bin/xcodebuildchecking for sdp... /usr/bin/sdpchecking for gmake... nochecking for make... /usr/bin/makechecking for X... nochecking whether the compiler supports -Wno-invalid-offsetof... yeschecking whether the compiler supports -Wno-variadic-macros... yeschecking whether the compiler supports -Werror=return-type... yeschecking whether ld has archive extraction flags... nochecking that static assertion macros used in autoconf tests work... yeschecking for 64-bit OS... yeschecking for Python version >= 2.5 but not 3.x... yeschecking for -dead_strip option to ld... yeschecking for ANSI C header files... yeschecking for working const... yes

•Once you have the code, you have to build it. Mozilla uses acombination of the standard Unix tools autoconf, configure and make.•Unfortunately, the build system is a big complicated mess. But I don’tknow of any large project for which this isn’t true, so there’s nothingsurprising there.•Actually doing a basic build isn’t that bad, though it can getcomplicated if you start mucking about with different configurations.The compilers used for typical builds are MSVC on Windows and GCCon Linux and Mac, though other compilers can be used with more orless effort. Building the browser from scratch takes about 25 minuteson my desktop machine which is a couple of years old.

19

Ubuntu + terminal + vim

•When it comes to actually modifying the code, different people usedifferent editors and IDEs.•I use a very simple environment, just using vim from the commandline. I mostly work on Ubuntu Linux, though I occasionally work on Macas well.

20



7. Release

•Once you’ve made a change, you need to test it.•For a program as big and complex as Firefox, thorough testing iscrucial.

21

hg push -f ssh://hg.mozilla.org/try

http://tbpl.mozilla.org/?tree=MozillaTry

• Firefox has a lot of tests, spread across a number of test suites. Tomake it easy to test on all the important platforms, there is a repositorycalled “try”, and you can trigger a full test run by pushing a change tothat repository. You can view the results through a website called the“tinderbox pushlog”.•On the left-hand side of this slide you can see the test results for asingle change, and on the right you can see a legend. Green lettersrepresent test suites that passed. Orange letters represent test suitesthat had one or more failures. Red entries indicate that the code failedto compile.•It’s fairly straightforward, except for those orange letters. With thismany tests, it’s inevitable that some tests will be prone to non-deterministic failure, and it can be diffcult to tell if it’s because the test isdefective or because Firefox is defective. In a typical run you’ll getseveral oranges, and understanding what went wrong can be difficult.Fortunately, there is a mechanism that matches these failuresmessages with existing bugs in Bugzilla. So you’ll usually get ansuggestion that a particular orange is one that’s been seen before. Youcan then mark it with a star, which indicates that it’s not your fault. Thisisn’t a perfect system, but it works pretty well most of the time.•Another thing worth noting is that full try server run results in over 100hours of machine time. Mozilla has thousands of machines just fortesting purposes, but load is still be a problem at times, so there is amechanism to say that you only want to run a subset of tests, or test ona subset of the platforms.

22

Fuzz Testing

•Another testing technique widely employed by Mozilla is fuzz testing,which involves generating randomized inputs and feeding them into aprogram to see what happens. If you get a crash or an assertionfailure, you’ve found a bug.•It sounds slightly ridiculous at first, but it actually works incredibly well,if your randomized inputs are good enough.•For example, we have a program called jsfunfuzz that cleverlygenerates randomized combinations of valid and invalid JavaScriptcode. This generated code looks completely unlike code that a humanwould write, and so it stresses the JavaScript engine in unusual ways.•The picture in this slide is hard to read, but the important thing to see isthat the huge list of blue numbers in the middle is the list of all bugs thatthis script has found in the JavaScript engine. There are 1,284 of them,and a lot of them would be memory-related bugs that are potentialsecurity holes.•Mozilla has several other fuzzers for other parts of the system, and acouple of guys run them on a couple of machines pretty much all thetime.

23

Performance

http://arewefastyet.com

•As well as testing correctness, you often want to test if a changeaffects performance. For example, if you’ve implemented anoptimization you’ll want to measure if you obtained a speed-up for therelevant operations. Alternatively, you might want to check that youhaven’t inadvertently caused a slow-down.•The tinderbox pushlog I showed earlier has some automated timingtests, and there are a few others. For example, this picture shows thegraph from a website called arewefastyet.com which the JavaScriptteam used to track the speed of the JavaScript engine on someimportant benchmarks.•This particular graph is for a benchmark called SunSpider, and youcan see how the purple line, which is Firefox, improved over a fewmonths last year to the point where Firefox was faster than both Safariand Google Chrome.•There’s a saying: “you make what you measure”, and in this case itwas very true. The JavaScript team focused very strongly on thesebenchmarks for several months and it really paid off, becauseJavaScript performance is something that gets a lot of attention thesedays.

24



7. Release

•Once you’re happy with the change, it need to undergo code review.

25

Attach the patch

•You attach the patch containing your changes to the bug, and asksomeone for a review.•No patch lands until it has passed review. For some parts of the code,you only need one review. For other parts of the code, you also needsomething called a super-review.

26

•Normal reviews cover all aspects of the code: formatting, complexity,performance, etc. Basically anything that could be improved.•Super-reviews are more limited in scope, and cover things like high-level architectural impact of any changes. There is a relatively smallnumber of people who can do super-review of each part of the browser.•Picking the right reviewer isn’t always easy. It’s one of those thingsthat you get better at with experience, as you come to understandwhich people understand which parts of the code.•The time it takes for someone to do a review varies greatly.Sometimes it might be done within a few minutes of you posting apatch. Sometimes it can take weeks, although everybody tries to avoidthat. A couple of days is typical, though it’s often faster for smallpatches and slower for large patches.•The delay can be annoying, but code reviews are really important.Most patches undergo at least some changes as a result of suggestionsmade during review, and a lot of problems are averted by a second pairof eyes.

27

r+ r-

•The usual outcomes of a review are either pass, pass with minorchanges, or fail. If it’s a fail, the reviewer should explain why, and whatchanges would be required for them to accept it, and any follow-uppatch will need to be reviewed as well.•The reviewer adds their review remarks as comments in the bug.

28



7. Release

•Once the change has passed review, it’s time to “land” it in theappropriate public repository, so it becomes visible to everyone elseusing that public repository.

29

> hg pushpushing to ssh://hg.mozilla.org/tracemonkey/searching for changesremote: adding changesetsremote: adding manifestsremote: adding file changesremote: added 2 changesets with 9 changes to 2 filesremote: Trying to insert into pushlog.remote: Please do not interrupt...remote: Inserted into the pushlog db successfully.

•You push to another repository with Mercurial’s “push”’ command.Then you need to watch the tinderbox pushlog for the next few hours.If the patch broke a build or caused some test failures, there are twooptions. If the breakage is small and easy to fix, you can push a smallfollow-up patch. If it’s more difficult than that, the patch needs to bebacked out.•While you’re doing that, there are a couple of important pieces of book-keeping.•First, you need to either mark the bug as fixed (if the patch has landedon mozilla-central) or add a marking to the field called “whiteboard”,such as “fixed-in-tracemonkey”.•Second, there is a web front-end to all the Mercurial repositories, andyou add a link to the URL containing the details of the patch.

30

Paper trail:bug to code

•By adding this link, it means you can see immediately from any bugwhich lines of code it affected.•This slide shows the link for the bug we’ve been following. The detailsare at the top, and the actual patch starts near the bottom. Lines ofcode that were removed are in red, and lines of code that were addedare in green.

31

> hg blame jsparse.cpp ...63368: JSParseNode *pn = condExpr1(); 3717: if (!pn) 3717: return NULL; 3717:63368: if (!tokenStream.isCurrentTokenType(TOK_ASSIGN)) {39559: tokenStream.ungetToken(); 3717: return pn; 3717: }

> hg log -r63368changeset: 63368:36a1aba3cd60parent: 63363:9335aa129fdcuser: Nicholas Nethercote <[email protected]>date: Wed Mar 23 18:00:04 2011 -0700summary: Bug 637549 - Speed up expression parsing, part 1: avoid ungetting/regetting many tokens. r=brendan,jimb.

Paper trail:code to bug

•This combines nicely with Mercurial’s ‘hg blame’ command, which youcan use to find out what revision number last modified every line ofcode in the repository.•When you combine this with the fact that we always put the bugnumber in a commit message, you can work out, for every line of code,which bug was responsible for last changing it.•So this gives us a bi-directional paper trail. For every bug, we candetermine which lines of code it changed, and for every line of code, wecan see which bug changed it. This is a very valuable thing to havewhen debugging, because you can see the full history of whathappened.

32



7. Release

•That ends the steps taken for a change by a normal developer. Butthere are a couple more steps that are done by others.•First, if the patch didn’t land on mozilla-central, it has to be mergedwith mozilla-central.

33

Merge tomozilla-central

•For each such repository, there is a single person in charge of doingthese merges, and they’ll merge in a big pile of patches periodically.•When they do so, they’ll update each merged bug with the URL link tomaintain the full paper trail, and mark the bug as fixed. Some of themergers have scripts that do this automatically, because there can behundreds of changes in a single merge and doing them by hand istedious.

34



7. Release

•The final step in the life cycle of a change is to release it into the handsof everyday users.•There is a lot of work and infrastructure that goes into creating andtesting releases, and making them available to hundreds of millions ofusers. This work is done by the release engineering team, and I won’ttalk about that because I don’t know much about it.•Instead, I’ll talk about when releases happen. As it happens, we’reright in the middle of a big transition in the way we do releases.

35

Old model

March 2011Firefox 4.0

January 2010Firefox 3.6

June 2009Firefox 3.5

June 2008Firefox 3.0

October 2006Firefox 2.0

November 2005Firefox 1.5

November 2004Firefox 1.0

Big, infrequent releases

•The model used up until now has been fairly traditional. Majorreleases would come out every 12 to 18 months or so. Each newrelease would have a lot of new features.•There would be several months of stabilization and beta testing beforeeach major release. For example, Firefox 4 had 12 beta releases.Each major release would be gated on a list of “blocking bugs”, whichare bugs that are severe enough that Firefox wouldn’t be released untilthey are fixed.•And once a major release was out, it would be updated with a minorrelease every few weeks or months. For example, Firefox 3.6 is up toversion 3.6.16. Each minor release would not include new features, butonly fixes of defects and security holes.•One problem with this model is that users only get new featuresoccasionally. This is a problem in a software domain as competitiveand fast-moving as the web, where a browser that’s only 6 months oldcan feel horribly out-of-date.•And on the development side, with releases so far apart, the cost offailing to get a feature into a release is very high -- if you just fall short,you have to wait another whole year. This creates an incentive to pushback a release to get more into it, which increases the chance of moreblocking bugs being introduced, which requires more stabilization, andso on, in a vicious cycle.•To give you an idea of what this means in practice, Firefox 4.0 wasoriginally scheduled to release in October 2010.

36

Newmodel

Small,frequentreleases

•In the new model, which we are literally right in the middle of changingto, the code in each major release goes through a series of stages:nightly, aurora, beta, release. Each stage takes about 6 weeks.•The code in each subsequent stage is intended to be more stable thanthe previous stage. New code only lands during the nightly stage.Changes that are found to have caused bugs, during any stage, will bebacked out or turned off.•Builds from the first three stages are made available regularly toanyone who wants to try them. The number of people using builds isexpected increase by roughly a factor of 10 as you go from stage tostage, starting with 100,000 using Nightly builds to hundreds of millionsusing normal releases. The Nightly and Aurora builds are not brandedas Firefox because they are less stable. Beta builds should be veryclose in quality to release builds.•The big advantage is that new code will make it to normal users inreleases within 3 to 4 months. Also, if a feature misses a release thecost is low, because the next cycle will start in only 6 weeks.•The possible disadvantage is that it might require more managementoverhead, but this is not clear yet because this new model so new.However, a similar release model seems to be working well for GoogleChrome; their approach heavily inspired this one and Mozilla peoplehave talked to Google people quite a lot about it.

37

Othermatters

•Now I briefly touch on some matters that I haven’t covered so far:documentation, communication and planning.

38

•Firefox has a lot of technical documentation. Almost all of that isstored on wikis, because that’s about the only form that makes sensefor such a large, distributed project. As always, wiki-baseddocumentation has its pros and cons.•The main advantage is that anyone can edit them. The maindisadvantage is that it takes effort to maintain them, and they will fallout of date if that doesn’t happen. Also, there’s so much of it and thestructure can be ad hoc, so it’s not always easy to find things. The bestway to find documentation is often by Googling for it, or by askingsomeone on IRC.•Finally, as is always the case, the source code itself is the ultimatetechnical documentation. There’s a website called mxr.mozilla.orgwhich allows you to do quite powerful searches through all the Mozillarepositories. Personally I find it’s usually overkill, and I find grep isgood enough 95% of the time.

39

IRC

Mailinglists

Blogs

•There is a lot of online communication. I’ve already shown youBugzilla, and a lot of online communcation happens through it, butthere’s a lot more.•One of the most important is IRC, which is a real-time instantmessaging protocol. There are lots of channels, and you can chatprivately with individuals as well. Pretty much everyone is logged in toIRC all the time they are working, so as long as you’re in a reasonabletimezone it’s usually not hard to find someone to answer a question.Being on IRC also has a social function, and is quite nice for peoplewho are working by themselves.•There are also many mailing lists, which are useful for more detailedand widespread conversations.•Finally, a lot of Mozilla contributors have blogs, and there’s a sitecalled planet.mozilla.org which aggregates all these blog posts.Reading planet is a good way to get a high-level view of what is goingon in the project.

40

Meetings

•There are also various kinds of face-to-face meetings. At the smallend of the scale, most Mozilla employees chat with their managersevery week or two on the phone or via Skype.•On a medium scale, there are various weekly meetings that are held inthe California office, that external employees can dial into, althoughtimezones can make this difficult. For example, Melbourne is 19 hoursahead of California during the southern summer and 17 hours ahead ofCalifornia during the southern winter, which means I can’t dial into anymeetings that are in the morning in California.•On a large scale, a few times a year there are larger meetings. Someof these involve all Mozilla employees, some of them involve justmembers of particular teams, and every two years there is somethingcalled the Mozilla Summit, which is a huge gathering in a resort inWhistler, Canada. Lots of non-employees get invited to that one andit’s a lot of fun.

41

Planning

•At this point you might be wondering how all this is co-ordinated. I’vetold you a lot about the nitty gritty details, such as the life cycle of anindividual bug. But where does the big picture come from?•Well, Mozilla does have some organisational structure. For example,there are various development teams that handle different parts of thesystem: JavaScript, content, layout, graphics, etc, and each of thesehas a manager, and there are managers above them in charge ofvarious aspects: engineering, marketing, evangelism, etc.•Each quarter, each team comes up with a high-level list of goals thatthey want to achieve, and you can see some examples on this slide.These goals are often decided upon during larger face-to-facemeetings. Also, there are certain things that are always considereddesirable to work on, such as improving performance and fixing defects.•But there’s not a great deal beyond that. Each developer has a greatdeal of autonomy within these goals, and the level of management isvery light. And non-employees obviously can work on whatever theywant, although they are encouraged to work on things that are widelythought to be important.•There’s not much in the way of formal documentation of the kind younormally associate with software engineering: specifications, designdocuments, testing plans.•It’s all very loose-knit and sometimes it’s hard to believe it actuallyworks. But it does, largely because the people involved are generallyvery good and everyone has similar aims.

42

Want to help?http://www.mozilla.org

http://www.mozilla.org/contribute/areas.html

https://developer.mozilla.org/en/Introduction

[email protected]

•I’ve covered a lot of ground today. All this can be a bit overwhelming,but that’s because I’ve dumped pretty much everything onto you atonce. The software engineering processes are really very light for aproject of this scale.•And Firefox is a great project to work on. It has lots of users so youcan make a big impact. There are lots of interesting and excitingtechnical challenges. And most people have heard of it -- I’ve had anumber of jobs in my career, but this is the first one where I can tell justabout anyone what I do in one sentence and they’ll understand it!•If you are interested in contributing to Firefox, I encourage you to startat these websites, email me, or talk to me afterwards. Thanks verymuch!

43

Credits

Browser market graph: http://gs.statcounter.com/Crowd photo: http://www.flickr.com/photos/anirudhkoul/3786725982/SLOC count: generated using David A. Wheeler's 'SLOCCount’Language chart: generated at www.wordle.netFlowchart: http://www.flickr.com/photos/thomasguest/3581215442/Wiring photo: http://www.flickr.com/photos/clonedmilkmen/4390901323/Mercurial logo: http://en.wikipedia.org/wiki/File:New_Mercurial_logo.svgCoding Quality cartoon: 2008 Focus Shift/OSNews/Thom HolwerdaFoxkeh pictures: Mozilla JapanMozilla summit photo: http://www.flickr.com/photos/gen/4784616521/

se4fx

Documents

bugzilla bug

bug report

term bug

volunteers firefox

users firefox

bugzilla entry

personyears personmonths

code changes