RelEng Docs · L10N change-sets Finalize and ship L10N Initiate release Start release viaShip-It!application Build every-thing Automation will build installers and updaters for all

RelEng Docs

Mozilla Release Engineering

Dec 22, 2020

OVERVIEW AND PROCEDURES

1 Release Workflows with Inter-Team Hand-offs 31.1 Firefox GA Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Firefox Beta Release . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Procedures 72.1 Firefox Release Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Release Duty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.3 Testing Autograph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.4 Tree Closing Window Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.5 Tree Closing Window Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372.6 Transitioning RelEng Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.7 How to automate nightly Google Play deployments . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.8 How to set up taskgraph for mobile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472.9 Access Bouncer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482.10 Analyze Update Verify Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482.11 Manually Generate Partials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482.12 Off-Cycle Partner Repacks and Funnelcakes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512.13 Puppet pyup merges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.14 Purging the Partials Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582.15 Rotate hg.m.o cert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582.16 Sharing your terminal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592.17 Upload to internal Pypi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612.18 Widevine updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

3 Releng Architecture 653.1 Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4 Best Practices 674.1 Coding Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.2 Documentation and Cross-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

5 Taskcluster Administration 715.1 Uploading an image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.2 Troubleshooting Workers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725.3 Scopes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

6 Hosts 776.1 hg.mozilla.org . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

7 Software 797.1 Taskcluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

i

7.2 BaseLogger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807.3 BaseConfig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 837.4 BaseScript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 857.5 Mozharness Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 877.6 legacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

8 Addons 938.1 Langpack Submission Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 938.2 XPI Signing Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

9 Balrog Articles and Documentation 1079.1 Balrog and Scheduled Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1079.2 Change Update to Last Known Good . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1109.3 win32-win64 migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

10 Signing Articles and Documentation 11310.1 Manually Sign Android APKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11310.2 Create a new release key to sign a new apk product . . . . . . . . . . . . . . . . . . . . . . . . . . . 11510.3 Notarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11610.4 Adhoc signing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

11 Releng Changelog 11911.1 During 82.0 >= 2020-09-21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11911.2 During 81.0 >= 2020-08-24 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11911.3 During 80.0 >= 2020-07-27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12011.4 During 75.0 >= 2020-04-06 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12011.5 During 74.0 >= 2020-03-10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12011.6 During 73.0 >= 2020-02-10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12011.7 During 71.0 >= 2019-10-14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12011.8 During 69.0 >= 2019-05-13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12111.9 During 67.0 >= 2019-03-11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12111.10 During 66.0 >= 2019-01-30 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12111.11 During 64.0 >= 2018-10-15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12211.12 During 63.0 >= 2018-09-05 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12211.13 During 62.0 >= 2018-06-26 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12211.14 During 61.0 >= 2018-05-09 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12211.15 During 60.0 >= 2018-03-13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12311.16 During 59.0 >= 2018-01-24 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12411.17 During 58.0 >= 2017-11-15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12411.18 During 57.0 >= 2017-09-27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12511.19 During 56.0 >= 2017-08-09 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

12 Releng Future Projects 12712.1 Ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12712.2 Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

13 Machine Users 12913.1 Mercurial Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

14 Troubleshooting Expirations Notifications 13114.1 Apple common types of error messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

15 Community Participation Guidelines 13315.1 How to Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

ii

16 Adding a new repo to Releng Account on Read-The-Docs 13516.1 Pre-requisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13516.2 Enable RTFD updates on github . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13516.3 Add repo to Releng Account on RTFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13616.4 Assign maintainers to the repo on RTFD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

17 Adding Sphinx docs to Existing Repositories 13717.1 Adding Sphinx to a project with no docs yet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13717.2 Adding autodoc to a project already using Sphinx . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13717.3 Tweaking the conf.py file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

18 Maintaining this Documentation 13918.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13918.2 Documenting Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

19 Topics needing authors 141

20 Indices and tables 143

Index 145

iii

RelEng Docs

Here is Mozilla’s Release Engineering Group’s more technical documentation. You might also be interested in ourmain web site.

Contents:

OVERVIEW AND PROCEDURES 1

https://wiki.mozilla.org/Release

RelEng Docs

2 OVERVIEW AND PROCEDURES

CHAPTER

ONE

RELEASE WORKFLOWS WITH INTER-TEAM HAND-OFFS

Contents:

1.1 Firefox GA Release

See also:

Disclaimer

3

RelEng Docs

4 Chapter 1. Release Workflows with Inter-Team Hand-offs

RelEng Docs

Name DescriptionDetermineL10N change-sets

Finalize and ship L10N

Initiate release Start release via Ship-It! applicationBuild everything Automation will build installers and updaters for all locales and all platforms. (Progress emails

are sent, some of which enable QE to begin phases of testing. That level of detail is not shownin this diagram.)

functional & up-dater testing

initiated by automated email QE tests all produced artifacts, obtained via internal links.

Decide buildwill be a GO

Manual email from QE initiates Decide if this build is acceptable, or another is needed.Restart process for new build.

push installers tomirrors

Manual email from RelMgmt initiates Push Installers and updaters to Mirrors

verify updaterlinks

initiated by automated email QE verifies installers are properly accessible, and updates areserved via normal mechanisms.

Set throttle % &go for live up-daters

Manual email from QE initiates Decide when the release should become visible to end users.

configure update%

Manual email from RelMgmt initiates Apply throttling

push updaterslive

Manual email from RelMgmt and completion of update_verify initiates Deploy the updaterartifacts to the production release site. End users will be offered updates at this point.

verify throt-tling and liveupdaters

Manual email from RelEng initiates QE verifies that update throttling is at the correct level,and end users will be served accordingly.

make fully visi-ble

Manual email from QE initiates Do final clean up of the release, including making visible onthe FTP servers.

Party! Everything completed for this release.

Note: All “RelEng” steps in the “Description” column above are taken from our checklist for Firefox Releases.

1.2 Firefox Beta Release

FULLY AUTOMATED

Caution: NOT AUTHORITATIVE

The authoritative procedures are maintained by each team. Where any discrepancies are noticed, the officialprocedures (which may include release specific changes) are authoritative.

For Release Engineering, the authoritative procedures are listed as checklists in the main wiki. See also the releasewarrior documentation and diagrams.

1.2. Firefox Beta Release 5

https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Preparation#L10N_Changesetshttps://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Starting_a_Release#Submit_to_Ship_Ithttps://ship-it.mozilla.com/https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates#Push_to_mirrorshttps://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#How_to_throttlehttps://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Publish_in_Balroghttps://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Publish_in_Balroghttps://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Desktop_post-releasehttps://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Desktop_post-releasehttps://wiki.mozilla.org/Releases/RelEngChecklist#Release_2https://bugzilla.mozilla.org/show_bug.cgi?id=1603106https://wiki.mozilla.org/Releases/RelEngChecklisthttps://github.com/mozilla/releasewarrior/blob/master/how-tos/relpro.mdhttps://github.com/mozilla/releasewarrior/blob/master/how-tos/relpro.md

RelEng Docs

6 Chapter 1. Release Workflows with Inter-Team Hand-offs

CHAPTER

TWO

PROCEDURES

Procedures that don’t easily fit elsewhere.

2.1 Firefox Release Process

2.1.1 Requirements and permissions to conduct a release

In order to initiate and ship a Firefox release, follow these steps to be granted the various permissions required.

2.1.2 Specific steps required to initiate and release Firefox

Once you have the permissions to conduct a release, the following is a walkthrough of how to do it.

High level

1. Connect to Shipit.

2. Initiate a new release

3. Trigger the various phases of the release

4. Sign off in Balrog

Connect to the VPN

Connect to the VPN. Visit https://shipit.mozilla-releng.net/. Sign in using your LDAP credentials.

Initiate a new release

From Shipit, click on “New Release”. Choose the target product, e.g. Firefox Desktop. Choose the target channel, e.g.Beta, and select the desired revision you would like to ship from.

Revision - you must use a valid revision from the chosen channel. e.g. if you selected Beta, you must choose a revisionfrom mozilla-beta.

Release ETA - If you are asked to give a “Release ETA”, you do not need to fill this out.

Partials - partials should be auto populated. The list of partials are previous released versions that users are usingcurrently that we want to provide a special update to that is small in download size and is a diff of the user’s current

7

https://hg.mozilla.org/releases/mozilla-beta

RelEng Docs

version and the new version of Firefox you are about to create and ship. The auto-populated versions are usually basedon which versions are currently the most used by users.

A note on release promotion: Releases do not create new builds of Firefox. Instead, the automation will take existingbuilds from the revision that was created and built when that revision was pushed (checked in). We refer to this as“Release Promotion”. For that reason, the revision must have builds started or complete and associated with the targetrevision. To see the builds of a given pushed revision, use Treeherder.

Version and build number - both of these are auto populated and can not be modified. The build number refers to howmany times we have had to create and recreate a release prior to shipping it to users. This often happens when therelease automation fails or or Firefox fails to pass QA.

Once the form is filled out, click Start tracking it. Note this does not actually kick off any release automation. Itmerely primes the release in Shipit so that you can start triggering the various phases.

For more on how to create a release in Shipit, including UI screenshots, see Relman’s documentation

Trigger the phases of the release

After creating a release in Shipit, you can see it via the “Releases” tab. From that tab, you can initiate a phase. Formost products, there are three phases: “promote”, “push”, and “ship”.

Promote - as mentioned above, Shipit and release automation do not actually create any new Firefox builds. It insteadtakes existing per-checkin builds from CI and signs, repacks, and publishes those to users. The promote phase consistsof release automation that does just that. Common tasks in this phase are signing, creating localization specific builds,uploading build artifacts to a staging location on http://archive.mozilla.org/ (S3), staging the release on our updateserver, Balrog, and running install and update testing.

Push - This phase takes all the build artifacts that were promoted and uploaded to the staging location on archive.m.oand pushes (copies) them to a final archive.m.o directory. It will also do some final update testing.

Ship - This final phase will make the release live and available on mozilla.org and start serving updates to existingusers via our update server Balrog.. See Types of releases below for more.

In most cases, you would trigger each phase individually, one at a time. Note: by design, it is possible to only click alater phase and Shipit is smart enough to backfill tasks from uninitiated earlier phases.

Once you trigger a phase, you can re-click on that phase to monitor the progress of the automation. This monitoringis through the Taskcluster’s Task Monitor UI. You have to keep an eye on these tasks. If any of them fail, escalate it tothe Release Engineering team via the proper communications

For more on how to trigger phases of a release in Shipit, including UI screenshots, see Relman’s documentation

Signing off in Balrog

If the channel you are releasing is Beta or a mobile product, no further action is required. If you are shipping a ESR orRelease release, you also need to sign off in Balrog itself via the Balrog Admin UI

To guard against bad actors and compromised credentials we require that any changes to primary release channels(beta, release, ESR) in Balrog are signed off by at least two people.

After the scheduled change has been created by the “updates” task in the “ship” phase that was triggered in Shipit, andprior to the desired release publish time.

• In context of the other rules, eg

– Firefox release: https://balrog.services.mozilla.com/rules?product=Firefox&channel=release&onlyScheduledChanges=1

8 Chapter 2. Procedures

https://treeherder.mozilla.orghttps://wiki.mozilla.org/Release_Management/Release_Process_Checklist_Documentation#Release_Taskshttps://firefox-ci-tc.services.mozilla.com/tasks/groupshttps://wiki.mozilla.org/Release_Management/Release_Process_Checklist_Documentation#Release_Taskshttps://balrog.services.mozilla.comhttps://balrog.services.mozilla.com/rules?product=Firefox&channel=release&onlyScheduledChanges=1https://balrog.services.mozilla.com/rules?product=Firefox&channel=release&onlyScheduledChanges=1

RelEng Docs

– Firefox ESR: https://balrog.services.mozilla.com/rules?product=Firefox&channel=esr&onlyScheduledChanges=1

Further details and examples can be found on the [[Balrog page|Balrog and Scheduled Changes]]

For more on how to sign off of a Balrog channel/rule change, including UI screenshots, see Relman’s documentation

How these taskcluster tasks are scheduled

Shipit uses Taskcluster actions and the “taskgraph” tool to schedule what tasks are needed in each phase. Taskclusteruses special release workers that are under Release Engineering control and locked down to do the actual work. SeeSource and under the hood below for more details.

2.1.3 Other Release Management Tasks

Release MGMT have a number of tasks and coordination items they need to address with each release. See an exampleof the 76 release checklist

2.1.4 Escalating issues and communications

For email, Slack, and Matrix communications with various release stakeholders, see the communications section.

For troubleshooting a release automation issue, contact Release Engineering via above.

For any coordination or product specific issue, contact Release Management via above.

2.1.5 Source and under the hood

Taskcluster

Firefox is released via the same tooling that’s used to build and test Firefox. We use our Mozilla in-house continuousintegration (CI) platform Taskcluster to drive the tasks and workers. The main service in this platform is the TaskclusterQueue. The queue takes requests of tasks and coordinates with a pool of workers to actually conduct the task work.The various scheduling and dependency logic is defined in taskgraph. The workers are trusted, locked down, andowned by Release Engineering. They are scriptworker based and the various implementations live here

Signing

We use signing scriptworkers that interface with Mozilla’s autograph service

Providing Updates

We use balrog scriptworkers that interface with Mozilla’s updater service, Balrog

2.1. Firefox Release Process 9

https://balrog.services.mozilla.com/rules?product=Firefox&channel=esr&onlyScheduledChanges=1https://balrog.services.mozilla.com/rules?product=Firefox&channel=esr&onlyScheduledChanges=1https://wiki.mozilla.org/Release_Management/Release_Process_Checklist_Documentation#Release_Taskshttps://docs.google.com/spreadsheets/d/1wRSzUShqVrg0O_5fa2Mr611R-DjhpcFk6swJvScj_mQ/edit#gid=221890705https://docs.taskcluster.net/docshttps://firefox-source-docs.mozilla.org/taskcluster/taskgraph.htmlhttps://github.com/mozilla-releng/scriptworkerhttps://github.com/mozilla-releng/scriptworker-scriptshttps://github.com/mozilla-services/autographhttps://mozilla-balrog.readthedocs.io/en/latest/

RelEng Docs

Shipit

Shipit is used to initiate, track, and sign off on Firefox releases for each of the various stages. Shipit is a web app.

2.2 Release Duty

Contents:

2.2.1 Updating one ESR version to another

Overview

When an ESR version is end-of-life’d we need to update still supported users to the next ESR version. This usuallyhappens with the x.3.0 release of the new ESR version, eg: 78.3.0esr. The general steps to this process are the sameevery time but the specifics change – so it’s important to check your work with other members of the team, as well asRelMan, and for QA to test.

Figure out what rules are needed

There’s two things you need to know before you can determine what rule changes will be needed:

1. Do we need to watershed at the final version of the older ESR?

2. Are there any platforms that the older ESR supports, that the newer one does not?

To answer the first question, look at the Firefox release channel rules on https://balrog.services.mozilla.com/rules?product=Firefox&channel=release. More specifically, you’re looking for any rules with a version specifier betweenthe older ESR version and the newer ESR version, and that are marked as a watershed (usually by the word “watershed”in the comment field).

To know whether ane platforms are being deprecated, talk to RelMan.

Adjust rules on the esr-localtest and esr-cdntest channels

This may be as little as deleting one existing rule, or as much as creating multiple new rules, depending on what youdiscovered in the previous section. This section will show examples of everything that may need to happen.

Add platform deprecation rules

If any platform are being deprecated, you need to create two new rules (per channel). The first will allow the deprecatedusers to update to their last support version, the second will block them from updating past that version. For example,


https://github.com/mozilla-releng/shipithttps://balrog.services.mozilla.com/rules?product=Firefox&channel=releasehttps://balrog.services.mozilla.com/rules?product=Firefox&channel=release

RelEng Docs

here is what we did for XP/Vista deprecation:

Add a watershed rule

If you need a watershed, you can re-use the existing rule for the older ESR version. Usually these are aliased assomething like esr68-localtest. You will need to add a version filter that catches any update requests lessthan the major version of the older ESR. For example, the esr68 watershed rule set version to

RelEng Docs

Adjust esr channel rules

After we ship the newer ESR version, we can carry the test channel rules that were made over to the main esr channel.The simplest way to do this is to take the esr-cdntest rules you created or changed, and update them to have achannel of esr* - which will cause them to apply to any channel starting with esr. These will be subject to signoff.Once they’ve been made live you should also delete the esr-localtest rules, as they have been superceded bythe new esr* rules.

The Snap case

In addition to Balrog, you need to update the Snap store too. For more information about Snap, see ubuntu-snap.Steps:

1. docker pull snapcore/snapcraft:stable

2. docker run -ti snapcore/snapcraft:stable

3. snapcraft login. If you don’t have credentials, use the team account.

4. snapcraft status firefox You should see this kind of output:

Track Arch Channel Version Revisionesr amd64 stable 78.3.0esr-1 426

candidate 78.2.0esr-1 413beta ^ ^edge ^ ^

latest amd64 stable 80.0.1-1 418candidate 81.0-2 425beta 82.0b1-1 427edge ^ ^

5. snapcraft close firefox esr/candidate and you will get:


candidate ^ ^beta ^ ^edge ^ ^

The esr/candidate channel is now closed.

Bouncer and EOL’ing the old ESR branch

This includes a couple of things, before and after building/shipping X.3.0esr.

Before GTB of the X.3.0esr release

1. The new ESR branch needs to become the new default in a couple of places. (e.g. UV configs, Snap configsand bouncer aliases). For example, when we switched over to ESR78.3.0, we had to land this on esr78 branch,in advance, to make sure the 78.3.0esr becomes both the current-esr and next-esr release returned bybedrock, we update the right Snap, etc.

2. The old ESR branch can be wiped off from central/beta/release/esr(NEW_ESR). During the 78.3.0 for example,we landed a patch similar to this to remove all occurrences of esr68 and adjust configuration.


ubuntu-snap.htmlhttps://phabricator.services.mozilla.com/D88591https://phabricator.services.mozilla.com/D88618

RelEng Docs

After X.3.0esr is shipped

1. Ship-it holds information that gets propagated to product-details which is consumed dynamically by a bunch ofplaces such as bedrock. So whenever we move to the new ESR, while EOL-ing the old one, we also need toupdate the configurations in Ship-it. When 78.3.0esr was released, we amended Ship-it here.

2. Cron jobs in the new ESR branch need to have their configs adjusted so that they query against the correct valuein bouncer. Until we branch ESR again, both current/next esr releases will point to a singular release so it needsto be properly updated to reflect the one one. For example once we shipped esr78, we pushed this on esr78.

3. EOL-ing the old ESR branch. Once X.3.0 is shipped, we can stop generating CI for the old branch. Thatincludes removing all the cron jobs, including but not limited to: periodic updates, nightlies, bouncer checks,searchfox, etc. For retiring and EOL-ing esr68 we’ve landed this.

4. Make sure to close the old ESR branch in TreeStatus as planned closure.

2.2.2 Desktop Flatpak Releases

Flatpak is a package format that targets to support every Linux distribution. We’ve made Firefox publicly available onhttps://flathub.org/apps/details/org.mozilla.firefox since Firefox 74.0.

Channels used

As of September, 2020 we build and publish two flatpaks: - mozilla-beta flatpak that is published to the Flathub betachannel. This does not have an UI and is used by ~1K users. It’s mainly for testing purposes - mozilla-release flatpakthat is published to the Flathub stable channel. This is the main flatpak that users get.

The flatpak store (Flathub) comes with the concept of tracks (à la Google Play Store). Release promotion automaticallyuploads to these tracks:

Brand name Track NotesFirefox stable Automatically shipped from mozilla-releaseFirefox beta beta Automatically shipped from mozilla-betaFirefox Developer Edition N/A Not supported yetFirefox Nightly N/A Not supported yetFirefox ESR N/A Not supported yet

Changes to Flatpak via web interface

As of September 2020, the Flatpak doesn’t have a formal UI to make changes to the flatpaks via authorized credentials.Most of the applications shipped to Flathub are done so as part of the Flathub’s CI on Github. Firefox is among thevery few applications that are being shipped from our automation via Flathub’s API.

For any urgent changes, Flathub administrators have leverage and can be contacted to help. Details on how to contactthem ca be found in the private repo within the flathub-store.txt file.

2.2. Release Duty 13

https://product-details.mozilla.org/1.0/firefox_versions.jsonhttps://product-details.mozilla.org/1.0/firefox_versions.jsonhttps://github.com/mozilla-releng/shipit/pull/458https://phabricator.services.mozilla.com/D88619https://phabricator.services.mozilla.com/D90994https://treestatus.mozilla-releng.net/static/ui/treestatus/https://flathub.org/apps/details/org.mozilla.firefox

RelEng Docs

## Refresh Flathub credentials

In order to publish to both channels, we use the flatpak scriptworker. The procedure to push is the same for bothchannels, only the token value depends. When these tokens expiry, they need to be refreshed. Specific instructionson how to do that lie within the flathub-store.txt in the private repo.

2.2.3 Desktop Releases

For an overview of how to do a release end-to-end, see How To Do A Release

Managing Different Types of Releases

Betas and Devedition

Aside from the first beta and devedition of a new cycle (e.g. 77.0beta1), Beta and Devedition releases are doneautomatically. No human or manual intervention is required. The releases are created via Taskcluster Cron, 3 timesa week (Mon, Wed, and Fri), at 03:00 UTC. These releases are still driven through Shipit but automation creates andtriggers the ship phase via the Shipit API.

As releaseduty, you are required to ensure that each beta and devedition release ships without bustage. In other words,ensure that the ship phase in Shipit has 100% green tasks in the associated Taskcluster graph. You are not expectedto actively monitor the graph, particularly if 03:00-06:00 UTC is not within our range of your normal working hours.That said, you are expected to confirm the release is unblocked and all green as you start or finish your day.

Sheriff support: Sheriffs actively monitor releases so they will often discover, rerun intermittents, and escalate to#releaseduty on Matrix if a release is stuck.

Release Candidate(RC) week and Beta 1: beta1 releases are initiated manually by Release Management. Automaticbetas can be disabled via Shipit. See Shipit documentation (TODO add link and docs) for disabling and re-enablingautomatic betas via the Shipit UI. When we create a Release Candidate (see below), we close mozilla-beta via Treesta-tus which stops developers from landing patches on the Beta based repository. We keep it closed for one week andRelman disable automatic betas from triggering. After one week, Release Management will re-open mozilla-beta,trigger the new cycle’s first Beta release (beta1) manually via Shipit, and then re-enable automatic betas so that Beta(and Devedition) based releases will start shipping automatically again.

Balrog signoffs: release updates for Betas and Devedition do not require human signoff in Balrog.

Release Candidates

Over the course of 1 week (the last week of each cycle), Release Management create a new Release Candidate (RC)release once a cycle. e.g. 76.0. This release is created manually in Shipit via Release Management. It is comprisedof 4 phases: promote, ship on beta, push, and ship on release. Each phase is triggered by Release Management whenthey are ready. Even though this release is intended for the “release” channel, we ship it to “beta” channel users to testit with the beta population. Often we will have many RC releases in a week and ship each one to the beta population.The reason this does not cause update conflicts to beta users is because we freeze actual beta releases during RC week.See above Beta section. So beta users prior to receiving say 76.0, will be on

RelEng Docs

Once we ship to “release” based users, releaseduty is required to signoff in Balrog. Balrog requires one person fromrelman and one person from releng to signoff on any changes to updates. See documentation on how to signoff inBalrog here (TODO add URL or docs)

ESR and Dot Releases

ESR and dot releases releases are similarly all initiated and triggered by Release Management. Dot releases arereleases on the release channel that are not full new major version based changes. e.g. 76.0.1 vs 76.0. In other words,they are not RC based releases. releaseduty is expected to monitor the graphs and ensure all tasks are green. Whenshipped, releaseduty is required to signoff in Balrog.

2.2.4 Run staging releases

Permissions

To do a staging release, you need to be able to push to try, access to the Mozilla VPN, and have staging shipit read/writeaccess.

See the production documentation for how to get Shipit and VPN access.

How-to

In order to prepare a smooth b1 and RC, staging releases should be run weekly or at least one week before RC week.In order for this to happen, we’re using staging releases submitted to try.

For central to beta migration

• hop on central repository

• make sure you’re up to date with the tip of the repo

• mach try release --version --migrationcentral-to-beta --tasks release-sim

For beta to release migration

• hop on beta repository

• make sure you’re up to date with the tip of the repo

• mach try release --version --migration beta-to-release--tasks release-sim

These will create try pushes that look-alike the repos once they are merged. Once the decision tasks of the newlycreated CI graphs are green, staging releases can be created off of them via the shipit-staging instance. For how tocreate a release via Shipit, refer to the production documentation. The same applies to staging, just ensure you areusing the staging instance (https://shipit.staging.mozilla-releng.net).

One caveat here is the list of partials that needs to be filled-in. :warning: The partials need to exist in S3 and be validreleases in Balrog staging.

Once the staging releases are being triggered, it’s highly recommended that at least a comment is being dropped toSheriffs team (e.g. Aryx) to let them know these are happening in order to: * avoid stepping on each others toes asthey may run staging releases as well * make sure we’re up-to-date to recent patches that they may be aware of

warning Allow yourself enough time to wait for these staging releases to be completed. Since they arerunning in try, they have the lowest priority even on the staging workers so it usually takes longerfor them to complete.


https://firefox-source-docs.mozilla.org/tools/try/selectors/release.htmlhttps://shipit.staging.mozilla-releng.net/https://shipit.staging.mozilla-releng.nethttp://ftp.stage.mozaws.net/pub/firefox/releases/https://balrog-admin-static-stage.stage.mozaws.net/

RelEng Docs

Staging scriptworkers

Reusing builds from a recent release

Outside of mergeduty, during development cycles, we often need to work around a single specific scriptworker,whether that entails changing the in-tree code or the *script itself. While triggering staging releases is a validsolution, it is often an expensive one as it generates an entire graph. In order to be more efficient, one can use thescriptworker selector which aims to run a selection of scriptworker tasks against builds from a recent release. Thereare a number of preset groups of tasks to run. The list is configured here and it get be extended for other tasks/products.To get the list of task sets, along with the list of tasks they will run:

mach try scriptworker list

The selector defaults to using tasks from the most recent beta.To use tasks from a different release, pass--release-type :

mach try scriptworker --release-type release linux-signing

Override workertype

One can extend the aforementioned behavior by overriding the worker type to use. This is particularly useful forstaging releases against the DEV scriptworker environment. Most of the workerType configs we have in-tree areconfigured as level-{1,3} for fake/production and level-1-dev for dev.

But the latter is not present in-tree by default so it needs to be amended. More information on this can befound in the scriptworker-scripts documentation. One can either manually change the intree kind’s config tothat specific worker-type, or can simply pass an argument to aforementioned command to make the replace-ment, e.g. mach try scriptworker TASK-TYPE --release-type beta --worker-override=, where TASK-TYPE is chosen from one of the mach try scriptworker list re-turns and alias comes from the taskcluster ci config file). For example, running the beetmover jobs against the mostrecent beta release, but on the DEV worker-type:

mach try scriptworker beetmover-candidates --release-type beta --worker-override→˓beetmover=gecko-1-beetmover-dev

2.2.5 Desktop Snap Releases

Snap is a package format supported by Canonical. It’s targeted to support every Linux distribution but it’s mainlyavailable on Ubuntu at the moment. We’ve made Firefox publicly available on https://snapcraft.io/firefox since Firefox59.0.

Channels used

The snap store comes with the concept of tracks (à la Google Play Store). For more explanation about them, seehttps://docs.snapcraft.io/reference/channels. Release promotion automatically uploads to these tracks:

Brand name Track NotesFirefox candidate A human has to manually promote the Snap to the stable channelFirefox Beta betaFirefox DeveloperEdition

N/A Not supported yet

Firefox Nightly N/A Not supported yetFirefox ESR esr aka esr/

stableWe plan to use esr /candidate whenever the next major ESRversion comes out


https://firefox-source-docs.mozilla.org/tools/try/selectors/scriptworker.html?highlight=scriptworkerhttps://hg.mozilla.org/mozilla-central/file/tip/tools/tryselect/selectors/scriptworker.py#l18https://scriptworker-scripts.readthedocs.io/en/latest/scriptworkers-dev.htmlhttps://hg.mozilla.org/mozilla-central/file/tip/taskcluster/ci/config.yml#l437https://snapcraft.io/firefoxhttps://docs.snapcraft.io/reference/channels

RelEng Docs

Promote a snap to the stable channel

Who?

Like for Google Play, Release Management is in change of deciding when they want to fully ship a Snap. ReleaseManagement has access to the web interface (and the CLI because credentials are the same) and performs the releaseaction (like Google Play). If needed, Release Engineering can help.

When?

Because, there is no roll out mechanism, Snaps are shipped to the entire population of a given channel. Unlike GooglePlay, we can roll back users to previous version, if needed. However, downgrades aren’t supported internally inFirefox. Based on these facts, we should ship when we have enough data of the stability on Linux.

How?

The easy way: via web interface

1. Connect to https://dashboard.snapcraft.io/snaps/firefox/. Your credentials will be asked, your 2FA code too. Ifit doesn’t, ‘select “Always” for “Require an authentication device”, and click “Update”’ like explained on thispage.

2. On the left side, click on the release you want to ship.

3. On the “channel” section, click on the link “Release”. If brings you to a new page. If the page remains blank,reload it.

4. Check the stable box (leave the candidate one checked) and click on Release.

The more complete one: via command line

1. Install snapcraft. The simplest way is via docker: docker pull snapcore/snapcraft:stable,then docker run -ti snapcore/snapcraft:stable bash * alternatively: apt-get installsnapcraft

2. snapcraft login. Your credentials will be asked, your 2FA code too. If it doesn’t, ‘select “Always” for“Require an authentication device”, and click “Update”’ like explained on this page.

3. snapcraft status firefox outputs something like:


candidate ^ ^beta ^ ^edge ^ ^

latest amd64 stable 60.0.1-2 89candidate 60.0.2-1 97beta 61.0b14-1 101edge ^ ^

1. Note the revision of the latest/candidate (aka candidate) snap. In this example: 97

2. If you don’t see the version you are expecting, list all available revisions by running snapcraftlist-revisions firefox | head


https://dashboard.snapcraft.io/snaps/firefox/https://help.ubuntu.com/community/SSO/FAQs/2FA#How_do_I_add_a_new_authentication_device_and_start_using_2-factor_authentication.3Fhttps://help.ubuntu.com/community/SSO/FAQs/2FA#How_do_I_add_a_new_authentication_device_and_start_using_2-factor_authentication.3Fhttps://hub.docker.com/r/snapcore/snapcraft/https://help.ubuntu.com/community/SSO/FAQs/2FA#How_do_I_add_a_new_authentication_device_and_start_using_2-factor_authentication.3F

RelEng Docs

3. snapcraft release firefox $REVISION stable, $REVISION being the number found in theprevious (e.g.: 97).

How to manually push a snap to the store, in case automation failed?

1. Install snapcraft and login (see previous paragraph)

2. snapcraft push target.snap --release $CHANNEL, $CHANNEL being one of esr,candidate, beta.

3. If you forgot --release in the previous command, you can still use snap release [...] (see previousparagraph) to make the snap available to a channel.

Refresh macaroons credentials

Snaps operate via macaroons to authenticate requests against the Store. These are used by k8s pushsnap scriptwork-ers to perform operations with the snaps.

When the macaroons token expiry, they need to be refreshed. Specific instructions on how to do that lie within theubuntu-store.txt in the private repo.

2.2.6 Mobile Releases

For an overview of how to do a release end-to-end, see How To Do A Release

Managing Different Types of Releases

Fennec vs Fenix

TODO

Betas

TODO - talk about how Fennec no longer is shipped on beta

Releases

TODO

2.2.7 Merge Duty

All code changes to Firefox land in the mozilla-central repository while the Fennec counterpart land to mozilla-esr68.

• The nightly releases are built from that repo twice a day.

• DevEdition and Beta releases are built from the beta repository

• Extended Support Releases follow-up from the relevant ESR repo, such as mozilla-esr68

• Release and Release Candidates are built from mozilla-release repository

How are those repositories kept in sync? That’s MergeDuty and is part of the releaseduty responsibility.


https://dashboard.snapcraft.io/docs/api/macaroon.htmlhttps://hg.mozilla.org/mozilla-centralhttps://hg.mozilla.org/releases/mozilla-esr68https://hg.mozilla.org/releases/mozilla-beta/https://hg.mozilla.org/releases/mozilla-esr68/https://hg.mozilla.org/releases/mozilla-release/

RelEng Docs

Overview of Procedure

MergeDuty consists of multiple separate days of work. Each day you must perform several sequential tasks. Thedays are spread out over nearly three weeks, with three major days of activity:

• Do the prep work a week before the merge

– File tracking migration bug

– Turn off beta l10n bumper on RC day

– Do migration no-op trial runs

– Sanity check no blocking migration bugs

– Land whatsnewpage list of locales

• On Merge day:

– Merge beta to release

– Reply migrations are complete

• A week after Merge day, bump mozilla-central:

– Merge central to beta

– Re-open trees

– Verify the l10n bumper output

– Tag central and bump versions

– Bump mozilla-esr

– Reply to RelMan that procedure is completed

– Update wiki versions

– Bump Nightly version in ShipIt

Historical context of this procedure:

Originally, the m-c -> m-b was done a week after m-b -> m-r. Starting at Firefox 57, Release Managementwanted to ship DevEdition b1 week before the planned mozilla-beta merge day. This meant Releng had to merge bothrepos at the same time. With 71.0, we’re back to the initial workflow with merging m-b -> m-r in the first week andthen m-c -> m-b in the follow-up week.

Do the prep work a week before the merge

File tracking migration bug

File a tracking migration bug if there isn’t one (e.g. bug 1412962)


https://bugzilla.mozilla.org/show_bug.cgi?id=1412962

RelEng Docs

Turn off beta l10n bumper on RC day

Write a patch against mozilla-beta that removes this line. This patch should only land on mozilla-beta, not mozilla-central. This patch should land before RC GTB.

The rationale: we no longer have Elmo for localization signoffs. Let’s turn off the automatic bumper on first RC, andonly update l10n-changesets manually if needed. By leaving central untouched, we will reenable beta l10n-bumper oncentral-to-beta merge day.

Do migration no-op trial runs

Doing a no-op trial run of each migration has one major benefit these days: you ensure that the migrations themselveswork prior to Merge day.

General steps

1. Go to Treeherder.

2. Select the repo depending on the merge you want to perform (central, beta or the ESR one).

3. On the latest push, click on the down arrow at the top right corner.

4. Select “Custom push action. . . ”

5. Choose merge-automation

mozilla-beta->mozilla-release migration no-op trial run

1. Follow the general steps hopping on beta

2. Insert the following payload and click submit.

force-dry-run: truebehavior: beta-to-releasepush: true

mozilla-central->mozilla-beta migration no-op trial run

1. Follow the general steps hopping on central


force-dry-run: truebehavior: central-to-betapush: true


https://hg.mozilla.org/releases/mozilla-beta/file/5e8e24da4af86d6f67e6145397bcb5e27dc09d89/.cron.yml#l287https://treeherder.mozilla.org/https://treeherder.mozilla.org/#/jobs?repo=mozilla-betahttps://treeherder.mozilla.org/#/jobs?repo=mozilla-central

RelEng Docs

mozilla-esr bump no-op trial run

1. Follow the general steps hopping on esr


force-dry-run: truebehavior: bump-esrpush: true

Diff should be similar to this esr68 one or this esr78 one.

Sanity check no blocking migration bugs

Make sure the bug that tracks the migration has no blocking items.

Land whatsnewpage list of locales

TODO - this needs to change, as the process no longer assumes this, but apply them; the l10n drivers provide the finallist of locales to receive the WNP on the Tuesday prior to the ship date.

1. For each release, there should already be a bug flying around named Setup WNP for users comingfrom < X and receiving the X release. Find it for the current release. e.g. Bug 1523699. Weshould always aim to chain this bug to our main mergeduty tracking bug. That is, block the WNP bug againstthe tracking XXX migration day. If not already, please do so. This way, it’s easier to find deps andnavigate via bugs.

2. By the Friday prior to merge day, the l10n (most likely Peiying Mo [:CocoMo]) team will have postedthe final list of locales for whatsnewpage. Double-check with them again to make sure that is the final list. Thelist of locales comes in two forms: attachment in bug directly to be hg imported, but also as a comment.Make sure to double-check they match as that’s generated automatically and sometimes there could be falloutresulting in mismatches.

3. Update the in-tree whatsnewpage list of locales on central and request an uplift of that to beta. Similar to thispatch. It will uplift to release when the merge happens on Monday

1. On development machine, update browser/config/whats_new_page.yml with the list of localesfrom the bug

2. Commit the change and create Phabricator patch request as usual

3. Once the patch request is approved, land the patch via lando

4. In Bugzilla edit the phabricator attachment and add a approval-mozilla-beta? flag similar to this

5. ensure someone from sheriffs or relman uplift this to Beta before Monday’s merge and RC go-to-build


https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr78https://hg.mozilla.org/releases/mozilla-esr68/rev/bf17c381b0615fba955f8998c89593b103f32ba1https://hg.mozilla.org/releases/mozilla-esr78/rev/5024137054922f8f9565a04a2fa4c5326ee1f190https://bugzilla.mozilla.org/show_bug.cgi?id=1523699https://hg.mozilla.org/mozilla-central/file/tip/browser/config/whats_new_page.ymlhttps://hg.mozilla.org/mozilla-central/rev/55c218c9489bhttps://hg.mozilla.org/mozilla-central/rev/55c218c9489bhttps://bugzilla.mozilla.org/show_bug.cgi?id=1616636#c7

RelEng Docs

Release Merge Day - part I

When: Wait for go from relman to [email protected]. Relman might want to do the migration in two steps.Read the email to understand which migration you are suppose to do, and then wait for second email. For date, seeRelease Scheduling calendar or check with relman

Merge beta to release

1. Close mozilla-beta. Check “Remember this change to undo later”. Please enter a good message as the reasonfor the closure, such as “Mergeduty - closing beta for $VERSION RC week”.

2. Run the m-b -> m-r no-op trial run one more time, and show the diff to another person on releaseduty.

3. The diff for release should be fairly similar to this, with updated the version change.

4. Submit a new task with force-dry-run set to false:

force-dry-run: falsebehavior: beta-to-releasepush: true

warning It’s not unlikely for the push to take between 10-20 minutes to complete.

warning If an issue comes up during this phase, you may not be able to run this command (or the no-opone) correctly. You may need to publicly backout some tags/changesets to get back in a known state.

1. Upon successful run, mozilla-release should get a version bump and branding changes consisting of acommit like this and a tag like this

2. In the same time mozilla-beta should get a tag like this

3. Verify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.

warning The decision task of the resulting pushlog in the mozilla-release might fail in the firstplace with a timeout. A rerun might solve the problem which can be caused by an unlucky slowinstance.

Reply to relman migrations are complete

Reply to the migration request with the template:

This is now complete:

* mozilla-beta is merged to mozilla-release, new version is XX.Y

* beta will stay closed until next week

Release Merge Day - part II - a week after Merge day

When: Wait for go from relman to [email protected]. For date, see Release Scheduling calendar or checkwith relman


mailto:[email protected]://calendar.google.com/calendar/embed?src=bW96aWxsYS5jb21fZGJxODRhbnI5aTh0Y25taGFiYXRzdHY1Y29AZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQhttps://treestatus.mozilla-releng.net/static/ui/treestatus/show/mozilla-betahttps://hg.mozilla.org/releases/mozilla-release/rev/0eae18af659f087056bce0f62a325e5e595fff72https://hg.mozilla.org/releases/mozilla-release/rev/0eae18af659f087056bce0f62a325e5e595fff72https://hg.mozilla.org/releases/mozilla-release/rev/be8c618fd8ad921642e04e1552fbad46a044fe9ehttps://hg.mozilla.org/releases/mozilla-beta/rev/d87f9b66ddd19a973ec3ef26a9163bab9383c438https://hg.mozilla.org/releases/mozilla-release/pushloghtmlhttps://treeherder.mozilla.org/#/jobs?repo=mozilla-releasemailto:[email protected]://calendar.google.com/calendar/embed?src=bW96aWxsYS5jb21fZGJxODRhbnI5aTh0Y25taGFiYXRzdHY1Y29AZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

RelEng Docs

Merge central to beta

1. Run the m-c -> m-b no-op trial run one more time, and show the diff to another person on releaseduty.

2. The diff generated by the task should be fairly similar to this.

3. Submit a new task with force-dry-run set to false:

force-dry-run: falsebehavior: central-to-betapush: true

warning It’s not unlikely for the push to take between 10-20 minutes to complete.

1. Upon successful run, mozilla-beta should get a version bump and branding changes consisting of acommit like this and a tag like this

2. In the same time mozilla-central should get a tag like this


warning The decision task of the resulting pushlog in the mozilla-beta might fail in the first placewith a timeout. A rerun might solve the problem which can be caused by an unlucky slow instance.

Re-opening the tree(s)

Reopen the mozilla-beta tree by restoring the previous state so that l10n bumper can run.

Verify the l10n bumper output

In theory, this happened during central-to-beta merges after 2020.09.30. Verify that browser/locales/l10n-changesets.json has revisions, not default, and/or verify that the merge task has l10n-bump in thelogs. The diff should look like this

Tag central and bump versions

What happens: A new tag is needed to specify the end of the nightly cycle. Then clobber and bump versions inmozilla-central as instructions depict.

1. Follow the general steps


force-dry-run: falsepush: truebehavior: bump-central

1. Upon successful run, mozilla-central should get a version bump consisting of a commit like this and atag like this



https://hg.mozilla.org/releases/mozilla-beta/rev/13d947f127a76828e19d8bb7f8f6353a7b3a0f6ehttps://hg.mozilla.org/releases/mozilla-beta/rev/13d947f127a76828e19d8bb7f8f6353a7b3a0f6ehttps://hg.mozilla.org/releases/mozilla-beta/rev/a6981603097c54950b3a00a6e7aa95f532947482https://hg.mozilla.org/mozilla-central/rev/6d98cc745df58e544a8d71c131f060fc2c460d83https://hg.mozilla.org/releases/mozilla-beta/pushloghtmlhttps://treeherder.mozilla.org/#/jobs?repo=mozilla-betahttps://treestatus.mozilla-releng.net/static/ui/treestatus/show/mozilla-betahttps://hg.mozilla.org/releases/mozilla-beta/rev/7564379b690bb9c24cb9a7a4bbb2552c9724c147https://hg.mozilla.org/mozilla-central/rev/b00860a2a28336267070c6fd882f0f5feabcebadhttps://hg.mozilla.org/mozilla-central/rev/0ab2bba66188606446c37868f4b01cdffebd0acchttps://hg.mozilla.org/mozilla-central/pushloghtmlhttps://treeherder.mozilla.org/#/jobs?repo=mozilla-central

RelEng Docs

Bump ESR version

Note: You could have one ESR to bump, or two. If you are not sure, ask.

Run the bump-esr no-op trial run one more time, and show the diff to another person on releaseduty.

Diff should be similar to this one.

Push your changes generated by the no-op trial run:

1. Follow the general steps - (As of 2020/04 this action hasn’t yet been uplifted to release or esr68, consider usingmozilla-central’s action, as the payload controls where the effects land)


force-dry-run: falsepush: truebehavior: bump-esr

Note This is currently set to esr78, the defaults can be overridden in-tree in taskcluster/ci/config.yml orspecified here as using an action payload such as:

force-dry-run: falsepush: truebehavior: bump-esrto-branch: esr88to-repo: https://hg.mozilla.org/releases/mozilla-esr88

1. Upon successful run, mozilla-esr${VERSION} should get a commit like this.

2. Verify new changesets popped on https://hg.mozilla.org/releases/mozilla-esr78/pushloghtml

Reply to relman central bump completed

Reply to the migration request with the template:

This is now complete:

* mozilla-central is merged to mozilla-beta, new version is XX.Y

* mozilla-central has been tagged and version bumped

* mozilla-esr has been version bumped

* newly triggered nightlies will pick the version change on cron-based schedule

Update wiki versions

1. Edit the new values manually:

• NEXT_VERSION

• CENTRAL_VERSION

• BETA_VERSION

• RELEASE_VERSION

• Next release date. This updates

– The next ship date

– The next merge date


https://hg.mozilla.org/releases/mozilla-esr78/rev/5024137054922f8f9565a04a2fa4c5326ee1f190https://hg.mozilla.org/releases/mozilla-esr78/rev/5024137054922f8f9565a04a2fa4c5326ee1f190https://hg.mozilla.org/releases/mozilla-esr78/pushloghtmlhttps://wiki.mozilla.org/Template:Version/Gecko/release/nexthttps://wiki.mozilla.org/Template:Version/Gecko/central/currenthttps://wiki.mozilla.org/Template:Version/Gecko/beta/currenthttps://wiki.mozilla.org/Template:Version/Gecko/release/currenthttps://wiki.mozilla.org/index.php?title=Template:NextReleaseDatehttps://wiki.mozilla.org/index.php?title=Template:FIREFOX_SHIP_DATEhttps://wiki.mozilla.org/index.php?title=Template:FIREFOX_MERGE_DATE

RelEng Docs

– The current cycle

Bump Nightly version and release dates in ShipIt

ShipIt currently hard-codes the version of Nightly that’s being released, as well as the release dates.

It doesn’t get automatically updated because it would need to know when a new nightly was available, not just whenthe version had been updated in-tree. Everything up to merging this pull request can be done early, but the PR mustnot be merged before the first nightly has been built and published with the new version.

1. git clone [email protected]:mozilla-releng/shipit.git

2. git checkout -b nightly_version_bump_${version}

3. Edit FIREFOX_NIGHTLY’s major version in https://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L49

4. Edit the LAST and NEXT known dates (all 6 of them) at https://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L54-L59

5. Commit, and submit a pull request

6. Merge the pull request after a new nightly version has been pushed to CDNs

2.2.8 Merge Duty for major ESR bump

Intro

This manual describes how to set up a new ESR branch. The same process can be applied for any branch set up, withslight modifications.

Example tracking bug: Bug 1334535 - tracking bug for build and release of Firefox 52.0esr

Internal changes

This section covers changes to buildbot-configs and tools.

See Bug 1339832 for the details and example changes.

Some highlights:

• Explicitly list the platforms (and use lock_platforms) since we don’t ship ESR to all mozilla-central/releaseplatforms (android, for example)

• Copy mozilla-beta configs, compare them to the previous ESR release and check if you understand what theystand for


https://wiki.mozilla.org/index.php?title=Template:CURRENT_CYCLEhttps://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L49https://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L49https://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L54-L59https://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L54-L59https://bugzilla.mozilla.org/show_bug.cgi?id=1334535https://bugzilla.mozilla.org/show_bug.cgi?id=1339832

RelEng Docs

tools changes

• copy the old patcher config, so we can generate partial updates against the previous ESR release.

• since the config patch introduces new configuration file(s) which are needed to be symlinked on thebuild/scheduler masters there is a need for a fabric methods to create the links

It’s important that the buildbot-configs changes are landed and in production prior to adding the symlinks and adjustingproduction-masters.json. When you land these two changes you must immediately run the fabric method to put thesymlinks in place to ensure the masters will continue to function correctly.

External systems

CI relies on multiple systems, supported by different teams. File bugs in advance to make sure other teams haveenough time to address the issue. Usually starting the whole process 2 weeks in advance of release builds (3 weeksbefore the release), gives enough time to everybody.

Tasks

Below is the list of bugs filed as a part of the ESR52 cycle. Go through the list and verify that they are still valid forthis ESR cycle and clone them if needed.

Bug Title1333745 Please add tracking-thunderbird_esr52 and status-thunderbird_esr52 to the tracking flags1335870 please create tracking-firefox-esr52 and status-firefox-esr52 flags1337061 Add mozilla-esr52 and comm-esr52 to release_repositories1337066 Please clone releases/mozilla-beta to releases/mozilla-esr521337087 update tree closure hooks for mozilla-esr521337090 Add mozilla-esr52 and comm-esr52 to treeherder1337091 Add mozilla-esr52 and comm-esr52 to treestatus1337093 Add mozilla-esr52 and comm-esr52 to orange factor1337366 Gecko-specific changes to support mozilla-esr521337489 HTTP 500 for all trees view due to missing comm-esr52 repo1339057 Please add the approval-esr52 flag1339074 Add mozilla-esr52 to the various lists in repository.py1339076 Make sure the right hooks are running on mozilla-esr521339832 Buildbot specific changes to enable mozilla-esr521340669 Update merge day and esr docs1341330 Please add mozilla-esr52 to DXR1341491 Set ESR_NEXT in ship-it config1342117 Set watch_all_branches to True for ESR521342204 Enable TC on mozilla-esr521342431 Turn off TC-based Android jobs on ESR521343097 Add mozilla-esr52 and comm-esr52 to release-notifications heroku app1343366 browser_parsable_css.js fails on ESR52 ASAN builds due to the ESR branding changes


https://bugzilla.mozilla.org/show_bug.cgi?id=1333745https://bugzilla.mozilla.org/show_bug.cgi?id=1335870https://bugzilla.mozilla.org/show_bug.cgi?id=1337061https://bugzilla.mozilla.org/show_bug.cgi?id=1337066https://bugzilla.mozilla.org/show_bug.cgi?id=1337087https://bugzilla.mozilla.org/show_bug.cgi?id=1337090https://bugzilla.mozilla.org/show_bug.cgi?id=1337091https://bugzilla.mozilla.org/show_bug.cgi?id=1337093https://bugzilla.mozilla.org/show_bug.cgi?id=1337366https://bugzilla.mozilla.org/show_bug.cgi?id=1337489https://bugzilla.mozilla.org/show_bug.cgi?id=1339057https://bugzilla.mozilla.org/show_bug.cgi?id=1339074https://bugzilla.mozilla.org/show_bug.cgi?id=1339076https://bugzilla.mozilla.org/show_bug.cgi?id=1339832https://bugzilla.mozilla.org/show_bug.cgi?id=1340669https://bugzilla.mozilla.org/show_bug.cgi?id=1341330https://bugzilla.mozilla.org/show_bug.cgi?id=1341491https://bugzilla.mozilla.org/show_bug.cgi?id=1342117https://bugzilla.mozilla.org/show_bug.cgi?id=1342204https://bugzilla.mozilla.org/show_bug.cgi?id=1342431https://bugzilla.mozilla.org/show_bug.cgi?id=1343097https://bugzilla.mozilla.org/show_bug.cgi?id=1343366

RelEng Docs

Merge

Once mozilla-release is merged from mozilla-beta you can run the script which pulls last changes from mozilla-release,updates some configs and replaces some branding bits.

Running gecko_migration.py for mozilla-esr

The script will tag mozilla-release. It will continue by pulling mozilla-release to mozilla-esrNN, adjusting brandingand changing some configs. Once the script finishes, run an hg diff to see the uncommitted changes that the scriptgenerated:

ESR_VERSION=51# checkout mozharness from the gecko tree using archiverwget https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_→˓client.pypython archiver_client.py mozharness --destination mozharness-esr$ESR_VERSION \

--repo releases/mozilla-esr$ESR_VERSION --rev default --debug# run the scriptpython mozharness-esr$ESR_VERSION/scripts/merge_day/gecko_migration.py \

-c merge_day/release_to_esr.pyhg -R build/mozilla-esr$ESR_VERSION diffpython mozharness-esr$ESR_VERSION/scripts/merge_day/gecko_migration.py -c \

merge_day/release_to_esr.py --commit-changes --push

The push should be available at Treeherder.

In case of failure, you can start again from the top; no need to clobber (the on-by-default clean-repos action will besufficient). It should be faster the second time, since you won’t be pulling in as many changesets from hg.m.o.

Release builds

Make sure to run a staging release.

Update this documentation

Keep this documentation up to date.

Ship it!

Close the bug and have some tea. :)

Releaseduty is a role within the Release Engineering team. It is conducted on a rolling rotation matching one personagainst one release cycle (currently 4 weeks). Below you will find a description of the expectations and resourcesneeded to do the role.


RelEng Docs

2.2.9 Expectations

If you’re reading this, it means you’re ramping up as an official releaseduty squirrel within Mozilla RelEng, so pleaseallow us to give you a warm welcome!

The role mainly involves handling all the coordination and communication with other teams as well as doing all theoperational tasks to make sure the release workflow is as smooth as possible.

While this role can get quite disruptive, we prefer this approach of assigning the responsibility to a small set of peoplewho will own all the tasks, while we shield the others in Release Engineering from interruptions.

Being ReleaseDuty means a couple of things:

• Communication and coordination with other teams

• Handle all incoming releases

• Fix and debug any potential errors in the automation

• Develop and improve the Release Automation process and tools

2.2.10 Communication

Slack Channels

Join Mozilla’s Slack network using information from the wiki

You ought to be present and pay attention to conversations happening in:

• #releaseduty-mobile (where mobile and other Github releng-backedup automation projects’s teams raise issues)

• #releng-notifications (where automation notifications are going, useful to know the state of our automationinfrastructure state)

• #taskcluster-cloudops (where CloudOps double-checks with RelEng whether it’s safe to deploy changes or noton their side)

Matrix Channels

Join Mozilla’s Matrix network using information from the wiki

You ought to be present and pay attention to conversations happening in:

• #sheriffs:mozilla.org (where CIDuty team helps with various hiccups that infra might encounter))

• #releaseduty:mozilla.org (main RelEng dedicated communication channel for releaseduty)

• #firefox-ci:mozilla.org

Email

As ReleaseDuty you need to subscribe to certain mailing lists:

• All types of sign-offs and approvals should go to release signoff mailing list

• All discussion topics should go to release drivers mailing list

These other mailing lists will likely have useful discussions and information:

• RelEng internal mailing list


https://wiki.mozilla.org/NDA-Slackhttps://wiki.mozilla.org/Matrixhttps://mail.mozilla.org/listinfo/release-signoffhttps://mail.mozilla.org/listinfo/release-driversmailto:[email protected]

RelEng Docs

• Thunderbird release drivers mailing list

• release-automation-notifications-thunderbird mailing list

• (optional) release-automation-notifications-dev mailing list

Meetings and Calendars

Regular meetings are a vital part of making sure all the teams are kept informed and consulted during the releaseprocess. To view those meetings in your calendar you need to subscribe/be added to the following calendar:

• Releases Scheduling (so that you can attend the Tuesday/Thursday channel meetings. You can add it followingRelMan’s docs) – If their instructions don’t work, try to the “Add to Google Calendar” button at the web versionof the calendar.

If you join a calendar and it’s blank, you may need to delete it and get a calendar invitation from an existingsubscriber

2.2.11 Permissions

Several tools for managing releases are protected or private. In order to do your job, you need to be granted access toa bare minimum:

• Access to the VPN

• A Bugzilla account

• Read/Write access to Balrog

• Read/Write access to Ship-it v2

How to get VPN access

First you need to be connected to the Mozilla VPN. See the mana page for how to get set up and started with connectingto the VPN.

How to get Ship-it access

You need to be added to the vpn_cloudops_shipit LDAP group, as well as be added to the allowedlist in the shipitconfig. Both of these can be done by filing a ticket requesting access.

File an Release Engineering bug under the Applications: Shipit (backend) component requesting to be granted shipitaccess.

Have someone in Release Engineering to vouch for you in the bug.


https://mail.mozilla.org/listinfo/thunderbird-drivershttps://mail.mozilla.org/listinfo/release-automation-notifications-thunderbirdhttps://groups.google.com/a/mozilla.com/forum/#!forum/release-automation-notifications-devhttps://calendar.google.com/calendar/embed?src=mozilla.com_dbq84anr9i8tcnmhabatstv5co@group.calendar.google.comhttps://wiki.mozilla.org/Release_Management#Calendar_Updatinghttps://calendar.google.com/calendar/embed?src=mozilla.com_dbq84anr9i8tcnmhabatstv5co@group.calendar.google.comhttps://calendar.google.com/calendar/embed?src=mozilla.com_dbq84anr9i8tcnmhabatstv5co@group.calendar.google.comhttps://mana.mozilla.org/wiki/display/SD/VPNhttps://bugzilla.mozilla.org/https://balrog.services.mozilla.com/https://shipit.mozilla-releng.net/https://mana.mozilla.org/wiki/display/SD/VPNhttps://bugzilla.mozilla.org/enter_bug.cgi?format=__default__&cloned_bug_id=1626312&product=Release%20Engineering&component=Applications%3A%20Shipit%20%28backend%29

RelEng Docs

How to get Balrog access

Similar to Shipit, you need to be added to the balrog and vpn_balrog LDAP groups and have someone add you as auser and attach you to the releng role through the Balrog admin UI.

For VPN group access, file a ticket with IT similar to this one. Have someone in Release Engineering to vouch foryou in the bug.

Next, file a Github issue within the Balrog repo. Ask to be added to the Balrog admin user list and attached to thereleng role. Also have someone vouch for you.

2.2.12 Tooling for debugging and rerunning tasks

Note many of the task-related operations can be conducted through Treeherder such as rerunning/retriggering a task.There is also a command line tool that can be used instead of the UI.

taskcluster

Release tasks are usually run through Taskcluster, which has a useful Command-line interface

• Download an appropriate binary from here

• Copy the binary somewhere useful, such as somewhere in your $PATH

• Make it executable, if using Mac or Linux: chmod a+x /path/to/taskcluster

• Run taskcluster signin to open a browser window and allow you to get temporary client credentials. Bydefault this is valid for 24 hours. The command will display two ``export`` commands you must copy/pasteinto your shell

• Familiarize yourself with the subcommands, starting with taskcluster help

in-bulk taskcluster operations

Sometimes operations need to be performed against a bulk of tasks. In order to do that, RelEng has developed ahandful of scripts to ease the operations. They lie around in the braindump repository.

hg clone https://hg.mozilla.org/build/braindump/cd braindump/taskclustermkvirtualenv workspacepip install taskcluster

Follow this resource to download the taskcluster-cli tool or read the previous section.

Once taskcluster-cli is installed, ensure right env vars are set, login and operate the tasks.

export TASKCLUSTER_ROOT_URL='https://firefox-ci-tc.services.mozilla.com/'eval taskcluster signinpython tc-filter.py --graph-id --state failed --action rerun

To speed things up even more, one can add this to zshrc for an alias to reduce the scopes and time-limit of the signin:

tc-relduty=$'eval `TASKCLUSTER_ROOT_URL=https://firefox-ci-tc.services.mozilla.com/→˓taskcluster signin --expires 1h -s "queue:rerun-task:*\nqueue:cancel-task:*"`'


https://bugzilla.mozilla.org/enter_bug.cgi?format=__default__&cloned_bug_id=1565283&product=Infrastructure%20%26%20Operations&component=Infrastructure%3A%20LDAPhttps://github.com/mozilla-releng/balrog/issueshttps://docs.taskcluster.net/https://github.com/taskcluster/taskcluster-clihttps://github.com/taskcluster/taskcluster-cli#installationhttp://www.linfo.org/path_env_var.htmlhttps://github.com/taskcluster/taskcluster/tree/main/clients/client-shell#installation

RelEng Docs

graph inspection

Inspecting a task group from the UI can sometimes be heavy for the browser. Instead, one can use a script to watchthe graph progress locally via poll & sleep. In the same braindump aforementioned directory, there is a graph-progressscript that can be run like:

bash graph-progress.sh

Firefox bookmarks

These bookmarklets should help you view tasks and taskgroups in Firefox.

• Go to Bookmarks -> Show All Bookmarks

• Gear symbol -> New Bookmark

• Name: task inspector Location: https://tools.taskcluster.net/tasks/%s ; Keyword: task

• Name: taskgroup inspector Location: https://tools.taskcluster.net/groups/%s ; Keyword:taskgroup

• Name: stop Location: javascript:stop();

• This can be used to stop further loading in the Task Group Inspector. It shouldn’t be used whenactively monitoring (i.e.: watching for failures), but it can greatly speed things up if you’re using itfor other reasons. Be sure to wait for the initial tasks to load before you use it.

Now if you go to your URL bar, you can type task TASKID or taskgroup TASKGROUPID and you’ll go to thattask or taskgroup in the inspector.

2.2.13 After ReleaseDuty

After your tour of releaseduty, In the past it was customary to fix any documentation or automation issues discovered.Now make sure you file any bugs so issues can be included in a future sprint or work cycle.

Ensure the next duty cycle have signed up to any phabricator reviews, such as the periodic file updates reviews.

2.2.14 Miscellaneous

• Bugzilla issues regarding specific releases/WNP are filed under Release Engineering:Release Requests

• Issues regarding automation are filed under Release Engineering:Release Automation

• The releng_changelog.md in the build-relengdocs repository contains a summary of larger changes made duringthe duty cycle.


https://tools.taskcluster.net/tasks/%shttps://tools.taskcluster.net/groups/%shttps://bugzilla.mozilla.org/enter_bug.cgi?format=__default__&product=Release%20Engineering&component=Release%20Requestshttps://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Release%20Automationhttps://github.com/mozilla/build-relengdocs/blob/master/releng_changelog.mdhttps://github.com/mozilla/build-relengdocs/tree/master

RelEng Docs

2.2.15 Hand Off

If a scheduled release has not completed its graphs prior releaseduty signing off, an explicit hand-off describingdescribing release state should be sent to individual folks in releng that are scheduled to come online next or will bearound for a while after you. #releaseduty in Matrix is best. A [email protected] email would be useful too.

2.2.16 Escalation

If a release is blocked. The normal flow is to:

1. confirm issue

2. determine what service, task kind, infrastructure, or external dependency is involved

3. file a ticket

4. determine which team(s) and person(s) should be escalated.

a. Searching phonebook is useful for org and ownership charts.

b. bugzilla and github history

c. source code history

5. escalate in the appropriate Slack and Matrix channel(s). At a minimum, #releaseduy@matrix.

6. determine who is available to help based on above. What hours they work, who is their manager, etc

7. ask for help if you can’t determine the above.

Good resources within releng:

• general release configuration (taskgraph): aki

• scopes / ciadmin: mtabara

• chainoftrust (cot): aki

• scriptworker (general): aki

• beetmoverscript / bouncer / artifact related: mtabara/aki

• signing / signingscript / autograph: aki

• balrog / balrogscript / updates related: bhearsum/mtabara

• l10n / treescript / addonscript: aki/mtabara

• pushapkscript / mozapkpublisher: mtabara

• shipit / shipitscript: bhearsum

2.2.17 Other useful resources

• More on Release Management


mailto:[email protected]:#releaseduy@matrixhttps://wiki.mozilla.org/Release_Management

RelEng Docs

2.2.18 Glossary

• WNP - The “What’s New Page” can be set to appear after an upgrade, to tell end-users of any changes in thebrowser they should be aware of.

• FF - Firefox

• TB - Thunderbird

• b1, b2, etc - beta release 1, beta release 2, etc

FAQ —

1. What does release-promotion refer to?

‘Release promotion’ is simply the idea that we take an already existing CI build from (e.g. beta) and promote that tobeing the build we release/ship to users. Prior to this approach, we had always rebuilt Firefox at the start of each newrelease. Long story short, release promotion entails taking an existing set of builds that have already been triggeredand passed QA and “promoting” them to be used as a release candidate. More on promotion can be found on our docshere

2. What is the train model?

Since 2012 Mozilla moved to a fixed-schedule release model, otherwise known as the Train Model, in which wereleased Firefox every six weeks to get features and updates to users faster and move at the speed of the Web. In 2020Mozilla switched to releasing every four weeks, hence, every four weeks the following merges take place: mozilla-beta=> mozilla-release mozilla-central => mozilla-beta

We used to have an intermediate branch named ‘aurora’ in between central and beta but that was brought to end-of-lifeduring April-May 2017. Instead, early beta releases are branded as ‘DevEdition’.

3. What is a partner repack change for FF?

Partner repacks refer to 3rd party customized branded versions of Firefox that Mozilla is taking care of for some of itsclients. With some exceptions, most of the partner reconfigs lie under private repositories. Mostly, the partner repacksdon’t need too much of RelEng interference as all bits are held under private git repos and are directly handled by thepartnering companies

4. Is there calendar-based release scheduled for Thunderbird as for Firefox?

No. It’s irregular. Conversations happen on #tbdrivers and TB mailing list and they trigger their release in Ship-it.

5. Why don’t I see update_verify_beta for dot releases?

From time to time, a handful of issues precipitate a dot release. When that happens, its behavior slightly varies from anormal release. A normal release (e.g. 43.0, 44.0, etc) has its RC shipped to beta channel first before making it to therelease channel - for testing purposes, update verify steps are taking place both ways, hence update_verify_release andupdate_verify_beta steps. Upon a successful testing phase we ship the RC on the beta channel and then on the releasechannel, following which we merge the code for the next release cycle so that the beta release bumps its version. Inthe lights of this logic, a dot release (e.g. 43.0.1 or 44.0.1) happens a certain amount of time after the official release.For that reason, a dot release can’t be tested in beta channel as the at-that-moment beta version is greater than the dotrelease version, hence the updater would refuse to downgrade. Therefore, there is only one cycle of update_verify fordot releases (update_verify_release == update_verify in this case).

6. Is there explicit signoff from RelMan for DevEdition builds?

No, after b1, there isn’t signoff from RelMan on DevEdition builds. QA only verifies the DevEdition builds every twoweeks. With the exception of b1, and assuming all the tasks complete as expected, the DevEdition builds should beshipped at the same time as we receive signoff for the corresponding desktop builds.

7. How do I coordinate with marketing on release day?

Join the #release-coordination channel on Mozilla Slack


https://firefox-source-docs.mozilla.org/taskcluster/release-promotion.html#in-depth-relpro-guidehttp://hg.mozilla.org/releases/mozilla-beta/http://hg.mozilla.org/releases/mozilla-release/http://hg.mozilla.org/mozilla-central/http://hg.mozilla.org/releases/mozilla-beta/

RelEng Docs

8. What is cdntest and localtest?

-cdntest and -localtest channels serve releases from the releases directory (mirrors or CDN) or candidatesdirectory depending on the release and channel. They are testing channels used before we serve from the real updatechannel, but they use the actual files that will be served once a release is published.

9. What’s the difference between Firefox and DevEdition?

In the beta cycle, Firefox and Devedition are different products built based on the same in-tree revision. Theirfunctionality is the same but branding options differ.

10. What do the terms ``releases directory``, ``mirrors`` and ``CDN`` mean?

releases directory, mirrors and CDN are different terms for the same concept - the CDN from whichshipped releases are served.

11. What does ``watershed`` mean?

watershed refers to a situation when we release a new version of a product (Firefox 57), but users on an olderversion (Firefox 53) are not able to update. This is a watershed and we would need to ensure we don’t serve invalidupdates.

2.3 Testing Autograph

We currently use Autograph to sign our files with a number of our signing formats. Occasionally the autograph teamwill ask us to test to make sure things are working properly.

2.3.1 Testing Autograph Stage

We currently use Autograph to sign our files with a number of our signing formats. In CI, we point at autograph-prod,to avoid having autograph-stage changes or availability affect production CI, nightlies, or releases.

However, sometimes the Autograph team needs to make changes to autograph-stage, and want us to verify that it stillworks for us. Here’s how.

Autograph-stage mar signing test

There is a tier 3 task that doesn’t run automatically. This task, mar-signing-autograph-stage-linux64-nightly/opt, can be added to a push that has run nightly builds and repackage tasks.

With indexes

First, go to the latest Firefox desktop index. Click on View Task. Click on Task Group at the top left to go tothe taskgroup view.

Make sure you’re signed in, then click on the three dots in the lower right hand corner. Add new jobs. Addmar-signing-autograph-stage-linux64-shippable/opt to the list of tasks in the input so it lookslike:

tasks: [mar-signing-autograph-stage-linux64-shippable/opt]times: 1

Click Add new jobs in the lower left. Once the task finishes, click on the log. You’ll see a line like:


https://mana.mozilla.org/wiki/display/SVCOPS/Autographhttps://mana.mozilla.org/wiki/display/SVCOPS/Autographhttps://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.latest.taskgraph/decision-nightly-desktop

RelEng Docs

[task 2020-09-08T20:12:48.698Z] Creating task with taskId Z6FmfYVCRpOimAafnoziKQ for→˓mar-signing-autograph-stage-linux64-shippable/opt

In which case, check task Z6FmfYVCRpOimAafnoziKQ for status. Report back to the bug.

With treeherder

First, go to mozilla-central treeherder. Make sure you’re logged in (top right).

Find the push with the most recent finished successful nightly. (It may be easiest to search for nightly).

On this push, click the down-arrow at the top right of the push. Choose Add new jobs. The push will now showall the unscheduled jobs.

Find the autograph-stage mar-signing test job. It will be in the Linux x64 opt row, with the ms-stage(N)symbol. Click on it. It should change colors to a dark grey.

Go back up to the top right of the push. Choose Trigger New Jobs. Assuming you’ve selected the right job, andyou have the right scopes, it should now create an add-new action task that triggers the autograph-stage mar-signingtask.

You can either watch treeherder or the taskcluster task. The task will be in the on-push graph. The treeherder viewmay require you to enable tier 3 in the Tiers menu at the top right of the page.

Once the mar-signing-autograph-stage-linux64-nightly/opt task goes green, we know thatautograph-stage has signed the mar, and signingscript has verified that the signature matches the autograph-stagemar public key. Report back to #autograph on irc that it has passed.

2.3.2 Testing Autograph Prod

Once we roll out to prod, we want to make sure everything still works.

Autograph-prod mar signing test

To test autograph-prod mar-signing, find a recent (but ideally not the most-recent) nightly graph. Find a mar-signingtask in that graph.

To retrigger it via treeherder:

• sign in to mozilla-central treeherder (top right of the page)

• click on the task

• find the ... menu in the lower left of the page, click on it

• choose Custom Action

• choose Retrigger

• click Trigger in the lower right

(A retrigger here is preferable to a rerun, because it won’t affect chain of trust verification for the rest of the graph.)

Make sure this task runs green. If it goes green, then the task signed the mar file(s) via autograph-prod, and verifiedthe signature matches the public key.

2.3. Testing Autograph 35

https://treeherder.mozilla.org/#/jobs?repo=mozilla-centralhttps://treeherder.mozilla.org/#/jobs?repo=mozilla-central

RelEng Docs

2.4 Tree Closing Window Planning

RelEng Docs · L10N change-sets Finalize and ship L10N Initiate release Start release viaShip-It!application Build every-thing Automation will build installers and updaters for all

Documents