Improvements to updates for Foxfooders

We've been providing on-device updates (that is to say: no flashing required) to users in the Foxfood program for nearly 6 months now. These updates are intended for users who are officially part of the Foxfooding program, but the way our update system works means that anyone who puts themselves on the right update channel can receive them. This makes things tough for us, because we'd like to be able to provide official Foxfooders with some extra bits and we can't do that while these populations are on the same update channel. Thanks to work that Rob Wood and Alexandre Lissy are doing, we'll soon be able to resolve this and get Foxfooders the bits they need to do the best possible testing.

To make this possible, we've implemented a short term solution that lets us only serve updates to official Foxfooders. When landed, they will send a hashed version of their IMEI as part of their update request. A list of the acceptable IMEI hashes will be maintained in Balrog (the update server), which lets us only serve an update if the incoming one matches one of the whitelisted ones.

To really make this work we need to detangle the current "dogfood" update channel. As I mentioned, it's currently being used in two distinct populations of users: those are part of the official program, and those who aren't. In order to support both populations of users we'll be splitting the "dogfood" update channel into two:

  1. The new "foxfood" channel will be for users who are officially part of the Foxfooding program. Users on this channel will be part of the IMEI whitelist, and could receive FOTA or OTA updates.
  2. The "dogfood" channel will continue to serve serve OTA updates to anyone who puts themself on it.

To transition, we will be asking folks who are officially part of the Foxfooding program to flash with a new image that switches them to the "foxfood" update channel. When this is ready to go, it will be announced and communicated appropriately.

Big thanks to everyone who was involved in this effort, particularly Rob Wood, who implemented the new whitelisting feature in Balrog, and Alexandre Lissy and Jean Gong, who went through multiple rounds of back and forth before we settled on this solution.

It's worth noting that this solution isn't ideal: sending IMEIs (even hashed versions) isn't something we prefer to do for both reasons of user privacy and protection of the bits. In the longer term, we'd like to look at a solution that wouldn't require IMEIs to be sent to us. This could come in the form of embedding or asking for credentials, and using those to access the updates. This type of solution would enhance user privacy and make it harder to get around the protections by brute forcing.

Going Faster with Balrog

Go Faster is a broad initiative at Mozilla that is focused on shipping things to users much faster than the current 6 week cycle. One important part of this project is having a mechanism to make Firefox aware of updates they need or may want to download. This is nothing entirely new of course - we've been shipping updates to users since Firefox 1.5 - but with Go Faster we will be updating bits and pieces of Firefox at a time rather than always updating the entire install. In this post I'm going to outline these new types of updates that we've identified, and talk about how things will work in the Glorious Future.

A Primer on Updates

Firefox updates work on a "pull" system, meaning Firefox regularly queries the update server (Balrog) to ask if there's an update available. For example, my Firefox is currently polling for updates by making a GET request to this URL:

All of the information in that URL is mapped against a set rules in Balrog, and eventually points to a single release. If that release is newer than the incoming one (based on the incoming version and buildid), Balrog returns the information necessary for the client to update to it:

    <update type="minor" displayVersion="41.0" appVersion="41.0" platformVersion="41.0" buildID="20150917150946" detailsURL="">
        <patch type="complete" URL="" hashFunction="sha512" hashValue="ea0757069363287f67659d8b7d42e0ac6c74a12ce8bd3c7d3e9ad018d03cd6f4640529c270ed2b3f3e75b11320e3a282ad9476bd93b0f501a22d1d9cb8884961" size="48982398"/>

It's important to note that Balrog only contains metadata about the update. The actual payloads of the updates are hosted on CDN networks.

New Types of Updates

We've identified three different new types of updates that we'll be implementing as part of Go Faster. They are:

  • System Addons: These are core (aka required) parts of Firefox that happen to be implemented as Addons.
  • Security Policy: This is a medium sized piece of JSON that instructs NSS about special security policies to enforce for various websites.
  • Optional Features: These are optional parts of Firefox that may be implemented as Addons or other means.

Each one of these will be implemented as an additional update request to Balrog (we may collapse these into a single request later). Eg, Firefox will look for new System Addons by making a GET request to an URL such as:

The responses will vary a bit depending on the type of update. More on that below.

System Addons

Seeing as Firefox can't function without them, System Addons may seem like a contradiction at first. The advantages are quite clear though: with them, we can ship updates to self contained pieces of Firefox at a substantionally faster rate. Shipping an update to all of Firefox takes nearly 24 hours (when we're moving as fast as we can); shipping an update to a System Addon could take as little as minutes.

Although they are implemented as Addons, we can't simply ship them through the AMO. Because Firefox cannot function without them we must ship them in the installers and full updates that happen every 6 weeks. This has the nice side effect of minimizing dependency problems -- we won't run into a case where Firefox updates but System Addons don't, which could cause incompatibilities. In between the 6 week cycles Firefox will poll Balrog for updates to System Addons and apply them as they become available. This graph may show this more clearly:

As you can see, Firefox 50.0 can be assumed to have any of Fizz 1.0, 1.1, 1.2, 1.3, or 2.0, while Firefox 51.0 is known to only have Fizz 2.0 (but may receive newer versions later).

When Firefox pings Balrog for System Addon updates, the response will look something like this:

        <addon id="" URL="" hashFunction="sha512" hashValue="abcdef123456" version="1.1"/>
        <addon id="" URL="" hashFunction="sha512" hashValue="abcdef123456" version="2.5"/>
        <addon id="" URL="" hashFunction="sha512" hashValue="abcdef123456" version="3.4"/>

Firefox will compare the list against its currently installed versions and update anything that's out of date. The exact details on where System Addons will live on disk are still being ironed out.

Security Policy

Every version of Firefox ships with the most up-to-date set of security policies that were available when it was built. However, these policies are updated more frequently than we ship, and it's crucial that we keep them up to date to keep our users safe. As with System Addons, Firefox will regularly ping Balrog to check for updated security policies. When one is found, Firefox downloads it from Kinto, which will serve it an incremental update to its security policies. The details of this process have been outlined in much more detail by the Cloud Services team.

The Balrog response for these updates is extremely simple, and simply contains a version that Firefox passes along to Kinto:

        <setting id="security" lastModified="129386427328"/>

Optional Features

These are parts of Firefox that are not core to the browser, but may be useful to subsets of users. For example: We currently ship a ton of hyphenation dictionaries as part of Firefox for Android. These are locale-specific, so only one ever gets used for each user. We can also distribute opt-in features that not everyone wants or needs, eg: Developer Tools may be a good candidate (there are no plans to do so at this time though).

Optional features may also be implemented in various ways. Hyphenation dictionaries are simple zip files, while something like Developer Tools would likely be an Addon. They will not be included in Firefox installers or update packages. Instead Firefox will regularly query Balrog to see what packages may be available to it. Some things may automatically install based on the user's environment (eg: hyphentation dictionaries for your locale), while other things may require opt-in (eg: optional features).

Balrog responses are not yet set in stone for these, but Kinto is likely to be involved, so the response may end up being similar to the one above for Security Policy updates.


While System Addons, Security Policy, and Optional Features overlap in some areas, each has its own unique combination of requirements. The chart below summarizes these:
Required? Shipped in Installer? Payload Type Payload Location
System Addons Yes Yes Addons CDN
Security Policy Yes Yes JSON Kinto
Optional Features No No Anything Kinto

Mozilla Software Release GPG Key Transition

Late last week we discovered the expiration of the GPG key that we use to sign Firefox, Fennec, and Thunderbird nightly builds and releases. We had been aware that this was coming up, but we unfortunately missed our deadline to renew it. This caused failures in many of our automated nightly builds, so it was quickly noticed and acted upon.

Our new GPG key is as follows, and available on keyservers such as and

pub   4096R/0x61B7B526D98F0353 2015-07-17
      Key fingerprint = 14F2 6682 D091 6CDD 81E3  7B6D 61B7 B526 D98F 0353
uid                            Mozilla Software Releases 
sub   4096R/0x1C69C4E55E9905DB 2015-07-17 [expires: 2017-07-16]

The new primary key is signed by many Mozillians, the old master key, as well as our OpSec team's GPG key. Nightlies and releases will now be signed with the subkey (0x1C69C4E55E9905DB), and a new one will be generated from the same primary key before this one expires. This means that you can validate Firefox releases with the primary public key in perpetuity.

We are investigating a few options to make sure key renewal happens without delay in the future.

Mozilla will stop producing automated builds of XULRunner after the 41.0 cycle

XULRunner is a runtime package that can be used to run XUL+XPCOM based applications. Automated builds of it have been produced alongside Firefox since 2006, but it has not been a supported or resourced product for many years. We've continued to produce automated builds of it because its build process also happens to build the Gecko SDK, which we do support and maintain. This will change soon, and we'll start building the Gecko SDK from Firefox instead (bug 672509). This work will land on mozilla-central during the 42.0 cycle, which means that when the 41.0 cycle ends (September 22, 2015), automated builds of XULRunner will cease.

If you are a consumer of the Gecko SDK this means very little to you -- we will continue to produce it with every Firefox release.

If you are a consumer of the XULRunner stub this means that you will no longer have a Mozilla produced version after 41.0. For folks in this group, you have two options:

  • Change your app to run through the stub provided by Firefox. Many apps will continue to work as before by simply replacing "xulrunner.exe application" with "firefox -app application.ini".
  • Build XULRunner yourself.

Buildbot <-> Taskcluster Bridge Now in Production

A few weeks ago I gave a brief overview of the Buildbot <->Taskcluster Bridge that we've been developing, and Selena provided some additional details about it yesterday. Today I'm happy to announce that it is ready to take on production work. As more and more jobs from our CI infrastructure move to Taskcluster, the Bridge will coordinate between them and jobs that must remain in Buildbot for the time being.

What's next?

The Bridge itself is feature complete until our requirements change (though there's a couple of minor bugs that would be nice to fix), but most of the Buildbot Schedulers still need to be replaced with Task Graphs. Some of this work will be done at the same time as porting specific build or test jobs to run natively in Taskcluster, but it doesn't have to be. I made a proof of concept on how to integrate selected Buildbot builds into the existing "taskcluster-graph" command and disable the Buildbot schedulers that it replaces. With a bit more work this could be extended to schedule all of the Buildbot builds for a branch, which would make porting specific jobs simpler. If you'd like to help out with this, let me know!

Buildbot Taskcluster Bridge - An Overview

Mozilla has been using Buildbot as its continuous integration system for Firefox and Fennec for many years now. It enabled us to switch from a machine-per-build model to a pool-of-slaves model, and greatly aided us in getting to our current scale. But it's not perfect - and we've known for a few years that we'll need to do an overhaul. Lucky for us, the FirefoxOS Automation team has built up a fantastic piece of infrastructure known as Taskcluster that we're eager to start moving to.

It's not going to be a small task though - it will take a lot more work than taking our existing build scripts and running them in Taskcluster. One reason for this is that many of our jobs trigger other jobs, and Buildbot manages those relationships. This means that if we have a build job that triggers a test job, we can't move one without moving the other. We don't want to be forced into moving entire job chains at once, so we need something to help us transition more slowly. Our solution to this is to make it possible to schedule jobs in Taskcluster while still implementing them in Buildbot. Once the scheduling is in Taskcluster it's possible to move individual jobs to Taskcluster one at a time. The software that makes this possible is the Buildbot Bridge.

The Bridge is responsible for synchronizing job state between Taskcluster and Buildbot. Jobs that are requested through Taskcluster will be created in Buildbot by the Bridge. When those jobs complete, the Bridge will update Taskcluster with their status. Let's look at a simple example to see see how the state changes in both systems over the course of a job being submitted and run:

Event Taskcluster state Buildbot state
Task is created Task is pending --
Bridge receives "task-pending" event, creates BuildRequest Task is pending Build is pending
Build starts in Buildbot Task is pending Build is running
Bridge receives "build started" event, claims the Task Task is running Build is running
Build completes successfully Task is running Build is completed
Bridge receives "build finished" event, reports success to Taskcluster Task is resolved Build is completed

The details of how this work are a bit more complicated - if you'd like to learn more about that I recommend watching the presentation I did about the Bridge architecture, or just have a read through my slides

A great tech support experience, from the most unlikely provider

After ranting and raving about them so much when trying to cancel my TV service, I feel like I should also share the really great experience I had with Bell recently. We've been having intermittent issues with our Internet connection. It wasn't clear whether or not it was our router or an issue with the DSL modem or line. I was dreading calling Bell's tech support because of so many bad experiences with tech support from all companies in the past.

To my great surprise, the call was one of the best tech support experiences I've ever had. The level 1 tech was the normal "turn it off, turn it on", but the level 2 seemed genuinely motivated to help me. She did not freak out and blame me when I told her I had my own router hooked up (something that Bell discourages). It was almost a brainstorming session for a few minutes of where the issue may be. I explained that the connection seems to drop when doing high bandwidth things like Skype or other videoconferencing. She thought for a second and then wondered if it only happened on wifi. That was a great insight, because Netflix (via our media centre PC, hooked up via ethernet) caused no issues at all. It was pretty clear to me that the problem was with my own router at that point, but she still insisted on running a line test to make sure there was nothing there. After that showed no issues I told her I was pretty sure the issue was with my own router, so I'd replace that. Even at that point she was insisting that I call back if I continue to experience issues, to the point that her supervisor called me back a few hours later to follow-up.

I don't know if this was a one-off experience or if something changed recently, but that was amazing tech support - even better than I received from geek-oriented ISPs like TekSavvy. Thank you Bell, keep it up.

Release Automation Futures: Seamless integration of manual and automated steps

I've written about the history of our Release Automation systems in the past. We've gone from mostly manual releases to almost completely automated since I joined Mozilla. One thing I haven't talked about before is Ship It - our web tool for kicking off releases:

It may be ugly, but having it has meant that we don't have to log on to a single machine to ship a release. A release engineer doesn't even need to be around to start the release process - Release Management has direct access to Ship It to do it themselves. We're only needed to push releases live, and that's something we'd like to fix as well. We're looking at tackling that and other ancillary issues of releases, such as:

  • Improving and expanding validation of release automation inputs (revisions, branches, locales, etc.)
  • Scripting the publishing of Fennec to Google Play
  • Giving release Release Managers more direct control over updates
  • Updating metadata (ship dates, versions, locales) about releases
  • Improving security with better authentication (eg, HSMs or other secondary tokens) and authorization (eg, requiring multiple people to push updates)

Rail and I had a brainstorming session about this yesterday and a theme that kept coming up was that most of the things we want to improve are on the edges of release automation: they happen either before the current automation starts, or after the current automation ends. Everything in this list also needs someone to decide that it needs to happen -- our automation can't make the decision about what revision a release should be built with or when to push it to Google Play - it only knows how to do those things after being told that it should. These points where we jump back and forth between humans and automation are a big rough edge for us right now. The way they're implemented currently is very situation-specific, which means that adding new points of human-automation interaction is slow and full of uncertainty. This is something we need to fix in order to continue to ship as fast and effectively as we do.

We think we've come up a new design that will enable us to deal with all of the current human-automation interactions and any that come up in the future. It consists of three key components:


A workflow is a DAG that represents an entire release process. It consists of human steps, automation steps, and potentially other types. An important point about workflows is that they aren't necessarily the same for every release. A Firefox Beta's workflow is different than a Fennec Beta or Firefox Release. The workflow for a Firefox Beta today may look very different than for one a few months from now. The details of a workflow are explicitly not baked into the system - they are part of the data that feeds it. Each node in the DAG will have upstreams, downstreams, and perhaps a list of notifications. The tooling around the workflow will respond to changes in state of each node and determine what can happen next. Much of each workflow will end up being the existing graph of Buildbot builders (eg: this graph of Firefox Beta jobs).

We're hoping to use existing software for this part. We've looked at Amazon's Simple Workflow Service already, but it doesn't support any dependencies between nodes, so we're not sure if it's going to fit the bill. We're also looking at Taskcluster which does do dependency management. If anyone knows of anything else that might be useful here please let know!

Ship It

As well as continuing to provide a human interface, Ship It will be the API between the workflow tool and humans/automation. When new nodes become ready it makes that information available to automation, or gives humans the option to enact them (depending on node type). It also receives state changes of nodes from automation (eg, build completion events). Ship It may also be given the responsibility of enforcing user ACLs.

Release Runner

Release Runner is the binding between Ship It and the backend parts of the automation. When Ship It is showing automation events ready to start, it will poke the right systems to make them go. When those jobs complete, it will send that information back to Ship It.

This will likely be getting a better name.

This design still needs some more thought and review, but we're very excited to be moving towards a world where humans and machines can integrate more seamlessly to get you the latest Firefox hotness more quickly and securely.

Signing Software at Scale

Mozilla produces a lot of builds. We build Firefox for somewhere between 5 to 10 platforms (depending how you count). We release Nightly and Aurora every single day, Beta twice a week, and Release and ESR every 6 weeks (at least). Each release contains an en-US build and nearly a hundred localized repacks. In the past the only builds we signed were Betas (which were once a week at the time), Releases, and ESRs. We had a pretty well established manual for it, but due to being manual it was still error prone and impractical to use for Nightly and Aurora. Signing of Nightly and Aurora became an important issue when background updates were implemented because one of the new security requirements with background updates was signed installers and MARs.

Enter: Signing Server

At this point it was clear that the only practical way to sign all the builds that we need to is to automate it. It sounded crazy to me at first. How can you automate something that depends on secret keys, passphrases, and very unfriendly tools? Well, there's some tricks you need to know, and throughout the development and improvement of our "signing server", we've learned a lot. In the post I'll talk about those tricks and show you how can use them (or even our entire signing server!) to make your signing process faster and easier.

Credit where credit is due: Chris AtLee wrote the core of the signing server and support for some of the signature types. Over time Erick Dransch, Justin Wood, Dustin Mitchell, and I have made some improvements and added support for additional types of signatures.

Tip #1: Collect passphrases at startup

This should be obvious to most, but it's very important not to store the passphrases to your private keys unencrypted. However, because they're needed to unlock the private keys when doing any signing the server needs to have access to them somehow. We've dealt with this by asking for them when launching a signing server instance:

$ bin/python tools/release/signing/ signing.ini
gpg passphrase: 
signcode passphrase: 
mar passphrase: 

Because instances are started manually by someone in the small set of people with access to passphrases we're able to ensure that keys are never left unencrypted at rest.

Tip #2: Don't let just any machine request signed files

One of the first problems you run into when you have an API for signing files is how to make sure you don't accidentally sign malicious files. We've dealt with this in a few ways:

  • You need a special token in order to request any type of signing. These tokens are time limited and only a small subset of segregated machines may request them (on behalf of the build machines). Since build jobs can only be created if you're able to push to, random people are unable to submit anything for signing.
  • Only our build machines are allowed to make signing requests. Even if you managed to get hold of a valid signing token, you wouldn't be able to do anything with it without also having access to a build machine. This is a layer of security that helps us protect against a situation where an evil doer may gain access to a loaner machine or other less restricted part of our infrastructure.

We have other layers of security built in too (HTTPS, firewalls, access control, etc.), but these are the key ones built into the signing server itself.

Tip #3: Use input redirection and other tricks to work around unfriendly command line tools

One of the trickiest parts about automating signing is getting all the necessary command line tools to accept input that's not coming from a console. Some of them are relative easy and accept passphrases via stdin:

proc = Popen(command, stdout=stdout, stderr=STDOUT, stdin=PIPE)

Others, like OpenSSL, are fussier and require the use of pexpect:

proc = pexpect.spawn("openssl", args)
proc.logfile_read = stdout
proc.expect('Enter pass phrase')

And it's no surprise at all that OS X is the fussiest of them all. In order to sign you have to unlock the keychain by hand, run the signing command, and relock the keychain yourself:

child = pexpect.spawn("security unlock-keychain" + keychain)
child.expect('password to unlock .*')
check_call(sign_command + [f], cwd=dir_, stdout=stdout, stderr=STDOUT)
check_call(["security", "lock-keychain", keychain])

Although the code is simple in the end, a lot of trial, error, and frustration was necessary to arrive at it.

Tip #4: Sign everything you can on Linux (including Windows binaries!)

As fussy as automating tools like openssl can be on Linux, it pales in comparison to trying to automate anything on Windows. In the days before the signing server we had a scripted signing method that ran on Windows. Instead of providing the passphrase directly to the signing tool, it had to typed into a modal window. It was "automated" with an AutoIt script that typed in the password whenever the window popped up. This was hacky, and sometimes lead to issues if someone moved the mouse or pressed a key at the wrong time and changed window focus.

Thankfully there's tools available for Linux that are capable of signing Windows binaries. We started off by using Mono's signcode - a more or less drop in replacement for Microsoft's:

$ signcode -spc MozAuthenticode.spc -v MozAuthenticode.pvk -t -i -a sha1 -tr 5 -tw 60 /tmp/test.exe
Mono SignCode - version
Sign assemblies and PE files using Authenticode(tm).
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Enter password for MozAuthenticode.pvk: 

This works great for 32-bit binaries - we've been shipping binaries signed with it for years. For some reason that we haven't figured out though, it doesn't sign 64-bit binaries properly. For those we're using "osslsigncode", which is an OpenSSL based tool to do Authenticode signing:

$ osslsigncode -certs MozAuthenticode.spc -key MozAuthenticode.pvk -i -h sha1 -in /tmp/test64.exe -out /tmp/test64-signed.exe
Enter PEM pass phrase:

$ osslsigncode verify /tmp/test64-signed.exe 
Signature verification: ok

Number of signers: 1
    Signer #0:
        Subject: /C=US/ST=CA/L=Mountain View/O=Mozilla Corporation/CN=Mozilla Corporation
        Issuer : /C=US/O=DigiCert Inc/ Assured ID Code Signing CA-1

Number of certificates: 3
    Cert #0:
        Subject: /C=US/O=DigiCert Inc/ Assured ID Root CA
        Issuer : /C=US/O=DigiCert Inc/ Assured ID Root CA
    Cert #1:
        Subject: /C=US/O=DigiCert Inc/ Assured ID Code Signing CA-1
        Issuer : /C=US/O=DigiCert Inc/ Assured ID Root CA
    Cert #2:
        Subject: /C=US/ST=CA/L=Mountain View/O=Mozilla Corporation/CN=Mozilla Corporation
        Issuer : /C=US/O=DigiCert Inc/ Assured ID Code Signing CA-1

In addition to Authenticode signing we also do GPG, APK, and couple of Mozilla-specific types of signing (MAR, EME Voucher) on Linux. We also sign our Mac builds with the signing server. Unfortunately, the tools needed for that are only available on OS X, so we have to run separate signing servers for these.

Tip #5: Run multiple signing servers

Nobody likes a single point of failure, so we've built support our signing client to retry against multiple instances. Even if we lose part of our signing server pool, our infrastructure stays up:
$ python --cachedir cache -t token -n nonce -c host.cert -H -H -H -H --formats dmgv2
2015-01-23 06:17:59,112 - ed40176524e7c197f4e23f6065a64dc3c9a62e71: processing on
2015-01-23 06:17:59,118 - ed40176524e7c197f4e23f6065a64dc3c9a62e71: connection error; trying again soon
2015-01-23 06:18:00,119 - ed40176524e7c197f4e23f6065a64dc3c9a62e71: processing on
2015-01-23 06:18:00,141 - ed40176524e7c197f4e23f6065a64dc3c9a62e71: uploading for signing
2015-01-23 06:18:10,748 - ed40176524e7c197f4e23f6065a64dc3c9a62e71: processing on
2015-01-23 06:19:11,848 - ed40176524e7c197f4e23f6065a64dc3c9a62e71: processing on
2015-01-23 06:19:40,480 - ed40176524e7c197f4e23f6065a64dc3c9a62e71: OK

Running your own signing server

It's easy! All of the code you need to run your own signing server is in our tools repository. You'll need to set-up a virtualenv and create your own config file, but once you're ready you can attempt to start it with the following command:

python signing.ini

You'll be prompted for the passphrases to your private keys. If there's any problems with your config file or the passphrases the server will fail to start. Once you've got it up and running you can use try signing! has an example of how to generate a signing token, and will take your unsigned files and give you back signed versions. Happy signing!

UPDATED: New update server is going live for release channel users on Tuesday, January **20th**

(This post has been updated with the new go-live date.)

Our new update server software (codenamed Balrog) has been in development for quite awhile now. In October of 2013 we moved Nightly and Aurora to it. This past September we moved Beta users to it. Finally, we're ready to switch the vast majority of our users over. We'll be doing that on the morning of Tuesday, January 20th. Just like when we switched nightly/aurora/beta over, this change should be invisible, but please file a bug or swing by #releng if you notice any issues with updates.

Stick around if you're interested in some of the load testing we did.

Shortly after switching all of the Beta users to Balrog we did a load test to see if Balrog could handle the amount of traffic that the release channel would throw at it. With just 10% of the release traffic being handled, it blew up:

We were pulling more than 150MBit/sec per web head from the database server, and saturating the CPUs completely. This caused very slow requests, to the point where many were just timing out. While we were hoping that it would just work, this wasn't a complete surprise given that we hadn't implemented any form of caching yet. After implementing a simple LRU cache on Balrog's largest objects, we did another load test. Here's what the load looked like on one web head:

Once caching was enabled the load was practically non-existent. As we ramped up release channel traffic the load grew, but in a more or less linear (and very gradual) fashion. At around 11:35 on this graph we were serving all of the release channel traffic, and each web head was using a meager 50% of its CPU:

I'm not sure what to call that other than winning.