Smaller & faster updates now accessible to more Firefox users

When a user receives an update to Firefox they get either a partial or complete. A complete is nearly identical to its associated installer, and can update any old Firefox version. A partial is a binary diff of a specific old version against a newer one and only compatible with that specific old version. The size difference a partial and a complete can be huge. For example, the complete MAR for 14.0.1, en-US, win32 was 20MB. The partial from 13.0.1 was 7.4MB (even smaller on other platforms, where PGO doesn't make diffing hard).

Until recently, we've only been able to produce partials against a single old version without a lot of extra time consuming and error prone manual work -- meaning that a lot of users who could benefit from a partial weren't receiving them.

With bug 773290 (multiple partial MAR support in release automation) resolved, we can now offer partial updates to many versions without the risk and time cost of doing them by hand. When we shipped 15.0 we offered partial updates to users on 14.0.1, 13.0.1, and 12.0 - which collectively represented just over 75% of our installed userbase. For future releases it's possible we'll offer partials to even more previous versions.

I'm sure some of you are asking why we can't just do partial updates for ALL old releases. There's a couple of reasons for that. Most importantly, there's big diminishing returns on partial updates. The 13.0.1 -> 15.0 partial was 12MB, the 12.0 -> 15.0 partial was 14MB, and the complete was 20MB. The further you go back the less you gain from getting a partial. Secondly, computing partial updates is not computationally cheap. We ship 88 locales across 4 platforms -- this works out to about 350 partials that need to be calculated. This can be made cheaper through caching and parallelization, but it still ends up adding about 45min to the running time of the release automation for every extra version we want partials for. In turn, this delays QA and other things in the critical path of shipping.

I also want to point out that this work does NOT apply to Nightly or Aurora. We have no plans to offer multiple partial updates on those channels at this time. Due to their relatively low userbase and very high frequency of change (almost every 24h), the cost/benefit just doesn't work. However, we will be looking at using this on Beta where the userbase is much larger and the rate of change slower (about once a week).

A huge thanks goes out to Rail Aliev and Nick Thomas, who helped work out the design, wrote some parts of it, and provided reviews. We couldn't have had this ready for 15.0 without their help.

Fresh-from-the-garden habanero hot sauce

WARNING: Do not touch your eyes, nose, mouth, or any other sensitive areas of your body when working with habaneros. You will regret it.

This year I tried to grow some habanero peppers in my garden. I only ended up with a single one from the entire plant, but even a tiny bit packs a punch. Because of that, the only real use for me is to make a sauce. I started by scaling back this recipe which calls for 5 habaneros. The sauce still ended up way too hot for me, so I further modified it to calm it down. The recipe I came up with is as follows:


  • 1 tablespoon olive oil
  • 1/4 cup chopped carrots
  • 1/4 cup chopped onion
  • 2 cloves garlic
  • 1 habanero pepper, seeds removed
  • 1 tbsp water
  • 3 tbsp lime juice
  • 1 tbsp white vinegar
  • 2 tomatoes
  • 1 tbsp sugar
  • salt and pepper to taste


  1. Saute the carrots, onion, and garlic in a saucepan until softened.
  2. Pour them into a blender along with the habanero, water, lime juice, vinegar, and tomatoes. Blend until smooth.
  3. Put the mixture back into the pan and bring to a simmer.
  4. Add the sugar and season with salt and pepper. Stir until combined.
  5. Reduce the mixture to your desired consistency.

The finished sauce should start off sweet and end up firey. If it's too hot more lime juice or sugar can be added to calm it down.

5 years with Mozilla

Today marks my 5 year anniversary of joining Mozilla full-time. (I was around the project for about a year prior to that, but it's easier to celebrate this :P.) So much has changed since I joined. Release Engineering has gone from a burnout position to a team of 14 people. The company as a whole has grown massively. We're shipping more products. We're moving faster. But Mozilla continues to be an amazing place to work, full of incredibly smart and awesome people.

My initial involvement with Mozilla was while I was a student at Seneca College when Rob Helmer mentored me through some Buildbot work that Mozilla was interested in. After that, I spent some time contracting with Mozilla during which I ported Talos to Linux (with guidance from Alice Nodelman) and created the initial implementation of the Try Server (with guidance from Paul Reed). As a novice programmer, being given the opportunity to work on important, high-impact work was inspiring. This is one of things that Mozilla does so so well.

Over the years I've worked on so many different things that I've long since lost track. There's always new challenges, new opportunities, and a never-ending queue of bugs. I'm looking forward to another 5 (or more likely, 35) years here. Y'all are amazing.

Lastly, it's important for me to note that Dave Humphrey was instrumental in my initial involvement with Mozilla. Without his work forming a relationship between Seneca and Mozilla I don't think I would've found my way here. I know you'll never take the credit you deserve Dave, but thank you. You've been instrumental in my and many others' careers.

Parsing Apache-like configs in Python

While working on partial updates for multiple old versions I had a need to parse our patcher config files with Python. These files are close to Apache-style configs except more liberal with the naming of nodes. I was pretty surprised when I couldn't quickly find a library that parsed them. I tried various XML/HTML parser (BeautifulSoup, lxml, minidom) but they all choked pretty quickly. Eventually I came across apache_conf_parser which sounded pretty promising! It complained at first about some of our node names, but a small code change allowed it to parse them correctly. I've published my changes to a git repository at in case anyone ever needs them.

B2G Desktop builds now (mostly) available

Hot on the heels of the Armv7a Gingerbread builds for B2G, we now have Mac and Windows Desktop builds of B2G+Gecko available for download. On TBPL, they show up as "Ng" cells in the existing OS X64 opt and Win opt rows, and you can download the builds from FTP in the nightly directories.

Please note that these are NOT full B2G builds. These are developer-targeted builds which only run on desktop machines, and cannot be flashed onto a phone or tablet. These builds are primarily useful to developers, QA and localizers working on Gaia. They can also be used by anyone who wishes to test out their websites or apps through a B2G-like client..

Linux versions of these builds are in the works, and will be available as soon as we fix an infrastructure issue preventing them from successfully building. We hope to have it fixed by the end of the week.


Edit: If you're confused about what to do with these builds even after they're running have a read over and

Our community is awesome (or, Mac builds now signed even better)

A few weeks ago a bug regarding the way we are signing our Mac builds was reported by a user of the Keychain Integration Services extension. I think this bug is an amazing example of how great our community is, and why they are so valuable to us.

This bug talks about a problem that exists on Snow Leopard and earlier where OS X will ask the user over and over again for permission for Firefox to access the system Keychain. The initial bug report has tremendous analysis of the problem, isolating it to specific OS X versions and comparing the signatures on them. Steven Michaud quickly jumped in to try to confirm and find a fix, but it wasn't until Julian Fitzell (the Keychain Integration Services developer) jumped in that the problem was fully understood. After he went to the effort of figuring out exactly what we needed to do it was relatively simple to fix the bug in a timely manner.

The reason this stands out so much to me is that because this is a problem that would only affect Keychain Integration Services users, it was (correctly) prioritized as low priority. (As much as we would like to fix every bug that is critical to everybody, it's not feasible.) Because Julian cared and was capable of helping us find the fix it was possible to fix this bug before Firefox 14 was released. Without his help it's very unlikely this would've been fixed so quickly.

So: thank you Julian and spinifer. You rock!

Armv7a Gingerbread Gecko builds (opt+debug) now enabled on project branches

Just a quick announcement that a couple of follow-up bugs to the work from landed:

  • bug 767503 added the new-style Armv7a GB-based Gecko builds to project branches
  • bug 767528 disabled the old, now-superfluous opt builds

All active project branches have had these builds enabled except for the elm and oak twigs. If you are the owner of a project branch and do not wish to have B2G builds on it, or do not feel they will be useful please file a RelEng bug to have them disabled. Do keep in mind that burning these builds when pushing to mozilla-central or mozilla-inbound will get you backed out.

Armv7a Gingerbread-based B2G Gecko debug builds now live

A few weeks ago armv7a Gingerbread-based B2G Gecko opt builds started running on most branches. As of a few hours ago we are now generating debug versions of these builds. We now have "Armv7a GB" rows for opt and debug with "Bg" (B2G Gecko Build) cells in them, and we'll add more build & test cells and platform rows as they arrive.

We know there's more B2G build requests coming, so as part of this, we also reworked our configs to make it easier to add additional B2G platforms and build types in the future.

For now, we've left the original "B2G" builds listed in the "Linux" row on TBPL. These are the initial opt builds that were set-up, and are exactly the same as the new Bg builds in the "Armv7a GB opt" row. This is transitional only - we'll be running them in parallel over the weekend and shutting the old builds off early next week after we're satisfied with the new ones.

For more details see bug 758425.

Update on Mac Code Signing #2

In my previous post I said that Mac signing had been turned on, and that it would stay on. Unfortunately, the following morning we caught some issues that only came up after updating. It took a bit of time to resolve those, but as of this morning we got them worked out, and signed mac builds are back. There are a couple of follow-up issues to address, but we're in a good enough state to backport signing to Aurora and Beta, and ship it in Firefox 13.0. On Saturday evening I'll be enabling it on Aurora, and on Monday morning on Beta - assuming no new issues come up.

If you see ANY issues related to updating on Mac please file a bug and cc me (:bhearsum). It's very important that any issues are brought up immediately, as we intend to ship this to the release channel on June 5th.

Huge thanks (again) to Steven Michaud, Ted Mielczarek, Erick Dransch, and anyone else that helped make this happen.

Update on Mac Code Signing of Firefox

Last week I talked about our plans around Mac build signing. In it, I said that I intended to have signed dep/try builds by the end of last week and signed Nightly builds early this week. Unfortunately, shortly after publishing I realized that due to some complicated infrastructure reasons, we couldn't turn on dep/try build signing prior to Nightly. However, I'm pleased to announce that all the necessary work for Mac builds signing is landed and in production. From now on, mozilla-central based builds will be signed. As other branches merge the necessary patch, they will be signed too. Nightly updates have been disabled for now, to give QA a chance to verify everything. Nightly updates should be re-enabled sometime tomorrow. This should be an invisible change to everyone, but please file bugs on any issues (especially those related to new installs or updates) and cc me (

Additionally, I want to correct one thing in my previous post. I said that applications 'will not run on 10.8 unless the user has allowed “applications downloaded from anywhere” to run'. However, it has come to my attention that this isn't entirely true. You can except specific applications from this policy if you ctrl+click the application and hit "open". This works for unsigned apps, self signed apps, and those signed with a Mac Development certificate.