Smaller & faster updates now accessible to more Firefox users

When a user receives an update to Firefox they get either a partial or complete. A complete is nearly identical to its associated installer, and can update any old Firefox version. A partial is a binary diff of a specific old version against a newer one and only compatible with that specific old version. The size difference a partial and a complete can be huge. For example, the complete MAR for 14.0.1, en-US, win32 was 20MB. The partial from 13.0.1 was 7.4MB (even smaller on other platforms, where PGO doesn’t make diffing hard).

Until recently, we’ve only been able to produce partials against a single old version without a lot of extra time consuming and error prone manual work — meaning that a lot of users who could benefit from a partial weren’t receiving them.

With bug 773290 (multiple partial MAR support in release automation) resolved, we can now offer partial updates to many versions without the risk and time cost of doing them by hand. When we shipped 15.0 we offered partial updates to users on 14.0.1, 13.0.1, and 12.0 – which collectively represented just over 75% of our installed userbase. For future releases it’s possible we’ll offer partials to even more previous versions.

I’m sure some of you are asking why we can’t just do partial updates for ALL old releases. There’s a couple of reasons for that. Most importantly, there’s big diminishing returns on partial updates. The 13.0.1 -> 15.0 partial was 12MB, the 12.0 -> 15.0 partial was 14MB, and the complete was 20MB. The further you go back the less you gain from getting a partial. Secondly, computing partial updates is not computationally cheap. We ship 88 locales across 4 platforms — this works out to about 350 partials that need to be calculated. This can be made cheaper through caching and parallelization, but it still ends up adding about 45min to the running time of the release automation for every extra version we want partials for. In turn, this delays QA and other things in the critical path of shipping.

I also want to point out that this work does NOT apply to Nightly or Aurora. We have no plans to offer multiple partial updates on those channels at this time. Due to their relatively low userbase and very high frequency of change (almost every 24h), the cost/benefit just doesn’t work. However, we will be looking at using this on Beta where the userbase is much larger and the rate of change slower (about once a week).

A huge thanks goes out to Rail Aliev and Nick Thomas, who helped work out the design, wrote some parts of it, and provided reviews. We couldn’t have had this ready for 15.0 without their help.

10 thoughts on “Smaller & faster updates now accessible to more Firefox users

  1. > We ship 88 locales across 4 platforms — this works out to about 350 partials that need to be calculated
    So locales should be Add-ons, such they wouldn’t affect updates and could be updated at their own pace, but could still be queued up (or even downloaded) by the installer. Making them restartless and even with a persona-like preview-on-rollover-install-page would be awesome, but probably infeasible. But then that suggests that add-ons should use partial updates.

    1. This is something that’s been talked about in the past, I don’t know that it’s been prioritized though. The fact that the stub installer (bug 322206) is now moving forward may make that easier. Only needing to produce 4 builds instead of 88*4 would certainly make it more feasible to generate more partials! Unfortunately, it would probably mean we can’t have partial updates of the localized addons. They might be trivially small enough that it doesn’t matter, though.

      1. I also thought that this could be related to the stub-installer, in that if the common-installer is seperated from the language specific-add-on, the sub-installer could start downloading the common-installer while it is showing the welcome/language-selection-screen, making the perceived download faster. But if the language selection was postponed to the first-run-screen (persona-like), it wouldn’t need to affect the installer at all (other than only the common installer needed to be made available — making the download page simpler). I wouldn’t, just as you mention, expect benefit from more partials since there are diminishing returns. But perhaps I underestimate the diff-algorithms.
        Are the size of diff(verX, verX+2) much smaller than diff(verX, verX+1)+diff(verX+1, verX+2), or could an option be to “chain” partials on the client (or alternatively on the server), but as one update UX-wise.

        > Unfortunately, it would probably mean we can’t have partial updates of the localized addons.
        How is that related? As I mentioned, add-ons (from amo) could be incremental, and even split into common and localized as the installer.

    1. If possible, you’d want to diff two unpacked Windows installers. Eg, grab the 14.0.1 and 15.0 en-US installer EXE, extract it (“7z x foo.exe”), and diff both directory structures. If you can only handle one file at a time on each side you should look at xul.dll and/or omni.ja from both installers – they’re two of the largest files in the packages. Feel free to find me on IRC in #build or elsewhere!

      1. Yup! Chris AtLee (, catlee on IRC) tried it out and found that it reduced partials a little bit. I believe he said that it’s only better for windows binaries though. Because omni.ja is a big part of our build, it doesn’t have as much win for us as we’d want.

  2. Is there no way to re-use the data from the previous PGO run when doing a chemspill update? Any possible penalty should be negligible in such a case.

    1. Release builds are always full-clobber, so I don’t think you could just drop in a xul.dll from a different one and have it work. Most chemspills will have modified code that goes into xul.dll anyways, so it wouldn’t even be potentially possible for those.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>