Shipping large software to end-user devices is a complicated process. Shipping large software securely to end-user devices is even more complicated. Signing the things that ship to end-user devices is one of those complications, and it gets even more complicated when you sign thousands of artifacts per day.
Mozilla has been signing Firefox in some form beginning with Firefox 1.0. This began with detached GPG signatures for builds, and progressed to Authenticode signing for Windows installers in Firefox 1.0.1. Since then it has evolved over time to encompass other platforms, other types of files within our products, and other ways that we ship (such as our own update packages). This post will provide a overview of the what, when, why, and how of code signing at Mozilla over the past ~20 years.
When we first began signing, it happened on a Windows machine. Late in the release process, after Windows installers had been built, we would download all of the release artifacts to this machine, sign the Windows installers, and generate detached GPG signatures for those before pushing the artifacts elsewhere.
At this time, the private keys and certificates were held on a USB stick that was kept removed from the machine at-rest. A Release Engineer needed to be physically present in Mountain View to perform this step. Once inserted, signing could be done via Remote Desktop rather than at the physical machine (but don't forget to remove the USB stick afterwards!).
GPG signing was done with the standard GPG tools, running in cygwin. Authenticode signing was also done with the standard (at the time) Microsoft 'signcode.exe' tool. An annoying fact about that tool, is that it only accepted the necessary passphrase from a GUI dialog. To work around this, we had an AutoIt script running in the background that injected the passphrase into this dialog whenever it popped up. This interesting way of automating the process meant that mouse movements or keyboard interaction at the wrong time could interfere with the signing process.
This process was partly scripted, but there was still a series of ~15 commands someone had to run by hand (and not mess up) to get everything done. You can see these commands for yourself in our now-ancient Unified Release Process documentation.
Careful readers may have noted that early Authenticode signing only covered the Firefox installer itself, not the EXEs and DLLs inside of it. At some point (I haven't gone to the effort of tracking down exactly where...) we started signing these inner files as well. This process seems to have been lost to the sands of time, but I seem to recall it worked very similarly to the installer signing process, but without the GPG parts.
The first notable improvement we had to this process was to automate most of the copy/pasting that was done from the wiki. This came in the form of a Makefile that with a few mere inputs, would download, sign, and re-upload the signed builds. The main benefit of this was reduced opportunity for human error.
Not long after that, we had our first real game-changing improvements. Chris AtLee wrote a nice post about it back in 2009, and how his changes took signing time from 8 hours all the way down to sub-15 minutes. This was accomplished through a combination of faster hardware, parallelized signing, in-process compression & decompression, and better caching. His changes also introduced a hefty amount of python into the signing process, which paved the way for the next big improvement...
Despite the signing process now being very quick once it gets started, it could still sometimes take hours or longer to begin the process. Typically this would happen if our build and repack processes finished at a time when no Release Engineer was around to begin signing. This was solved with what we called "autosign". Rather than require a Release Engineer to be around at the right moment, we adjusted our scripts to allow them to be started ahead of time, and be smart enough to know when all of the files it needs to sign are ready. This work eliminated all wait time between builds being ready and signing running.
In 2011, signing was rearchitected altogether. In short, the idea was to move signing to a highly secured Linux server, and sign builds through an API as part of the build process. This allowed builds to be signed as they were produced, and reduced the number of times builds had to move from one server to another before they shipped. An obvious question here is how we would manage to sign Windows binaries on Linux...as it turns out, the mono project had its own version of signcode that ran natively on Linux that we were able to make use of.
Shortly after (and perhaps even motivating - I'm not sure at this point) the aforementioned signing server work, we began signing our MAR (Mozilla ARchive) packages that update users from an older version of Firefox to a newer version. Thanks to the earlier work, it was fairly trivial to use the same architecture to sign these files.
The idea of signing .app bundles for macOS was filed all the way back in 2007. There was some initial work on this in 2010, but we were unable to land it at that time. Around the same time that MAR signing was happening in 2012, we picked up this work again and managed to drive it home this time.
Unfortunately, there were no tools available at the time to sign macOS builds on anything except a fairly modern macOS machine. For this reason, we had to run additional copies of our signing server on macOS and sign those builds with them. (If you've ever had to run macOS as a server you'll know just how unfortunate this was...)
In 2018, Mozilla migrated its CI and Release automation from our aging Buildbot systems to Taskcluster. As part of this, signing tasks moved to specialized Taskcluster workers known as "signingscript" and "iscript", used for signing non-macOS and macOS builds respectively. These specialized workers continued to outsource the actual work of signing to the previously discussed signing servers.
An important part of this change is the introduction of Chain of Trust, a significant security enhancement that helps ensure that only authentic artifacts are signed to this day.
Autograph is Mozilla's modern code signing service. It was built specifically to provide a signing service that allowed us to keep private key material in Hardware Security Modules (HSMs). Migrating release signing to it was a huge improvement over the existing signing server where Release Engineers had direct access to such things. It was initially used for signing XPIs and APKs, but by the end of 2019 we had migrated all non-macOS signing to it and retired the old Linux signing servers.
In addition to the security enhancements it brought, we saw great performance wins with it as well, largely in thanks to it's support for only requiring a hash of the bytes being signed to be sent over the wire. (This requires that the client has some advanced knowledge of the file being signed, but it saves a tremendous amount of network traffic at our scale.)
I've mentioned a number of tools and technology that we use as part of signing, but I've purposely glossed over some details in the interest of brevity. The following section is a glossary of sorts, and introduces some more under the hood tools that we use as part of signing. If you're interested in the gory details, the links below should be enough to find them for yourself! Or you can stop by #firefox-ci on Matrix to ask questions!
osslsigncode is a tool that implements parts of Microsoft's signtool.exe. In the past, we used it to directly sign PE files. These days, we use it's support for attaching the signatures that Autograph makes to them.
Winsign is a python library for signing and manipulating Authenticode signatures. It relies on osslsigncode for writing signatures, and supports signing directly with a private key, or outsourcing the signing process to a passed in function. The latter is what we use, and it's how we inject a call to Autograph into the signing process.
In 2021 we began shipping Firefox as an MSIX package. As part of this we discovered that osslsigncode does not support signing MSIX packages. Luckily for us, Microsoft's MSIX packaging tools are open source and run on Linux, and we found a fork that contained most of what was needed to support signing. With a few additional modifications, we were able to support signing these packages in our existing systems.
apple-codesign is a very exciting project from Gregory Szorc which provides 3rd party tools capable of signing, notarizing, and stapling .app bundles and other Apple formats such as .pkg and .dmg. These tools run on Linux, and as noted above, we're already making use of them to notarize and staple our .app bundles. We're extremely excited about this project, and grateful to Gregory Szorc for all the effort he's bit into it. In the future we're looking forward to migrating our actual code signing to these tools which would (finally) allow us to retire our dedicated macOS signing machines.
mardor is a python tool to manage, and most importantly, sign, MAR files. In the days before Autograph it was used to directly sign MAR files. These days we only use it to inject signatures made by Autograph into the files, similar to our usage of osslsigncode.
signingscript is the glue between our CI system (Taskcluster) and Autograph. Through a combination of the tools listed above, custom code in signingscript itself, and communication with Autograph it produces signed builds. It is additionally responsible for notarizing and stapling our macOS builds.
iscript is essentially a pared down version of signingscript (in fact their code is both derived from our early signing server code), and is responsible for signing our macOS builds. iscript runs on a small cluster of mac minis, which are a huge pain in the butt to manage.
What a ride it's been over the last 20 years! We've gone from signing nothing to signing nearly everything in some form. Signing started off as a very manual process, and now happens seamlessly thousands of times per day.
I don't think it would be possible to name everyone that contributed to this, but it took the ideas and efforts of tens, if not hundreds, of people to get to this point: release engineers, build system experts, security folks, and many others were all critical to getting us where we are today.
I've got this post as brief as possible, but if you're interested in more details on any parts here feel free to reach out!