Apple recently announced changes to how OS X applications must be packaged and signed in order for them to function correctly on OS X 10.9.5 and 10.10. The tl;dr version of this is “only mach-O binaries may live in .app/Contents/MacOS, and signing must be done on 10.9 or later”. Without any changes, future versions of Firefox will cease to function out-of-the-box on OS X 10.9.5 and 10.10. We do not have a release date for either of these OS X versions yet.
* Move all non-mach-O files out of .app/Contents/MacOS. Most of these will move to .app/Contents/Resources, but files that could legitimately change at runtime (eg: everything in defaults/) will move to .app/MozResources (which can be modified without breaking the signature): https://bugzilla.mozilla.org/showdependencytree.cgi?id=1046906&hide_resolved=1. This work is in progress, but no patches are ready yet.
* Add new features to the client side update code to allow partner repacks to continue to work. (https://bugzilla.mozilla.org/show_bug.cgi?id=1048921)
* Create and use 10.9 signing servers for these new-style apps. We still need to use our existing 10.6 signing servers for any builds without these changes. (https://bugzilla.mozilla.org/show_bug.cgi?id=1046749 and https://bugzilla.mozilla.org/show_bug.cgi?id=1049595)
* Update signing server code to support new v2 signatures.
We are intending to ship the required changes with Gecko 34, which ships on November 25th, 2014. The changes required are very invasive, and we don’t feel that they can be safely backported to any earlier version quickly enough without major risk of regressions. We are still looking at whether or not we’ll backport to ESR 31. To this end, we’ve asked that Apple whitelist Firefox and Thunderbird versions that will not have the necessary changes in them. We’re still working with them to confirm whether or not this can happen.
This has been cross posted a few places – please send all follow-ups to the mozilla.dev.platform newsgroup.
In my previous post I said that Mac signing had been turned on, and that it would stay on. Unfortunately, the following morning we caught some issues that only came up after updating. It took a bit of time to resolve those, but as of this morning we got them worked out, and signed mac builds are back. There are a couple of follow-up issues to address, but we’re in a good enough state to backport signing to Aurora and Beta, and ship it in Firefox 13.0. On Saturday evening I’ll be enabling it on Aurora, and on Monday morning on Beta – assuming no new issues come up.
If you see ANY issues related to updating on Mac please file a bug and cc me (:bhearsum). It’s very important that any issues are brought up immediately, as we intend to ship this to the release channel on June 5th.
Huge thanks (again) to Steven Michaud, Ted Mielczarek, Erick Dransch, and anyone else that helped make this happen.
Last week I talked about our plans around Mac build signing. In it, I said that I intended to have signed dep/try builds by the end of last week and signed Nightly builds early this week. Unfortunately, shortly after publishing I realized that due to some complicated infrastructure reasons, we couldn’t turn on dep/try build signing prior to Nightly. However, I’m pleased to announce that all the necessary work for Mac builds signing is landed and in production. From now on, mozilla-central based builds will be signed. As other branches merge the necessary patch, they will be signed too. Nightly updates have been disabled for now, to give QA a chance to verify everything. Nightly updates should be re-enabled sometime tomorrow. This should be an invisible change to everyone, but please file bugs on any issues (especially those related to new installs or updates) and cc me (firstname.lastname@example.org).
Additionally, I want to correct one thing in my previous post. I said that applications ‘will not run on 10.8 unless the user has allowed “applications downloaded from anywhere” to run’. However, it has come to my attention that this isn’t entirely true. You _can_ except specific applications from this policy if you ctrl+click the application and hit “open”. This works for unsigned apps, self signed apps, and those signed with a Mac Development certificate.
A few weeks ago a new Developer Preview of OS X 10.8 was released and it was discovered that as things stand now, Firefox will not run on it. With the current default settings, 10.8 will not allow any software to run unless it’s signed with an Apple Developer ID (essentially, a certificate issued by a particular Apple Root CA). We don’t know exactly when 10.8 will be released to the public but some have speculated that it could be as early as the week of June 11th at WWDC 2012. We must have a signed and released Firefox out there before the general public starts upgrading and we’ve been working hard to make that happen as soon as possible. This post will give a short history of Mac signing at Mozilla and talk about our timeline for enabling it.
Code signing of Mac builds has been on our radar for a long time. Bug 400296 was originally filed in 2007. In late 2010 Syed Albiz did a ton of great work figuring out the Apple tools and how we can integrate them into our automation. That work didn’t quite get finished before his internship was completed and the bug stagnated for some time afterwards. At the start of this year there was renewed energy when Erick Dransch picked up the bug. We attempted to land his work and enable signing on nightlies in mid-April, but that ended up bouncing due to some conflicts with our upgrade to 10.7-based build machines. Erick’s internship expired before everything could be fixed up, and the bug fell to me.
After gaining access to Mozilla’s Apple Developer account on Monday there was a lot of early iteration before we got to the point where we could sign a build in a way that Mountain Lion liked. There’s multiple certificates types that one can get from Apple (“Development Certificate”, “Mac App Certificate”, “Developer ID Certificate”) and multiple versions of OS X and XCode (each with their own quirks) that one can sign with. Mostly thanks to Steven Michaud’s knowledge and assistance we figured out exactly what combination of these we’ll need to use to have signed Firefox builds that work everywhere.
Where we’re at now
At this point in time we’ve got all the tools we need to sign all Mac Firefox builds. The only blocking issue at this point is figuring out access restrictions to our Apple Developer Account, so that we can generate our final Developer ID certificates.
Like with Windows Authenticode signing we will have 3 different certificates for different types of Firefox builds. Dep and try builds are at the lowest level of trust and have no regular users and therefore will be signed with a self signed certificate. This means that they will not run on 10.8 unless the user has allowed “applications downloaded from anywhere” to run (which is not the default). Nightly and Aurora are at an elevated level of trust and have a userbase. These will be signed with their own Developer ID certificate. Finally, Beta and Release builds are at the highest level of trust and oversight, and represent the majority of our users. They will be signed with a separate Developer ID certificate. From a user standpoint, Nightly, Aurora, Beta and Release will all look the same but using separate certificates gives us some degree of isolation in terms of certificate revocation.
I intend to have dep and try builds signed by the end of the week. After we figure out the access restrictions to our Developer Account we will turn on signing of Nightly builds, hopefully early/mid next week. After letting those settle for a day or two we will turn on signing of Aurora and Beta builds, hopefully by the end of next week.
If you’re interested in the technical details of signing Mac builds Erick wrote an excellent blog post detailing the trials and tribulations of writing tools around them.
Our intern Erick has been doing some great work reviving, polishing and finalizing the patches that will allow us to start signing OS X builds (more to come on that in his blog!). When we do start signing them we’re planning to use our existing set of code signing certificates for them rather than buy new ones. I thought it would be a simple task to convert them so I set off to convert our internal, self-generated ones. After hours and hours of head scratching and frustration I learned that some versions of Microsoft’s “makecert” tool are broken, and generate invalid PKCS7 certs that openssl can’t cope with properly. From the OpenSSL PKCS#12 FAQ:
Q. What are SPC files?
A. They are simply DER encoded PKCS#7 files containing the certificates. Well they are in the newer versions of the tools. The older versions used an invalid PKCS#7 format.
The end result of all my attempts ended up being a PKCS#12 certificate that Apple’s codesign tool claimed couldn’t be used to do code signing.
After finding that FAQ, I decided to try to convert our Nightly code signing certificate instead. Following the great instructions found on Marc Liyanage’s blog I managed to successfully convert the certificate, import it into a Keychain, and successfully sign something! Here’s the shortened version of what I did. Note that it requires the PVK tool found here:
~/pvk.exe -in Nightly.pvk -out Nightly.key.pem
openssl pkcs7 -inform der -print_certs < Nightly.spc > Nightly.cert.pem
openssl pkcs12 -export -inkey Nightly.key.pem -in Nightly.cert.pem -out Nightly.p12
I hope this helps someone else avoid the same frustration!