Some of you may have noticed bug 448616. It turns out that we have not had proper breakpad symbols for OS X builds *ever* on 3.1 builds. This caused our OS X crash reports to be sigificantly less useful We've had a heck of a time isolating the problem. It took quite a few manual builds before we could even reproduce the problem. There were a few rounds of diagnosis involved before the problem became clear: Running 'make package' before 'make buildsymbols' breaks OS X symbols. It seems that 'make package' on Mac strips objdir/ppc/dist/universal, which is where we generate our symbols from! With that knowledge in hand I deployed a fix....and unwittingly broke Linux nightlies. As part of my fix, I had changed around the order in which we do symbols, updates, and packaging. Doing so had tripped another weirdness in our build system: On Linux, 'make -C tools/update-packaging' (creating a full mar) *must* be done after 'make package'. On Mac and Windows we're able to create a mar at any point, however. So, the correct order to do post-build processing in is thus: Symbols, Packaging, Updates. This fix is now deployed in production and nightly builds have been re-spun. The latest Mac nightly of 3.1 should have proper breakdpad symbols (and in turn, make our crash reports useful). (Unfortunately, due to bug 454198 line numbers are only available in the raw dump, but still, this is a big improvement.) Big thanks to Lukas and Ted for all their help here. Any regressions due to this should be filed in mozilla.org:Release Engineering. As a side note, we ended up upgrading to Xcode 3.1 in a failed attempt to fix this bug. Included with it is GCC 4.2, which sadly bails out very early in our build process.