Upcoming changes to nightly update generation

Don't panic. If all goes well nothing will change from a nightly-build-user perspective. We are going to be moving to a more sane system for the generation of nightly .mar files and AUS2 snippets. Details below, but first, some background.

Rob Helmer talked a lot about AUS and updates, mostly regarding releases. What he did not mention the silly path that our nightly updates take.

All of our nightly updates currently take the following path:

  1. Nightly build happens - this includes a complete MAR and complete AUS snippet
  2. Cronjob on a specific build machine performs some magic and generates a partial MAR and partial AUS snippet

Once these changes land our 3.1 builds will take the following path:

  1. Nightly build happens - this includes complete AND partial MAR, complete AND partial AUS snippet

(We could probably support Tinderbox driven builds (2.x, 3.x) pretty easily too, if someone wants to write the patch.)

I think it's pretty obvious that this is a more sensible way to do things. As it stands now if we lose the VM that generates partials we lose all nightly updates. The hidden benefit here is that updates for releases and nightlies will have fewer differences between them. This means that problems with that system will be caught in the nightlies and not during a live release. (NB: We will still need to do snippets for older than n-1 builds, eg. 3.0->3.0.2 during the release process).

I'm still doing testing of this but I hope to land these changes in mozilla-central/tools/update-packaging by the end of next week.

Special note for build folks from SeaMonkey and Thunderbird: I will be replacing the CreateCompleteUpdateSnippet and snippet uploading ShellCommands with a couple Makefile targets - you may want to do the same.

Useful symbols for OS X builds!

Some of you may have noticed bug 448616. It turns out that we have not had proper breakpad symbols for OS X builds ever on 3.1 builds. This caused our OS X crash reports to be sigificantly less useful We've had a heck of a time isolating the problem. It took quite a few manual builds before we could even reproduce the problem. There were a few rounds of diagnosis involved before the problem became clear: Running 'make package' before 'make buildsymbols' breaks OS X symbols. It seems that 'make package' on Mac strips objdir/ppc/dist/universal, which is where we generate our symbols from!

With that knowledge in hand I deployed a fix....and unwittingly broke Linux nightlies. As part of my fix, I had changed around the order in which we do symbols, updates, and packaging. Doing so had tripped another weirdness in our build system: On Linux, 'make -C tools/update-packaging' (creating a full mar) must be done after 'make package'. On Mac and Windows we're able to create a mar at any point, however.

So, the correct order to do post-build processing in is thus: Symbols, Packaging, Updates.

This fix is now deployed in production and nightly builds have been re-spun. The latest Mac nightly of 3.1 should have proper breakdpad symbols (and in turn, make our crash reports useful). (Unfortunately, due to bug 454198 line numbers are only available in the raw dump, but still, this is a big improvement.) Big thanks to Lukas and Ted for all their help here. Any regressions due to this should be filed in mozilla.org:Release Engineering.

As a side note, we ended up upgrading to Xcode 3.1 in a failed attempt to fix this bug. Included with it is GCC 4.2, which sadly bails out very early in our build process.

Mozilla Scheduled Downtime for Tinderbox Moving - 07/04/2008, 5am - 10am PDT (approx)

During this time we will be moving around tinderboxes (as discussed in the "Moving Tinderboxes" thread). After this maintenance is done, the following changes will have been made:

  • All Firefox 3.0.x/Gecko 1.9.0.x/CVS trunk tinderboxes will report to the soon-to-be-created "Firefox3.0" tree. (http://tinderbox.mozilla.org/Firefox3.0/)

  • All Firefox 3.1/Gecko 1.9.1/mozilla-central tinderboxes will report to the "Firefox" tree.


  • The 'Mozilla2' tree will be deleted.

VMware ESX

As has been previous documented, we have a lot of VMs for Release Engineering purposes. We often need to start, stop, edit, or do other management things to them. For a long time this meant using the Virtual Infrastructure client, which is...quite buggy, and only runs on Windows. Awhile ago I discovered the VMware Web Interface, which let me start and stop VMs. Just now, I realized that I can actually edit settings, too. This means that I no longer have to start up a Windows VM to do 99% of the things I need to.

VMware, you're web interface kicks ass. Now give me an MKS plugin for Firefox on Mac and I'm all set!

You! Can help make the Try Server better.

Once or twice a month a bug is filed on the try server. I see all of these go by and typically say something like "this would be cool, I don't have time to do it". Someone mentioned to me that they were going to blog about a specific try server to try and get some traction on it and I thought, "Why didn't I think of that?". So, the rest of this post is just a list of try server bugs that are relatively simple try server bugs that shouldn't be too difficult to fix.

Bug 391364 - Tryserver should have the option of building specified platforms

Requires changes to: CGI script/UI, master.cfg/Buildbot steps

Bug 411735 - Client side form validation

Requires changes to: CGI script/UI

Bug 411735 - Support building from an alternate revision/branch/tag in a mercurial based try server build

Requires changes to: CGI script/UI, master.cfg/Buildbot steps

Bug 430942 - If bug # given, automatically add comment to bugzilla when try build finishes

Requires new Buildbot Status plugin

Bug 431375 - Try server needs to be able to handle non-Firefox builds

Requires changes to: CGI script/UI, master.cfg/Buildbot steps

Bug 431400 - Try server should support downloading bugzilla attachments as well as receiving uploaded patches

Requires changes to: CGI script/UI

I am happy to give guidance to and assist anyone who wants to work on any of these bugs. The CGI/UI can be found here: http://lxr.mozilla.org/mozilla/source/webtools/buildbot-try, the buildbot master.cfg is here: http://lxr.mozilla.org/mozilla/source/tools/buildbot-configs/tryserver/, and classes supporting it are here: http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/tryserver/.