Buildbot + Mozilla

Over the past year or so Mozilla has increasingly relied on Buildbot for our Building, Releasing, Testing, and l10n processes. For awhile, everyone that touched Buildbot did things their own way. As a result we had a ton of code (and bug) duplication. There is now a heavy push within the Release Engineering group to rectify this. We've slowly but surely been moving code into our "buildbotcustom". This module contains many different things that are not appropriate to put in upstream-Buildbot, but are very appropriate and useful to those working with Mozilla. We've done our best to keep things as generic as possible. Some things (tryserver and unittest) were moved as-is into buildbotcustom to make it easier to generalize them in the future.

The rest of this post is aimed towards people who want to build Mozilla products with Buildbot and talks about some of the custom classes in more detail.

Change Sources

TinderboxPoller

TinderboxPoller watches a Tinderbox quickparse file (such as http://tinderbox.mozilla.org/Firefox/quickparse.txt) for new builds from a specific builder (each column on the Tinderbox waterfall is a builder). When a new build is detected it kicks off a "build" (which isn't necessarily a compilation process) on the specified "branch" (which in this case is simply a string that matches the TinderboxPoller to a specific builder or builders. For example, we have a TinderboxPoller that watches fx-linux-tbox for new builds. Whenever it sees one it kicks off Talos runs ("builds") on a number of different machines. To the best of my knowledge, the only thing we use this for is Talos - but there may be things I'm unaware of.

Build Steps

TinderboxShellCommand

This is a BuildStep that simply executes a regular Buildbot ShellCommand and ignores the result. Those familiar with Tinderbox client will understand the naming of it.

GetBuildID

This step attempts to pull the BuildID from the objdir of a build and set it as a BuildProperty. It does so through printconfigsetting.py and application.ini. If you are using an objdir you must pass the path to it, relative to the builddir. If you are not using one you do not need to pass anything. Once this step is run you can refer to it through WithProperties.

SetMozillaBuildProperties

This step does the job of GetBuildID and more (oh so much more). However, it depends on a slave-side command that is only in Mozilla's Buildbot tree. To use it, you must be using Buildbot checked out from mozilla/tools/buildbot. (Technically, you could copy and paste the slave command into an upstream Buildbot, but let's keep it simple). In addition to gathering the BuildID and setting it as a BuildProperty it does the same for the following: appVersion (from application.ini, falling back on platform.ini), the filename, size (in bytes), and sha1 hash of the package, installer, and complete mar are gathered. Please see the in-code documentation for the names of the BuildProperties. Obviously, this step should not be run until the package and/or installer and/or complete mar are created, if you care about their metadata.

As with GetBuildID it needs to know where the objdir is rooted (relative to builddir). Again, if there is no objdir you can simply ignore this parameter.

MozillaStageUpload

(I'm not sure how useful this is to non-Firefox projects as it contains some business logic that may be specific to Fx, but it performs an important function so I figure I'll include it here.)

This is a bit of a kludgy class as it ends up generating one huge line of bash that gets executed. It was written to be a replacement for the

Tinderbox client 'pushit()' function. It's job is simple: upload packages and mars to a server (for example, stage.mozilla.org). Because it was written to replicate some Tinderbox logic it's got a lot of different options that I don't need to go through here. The in-code documentation for it is quite complete and available here: http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/steps/transfer.py

One note: your BuildSlaves will need passwordless logins to remoteHost with the ssh host key accepted, or else this step will hang. You can test this by doing 'ssh username@remoteHost' on your slaves.

In the coming weeks I intend to write a few more posts about Mozilla's Buildbot usage.

Nightly updates on Mozilla2 builds.

As of today, anyone running a mozilla-central nightly build (all zero of you) will start receiving nightly updates for it. There isn't a whole lot going on there right now but this is another step is getting 1.9 parity on Mozilla2. Many many thanks to nthomas for teaching me so much about the update system while getting this going.

At some point in the future we plan to enable updates for other branches as well, such as actionmonkey.

Try Server Symbol Server

As of earlier today, the try server began uploading windows debugging symbols to a symbol server. For those of you interested in debugging your Windows try server builds you can follow the same instructions listed on devmo replacing http://symbols.mozilla.org/firefox with http://build.mozilla.org/tryserver-symbols. (For posterity, this information is also recorded on the try server wiki page. Any problems discovered should be filed (as always) in Webtools:Try Server.

Mozilla2, Linux 64-bit, and Mobile

Over the past month there's been some new builds set-up. We now have Mobile and Mozilla 2 tinderboxes, as well as Linux 64-bit builds.

Mobile builds are done on Linux ARM, inside of Scratchbox. For more details about the Mobile platform setup, see the ref platform doc. Builds are not yet being published, but we hope to get them going soon. Details are in bug 418852.

We have very very basic Mozilla2 infrastructure up. Linux and Windows builds are going, Mac is pending. Currently, actionmonkey and mozilla-central are built. There is no unit or perf testing as of yet. You can find these builds here and here. These builds are labeled as Firefox 4.0a1pre to avoid confusion with the current CVS trunk but that's the only reason they are named as such. We are a ways off from Firefox 4 and these builds should not be considered as previews of it. I've seen some blog posts about "Firefox 4 Alpha 1" being released and I want to stress that this is NOT the case. The 'pre' in the version strings means 'prerelease' - aka. not yet released. For all of the nitty gritty details on Moz2 builds, see this tracking bug.

Last but not least, Linux 64-bit builds have been set-up. They are built on CentOS 5.0 with GCC 4.2.3 and follow the same mozconfig as 32-bit dep/nightly builds. You can see them on the MozillaExperimental tinderbox for now, but they should be pushed to the Firefox tree by the end of the week. You can find the hourly builds here and the nightly builds here (the same place as 32-bit Firefox builds). These builds are being done primarily to "catch bugs in visibility handling" and make sure we don't break x86_64. At this time, there is no plans to do 64-bit releases.

Try Server Updates

There's been some small but important updates to the Try Server the past week. Here's a list!:

Linux Reference Platform RC2

Thanks to those people that downloaded and RC1 of the Linux ref platform. There was one small bug found (the wrong GCC version is used), so I've spun an RC2. As before you can find it on ftp.mozilla.org.

In much more exciting news, Chris Tyler made available a KVM image of the ref platform. An image should be available on ftp.mozilla.org very soon, but for now here's instructions on how to convert the VMware image to KVM format.

There was a request to host a torrent of this image in my last post. Due to the infrequency of these releases, relatively low interest, and the fact that we don't have any existing infrastructure for hosting torrents I've decided against doing this.

As before, any bugs found should be reported in mozilla.org :: Build & Release.

Happy building!

Stuck on Build

As was announced at the Weekly Meeting yesterday my permanent, full-time position is now in Build.

I'm pretty excited by this move. The first thing I did for Mozilla was write a couple of Buildbot plugins. I've mostly been doing Build work since then, first as schoolwork, then as a contractor, and now as an employee. When I got hired for QA I was happy with the change, and enjoyed all of the work I did there (despite Talos causing me sadfaces at times).

I spent the greater part of November and December on loan to Build while they were short-handed. When I was given the opportunity to have a full-time Build position it took some consideration but I did realize that I preferred the work here. (There's been some jokes about me having masochistic tendencies.....given our build system it's hard to argue with that!)

In the next few months I'll likely be focusing on (but not limited to): trunk release automation polish, try server Talos, nagios for the Build network, and various Buildbot related things. I feel like the de-facto owner for the Linux Reference Platform, so I will be taking care of that. (Although, I don't expect another release anytime soon.) And along with Ted, I will be taking care of MozillaBuild. I'm sure people will find other things for me to do, too ;-).