One of the the most important systems I work is the release automation for Firefox and Thunderbird. The process behind the automation long predates me, but I’ve been deeply involved in automating, refining, and optimizing it. It shouldn’t come as any surprise that one of the biggest challenges of working on such a complex system is understanding how the smaller pieces fit together to make the whole system. For the release automation we have an advantage though: the smaller pieces are generally Buildbot Builders, and the things that fit them together are generally Buildbot Schedulers. Awhile ago I was improving parallelism for l10n repacks and found it extremely difficult to reason about whether or not my changes would actually create the desired Builders and string them together correctly. I threw together some (terrible) code that spat out a digraph of the release automation’s Builders and Schedulers. By comparing the before and after graphs I was able to iterate on some parts of my code without spending hours and hours testing.
This week I finally got around to tidying up and packaging this code as a more general purpose tool. It’s not nearly complete and has many rough edges, but as a very basic tool to help you understand non-trivial Buildbot installations, I think it’s wonderful. It’s pip installable (“buildbot-scheduler-graph”) and available on Github. Once you’ve got it, try it out with “buildbot-scheduler-graph /path/to/your/master.cfg /path/to/output-dir”. Here’s what Mozilla’s scheduler graphs looks like. What do yours look like?
I’ve been hacking on a tool on and off that needs to load Python code from badly named files (eg, “master.cfg”). To my surprise, there wasn’t an obvious way to do this. My “go to” method of doing this is with execfile. For example, this will load the contents of master.cfg into “m”, with each top level object as a key:
m = {}
execfile("master.cfg", m)
This works well enough for simple cases, but what happens when you try to load a module that loads other modules? It turns out that execfile has a nasty limitation of requiring modules that aren’t in sys.pathto be in the same directory as the file that calls execfile. You can’t even chdir your way around this, you have to copy the files you need to the caller’s directory. (We actually have some production code that does this.
Someone in #python on Freenode suggested using importlib. That seemed like a fine idea, especially after recently watching Brett Cannon’s“How Import Works” talk. Unfortunately, Python 2.7′s importlib only has a single method which can only load a module by name.
Eventually I came across a Stack Overflow post that pointed me at imp.load_source. This function is similar to execfile in that it loads Python code from a named file. However, it properly handles imports without the need to copy files around. It also has the nice added bonus of returning a module rather than throwing objects into a dict. I ended up with code like this, to load the contents of “foo/bar/master.cfg”:
>>> import os, sys
>>> os.chdir("foo/bar")
>>> sys.path.insert(0, "") # Needed to ensure that the current directory is looked at when importing
>>> m = imp.load_source("buildbot.master.cfg", "master.cfg")
Over the years we’ve had quite a few run-ins with Windows’ maximum path and command line lengths. These problems exhibit themselves with a cryptic “Bad file number” error and usually leave someone scratching their head for a bit. Due to the variable length of our build directories these problems are prone to biting us at inconvenient times – in particular, during release builds. They also have the potential to bite us on merge day or anytime cross-branch merging happens.
To prevent this from happening in the future I’m going to be landing a change to our build directory naming this week. After it lands we’ll be padding build directories with “0″ until they reach the required length. For example, “try-w32″ becomes “try-w32-0000000000000000000000″ and “rel-m-beta-w32_bld” becomes “rel-m-beta-w32_bld-00000000000″. Because all build directories across all branches (including try) will be consistent, this means that any changes that would’ve eventually failed because of path or command line length, will now fail during the original landing.
This change should be mostly invisible to developers, except for the extra characters in paths when looking at build logs.
A few weeks ago we had a very strange problem with a release. We didn’t have the time to investigate it, so we added a hack to workaround it and carried on. I’ve been slowly digging deeper into the issue this week and finally got to the bottom of it. As it turns out the root problem was that our release builds weren’t using the same “make” as the non-release builds! This is absolutely unintended, and could’ve been the source of even more, subtler errors. Eventually, we would’ve hit a problem that we couldn’t have hacked around, and would’ve had to make scary changes under time pressure to fix it.
This is a great example of why it’s important to remove hacks from your code and take the time to understand the errors you hit. You can clean your house all you want but if you don’t find out where the mess is coming from you’ll never keep up with it.
With apologies for the short notice, this post is to let you all know that 32-bit linux tests will be completely disabled on mozilla-central, mozilla-inbound, try, and all other mozilla-central-based branches for the next few days. Our B2G developers are working very hard to finish up 1.0, and need every cycle they can get on these machines to aid them in that.
A few notes, for clarity:
* 32-bit Linux builds are completely unaffected
* “make check” tests will still be run, because they run as part of the build job
* All of the disabled jobs will be re-enabled next week
Again, apologies for the short notice and we appreciate your understanding.
For as long as I can remember #build has been the Release Engineering (née Build & Release) home on IRC. We’ve learned over the years that most people who stop by #build randomly ask questions related to the build _system_, not the infrastructure. This ends up with us redirecting them to #pymake, #developers, or other channels where they’re more likely to get help. In order to help avoid this confusion we’ve decided to move RelEng to #releng and let #build be the home for the build system, likely to replace #pymake.
If you’ve got build infrastructure questions or just like to hang out with us, join us in #releng!
As I mentioned last week, we’ve been working hard to get multilocale B2G builds going. This morning we flipped the switch and turned on multilocale Gaia across the board. The existing desktop and device builds on TBPL will now include a Gaia profile with Arabic, English, Spanish, French, Brazilian Portuguese, and Mandarin Chinese (zh-TW).
Additionally, we now have new desktop builds with all of the locales listed in this file available — meaning that most people localizing B2G can now test their translations in-app. These are available for Linux, Mac, and Windows in the “localizer” packages in this directory.
A few notes:
Linux users who have run a B2G desktop build in the past may need to rm -rf ~/.mozilla/b2g before these new builds will function correctly.
Gecko is still en-US only. Expect network errors and other such things to be in en-US for now. Multilocale gecko for B2G work is being tracked in bug 817197 and the bugs that it blocks.
All of the desktop builds will be updated on a nightly basis. However, there are no automatic updates for them – you must download by hand whenever you want to test newer code or translations.
Currently, all of the b2g builds on TBPL are using a fixed set of locales which are built into the Gaia repository. These were OK at first, but we’re at the point now where we need to be able to test the languages that we’ll be initially shipping, as well as provide some way for localizers to test out their work. For these reasons, the following changes will be made to the B2g desktop and device builds:
* Unagi, otoro, panda, and the current desktop b2g builds will include 6 locales (instead of the 4 they currently do): Arabic (ar), English (en-US), French (fr), Spanish (es), Brazilian Portuguese (pt-BR), and Mandarin Chinese (zh-TW). These builds are intended for developer consumption and help to test out a wide array of features (rtl languages, languages with long strings, unicode characters, etc).
* We will be adding new desktop b2g builds that contain all languages that Gaia is available in (see https://github.com/mozilla-b2g/gaia/blob/master/shared/resources/languages-all.json for the full list). These builds are intended for localizers to see how their translations look and feel.
These changes should be taking place sometime this week. Big thanks to Staś Małolepszy for adding support for this to the Gaia build system.
Note that for now, the Gecko portions (for example, network error pages in the Browser) will not be localized. We will be enabling localization for Gecko as soon as we can, but it’s not quite ready yet.
It’s not uncommon to see unit test failures, performance issues, or build problems that seem to happen exclusively on our pool of build & test machines. If you’ve ever been plagued by one of these you know what a pain in the butt it is to debug such a thing. Because of this, RelEng is always willing to loan out slaves to anyone who needs to do on-machine debugging. All you need to do is file a bug! These requests can usually be turned around in the same business day. If you think having access will save you time, do not hesitate – file today!
I’m an avid fisherman, a beginner gardener and in general someone who loves to do things for himself. Hunting is something I’ve wanted to try for a very long time (minus the 5 years that I was a vegetarian), and this year I finally got around to getting the proper licenses and doing it.
Preparation
Here in Ontario it’s not a trivial process to be licensed to hunt with a gun. One must first pass the RCMP’s course for unrestricted firearms, then pass the Ontario Hunter Education Program, and _then_ apply to multiple government agencies for licenses. The background check for getting a gun license is at least as extensive (and probably more) than for getting a passport. The hunting license is pretty trivial once you have that, but there’s an array of licensing options to sort through. I took my training courses back in May only finished getting all of my licenses in mid-October — two weeks before deer season opened. (My experience was a bit non typical because I needed an extra medical clearance, I expect it’s a bit quicker for folks who don’t.)
After getting my licenses it was time to gear up. There’s a lot you need to be properly equipped to hunt. In the end I purchased:
Sweat whisking pants/shirt
Long, cotton underwear+top
Two pairs of thick socks
Blaze orange hat/vest
Two-layer camo jacket
Camo gloves
Thin undergloves, for extra warmth
Multitool
Hunting knife
Compass
Small maglite
…and still had to borrow waterproof camo pants, a rifle and misc. things (ground blind, chair, gun stand, etc.) in order to be fully equipped. I decided to buy a gun for myself this year to get some more hands on time with my friend’s gun to figure out my preferences a bit. Excluding the course/license fees I spent about $250 on gear.
I set aside 3 days for hunting this year, and one for getting comfortable with the gun I’d be using. I rolled into town Sunday afternoon and spent some time with my friend’s Remington 700, 7mm rifle. It had already been sighted-in so all I needed to do was get comfortable with it. The first thing that was surprising to me was how heavy it was — a 7mm is NOT a small gun. I don’t know the exact weight with the scope but even in a seated position with a gunstand it was impossible for me to hold it still. With that said, I was surprised at how easy it was to shoot and at my accuracy (at a paltry 50 yards). With a 3 shot clip I had about a 3″ spread — for a pro that would be terrible but given that it was my first time shooting a high powered rifle I was quite pleased. I later learned that a 7mm is generally considered hugely overpowered for deer hunting, so next year I’ll likely buy something more reasonable and lighter. I shot off a couple of clips without issue, but on the last shot I managed to get whacked by the scope from the recoil. I had a been warned this would happen to be at least once, and then I would get laughed at — both of those were true.
This picture doesn’t do the wound justice — it bled for 15min and stayed red for days.
The Hunt
The person that I was hunting with is good friends with some folks who own a good portion of land in a rural area of eastern Ontario. They were kind enough not only to let us hunt on their land, but to feed us a huge breakfast every day.
In Ontario, you may hunt between 30min prior to sunrise until 30min after sunset. Factoring in time to shower, get ready, drive over, walk out & get comfortable this meant that we were getting up at 4am and leaving the house at 5. Each day we were sitting down in the blind before 6am and ready to load our guns as soon as it was legal to (you can’t load your gun until 30min before sunrise, either). It was below zero every day but other than the tips of my toes and fingers I stayed quite warm by wearing about 5 layers of clothing.
All three days of hunting went pretty much the same. We walked in prior to hunting time, made calls until 9am or so, and then headed in for breakfast. We’d come back out around 10/10:30 and spend the afternoon watching the birds, napping on and off, and just relaxing. (Let me tell you, if you’ve never had a nap while bundled up at -3C in the middle of the woods, you’re missing out. It is freaking amazing.) Around 3pm we’d start focusing again, as dusk is when the deer will start to become more active again. We’d sit as quietly as possible, make calls, and do whatever else we can to bring the deer in. On the final day we found some fresh tracks in the afternoon and were very hopeful that it would return that night. Unfortunately, each day ended without even seeing a deer.
Going in I expected to be very disappointed if I came out empty handed, but that was not the case at all. Even though I regularly fish, have a yearly garden, and occasionally hike I haven’t felt this close to nature since I was a cub scout. Spending nearly 12h just sitting, observing, and thinking about nature gives me a new level of appreciation for it. Whether you hunt or not, I can’t recommend the experience of observing nature enough.
