Today I successful landed the first part of bug 505512, which lays the ground work for catching all sorts of build problems and turning them purple, instead of red. As part of this initial work we'll now be catching most problems when cloning Mercurial repositories, turning the builds purple, and automatically retrying them.
In the next week or two I'm going to add similar behaviour for at least the following:
- Graph Server post failures
- Slave disconnections
- Sendchange failures
- out of disk issues
- CVS checkout failures (yes, we still use CVS....)
If there's other things people can think of that should be flagged as infra problems, or that should cause builds to be retried, please add them to this Etherpad: http://etherpad.mozilla.com:9000/build-infra-errors. Bonus points if you write the regular expression that catches it :-).
Currently, the purpleness is only visible on plain Tinderbox, but once bug 592340 is resolved, TBPL will support it as well.