One of the the most important systems I work is the release automation for Firefox and Thunderbird. The process behind the automation long predates me, but I've been deeply involved in automating, refining, and optimizing it. It shouldn't come as any surprise that one of the biggest challenges of working on such a complex system is understanding how the smaller pieces fit together to make the whole system. For the release automation we have an advantage though: the smaller pieces are generally Buildbot Builders, and the things that fit them together are generally Buildbot Schedulers. Awhile ago I was improving parallelism for l10n repacks and found it extremely difficult to reason about whether or not my changes would actually create the desired Builders and string them together correctly. I threw together some (terrible) code that spat out a digraph of the release automation's Builders and Schedulers. By comparing the before and after graphs I was able to iterate on some parts of my code without spending hours and hours testing. This week I finally got around to tidying up and packaging this code as a more general purpose tool. It's not nearly complete and has many rough edges, but as a very basic tool to help you understand non-trivial Buildbot installations, I think it's wonderful. It's pip installable ("buildbot-scheduler-graph") and available on Github. Once you've got it, try it out with "buildbot-scheduler-graph /path/to/your/master.cfg /path/to/output-dir". Here's what Mozilla's scheduler graphs looks like. What do yours look like?