Yesterday marked a big milestone for the Balrog project when we made it live for Firefox and Thunderbird Beta users. Those with a good long term memory may recall that we switched Nightly and Aurora users over almost a year ago. Since then, we've been working on and off to get Balrog ready to serve Beta updates, which are quite a bit more complex than our Nightly ones. Earlier this week we finally got the last blocker closed and we flipped it live yesterday morning, pacific time. We have significantly (~10x) more Beta users than Nightly+Aurora, so it's no surprise that we immediately saw a spike in traffic and load, but our systems stood up to it well. If you're into this sort of thing, here are some graphs with spikey lines: The load average on 1 (of 4) backend nodes: The rate of requests to 1 backend node (requests/second): Database operations (operations/second): And network traffic to the database (MB/sec): Despite hitting a few new edge cases (mostly around better error handling), the deployment went very smoothly - it took less than 15 minutes to be confident that everything was working fine. While Nick and I are the primary developers of Balrog, we couldn't have gotten to this point without the help of many others. Big thanks to Chris and Sheeri for making the IT infrastructure so solid, to Anthony, Tracy, and Henrik for all the testing they did, and to Rail, Massimo, Chris, and Aki for the patches and reviews they contributed to Balrog itself. With this big milestone accomplished we're significantly closer to Balrog being ready for Release and ESR users, and retiring the old AUS2/3 servers.