Update on recent Tinderbox issues

My last post talked about the issues we've been having with load on the Tinderbox server and some ways we could fix it. I'm happy to report that two things were completed yesterday that should keep the load under control for the foreseeable future.

One of the things mentioned in my previous post, splitting incoming build processing from the rest of Tinderbox (bug 585691), was completed very late last night. Additionally, Nick Thomas discovered that we had lost the cronjob that takes care of cleaning out old builds from Tinderbox's memory. That script was re-enabled and a one time clean up removed 64GB of old build data. Both of these were completed around 4am PDT this morning and load is looking much better.

Especially because we're now running cleanup scripts on a regular basis again, I believe that this should get as back to good state.

Everyone should feel free to send justdave their thanks for staying up reaaaaallllly late last night to get us back to a good state.