This week in Mozilla RelEng – February 21th, 2014
Highlights:
- Rail spent some time optimizing spot instance bidding, which helps lower our AWS bill.
- Massimo improved one of our major AWS usage reports, which will also help us be more efficient with our usage.
- Ahal got us uploading qemu.logs to blobber, making it easier to debug some types of failures.
- Mike got his patch for "hg purge" landed, which has improved try build times!
- Callek moved all of our in-house test machines onto in-house masters, which takes additional load off of our AWS VPN link (which reduces the chance of tree closures due to it failing).
- Dan continued his work on moving MozBase tests out of the build jobs, which improves build and test turnaround time.
- John started figuring out how to make our signing servers support 64-bit Windows binaries, which is one thing (out of many) that we need to do before shipping 64-bit Windows builds.
- Catlee continued work on Jacuzzi implementation, making good progress on some of the Buildbot implementation that we need.
- Nick made additional progress on fixing our release automation to support shipping Release builds to Beta users for extra testing before we ship for real.
- Buildduty
- mac-signing4's nagios is having issues
- Report impaired AWS non slave machines
- upload a new talos.zip to capture recent talos changes
- Add swap to linux build machines with
- builds-4hr.js.gz not updating, all trees closed
- Please upload mozfile-1.1 to http://pypi.pub.build.mozilla.org/pub
- some test machines unable to connect to their masters
- All trees closed due to timeouts as usw2 slaves download or clone
- General Automation
- Need helix-eng builds for test automation
- Rooting analysis mozconfig should be in the tree
- Periodic PGO and non-unified builds shouldn't be running again on pushes that already have them
- Implement ghetto "gaia-try" by allowing test jobs to operate on arbitrary gaia.json
- Make Pine use mozharness production & limit the jobs run per push
- Save qemu.log to blobber
- decommission final set of KVM buildbot masters back to IT
- Compression for blobber
- Temporarily revert the change to m3.medium AWS instances to see if they are behind the recent increase in test timeouts
- Do our own hg purge
- Lots of Command Queue nagios alerts from new AWS masters
- Loan Requests
- Please loan t-xp32-ix- instance to dminor
- Loan glandium a m3.medium test slave
- Loan jgilbert a Linux M1/R test slave
- Need a slave for bug 818968
- Loan Win8 test slave to Jim Mathies for soft keyboard investigation work
- Slave loan request for a tst-linux64-ec2 vm
- Other
- Platform Support
- release-automation
- Releases
- Releases: Custom Builds
- Repos and Hooks
- Tools
- Balrog: Backend
- Buildduty
- Setup tegras that are returning from loan
- Deploy python 2.7.3 to all build machines
- rewrite watch pending to cope better with spot requests that aren't being fulfilled
- General Automation
- mozilla-central should contain a pointer to the revision of all external B2G repos, not just gaia
- Don't reset mock environments if we don't have to
- Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
- Schedule multimedia b2g mochitests for emulator-jb on Cedar
- Provision enough in-house master capacity
- Provide B2G Emulator builds for Darwin x86
- Do debug B2G desktop builds
- keep buildbot master twistd logs longer
- [tracking bug] Run desktop unittests on Ubuntu
- Add support for add-if-not instruction added by bug 759469 to the mar generation scripts
- Configure holly like Aurora properly
- [Meta] Some "Android 4.0 debug" tests fail
- b2g build improvements
- Run mozbase unit tests from test package
- fx desktop builds in mozharness
- Move Firefox Desktop repacks to use mozharness
- add --dump-config and --dump-config-hierarchy to mozharness
- port MockReset to mozharness
- allow for multiple basedirs to be passed to PurgeMixin.purge_builds()
- allow mozharness's tbox_print_summary() to be used outside of unittests
- Use spot instances for regular builds
- Create SpiderMonkey builds for Windows on TBPL again
- Do nightly builds with profiling disabled
- Windows ix test machines don't always start buildbot
- Don't clobber the source checkout
- Add Linux32 debug SpiderMonkey ARM simulator build
- aws sanity check shouldn't report instances that are actively doing work as long running
- Allow the possibility of spidermonkey builds on more platforms
- [tarako][build]create "tarako" build
- move off of dev-master01
- Schedule JB emulator builds and tests on cedar
- [tracking bug] migrate hosts off of KVM
- Run jit-tests from test package
- Intermittent Linux spot builder "command timed out: 2700 seconds without output, attempting to kill" while trying to install mock
- Loan Requests
- Loan felipe an AWS unit test machine
- need a loaner bld-linux64-ec2 instance
- Slave loan request for Matt Woodrow
- Loan an ami-6a395a5a instance to Aaron Klotz
- Please lend Andreas Tolfsen a linux64 EC2 m3.medium slave
- Please loan dminor instance to build Android 2.3 Emulator
- Loan :sfink a linux64 bld-centos6-hp-005
- Slave loan request for a tst-linux64-ec2 machine
- Other
- Platform Support
- Migrate ESR branches to win64-rev2
- Setup in-house buildbot masters for remaining in-house testers
- signing win64 builds is busted
- Windows slaves often get permission denied errors while rm'ing files
- Deploy ndk-stack on foopies
- do initial planning on "chunks" of machines to move.
- slave pre-flight tasks
- Deploy patched version of Mesa 8.0.4
- release-automation
- Support modifying update-settings.ini when doing update verify
- Create SSL products in bouncer as part of release automation
- Figure out how to offer release build to beta users
- cache MAR + installer downloads in update verify
- Releases: Custom Builds
- Repos and Hooks
- Tools
- [Tracking bug] - Assisted/Auto Landing from Bugzilla to tip of $branch
- Store the timing of actions in slaveapi
- Move trychooser hg extension into the new https://hg.mozilla.org/hgcustom/version-control-tools repo
- Use MozillaPulse from pypi
- cut over build/* repos to the new vcs-sync system
- slaveapi still files IT bugs for some slaves that aren't actually down
- Replace dev-master01 for dev-master1 in our repos