This week in Mozilla RelEng – May 16th, 2014
Major highlights:
- Aki is working on changing how we chunk our unittest builders. Because of some limits in the way we currently scale with Buildbot, this has became extremely important.
- Kim enabled a bunch of new Firefox for Android tests on emulators. These will let us move some on device tests into more scalable and reliable solutions.
- Andrew moved a ton of Mozharness test configs in-tree. Having them there reduces a lot of pain by giving us greater flexibility in branch specific configuration, and making it easier for test changes to follow code merges.
- Catlee changed our job scheduling in Buildbot to coalesce a maximum of 3 jobs together. This should help reduce the bisection windows when bad pushes are made. He wrote a great blog post detailing it here.
- Anhad (our first intern of the year), started detailing a plan to build a partial MAR generation service that we've wanted for a long time. When completed, it will allow us to serve many more users more efficient updates.
- Buildduty
- No Tarako Build Today on 5/15/2014
- Please block/disable/stop updates to today's Thunderbird nightly builds
- running out of disk space during linux64_gecko-debug jobs
- upload https://github.com/vitillo/python_cache_flusher to http://pypi.pub.build.mozilla.org/pub
- 10.5 hour wait time for mtnlion try test builds due to around 50 mtnlion slaves breaking last night
- Respin B2G Trunk Hamachi Builds Due to Critical Regressions from Bug 906164
- A lot of XP machines out of action
- General Automation
- The b2g desktop /latest/ directory contains way out of date builds
- Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
- nuke lockfiles in .repo dir in b2g_build.py
- upload mars for other b2g device builds that need updates
- upload flame gecko/gaia mars to public ftp
- make it possible to post b2g mar info to balrog
- Mozharness test configs should live in-tree
- Jetpack jobs failing with "IOError: [Errno 13] Permission denied: '/etc/instance_metadata.json'"
- Don't do Thunderbird builds on comm-* branches for non-Thunderbird pushes
- Stop running tests on 10.8 on B2g28-v1.3 and B2g26-v1.2
- Need Tarako 1.3t FOTA updates for testing purposes
- Move Android 2.3 reftests to ix slaves (ash only)
- Changes introduced for Bug 970918 break mozharness code when mock_target is not defined
- Reduce log retention on buildbot masters from 200 twistd.log files to 100 twistd.log files
- We need to pull in merge commit refs for gaia-try runs
- Limit coalescing
- Get e10s tests running on inbound, central, and try
- Make it possible to run gaia try jobs *without* doing a build
- inari eng nightlies failing because stage-update.py can't find a mar
- Don't enable sccache on PGO builds
- Disable branches to reduce builder count
- blobber TinderboxPrints could be tidier
- Run more Android 4.0 Debug tests on Cedar
- Schedule Android 2.3 crashtests, js-reftests, plain reftests, and m-gl on all trunk trees and make them ride the trains
- add b2gbld password to BuildSlaves.py
- Add MySQL specific database schema dumps to the BuildAPI repo
- Configure Elm for specific builds
- Loan Requests
- Slave loan request for a VS2013 build machine
- Slave loan request for a t-w732-ix machine
- Loan glandium a (the) linux build slave in scl3
- Other
- Platform Support
- mh and device changes for relocated p3 panda racks
- Create a Windows-native alternate to msys rm.exe to avoid common problems deleting files
- release-automation
- Release tagging should use purge_builds.py
- release jobs don't pay attention to jacuzzi allocations
- release repacks needs to submit data to balrog
- Repos and Hooks
- Tools
- Buildduty
- General Automation
- Run Android 2.3 tests against armv6 builds, on Ash only
- make mozharness test scripts easier to run standalone
- kill inari, leo builds
- Provide B2G Emulator builds for Darwin x86
- Figure out the correct path setup and mozconfigs for automation-driven MSVC2013 builds
- set-up initial balrog rules for b2g updates
- Add a MOZ_AUTOMATION environment to all builds
- Manage repo checkout directly from b2g_build.py
- Triggering arbitrary jobs gets branch wrong
- b2g balrog submission should point at dated dirs, not latest-*
- Create 4 more linux64 tests masters
- hamachi device builds submitting bad urls to balrog
- [Meta] Some "Android 4.0 debug" tests fail
- /builds/slave is read only for cltbld
- allow mozharness scripts to call other mozharness scripts
- [Flame] Please update the blobs to use blobs from 10F-3
- Add the build step or else process name to buildbot's generic command timed out failure strings
- switch b2g builds to use aus4.mozilla.org as their update server
- Run Gaia unit oop and reftest sanity oop for b2g desktop on trunk
- Stop running cppunit and jittest on OS X 10.8 on all branches
- eng+noneng nightlies upload MARs (and maybe other files) that stomp on each other
- Please add non-unified builds to mozilla-central
- Do debug B2G desktop builds
- change how we chunk in desktop_unittest.py
- include device in fota mar filenames
- Add 'hsb' to the Firefox build
- Upload the list of all functions from hazard analysis
- Start doing mulet builds
- Don't require puppet or DNS to launch new instances
- Put ccache on SSDs
- Add support for webapprt-test-chrome test jobs & enable them per push on Cedar
- Loan Requests
- Other
- Deprecate tinderbox-builds/old directories for desktop & mobile
- [tracking] infrastructure improvements
- Switch in house try builds to S3 for sccache
- Platform Support
- Cleanup temporary files on boot
- evaluate mac cloud options
- cancelled 2.3 mochitest jobs put ix slaves into weird state
- signing win64 builds is busted
- Windows slaves often get permission denied errors while rm'ing files
- slave pre-flight tasks
- Run unittests/talos on OS X 10.9 Mavericks
- Update version of pip installed on automation machines from 0.8.2 to 1.5.4+
- release-automation
- release automation can't update balrog blobs during the update step
- Figure out how to offer release build to beta users
- Releases
- tracking bug for build and release of Firefox and Fennec 30.0
- Trim rsync modules (May 2014 OMG we still have to do this edition)
- Repos and Hooks
- Tools
- b2g tagging script
- AWS Sanity Check lies about how long an instance was shut down for...
- implement "disable" action in slaveapi
- tegra/panda health checks (verify.py) should not swallow exceptions
- Setup docker apps for buildbot, buildapi and redis
- Blobber upload files not served with correct content type