This week in Mozilla RelEng – March 28th, 2014
Major Highlights:
- Catlee wrote a great blog post about the size of our updates and how we can reduce them.
- Joel discovered that test screenshots on some of our Linux test machines were broken, and Armen deployed a change to fix it. Debugging hard to reproduce failures one these machines should be easier again.
- Aki enabled nightly debug b2g emulator builds, which give QA much smaller regression windows for some types of failures. You can find these builds in directories like this.
- Kim has been working on enabling a few new types of tests.
- I enabled early testing of our new update server (Balrog) for Beta releases. If you'd like to give it a whirl following these instructions.
- Catlee enabled builds for our new 128MB phone (Tarako).
- Callek landed his work on SlaveAPI to support Panda boards. With this done, sheriffs have more tools to deal with flaky test machines.
- Buildduty
- Set value of lastgood to False by default
- kmoir needs access to AWS console
- /data over 85 full (85%) on ip-10-39-70-110.ec2.internal
- Windows 8 tests failing across all branches
- watch pending and misc.py scheduling make different decisions about which slave to use for a retried job
- Android 2.3 tests not starting on Ash
- Integration Trees closed, high number of pending linux/android compile jobs about 1 hour backlog
- Please upload new xre.zip to tooltool
- Please upload new xre.zip to tooltool
- Determine RFO for 5 hour tree closure: No Linux64 test slaves running, 3+ hour pending
- All trees closed - mozpool issues with "We could not request the device: mozpool status not ok, code 500"
- nightly builds failing because of switch to ftp-ssl
- General Automation
- Stop setting JAVA_HOME in buildbot and mozharness to a dir that does not exist
- [tracking bug] Use AWS spot instances
- ASan is the only Linux platform running the separate jittest suite on try
- Tracking bug for 17-mar-2014 migration work
- Increase number of b2g reftest on EC2 chunks from 10 to 15
- Pass tooltool_script into misc.py, and make it work
- Increase free space requirement for Thunderbird l10n builds on central & aurora
- [tracking bug] Run desktop unittests on Ubuntu
- gaia-ui tests need to dump a stack when the process crashes
- b2g build improvements
- Please schedule linux jit-tests in two chunks
- take ownership of and fix relman merge scripts
- Drop the lastgood requirement for nightlies off m-c
- bld-lion-r5-037 debug failures may be showing a problem with bug 969689
- [tracking bug] Run b2g emulator unittests on Ubuntu
- mozharness reports success when build fails in some cases
- B2G desktop builds with git tryserver syntax in gaia.json
- generate signing token before doing build
- ccache sometimes hits fatal permission denied errors when trying to create temp files
- Add support for non-unified builds in mozharness desktop builds
- prep buildbot.py for desktop builds in mozharness
- Enable ASAN tests on the Jetpack tree
- [tarako][build]create "tarako" build
- Speed up reconfigs on linux test masters
- Improve prioritizeBuilders
- Add web-platform-tests to buildbot config
- tag_spot_instances.py exceptions
- Prepare infra to handle spot instances
- Increase maxtime for browser-chrome-3
- submit release builds to balrog automatically
- Stop running builds and tests on pushes which only include changes to the b2g or mobile or browser directories
- Loan Requests
- Loan an ami-6a395a5a instance to Dan Glastonbury
- borrow a linux ix multinode host to test CentOS-6.5
- Please loan dminor t-xp32-ix- instance
- Other
- Platform Support
- Disable deja-dup-monitor and jockey-gtk services from autostarting on Linux ubuntu slaves
- Switch 32-bits linux builds to x86-64 mock environments
- Update builders to Android SDK platform 17
- release-automation
- run clobbers after release
- release runner attempts to tag with the same tag multiple times in some situations
- make it possible to control whatsnew page showing or not via ship it, per-locale
- Releases
- Repos and Hooks
- Tools
- Balrog: Frontend
- Buildduty
- General Automation
- Make blobber uploads discoverable
- Add ssltunnel to xre.zip bundle used for b2g tests
- Enable all mozharness desktop build linux variants on Cedar
- Add a TURN server for running mochitest automation under a VPN
- Requesting OMTC Reftest (Ro) variant for Windows
- Add tooltool support to Windows builds
- decommission (releng side) aws buildbot masters migrated to scl3
- Show SM(Hf) builds on mozilla-aurora, mozilla-beta, and mozilla-release
- Add configs for debug B2G desktop builds & enable on mozilla-central and mozilla-aurora
- Please remove hamachi.zip and the out folder from the ril build for hamachi
- Mozharness test configs should live in-tree
- Bengali in 1.3t builds
- Make --enable-debug gecko device builds for B2G
- fx desktop builds in mozharness
- Run desktop mochitests-browser-chrome on Ubuntu
- Do nightly builds with profiling disabled
- Install ant on builders
- prioritize l10n nightlies below everything, regardless of branch
- Please schedule desktop jit-tests on all trunk trees and make them ride the trains
- Run additional hidden 3 debug mochitest-browser-chrome chunks and b2g reftests
- Please add non-unified builds to mozilla-central
- Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with "command timed out: 1200 seconds without output, attempting to kill", "command timed out: 1800 seconds without output, attempting to kill"
- Schedule Android 4.0 Debug M4,M5,M6,M7,J1,J2,J3 (only) on all trunk trees and make them ride the trains
- Updates not properly signed on the nightly-ux branch: Certificate did not match issuer or name.
- Schedule (mostly) B2G tests on Mozilla-B2g28-v1.3t
- Schedule desktop tests only on Holly
- Split up mochitest-bc on desktop into ~30minute chunks
- Run jit-tests from test package
- Loan Requests
- Loan t-w864-ix-003 to jrmuizel
- Please loan shu OS X 10.6 test runner
- Slave loan request for a t-w864-ix machine
- Other
- Use mozmake for some windows builds
- Upgrade ASan Clang in Q1
- [tracking] infrastructure improvements
- s/m1.large/m3.medium/
- Platform Support
- [tracker] Move away from the rev3 minis
- evaluate mac cloud options
- Experiment with other instance types for Android 2.3 jobs
- aws_create_ami.py should generate HVM and PV AMIs
- Releng Config/Automation support of staging run of scl1->scl3 move
- Upgrade Windows 8 Nvidia drivers to 335.23
- release-automation
- Releases
- Releases: Custom Builds
- Repos and Hooks
- Tools
- update relman scripts based on notes from first usage
- Add jittest to trychooser
- Store the timing of actions in slaveapi
- Setup docker apps for buildbot, buildapi and redis
- Add Windows support to aws_stop_idle.py
- Check mozilla bitbucket account is functioning correctly and that access key is still valid
- Create a Comprehensive Slave Loan tool