Major Highlights:

• Massimo finished converting the remainder of our AWS Buildbot masters to cheaper & faster instance types.
• Aki is making steady progress on Try support for Gaia changes.
• Armen turned off the last jobs that were run on our rev 3 machines, which finally lets us retire them.
• Aki and Catlee got initial builds for the FirefoxOS "Flame" reference device going.
• Pete made more progress on scripts to do fully automated buildbot master reconfigs, which should help us land changes to them more quickly and efficiently.

Completed work (resolution is 'FIXED'):

• Balrog: Backend
• balrog not substituting %locale% in detailsUrl
• Buildduty
• tons of unauthorized slaves on panda masters
• General Automation
• Split up test suites that take longer than N minutes to run
• remove machines that don't exist from buildbot-configs
• investigate increase in certificate attribute check failures since bug 803531
• generate "FxOS Simulator" builds for B2G
• Spot instances don't have security group set properly
• Rev4 machines have Puppet disabled which can lose their name and burn talos jobs because they end up with a name like client-builders-mac-mini-10
• Add mbrubeck to bors reviewers for servo
• turn off some b2g builds
• [Flame]figure out how to create "Flame" builds manually
• Create retry() context manager
• Bump MAX_BROKER_REFS to 4096
• Turn off Fedora b2g reftests on trunk branches
• Provide native js shell to tarako b2g builds
• Split up mochitest-bc on desktop into ~30minute chunks
• Bengali in 1.3t builds
• Loan Requests
• please loan jmaher an ec2 machine equal to a linux32-spot instance
• Slave loan request for Matt Woodrow
• borrow a linux ix multinode host to test CentOS-6.5
• Need a win32 slave to debug Lightning pymake issues
• Slave loan request for a w64-ix machine
• Slave loan request for a t-w864-ix machine
• Slave loan request for a bld-lion-r5 machine
• Other
• Figure where to put sccache config for ceph
• s/m1.large/m3.medium/
• Platform Support
• [tracker] Move away from the rev3 minis
• Experiment with other instance types for Android 2.3 jobs
• release-automation
• No Tarako 1.3 build available to smoketest
• staging release runner instructions don't work with new dev-master1/ssh key management
• bouncer submitter doesn't work in staging
• ccache enabled in nightly mozconfigs but not beta or release
• kill release downloader
• Need an update watershed in 29.0bN for beta channel
• Releases
• do a staging release of Firefox and Fennec 30.0b1
• Repos and Hooks
• Request for a new git repository: "fuzz-tools" (public)
• Mirror bmo to github
• Mirror upstream Bugzilla to github
• Tools
• Make mozharness+buildbot configuration changes necessary for new test suite mochitest-dt
• Add mochitest-dt to Trychooser
• Slaveapi should support mobile devices for reboots
• Slaveclass warning on slave health
• Cron /home/buildduty/buildfaster/bin/buildfaster_report.sh
• buildbot wrangler should notice failed reconfigs
• Create instructions for Windows on EC2

In progress work (unresolved and not assigned to nobody):

• Balrog: Frontend
• balrog's rules page is excessively wide
• Buildduty
• Reconfigs should be automatic, and scheduled via a cron job
• General Automation
• Make it possible to run gaia try jobs *without* doing a build
• Updates not properly signed on the nightly-ux branch: Certificate did not match issuer or name.
• Schedule Android 4.0 Debug M1,M2,M3,M8 on all trunk trees and let them ride the trains
• branch gaia l10n repos
• Move Android 2.3 reftests to ix slaves (ash only)
• revamp b2g upload configs
• Add ssltunnel to xre.zip bundle used for b2g tests
• Do debug B2G desktop builds
• Run unittests on Win64 builds
• Need Tarako 1.3t FOTA updates for testing purposes
• Add the build step or else process name to buildbot's generic command timed out failure strings
• Fully decommission no longer used buildbot masters
• point [xxx] repository at new bm-remote webserver cluster to ensure parity in talos numbers
• Intermittent Android 2.3 install step hang
• Configure the MLS key for pvt builds
• Loan Requests
• Loan t-w864-ix-003 to jrmuizel
• Slave loan request for a tst-linux64-ec2 vm
• Slave loan request for dtownsend@mozilla.com
• Need a slave for bug 818968
• Slave loan request for dtownsend@mozilla.com
• Loan t-w732-ix-003, t-w732-ix-004 to Q
• Loan mshal an aws builder
• Loan :sfink a linux64 bld-centos6-hp-005
• Request Loan Machine for tst-linux64-spot
• Other
• Deprecate tinderbox-builds/old directories for desktop & mobile
• Platform Support
• Deploy NSIS 3.0a2 or MozillaBuild 1.9 (includes NSIS 3.0a2) to buildslaves
• signing win64 builds is busted
• Cleanup temporary files on boot
• evaluate mac cloud options
• Create a Windows-native alternate to msys rm.exe to avoid common problems deleting files
• release-automation
• Thunderbird Windows L10n repack builds failed with pymake enabled
• Figure out how to offer release build to beta users
• cache MAR + installer downloads in update verify
• Releases
• tracking bug for build and release of Firefox and Fennec 30.0
• tracking bug for build and release of Firefox 24.6.0 ESR
• Add SeaMonkey 2.26 Beta 2 to bouncer
• tracking bug for build and release of Thunderbird 24.6.0
• do a staging release of Firefox and Fennec 31.0b1
• Tools
• cut over gecko.git to the new vcs-sync system
• cut over l10n repos to the new vcs-sync system
• Blobber upload files not served with correct content type
• update relman scripts based on notes from first usage
• tool to compare different sources of slave and master data
• Add Windows support to aws_stop_idle.py
• Create a Comprehensive Slave Loan tool

Major highlights:

• Armen and Bill McCloskey got an initial set of tests running with e10s enabled, which is a big step towards stabilizing it.
• Hal helped run the shortest tree closure window to date (less than 2 hours!).
• Massimo switched over some of our EC2 machines to SSD instance storage, which has improved Linux build times on try.
• Aki attended the first TRIBE session.
• Armen talked about some of our cost cutting measure in an air.mozilla.org presentation.

Completed work (resolution is 'FIXED'):

• Balrog: Backend
• Fix up Vagrant for balrog
• Buildduty
• Please add t-mavericks-r5-002 and t-mavericks-r5-003 to graphserver
• Reconfig bustage - temporary fix for "exceptions.KeyError: 'tst-linux64-ec2-300'"
• Reimage remaining win64-rev1 machines as win64-rev2
• Testpool connection issues
• General Automation
• Provide native js shell to tarako b2g builds
• Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out, as does downloading/unzipping test zips
• add missing pandas/tegras to buildbot configs
• b2g build improvements
• generate "FxOS Simulator" builds for B2G
• Run additional hidden 3 debug mochitest-browser-chrome chunks and b2g reftests
• prioritize l10n nightlies below everything, regardless of branch
• Enable all mozharness desktop build linux variants on Cedar
• Run desktop C++ unit tests from test package
• Increase maxtime for browser-chrome-2. Again.
• Perma-fail on Linux64 slaves - test_hosted.xul,test_packaged.xul,test_packaged_launch.xul | Error during test: [Exception... "Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIFileOutputStream.init]" nsresult: "0x80520012
• Increase maxtime for browser-chrome-1
• Get e10s tests running on inbound, central, and try
• [tarako][build]create "tarako" build
• post_upload.py should use hardlinks where possible
• disable confusing double runs of mochitest-bc for linux debug
• Enable updates for tarako on 1.3t branch
• Run jit-tests from test package
• Kill spot vs on-demand retry logic
• logrotate aws_watch_pending.py
• Loan Requests
• please loan jmaher an ec2 machine equal to a linux32-spot instance
• Need a win32 unittest slave for debugging packaged tests
• Loan an ami-3ac4aa0a instance to Tim Abraldes
• Need a win32 slave to debug Lightning pymake issues
• please loan jmaher an ec2 machine equal to a linux32-spot instance
• Please loan dminor t-xp32-ix- instance
• Other
• Use mozmake for some windows builds
• stage NFS volume about to run out of space
• Add dev-master1:8951 master to slavealloc
• Platform Support
• Please update libxcb on the linux test slaves to a more recent release
• release-automation
• Minor documentation improvements on the ship-it form
• Add documentation about the beta release of Fennec
• Strip the firefox locale string
• deploy ship it patches from a few bugs
• Documentation about release-kickoff
• release repacks needs to submit data to balrog
• Releases
• tracking bug for build and release of Firefox 24.4.0 ESR
• Add SeaMonkey 2.26 Beta 1 to bouncer
• Releases: Custom Builds
• FF 28 config change for yahoo
• rebuild FF 28 yahoo
• Repos and Hooks
• enable git mirrors for all of /gaia-l10n/*
• New git repos to mirror
• Tools
• gaia github->git.m.o sync is broken
• legacy vcs-sync doesn't AUTOFIX git hangs

In progress work (unresolved and not assigned to nobody):

• Balrog: Backend
• Support new attributes for update.xml in Balrog
• Balrog: Frontend
• balrog's rules page is excessively wide
• Buildduty
• Reconfigs should be automatic, and scheduled via a cron job
• General Automation
• Port TryBuildFactory to Mozharness for fx desktop builds
• Add ssltunnel to xre.zip bundle used for b2g tests
• Disable mochitest-metro on Cedar
• m-c b2g version should be 2.0, not 1.5
• Add configs for debug B2G Mac OS X desktop builds & enable on mozilla-central and mozilla-aurora
• Bengali in 1.3t builds
• fx desktop builds in mozharness
• Spot instances don't have security group set properly
• Add the build step or else process name to buildbot's generic command timed out failure strings
• Need Tarako 1.3t FOTA updates for testing purposes
• Don't do Thunderbird builds on comm-* branches for non-Thunderbird pushes
• [Flame]figure out how to create "Flame" builds manually
• Move Android 2.3 reftests to ix slaves (ash only)
• Turn off Fedora b2g reftests on trunk branches
• Please add non-unified builds to mozilla-central
• point [xxx] repository at new bm-remote webserver cluster to ensure parity in talos numbers
• remove machines that don't exist from buildbot-configs
• Make it possible to run gaia try jobs *without* doing a build
• Updates not properly signed on the nightly-ux branch: Certificate did not match issuer or name.
• Decommission the ionmonkey tree
• Configure the MLS key for pvt builds
• Split up mochitest-bc on desktop into ~30minute chunks
• Loan Requests
• Slave loan request for a VS2013 build machine
• Loan an ami-6a395a5a instance to Aaron Klotz
• Please loan tst-linux64- instance to dminor
• borrow a linux ix multinode host to test CentOS-6.5
• Need a bld-lion-r5 to test build times with SSD
• Request Loan Machine for tst-linux64-spot
• Other
• Figure where to put sccache config for ceph
• Deprecate tinderbox-builds/old directories for desktop & mobile
• s/m1.large/m3.medium/
• Platform Support
• [tracker] Move away from the rev3 minis
• release-automation
• cache MAR + installer downloads in update verify
• Releases: Custom Builds
• rebuild FF 28 yahoo
• Repos and Hooks
• find RFO for git.m.o OOM condition in bug 985864
• Request for a new git repository: "fuzz-tools" (public)
• Tools
• Update SlaveAPI sphynx docs to reflect recent changes
• New tooltool deployment
• Set up auto publish of doc index per Boston workweek agreement
• Using slaveapi to reboot a machine on the slave health page results in "list index out of range" under 'output'
• tool to compare different sources of slave and master data
• slaveapi's shutdown_buildslave action doesn't cope well with a machine that isn't connected to buildbot
• Add Windows support to aws_stop_idle.py
• db-based mapper on web cluster
• Create a Comprehensive Slave Loan tool
• SlaveAPI should be more precise on request TS
• vcs-sync needs to populate mapper db once it's live

Major highlights:

• Kim started evaluating options for hosted Mac hardware.
• Aki started standing up builds for "Flame", the first Firefox reference device.
• Rail long standing manual parts of our Firefox release process.
• Massimo revamped our builder prioritization to make it easier to prioritize l10n builds properly, and make prioritization more dynamically adjustable.

Completed work (resolution is 'FIXED'):

• Balrog: Backend
• No update served while aurora update frozen
• Buildduty
• Windows 8 tests failing across all branches
• EC2 test spot instances with 15GB disk running low on (or completely out of) available disk when checking out gaia-1_4, gaia-1_2 and gaia-central under /builds/hg-shared/integration
• General Automation
• Android clobber clobbers things such that ant can't be found
• Change birch config and add jacuzzis
• Add ssltunnel to xre.zip bundle used for b2g tests
• Schedule Android 4.0 Debug M4,M5,M6,M7,J1,J2,J3 (only) on all trunk trees and make them ride the trains
• mozharness' unit.sh should list 'mock' as a dependency
• Schedule Android 2.3 mochitest,xpcshell,robocop tests (only) on all trunk trees and make them ride the trains
• mozharness should treat warnings containing "SyntaxError" keywords as warnings
• decommission (releng side) aws buildbot masters migrated to scl3
• Add configs for debug B2G Mac OS X desktop builds & enable on mozilla-central and mozilla-aurora
• Please remove hamachi.zip and the out folder from the ril build for hamachi
• Make --enable-debug gecko device builds for B2G
• Mozharness install step should catch failures due to insufficient disk space
• Rename cppunittest to cppunit in mobile_config.py
• AWS manager can't reach buildbot masters
• [Flame]figure out how to create "Flame" builds manually
• Please schedule desktop jit-tests on all trunk trees and make them ride the trains
• stop importing WithProperties in buildbot-configs
• Add configs for debug B2G linux64 desktop builds
• enable_nightly is in birch twice
• Schedule desktop tests only on Holly
• Please build b2g and gaia parts for the Tarako build based on bug 956631 output
• Add tooltool support to Windows builds
• Increase free space requirement for linux debug builds
• purge_builds needs to know about different partitions
• Loan Requests
• please loan jmaher an ec2 machine equal to a linux32-spot instance
• Other
• Stage MozillaBuild 1.9.0
• releng jenkins is down
• Upgrade ASan Clang in Q1
• ReadTheDocs not regenerating slaveapi documentation
• Stage MozillaBuild 1.8.0
• Use pymake by default on Windows
• Use mozmake for b2g win32 builds
• Use mozmake for some windows builds
• Platform Support
• Switch 32-bits linux builds to x86-64 mock environments
• do initial planning on "chunks" of machines to move.
• Releng Config/Automation support of staging run of scl1->scl3 move
• release-automation
• xr_postrelease failed for 29.0b4
• Generate 1.3T Tarako builds twice a day (9am China time - UTC+8)
• Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
• run xulrunner post-release task automatically after Firefox post-release
• Releases
• Bad link on ftp.mozilla.org to xulrunner builds
• Show Australis tour page upon update for en-US on Beta
• Releases: Custom Builds
• Version bumps for FF 28 builds
• Repos and Hooks
• Create hg mirror of gaia 1.3t branch
• Tools
• Can't assimilate loaner instance on aws-manager1.srv.releng.scl3.mozilla.com
• Add jittest to trychooser
• update beta2release to current working version
• Display Most recent 10 slaveapi calls, not oldest 10
• self-serve agents eat exceptions, causing stalls
• Remove cppunittest for Android as option in TryChooser
• host instance: 'None' stopped for 1348 hours.

In progress work (unresolved and not assigned to nobody):

• Buildduty
• Please add t-mavericks-r5-002 and t-mavericks-r5-003 to graphserver
• General Automation
• Make blobber uploads discoverable
• Port TryBuildFactory to Mozharness for fx desktop builds
• branch gaia l10n repos
• Do debug B2G desktop builds
• m-c b2g version should be 2.0, not 1.5
• disable confusing double runs of mochitest-bc for linux debug
• Mozharness test configs should live in-tree
• Bengali in 1.3t builds
• fx desktop builds in mozharness
• generate "flatfish" builds for B2G
• Fully decommission no longer used buildbot masters
• prioritize l10n nightlies below everything, regardless of branch
• revamp b2g upload configs
• Run additional hidden 3 debug mochitest-browser-chrome chunks and b2g reftests
• Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with "command timed out: 1200 seconds without output, attempting to kill", "command timed out: 1800 seconds without output, attempting to kill"
• Enable all mozharness desktop build linux variants on Cedar
• Get e10s tests running on inbound, central, and try
• Make it possible to run gaia try jobs *without* doing a build
• Updates not properly signed on the nightly-ux branch: Certificate did not match issuer or name.
• Don't require puppet or DNS to launch new instances
• Decommission the ionmonkey tree
• Split up mochitest-bc on desktop into ~30minute chunks
• Run jit-tests from test package
• Loan Requests
• Slave loan request for a VS2013 build machine
• Loan t-w864-ix-003 to jrmuizel
• Slave loan request for a tst-linux64-ec2 machine
• Slave loan request for a w64-ix machine
• Request Loan Machine for tst-linux64-spot
• Other
• [tracking] infrastructure improvements
• s/m1.large/m3.medium/
• Platform Support
• Upgrade Valgrind on build machines to 3.9.0
• [tracker] Move away from the rev3 minis
• Run unittests/talos on OS X 10.9 Mavericks
• clean up mobile_config.py
• release-automation
• Thunderbird Windows L10n repack builds failed with pymake enabled
• cache MAR + installer downloads in update verify
• release repacks needs to submit data to balrog
• Releases
• Add SeaMonkey 2.26 Beta 1 to bouncer
• Repos and Hooks
• find RFO for git.m.o OOM condition in bug 985864
• Tools
• update relman scripts based on notes from first usage
• vcs-sync needs to populate mapper db once it's live
• Update SlaveAPI sphynx docs to reflect recent changes
• Add an option to trychooser to select Talos profiling options
• Using slaveapi to reboot a machine on the slave health page results in "list index out of range" under 'output'
• tool to compare different sources of slave and master data
• slaveapi's shutdown_buildslave action doesn't cope well with a machine that isn't connected to buildbot
• Add Windows support to aws_stop_idle.py
• db-based mapper on web cluster
• legacy vcs-sync process doesn't auto retry when it should
• SlaveAPI should be more precise on request TS
• legacy vcs-sync doesn't AUTOFIX git hangs

Major Highlights:

• Catlee wrote a great blog post about the size of our updates and how we can reduce them.
• Joel discovered that test screenshots on some of our Linux test machines were broken, and Armen deployed a change to fix it. Debugging hard to reproduce failures one these machines should be easier again.
• Aki enabled nightly debug b2g emulator builds, which give QA much smaller regression windows for some types of failures. You can find these builds in directories like this.
• Kim has been working on enabling a few new types of tests.
• I enabled early testing of our new update server (Balrog) for Beta releases. If you'd like to give it a whirl following these instructions.
• Catlee enabled builds for our new 128MB phone (Tarako).
• Callek landed his work on SlaveAPI to support Panda boards. With this done, sheriffs have more tools to deal with flaky test machines.

Completed work (resolution is 'FIXED'):

• Buildduty
• Set value of lastgood to False by default
• kmoir needs access to AWS console
• /data over 85 full (85%) on ip-10-39-70-110.ec2.internal
• Windows 8 tests failing across all branches
• watch pending and misc.py scheduling make different decisions about which slave to use for a retried job
• Android 2.3 tests not starting on Ash
• Integration Trees closed, high number of pending linux/android compile jobs about 1 hour backlog
• Please upload new xre.zip to tooltool
• Please upload new xre.zip to tooltool
• Determine RFO for 5 hour tree closure: No Linux64 test slaves running, 3+ hour pending
• All trees closed - mozpool issues with "We could not request the device: mozpool status not ok, code 500"
• nightly builds failing because of switch to ftp-ssl
• General Automation
• Stop setting JAVA_HOME in buildbot and mozharness to a dir that does not exist
• [tracking bug] Use AWS spot instances
• ASan is the only Linux platform running the separate jittest suite on try
• Tracking bug for 17-mar-2014 migration work
• Increase number of b2g reftest on EC2 chunks from 10 to 15
• Pass tooltool_script into misc.py, and make it work
• Increase free space requirement for Thunderbird l10n builds on central & aurora
• [tracking bug] Run desktop unittests on Ubuntu
• gaia-ui tests need to dump a stack when the process crashes
• b2g build improvements
• Please schedule linux jit-tests in two chunks
• take ownership of and fix relman merge scripts
• Drop the lastgood requirement for nightlies off m-c
• bld-lion-r5-037 debug failures may be showing a problem with bug 969689
• [tracking bug] Run b2g emulator unittests on Ubuntu
• mozharness reports success when build fails in some cases
• B2G desktop builds with git tryserver syntax in gaia.json
• generate signing token before doing build
• ccache sometimes hits fatal permission denied errors when trying to create temp files
• Add support for non-unified builds in mozharness desktop builds
• prep buildbot.py for desktop builds in mozharness
• Enable ASAN tests on the Jetpack tree
• [tarako][build]create "tarako" build
• Speed up reconfigs on linux test masters
• Improve prioritizeBuilders
• Add web-platform-tests to buildbot config
• tag_spot_instances.py exceptions
• Prepare infra to handle spot instances
• Increase maxtime for browser-chrome-3
• submit release builds to balrog automatically
• Stop running builds and tests on pushes which only include changes to the b2g or mobile or browser directories
• Loan Requests
• Loan an ami-6a395a5a instance to Dan Glastonbury
• borrow a linux ix multinode host to test CentOS-6.5
• Please loan dminor t-xp32-ix- instance
• Other
• Use pymake by default on Windows
• Merge git build-cloud-tools to official hg cloud-tools
• Platform Support
• Disable deja-dup-monitor and jockey-gtk services from autostarting on Linux ubuntu slaves
• Switch 32-bits linux builds to x86-64 mock environments
• Update builders to Android SDK platform 17
• release-automation
• run clobbers after release
• release runner attempts to tag with the same tag multiple times in some situations
• make it possible to control whatsnew page showing or not via ship it, per-locale
• Releases
• Show Australis tour page upon update for en-US on Beta
• Repos and Hooks
• b2g branching automation
• Tools
• Slaveapi should abstract out Machine from Slave
• Slaveapi should support mobile devices for reboots
• slaveapi should have a seperate knob for concurrency on buildapi calls alone

In progress work (unresolved and not assigned to nobody):

• Balrog: Frontend
• balrog's rules page is excessively wide
• Buildduty
• Reconfigs should be automatic, and scheduled via a cron job
• General Automation
• Make blobber uploads discoverable
• Add ssltunnel to xre.zip bundle used for b2g tests
• Enable all mozharness desktop build linux variants on Cedar
• Add a TURN server for running mochitest automation under a VPN
• Requesting OMTC Reftest (Ro) variant for Windows
• Add tooltool support to Windows builds
• decommission (releng side) aws buildbot masters migrated to scl3
• Show SM(Hf) builds on mozilla-aurora, mozilla-beta, and mozilla-release
• Add configs for debug B2G desktop builds & enable on mozilla-central and mozilla-aurora
• Please remove hamachi.zip and the out folder from the ril build for hamachi
• Mozharness test configs should live in-tree
• Bengali in 1.3t builds
• Make --enable-debug gecko device builds for B2G
• fx desktop builds in mozharness
• Run desktop mochitests-browser-chrome on Ubuntu
• Do nightly builds with profiling disabled
• Install ant on builders
• prioritize l10n nightlies below everything, regardless of branch
• Please schedule desktop jit-tests on all trunk trees and make them ride the trains
• Run additional hidden 3 debug mochitest-browser-chrome chunks and b2g reftests
• Please add non-unified builds to mozilla-central
• Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with "command timed out: 1200 seconds without output, attempting to kill", "command timed out: 1800 seconds without output, attempting to kill"
• Schedule Android 4.0 Debug M4,M5,M6,M7,J1,J2,J3 (only) on all trunk trees and make them ride the trains
• Updates not properly signed on the nightly-ux branch: Certificate did not match issuer or name.
• Schedule (mostly) B2G tests on Mozilla-B2g28-v1.3t
• Schedule desktop tests only on Holly
• Split up mochitest-bc on desktop into ~30minute chunks
• Run jit-tests from test package
• Loan Requests
• Loan t-w864-ix-003 to jrmuizel
• Please loan shu OS X 10.6 test runner
• Slave loan request for a t-w864-ix machine
• Other
• Use mozmake for some windows builds
• Upgrade ASan Clang in Q1
• [tracking] infrastructure improvements
• s/m1.large/m3.medium/
• Platform Support
• [tracker] Move away from the rev3 minis
• evaluate mac cloud options
• Experiment with other instance types for Android 2.3 jobs
• aws_create_ami.py should generate HVM and PV AMIs
• Releng Config/Automation support of staging run of scl1->scl3 move
• Upgrade Windows 8 Nvidia drivers to 335.23
• release-automation
• Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
• Releases
• tracking bug for build and release of Firefox and Fennec 29.0
• Releases: Custom Builds
• FF 28 config change for yahoo
• Repos and Hooks
• Request for a new git repository: "fuzz-tools" (public)
• Tools
• update relman scripts based on notes from first usage
• Add jittest to trychooser
• Store the timing of actions in slaveapi
• Setup docker apps for buildbot, buildapi and redis
• Add Windows support to aws_stop_idle.py
• Check mozilla bitbucket account is functioning correctly and that access key is still valid
• Create a Comprehensive Slave Loan tool

Major Highlights:

• Pete has been working away at getting a production grade hg git revision mapper to replace the current quick&dirty one. This is an important service because we how we build B2G out of a combination of Mercurial and Git repositories.
• Jordan has been making great strides on moving Firefox Desktop builds to Mozharness, and Aki has been helping him keep pace with quick reviews. Once these are moved, RelEng Firefox builds will be much easier to reproduce outside of Buildbot, making it easier to debug problems that only happen on our machines.
• Kim presented her "Built to Scale" talk at EclipseCon!
• Armen enabled B2G reftests on EC2 machines, getting us one step closer to retiring our aging Fedora hardware.
• Hal helped us take ownership of and improve the merge day scripts. This should help merge days happen more efficiently in the future, and eventually be fully automated.
• Callek made great strides in getting SlaveAPI to support panda boards. When finished, sheriffs will have more control to deal with problematic pandas.
• Nick added support for locale-specific actions for updates. This will allow us improve user experience by doing things like only showing users "what's new" pages if it's been translated to their locale.

Completed work (resolution is 'FIXED'):

• Balrog: Backend
• Updates not found since 2014-03-13 Firefox for Android Nightly/Aurora
• Support multiple products in a permission options
• Buildduty
• Integration Trees closed, high number/time backlog of pending jobs
• buildbot-master54's buildbot process got OOM'ed
• General Automation
• Enable EC2 b2g reftests and Ubuntu EC2 debug mochitest-b-c on try
• support desktop build builders being created from Mozharness (SigningScriptFactory) or through Buildbot (MBF)
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• Desktop UnitTests should not rely on PyPi as a repo
• submissions to balrog shouldn't silently fail
• Parallelize spot instance tagging
• Turn off PGO, talos and debug builds on UX project branch
• b2g emulator tests not running since ~9am 20140319, all trunk and b2g-building release trees closed
• aws sanity check shouldn't report instances that are actively doing work as long running
• Non-Fast Forward tag operation on git.mozilla.org/releases/gecko.git
• take ownership of and fix relman merge scripts
• Allow to run retried jobs on spot instances
• Bump MAX_BROKER_REFS to 4096
• ril repack failure - merge day failure
• Fix compression for non-compressed uploads
• Do not use spot instances for some builders
• Stop running builds and tests on pushes which only include changes to the b2g or mobile or browser directories
• emulator-kk builds
• Request: Nightly FirefoxOS engineering builds for nexus-4, please
• Increase the 9900 second time limit for browser-chrome
• Loan Requests
• Slave loan request for a t-w732-ix machine
• Slave loan request for gbrown
• Please lend Andreas Tolfsen a linux64 EC2 m3.medium slave
• Loan felipe an AWS unit test machine
• Other
• please add a --bootstrap cli flag to reftest/crashtest/jsreftest for android
• Platform Support
• deploy ccache 3.1.9 to linux builders
• Migrate ESR, b2g26_v1_2, b2g28_v1_3 branches to win64-rev2
• Update builders to Android SDK platform 17
• Add support for win64+gpo platform
• release-automation
• Create SSL products in bouncer as part of release automation
• staging release runner instructions don't work with new dev-master1/ssh key management
• Installer bouncer entries should be ssl-only
• turn off js/src/config/milestone.txt bumping for tb beta + add comment in esr24
• Releases
• tracking bug for build and release of Firefox and Fennec 28.0
• Thunderbird SSL-only Bouncer product
• do a staging release of Firefox and Fennec 29.0b1
• 29.0b1 failures for some mac 'or' locale
• Wrong link latest-esr/. Points to 24.3.0 instead of 24.4.0
• Stub Installer is serving Firefox 27.0.1
• Disable Aurora 30 daily updates until merge to mozilla-aurora has stabilized
• Please sign the hotfix to temporarily disable the malware blocklist (v20140319.01)
• Add SeaMonkey 2.25 to bouncer
• Releases: Custom Builds
• Version bumps for FF 28 builds
• Partner repack changes for web.de, gmx, and mail.com (Firefox 28)
• Repos and Hooks
• Create a script to import one Mercurial repository into a subdirectory of another, with linear history
• Pushing to projects/metro fails due to "Error accessing https://treestatus.mozilla.org/metro?format=json"
• Tools
• Remove android-noion from trychooser
• Add emulator-kk builds to trychooser
• Clean up trychooser's tryload
• update merge_helper.py to work in releng environment
• Add gaia-integration to trychooser
• update beta2release to current working version
• Update b2g cronjobs to keep the next 12 weeks worth of builds after 3/17's merge day
• Add b2g desktop (win32)/win32_gecko to trychooser
• Spot bidding should be able to filter out AZs with bad conditions

In progress work (unresolved and not assigned to nobody):

• Balrog: Frontend
• balrog's rules page is excessively wide
• Buildduty
• Reconfigs should be automatic, and scheduled via a cron job
• Determine RFO for 5 hour tree closure: No Linux64 test slaves running, 3+ hour pending
• General Automation
• Stop setting JAVA_HOME in buildbot and mozharness to a dir that does not exist
• Make blobber uploads discoverable
• ASan is the only Linux platform running the separate jittest suite on try
• Tracking bug for 17-mar-2014 migration work
• Enable all mozharness desktop build linux variants on Cypress
• Add a TURN server for running mochitest automation under a VPN
• Pass tooltool_script into misc.py, and make it work
• Requesting OMTC Reftest (Ro) variant for Windows
• Add tooltool support to Windows builds
• [tracking bug] Run desktop unittests on Ubuntu
• Improve prioritizeBuilders
• b2g build improvements
• [tracker] run Android 2.3 test jobs on EC2
• Use standard release channel names for B2G builds, remove slashes
• Run desktop mochitests-browser-chrome on Ubuntu
• Do nightly builds with profiling disabled
• Install ant on builders
• make mozharness test scripts easier to run standalone
• B2G desktop builds with git tryserver syntax in gaia.json
• Run additional hidden 3 debug mochitest-browser-chrome chunks on m-i, and then switch on all trunk branches
• Please add non-unified builds to mozilla-central
• Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with "command timed out: 1200 seconds without output, attempting to kill", "command timed out: 1800 seconds without output, attempting to kill"
• Speed up reconfigs on linux test masters
• Schedule Android 4.0 Debug M4,M5,M6,M7,J1,J2,J3 (only) on all trunk trees and make them ride the trains
• Turn off Metro Browser Chrome tests on inbound, central, cedar, ash, fx-team, etc.
• Mozharness test configs should live in-tree
• Avoid unnecessary buildbot builds for more parts of the tree
• add failure_config parameter to mozharness's run_command()
• Bengali in 1.3t builds
• submit release builds to balrog automatically
• Split up mochitest-bc on desktop into ~30minute chunks
• [tarako][build]create "tarako" build
• Loan Requests
• Loan an ami-6a395a5a instance to Aaron Klotz
• Loan an ami-6a395a5a instance to Dan Glastonbury
• Loan an ami-3ac4aa0a instance to Tim Abraldes
• Slave loan request for a t-w864-ix machine
• Other
• [tracking] infrastructure improvements
• s/m1.large/m3.medium/
• Merge git build-cloud-tools to official hg cloud-tools
• Platform Support
• [tracker] Move away from the rev3 minis
• Ensure that AWS Windows Server 2012 instances are running the Windows Default theme
• signing win64 builds is busted
• Experiment with other instance types for Android 2.3 jobs
• win32file import failure on win64
• disable deja-dup-monitor and jockey-gtk services from autostarting on linux ubuntu hardware slaves
• [tracker] Determine how to support and roll out darwin-on-linux in production
• Releng Config/Automation support of staging run of scl1->scl3 move
• Upgrade Windows 8 Nvidia drivers to 335.23
• Setup Diamond to send metrics to hosted graphite
• AWS Windows Server 2012 tests failing due to start screen
• release-automation
• make it possible to control whatsnew page showing or not via ship it, per-locale
• Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
• Releases
• tracking bug for build and release of Firefox 24.4.0 ESR
• tracking bug for build and release of Firefox and Fennec 29.0
• Show Australis tour page upon update for en-US on Beta
• Releases: Custom Builds
• FF 28 config change for yahoo
• Tools
• Buildduty Report sorting should not error out (JS) if a section has no bugs
• Slaveapi should abstract out Machine from Slave
• Slaveapi should support mobile devices for reboots
• slaveapi should have a seperate knob for concurrency on buildapi calls alone
• permanent location for vcs-sync mapfiles, status json, logs
• Move mqext into the new https://hg.mozilla.org/hgcustom/version-control-tools repo
• self-serve agents eat exceptions, causing stalls
• db-based mapper on web cluster

Major Highlights:

• John Zeller made it possible to trigger builds/tests on arbitrary revisions. This is a feature with HUGE impact -- it will allow us to bisect problems much more easily, and possibly even do so automatically.
• Kim is happy to be back from parental leave and has been preparing to spread the word of Mozilla RelEng to EclipseCon next week and at Google's RelEng Workshop in April.
• John Hopkins has been continuing his work on helping get tests on 64-bit Windows EC2 machines green, which will help us scale to the cloud for those tests and gets us closer to shipping a 64-bit Windows binary.
• Armen enabled mozilla-inbound desktop browser-chrome and b2g reftests on EC2 test machines. While they're still hidden for now, it's gotten us much closer to making sure we keep turnaround times great for those tests.
• Catlee figured out that l10n nightlies were a big part of the reason we've had tree closures due to pending jobs, and we've moved those builders to jacuzzis to fix that.
• Taras and Rail wrote a better algorithm for bidding on EC2 spot instances. Because of that we're now running more jobs than ever on spots, and well under 1% of them are killed off because of a higher bid.

Completed work (resolution is 'FIXED'):

• Balrog: Backend
• /rules/:id endpoint needs to return data_version/csrf token
• exception in getLocaleOrTopLevelParam since latest balrog push to prod
• Balrog: Frontend
• need to be able to delete rules from admin ui
• balrog revisions pages never load sometimes
• fix balrog admin ui title not to read "AUS3"
• Buildduty
• Heavy Tegra Backlog on all trees
• servo linux slaves couldn't successfully run puppet
• Reimage remaining win64-rev1 machines as win64-rev2
• nightly builds failing because of switch to ftp-ssl
• Trigger b2g Hamachi nightly build
• General Automation
• remove old, now unused balrog credentials from BuildSlaves.py
• Make B2G device builds periodic
• Schedule multimedia b2g mochitests for emulator-jb on Cedar
• b2g_bumper can add extra comments
• add jacuzzis for linux l10n builders and more b2g-inbound builders
• gdb not installed in idle-mock-hw
• Schedule mochitest-browser-chrome on EC2 slaves on all branches
• Compression for blobber
• mozharness should use --manifest instead of --test-manifest for emulator-jb mochitests
• emulator-kk builds
• Add support for add-if-not instruction added by bug 759469 to the mar generation scripts
• Make TBPL gaia-integration download node modules from git mirror
• Remove the version 1 manifest from the mar creation scripts
• Loan Requests
• Request for tst-linux64-ec2-XXX machine to run b2g-emulator mochitests
• Please loan dminor Android 2.3 Emulator test instance
• loan ec2 linux32 and linxu64 slaves to jmaher
• Slave loan request for a t-xp32-ix machine
• Loan of linux64 test machine to Steve Fink for M-oth
• Need a slave for bug 818968
• Other
• Recent builds missing from http://inbound-archive.pub.build.mozilla.org
• Platform Support
• The clock of some Fedora machines is out of sync
• AWS machines should run b2g emulator reftests with GALLIUM_DRIVER=softpipe
• Intermittent Android x86 We have not been able to establish a telnet connection with the emulator
• Please update libxcb on the linux test slaves to a more recent release
• Upgrade ec2 test instances mesa versions from 8.0.2-0ubuntu3(32-bit hosts)/8.0.4-0ubuntu0.2(64-bit hosts) to 8.0.4-0ubuntu0.6mozilla1
• Ensure way to create VM that can generate our current Android 2.3 armv7 emulator
• Releases
• Add SeaMonkey 2.25 Beta 2 to bouncer
• Releases: Custom Builds
• Fix outdated and duplicated landing pages for MSN/Bing builds
• move inactive partner builds out of the build process
• Partner repack changes for Onet
• Update Rambler pack with updated extension 4.9.3
• Repos and Hooks
• New git repositories to mirror (for tarako)
• Tools
• beta2release_l10n.sh does not handle merged locales properly
• linux_gecko should be linux32_gecko in trychooser
• Spot bidding should be able to filter out AZs with bad conditions
• b2g_bumper should check tree status
• Integrate slaveapi into slave heath

In progress work (unresolved and not assigned to nobody):

• Balrog: Frontend
• balrog's rules page is excessively wide
• General Automation
• support desktop build builders being created from Mozharness (SigningScriptFactory) or through Buildbot (MBF)
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• Tracking bug for 17-mar-2014 migration work
• Schedule Android 4.0 Debug M4,M5,M6,M7,J1,J2,J3 (only) on all trunk trees and make them ride the trains
• Stop running mochitest-browser-chrome on B2G branches
• Pass tooltool_script into misc.py, and make it work
• Requesting OMTC Reftest (Ro) variant for Windows
• Add tooltool support to Windows builds
• [tracking bug] Run desktop unittests on Ubuntu
• [Meta] Some "Android 4.0 debug" tests fail
• [tracker] run Android 2.3 test jobs on EC2
• b2g_build.py steps aren't independent
• Please schedule linux jit-tests in two chunks
• take ownership of and fix relman merge scripts
• Run desktop mochitests-browser-chrome on Ubuntu
• Rename cppunittest to cppunit in mobile_config.py
• Do not use spot instances for some builders
• Do nightly builds with profiling disabled
• Install ant on builders
• prep buildbot.py for desktop builds in mozharness
• Run additional hidden 3 debug mochitest-browser-chrome chunks on m-i, and then switch on all trunk branches
• Please add non-unified builds to mozilla-central
• Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with "command timed out: 1200 seconds without output, attempting to kill", "command timed out: 1800 seconds without output, attempting to kill"
• Prepare infra to handle spot instances
• Fix compression for non-compressed uploads
• Bengali in 1.3t builds
• submit release builds to balrog automatically
• [tarako][build]create "tarako" build
• Intermittent Linux spot builder "command timed out: 2700 seconds without output, attempting to kill" while trying to install mock
• Loan Requests
• Slave loan request for a windows 7 32 bit machine
• Loan felipe an AWS unit test machine
• loan linux32 (hardware) slave to jmaher
• Need a slave for bug 818968
• Loan talos-linux64-ix-001 to dminor, ctalbert & gbrown
• Please loan dminor t-xp32-ix- instance
• Slave loan request for a tst-linux64-ec2 machine
• Slave loan request for a t-w864-ix machine
• Slave loan request for a t-w732-ix machine
• Loan an ami-6a395a5a instance to Dan Glastonbury
• Other
• Make AWS node type available to graphite & build metadata
• s/m1.large/m3.medium/
• Allow non-RelEng folk to enable/disable snippets and move/copy them between channels
• Platform Support
• [tracker] Move away from the rev3 minis
• Update builders to Android SDK platform 17
• Experiment with other instance types for Android 2.3 jobs
• Add msttcorefonts package to Servo buildbots
• Upgrade Windows 8 Nvidia drivers to 335.23
• figure out why test machines can't set resolution sometimes
• release-automation
• Thunderbird Windows L10n repack builds failed with pymake enabled
• staging release runner instructions don't work with new dev-master1/ssh key management
• Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
• Releases
• do a staging release of Firefox and Fennec 29.0b1
• tracking bug for build and release of Firefox and Fennec 29.0
• Add SeaMonkey 2.25 Beta 3 to bouncer
• Yandex partner repack changes for Fx 28 release
• tracking bug for build and release of Firefox and Fennec 28.0
• Tools
• Store the timing of actions in slaveapi
• more vcs-sync email tweaks
• Add an option to trychooser to select Talos profiling options
• self-serve agents eat exceptions, causing stalls
• self-serve agents' error handling needs more care
• db-based mapper on web cluster

Major Highlights:

• Armen has been working on upgrading the Mesa libraries on our EC2 test instances. When completed, this will let us scale B2G reftests to EC2!
• Coop moved fuzzing jobs off of our very old Linux VMs, which lets us retire that platform.
• Nick and I got many Balrog bugs ready to land. Once deployed we'll be able to start dogfooding it with Firefox Betas.
• Callek added Tegra support to SlaveAPI, which will let it try to automatically fix broken Tegras, which helps keep test turnaround time fast.
• Massimo continued his work on moving some of our AWS machines from m1 instances to m3, which saves us money without sacrificing performance.
• Mike is working on adding tooltool support to Windows builds. When completed, it will make it possible for developers to self serve some build tool upgrades and test them on try.
• Rail, Armen, Jordan, and Gregory met and hammered out a great plan for a better interface between RelEng automation and the build system.

Completed work (resolution is 'FIXED'):

• Balrog: Backend
• send cef events to syslog's local4
• Buildduty
• Add swap to linux build machines with
• servo linux slaves couldn't successfully run puppet
• Pandas configured to connect to non-running / non-existing buildbot masters 39, 40, 41
• General Automation
• use tbirdbld account to submit thunderbird data to balrog
• Increase filesize limit for mac signing (Polish mac l10n nightly updates missing since 2014-02-28)
• run tests on ASAN builds
• Run win64 unit tests at bootup on tst-w64-ec2-xxx fork
• Remove --no-update from B2G media test invocation
• Jacuzzis for mozilla-inbound
• Refactor cloud-tools a bit
• Change birch config and add jacuzzis
• Install b2g desktop builds using mozinstall for gaia-integration
• allow mozharness's tbox_print_summary() to be used outside of unittests
• Create 'win64-debug' platform, and enable on date and try
• allow for multiple basedirs to be passed to PurgeMixin.purge_builds()
• port MockReset to mozharness
• Schedule gaia-integration tests on OSX on cedar
• add --dump-config and --dump-config-hierarchy to mozharness
• Schedule gaia-ui-tests on cedar against emulator builds
• Add support for add-if-not instruction added by bug 759469 to the mar generation scripts
• Tweak buildername for Android l10n nightlies to make them more consistent with other multi-part jobs
• Remove the version 1 manifest from the mar creation scripts
• Loan Requests
• mac minis for new puppet version
• Connect a linux and mac slave to dev-master1:8950 for pkewisch
• Loan a B2G emulator machine to baku
• Please loan dminor instance to build Android 2.3 Emulator
• Other
• Disable aurora daily updates (26) until merge to mozilla-aurora has stabilized
• give larsberg access to servo machines
• spot bidding library
• Platform Support
• Intermittent Android x86 We have not been able to establish a telnet connection with the emulator
• release-automation
• replace TuxedoEntrySubmitterFactory with a mozharness script
• Releases
• Please make FF24.0 release updates manual-only (likely Thursday - 9/19 afternoon PT)
• Repos and Hooks
• New upstream git repos to mirror
• Tools
• Release and update slaveapi to support tegras
• Store the timing of actions in slaveapi
• Add cppunittests as option in TryChooser
• cut over build/* repos to the new vcs-sync system
• slaveapi still files IT bugs for some slaves that aren't actually down
• Self serve API for oak not building Oak nightly builds even know there is a nightly from 2 days ago

In progress work (unresolved and not assigned to nobody):

• Balrog: Backend
• Support new attributes for update.xml in Balrog
• /rules/:id endpoint needs to return data_version/csrf token
• exception in getLocaleOrTopLevelParam since latest balrog push to prod
• Balrog: Frontend
• /revisions/ page doesn't show who made the most recent change to a rule
• balrog's rules page is excessively wide
• need to be able to delete rules from admin ui
• rules page doesn't update properly after new rule creation or existing rule updates
• fix balrog admin ui title not to read "AUS3"
• General Automation
• support desktop build builders being created from Mozharness (SigningScriptFactory) or through Buildbot (MBF)
• ASan is the only Linux platform running the separate jittest suite on try
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• b2g_bumper can add extra comments
• Tracking bug for 17-mar-2014 migration work
• Do debug B2G desktop builds
• Requesting OMTC Reftest (Ro) variant for Windows
• Add tooltool support to Windows builds
• decommission (releng side) aws buildbot masters migrated to scl3
• Add configs for debug B2G desktop builds & enable on mozilla-central and mozilla-aurora
• Run mozbase unit tests from test package
• Please schedule linux jit-tests in two chunks
• fx desktop builds in mozharness
• bld-lion-r5-037 debug failures may be showing a problem with bug 969689
• mozharness should use --manifest instead of --test-manifest for emulator-jb mochitests
• Please add non-unified builds to mozilla-central
• Do nightly builds with profiling disabled
• Install ant on builders
• for mozharness desktop builds, replace usage of client.mk with upcoming mach/python script
• Add support for non-unified builds in mozharness desktop builds
• port mozharness desktop build cfgs from mozharness to in-tree
• prep buildbot.py for desktop builds in mozharness
• [tarako][build]create "tarako" build
• Enable all mozharness desktop build linux variants on Cypress
• Run unittests on Win64 builds
• Device Builds Tests/Builds for mozilla-inbound
• Make TBPL gaia-integration download node modules from git mirror
• Increase free space requirement for linux debug builds
• need android build slave for Servo
• Intermittent Linux spot builder "command timed out: 2700 seconds without output, attempting to kill" while trying to install mock
• Loan Requests
• Please loan a b2g emulator testbot to debug b2g emulator failures on https://tbpl.mozilla.org/?tree=Try&rev=aebcb16d05fb
• Loan t-w864-ix-003 to jrmuizel
• Need a win32 unittest slave for debugging packaged tests
• Slave loan request for gbrown
• loan osx 10.8 slave to jmaher
• Please loan dminor t-xp32-ix- instance
• Loan of linux64 test machine to Steve Fink for M-oth
• Other
• Upgrade ASan Clang in Q1
• Decide on document workflow for releng docs
• s/m1.large/m3.medium/
• Make AWS node type available to graphite & build metadata
• Platform Support
• [tracker] Move away from the rev3 minis
• Update builders to Android SDK platform 17
• Experiment with other instance types for Android 2.3 jobs
• Add msttcorefonts package to Servo buildbots
• Upgrade ec2 test instances mesa versions from 8.0.2-0ubuntu3(32-bit hosts)/8.0.4-0ubuntu0.2(64-bit hosts) to 8.0.4-0ubuntu0.6mozilla1
• Upgrade Windows 8 Nvidia drivers to 334.89
• Setup Diamond to send metrics to hosted graphite
• Create a Windows-native alternate to msys rm.exe to avoid common problems deleting files
• release-automation
• Thunderbird Windows L10n repack builds failed with pymake enabled
• Bouncer submitter should print returned page content if hits a http error
• Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
• Releases
• tracking bug for build and release of Firefox and Fennec 28.0
• do a staging release of Firefox and Fennec 29.0b1
• Show Australis tour page upon update for en-US on Beta
• Yandex partner repack changes for Fx 28 release
• tracking bug for build and release of Firefox and Fennec 29.0
• tracking bug for build and release of Thunderbird 24.5.0
• tracking bug for build and release of Firefox 24.5.0 ESR
• do a staging release of Firefox and Fennec 30.0b1
• Repos and Hooks
• Fix documentation for hgrc [extension] paths for Windows
• Tools
• Set up auto publish of doc index per Boston workweek agreement
• b2g_bumper should check tree status
• build-tools: Can't push /opt/vcs2vcs/vcs_sync/build/conversion/build-tools to build-tools-github!
• Slaveapi should support mobile devices for reboots
• Add an option to trychooser to select Talos profiling options
• Move mqext into the new https://hg.mozilla.org/hgcustom/version-control-tools repo
• tracker for all pending releng nagios work
• Spot bidding should be able to filter out AZs with bad conditions

Highlights:

• Catlee worked on a lot of tooling for Jacuzzis, and we enabled them for more builders, which is greatly speeding up turnaround time for them.
• He also changed the algorithm used when spinning up AWS machines which has significantly reduced the number of unused hours our machines run for.
• Armen submitted a talk proposal for next year's RelEng Workshop.
• John Hopkins got a large portion of machines sending data to hosted Graphite service which will help us analyze utilization more easily.
• John Zeller finished work on a patch that allows developers and sheriffs to trigger builds for any revision. This is a feature that's been wanted for quite awhile, and helps pave the way for things like automatic bisection.
• Pete and Simone learned the basics of Buildduty from Coop, which should help us be more responsive to infrastructure issues during the European working day.

Completed work (resolution is 'FIXED'):

• Balrog: Backend
• API for manipulating balrog rules
• balrog should return empty snippets instead of 500s
• Balrog shouldn't serve updates to older builds
• Buildduty
• buildbot-master70 is having trouble with puppet
• Integration Trees closed, high number of pending linux compile jobs about ±2 hour backlog
• What happened at 18:00 PT that disconnected Windows builds across all trees?
• upload mozlog-1.5 to http://pypi.pub.build.mozilla.org/pub
• Deploy python 2.7.3 to all build machines
• upload mozinfo-0.7 to http://pypi.pub.build.mozilla.org/pub
• General Automation
• B2G sources.xml (or ) should state which github repos map to the repos/revisions listed
• Rooting analysis mozconfig should be in the tree
• [Meta] Tree closing/reliability issues with builds-pending.js, builds-running.js, builds-4hr.js.gz or buildapi in general
• aws_watch_pending.py should use jacuzzis
• Set --timeout 10000 on gaia-ui-test command line
• Don't reset mock environments if we don't have to
• Schedule multimedia b2g mochitests for emulator-jb on Cedar
• Do our own hg purge
• aws sanity check shouldn't report instances that are actively doing work as long running
• move off of dev-master01
• Update b2g_bumper.py with a git dict
• Remove 'update_files' logic from B2G unittest mozharness scripts
• rewrite watch pending to cope better with spot requests that aren't being fulfilled
• Use spot instances for regular builds
• write mozconfigs for non-unified builds
• Mozharness swallows return_code of a talos run if it is 1
• [tracking bug] migrate hosts off of KVM
• Create IAM roles for EC2 instances
• Configure holly like Aurora properly
• Remove linux-opt arm-sim builder
• Loan Requests
• Please loan dminor Android 2.3 Emulator test instance
• Please loan dminor instance to build Android 2.3 Emulator
• Linux 64-bit test slave for bug 926264
• loan linux slave to jmaher
• Other
• buildbot master manifests should ensure firewall is disabled
• Create (private?) repo holding TEF customization data
• The stage.m.o:/mnt/cm-ixstore01 mount is filling up
• Need to increase file size limit for mar files to enable linux+mac mar signing
• selfserve agent drops requests
• Platform Support
• Re-allocate tegras that were on decommissioned foopy118
• [tracker] move jobs away from obsoleted machines
• remove joduinn from puppet controlled access
• Setup in-house buildbot masters for remaining in-house testers
• release-automation
• replace TuxedoEntrySubmitterFactory with a mozharness script
• Support modifying update-settings.ini when doing update verify
• Create SSL products in bouncer as part of release automation
• Request notification of events affecting release timelines
• Repos and Hooks
• Request for a /gaia-l10n/lt repository
• Please create git.mozilla.org mirror for https://github.com/mozilla-b2g/gaia-node-modules
• Tools
• Use MozillaPulse from pypi
• raise slaverebooter's parallelization
• Replace dev-master01 for dev-master1 in our repos
• Investigate seemingly incorrect CVS -> git commit
• buildfaster report doesn't know about some builders/OS

In progress work (unresolved and not assigned to nobody):

• Balrog: Backend
• /rules/:id endpoint needs to return data_version/csrf token
• Balrog: Frontend
• balrog's rules page is excessively wide
• Buildduty
• Add swap to linux build machines with
• General Automation
• Override ASan malloc_context_size option
• ASan is the only Linux platform running the separate jittest suite on try
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• Provision enough in-house master capacity
• Tracking bug for 17-mar-2014 migration work
• Change birch config and add jacuzzis
• Provide B2G Emulator builds for Darwin x86
• Do debug B2G desktop builds
• decommission (releng side) aws buildbot masters migrated to scl3
• Add support for add-if-not instruction added by bug 759469 to the mar generation scripts
• Tweak buildername for Android l10n nightlies to make them more consistent with other multi-part jobs
• use tbirdbld account to submit thunderbird data to balrog
• [Meta] Some "Android 4.0 debug" tests fail
• Remove --no-update from B2G media test invocation
• Normalize builder names
• [Meta] Fix + unhide broken testsuites or else turn them off to save capacity
• b2g_build.py steps aren't independent
• Please schedule linux jit-tests in two chunks
• add --dump-config and --dump-config-hierarchy to mozharness
• port MockReset to mozharness
• allow for multiple basedirs to be passed to PurgeMixin.purge_builds()
• allow mozharness's tbox_print_summary() to be used outside of unittests
• Schedule gaia-integration tests on OSX on cedar
• Create SpiderMonkey builds for Windows on TBPL again
• desktop_unittest.py should allow using existing files
• Do nightly builds with profiling disabled
• Install ant on builders
• Add Linux32 debug SpiderMonkey ARM simulator build
• Run mozbase unit tests from test package
• Allow the possibility of spidermonkey builds on more platforms
• Please add non-unified builds to mozilla-central
• Device Builds Tests/Builds for mozilla-inbound
• submit release builds to balrog automatically
• Intermittent Linux spot builder "command timed out: 2700 seconds without output, attempting to kill" while trying to install mock
• Loan Requests
• loan windows 7 slave to jmaher
• Loan a B2G emulator machine to baku
• Loan t-w864-ix-003 to jrmuizel
• Loan t-w732-ix-003, t-w732-ix-004 to Q
• Loan felipe an AWS unit test machine
• Other
• give larsberg access to servo machines
• [tracking] infrastructure improvements
• Decide on document workflow for releng docs
• s/m1.large/m3.medium/
• stage NFS volume about to run out of space
• Make AWS node type available to graphite & build metadata
• Platform Support
• Deploy git to windows builders
• Update wget to 1.15 on foopies
• Migrate ESR, b2g26_v1_2, b2g28_v1_3 branches to win64-rev2
• Update builders to Android SDK platform 17
• signing win64 builds is busted
• Update VC++2010 runtime to MS11-025 updated version
• Add msttcorefonts package to Servo buildbots
• Produce crash report symbols for darwin-on-linux builds
• foopy watch_devices does not clear lockfile when it fails to get data from slavealloc
• Configure VS2012 builds in staging
• Deploy new version of MozillaBuild from bug 927213 on windows build slaves
• [tracker] Determine how to support and roll out darwin-on-linux in production
• deploy mac on linux build tools to tooltool
• puppet should setup directory structure on foopies
• Update all foopies to Python 2.7.3
• Deploy patched version of Mesa 8.0.4
• Upgrade Windows 8 Nvidia drivers to 334.89
• Setup Diamond to send metrics to hosted graphite
• Create a Windows-native alternate to msys rm.exe to avoid common problems deleting files
• release-automation
• Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
• Releases
• Add SeaMonkey 2.25 Beta 2 to bouncer
• Releases: Custom Builds
• Fix outdated and duplicated landing pages for MSN/Bing builds
• Update Rambler pack with updated extension 4.9.3
• Repos and Hooks
• Create a script to import one Mercurial repository into a subdirectory of another, with linear history
• Tools
• more vcs-sync email tweaks
• build-tools: Can't push /opt/vcs2vcs/vcs_sync/build/conversion/build-tools to build-tools-github!
• Slaveapi should support mobile devices for reboots
• Add an option to trychooser to select Talos profiling options
• Create a new landing page for releng
• slaveapi still files IT bugs for some slaves that aren't actually down
• legacy vcs-sync process doesn't auto retry when it should
• Add buildbot database sources into Tableau

Like most other engineering groups at Mozilla, Release Engineering uses Bugzilla to track most of the work we do. Our bugs and patches use many of the same flags as a Firefox bug, so it's no surprise that many people expect our bugs to be like Firefox bugs in other ways. I've noticed a few instances of confusion recently, so I'd like to try to clear things up a little bit by talking about what's different about ours.

In order to understand our bugs, a brief overview of our repositories is required. Unlike Firefox, RelEng hosts its code in a number of different Mercurial repositories. Most of these repositories have (or should have) a "production" branch in addition to the "default" one. Our production systems (Buildbot masters/slaves, AWS management machines, signing servers, etc.) track the production branches of our repositories. This is important for a few reasons:

• Some patches can depend on other patches in a different repository
• Some patches need to be deployed at specific times
• Some systems cannot automatically use new code
• Every commit has the potential to close the trees

Once a patch has been reviewed it will be landed on the default branch of its repository. At this time the checked-in attachment flag is set to "+". Unlike Firefox bugs, the bug stays open at this point. Many of our repositories have their own continuous integration tests that watch the default branches and do as much up-front verification as they can. Like mozilla-central, we do our best to stay in a shippable state at all times - so if we have test failures, they get fixed quickly or backed out. Sometime later (usually about once per day), someone will merge all of the pending patches to the production branches and make sure the production systems pick them up. Once this is done, they will update the Maintenance page and add a note like "In production" to all of the bugs that had patches merged. Any new jobs that start after this point will be using all of the new code. In most cases bugs will be left open, and closed by their assignee when appropriate.

I hope this helps the next time you're confused about the state of some RelEng work!

Highlights:

• Rail spent some time optimizing spot instance bidding, which helps lower our AWS bill.
• Massimo improved one of our major AWS usage reports, which will also help us be more efficient with our usage.
• Ahal got us uploading qemu.logs to blobber, making it easier to debug some types of failures.
• Mike got his patch for "hg purge" landed, which has improved try build times!
• Callek moved all of our in-house test machines onto in-house masters, which takes additional load off of our AWS VPN link (which reduces the chance of tree closures due to it failing).
• Dan continued his work on moving MozBase tests out of the build jobs, which improves build and test turnaround time.
• John started figuring out how to make our signing servers support 64-bit Windows binaries, which is one thing (out of many) that we need to do before shipping 64-bit Windows builds.
• Catlee continued work on Jacuzzi implementation, making good progress on some of the Buildbot implementation that we need.
• Nick made additional progress on fixing our release automation to support shipping Release builds to Beta users for extra testing before we ship for real.

Completed work (resolution is 'FIXED'):

• Buildduty
• mac-signing4's nagios is having issues
• Report impaired AWS non slave machines
• upload a new talos.zip to capture recent talos changes
• Add swap to linux build machines with
• builds-4hr.js.gz not updating, all trees closed
• Please upload mozfile-1.1 to http://pypi.pub.build.mozilla.org/pub
• some test machines unable to connect to their masters
• All trees closed due to timeouts as usw2 slaves download or clone
• General Automation
• Need helix-eng builds for test automation
• Rooting analysis mozconfig should be in the tree
• Periodic PGO and non-unified builds shouldn't be running again on pushes that already have them
• Implement ghetto "gaia-try" by allowing test jobs to operate on arbitrary gaia.json
• Make Pine use mozharness production & limit the jobs run per push
• Save qemu.log to blobber
• decommission final set of KVM buildbot masters back to IT
• Compression for blobber
• Temporarily revert the change to m3.medium AWS instances to see if they are behind the recent increase in test timeouts
• Do our own hg purge
• Lots of Command Queue nagios alerts from new AWS masters
• Loan Requests
• Please loan t-xp32-ix- instance to dminor
• Loan glandium a m3.medium test slave
• Loan jgilbert a Linux M1/R test slave
• Need a slave for bug 818968
• Loan Win8 test slave to Jim Mathies for soft keyboard investigation work
• Slave loan request for a tst-linux64-ec2 vm
• Other
• Grant pkewisch access to dev-master1
• Platform Support
• Intermittent Android x86 We have not been able to establish a telnet connection with the emulator
• release-automation
• release builds not using space requirement of platform config
• Releases
• tracking bug for build and release of Firefox 27.0.1
• Releases: Custom Builds
• Clean up MSN bundles and restore MSN add-on to the correct URL
• Repos and Hooks
• Determine why github mozilla/mozilla-central has been removed from service by github
• Tools
• Self-serve nightly wildcard matching is not strict enough (can't retrigger UX nightlies)
• Create a staging environment for buildapi

In progress work (unresolved and not assigned to nobody):

• Balrog: Backend
• API for manipulating balrog rules
• Buildduty
• Setup tegras that are returning from loan
• Deploy python 2.7.3 to all build machines
• rewrite watch pending to cope better with spot requests that aren't being fulfilled
• General Automation
• mozilla-central should contain a pointer to the revision of all external B2G repos, not just gaia
• Don't reset mock environments if we don't have to
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• Schedule multimedia b2g mochitests for emulator-jb on Cedar
• Provision enough in-house master capacity
• Provide B2G Emulator builds for Darwin x86
• Do debug B2G desktop builds
• keep buildbot master twistd logs longer
• [tracking bug] Run desktop unittests on Ubuntu
• Add support for add-if-not instruction added by bug 759469 to the mar generation scripts
• Configure holly like Aurora properly
• [Meta] Some "Android 4.0 debug" tests fail
• b2g build improvements
• Run mozbase unit tests from test package
• fx desktop builds in mozharness
• Move Firefox Desktop repacks to use mozharness
• add --dump-config and --dump-config-hierarchy to mozharness
• port MockReset to mozharness
• allow for multiple basedirs to be passed to PurgeMixin.purge_builds()
• allow mozharness's tbox_print_summary() to be used outside of unittests
• Use spot instances for regular builds
• Create SpiderMonkey builds for Windows on TBPL again
• Do nightly builds with profiling disabled
• Windows ix test machines don't always start buildbot
• Don't clobber the source checkout
• Add Linux32 debug SpiderMonkey ARM simulator build
• aws sanity check shouldn't report instances that are actively doing work as long running
• Allow the possibility of spidermonkey builds on more platforms
• [tarako][build]create "tarako" build
• move off of dev-master01
• Schedule JB emulator builds and tests on cedar
• [tracking bug] migrate hosts off of KVM
• Run jit-tests from test package
• Intermittent Linux spot builder "command timed out: 2700 seconds without output, attempting to kill" while trying to install mock
• Loan Requests
• Loan felipe an AWS unit test machine
• need a loaner bld-linux64-ec2 instance
• Slave loan request for Matt Woodrow
• Loan an ami-6a395a5a instance to Aaron Klotz
• Please lend Andreas Tolfsen a linux64 EC2 m3.medium slave
• Please loan dminor instance to build Android 2.3 Emulator
• Loan :sfink a linux64 bld-centos6-hp-005
• Slave loan request for a tst-linux64-ec2 machine
• Other
• [tracker] Machine move SCL1 -> SCL3
• [tracking] infrastructure improvements
• s/m1.large/m3.medium/
• Platform Support
• Migrate ESR branches to win64-rev2
• Setup in-house buildbot masters for remaining in-house testers
• signing win64 builds is busted
• Windows slaves often get permission denied errors while rm'ing files
• Deploy ndk-stack on foopies
• do initial planning on "chunks" of machines to move.
• slave pre-flight tasks
• Deploy patched version of Mesa 8.0.4
• release-automation
• Support modifying update-settings.ini when doing update verify
• Create SSL products in bouncer as part of release automation
• Figure out how to offer release build to beta users
• cache MAR + installer downloads in update verify
• Releases: Custom Builds
• Update Rambler pack with updated extension 4.9.3
• Repos and Hooks
• Request for a /gaia-l10n/lt repository
• disable & delete mirror of gaia-ui-tests
• Tools
• [Tracking bug] - Assisted/Auto Landing from Bugzilla to tip of $branch
• Store the timing of actions in slaveapi
• Move trychooser hg extension into the new https://hg.mozilla.org/hgcustom/version-control-tools repo
• Use MozillaPulse from pypi
• cut over build/* repos to the new vcs-sync system
• slaveapi still files IT bugs for some slaves that aren't actually down
• Replace dev-master01 for dev-master1 in our repos