Monthly Archives: September 2012

Python and SSL Certificate Verification

We’re in the process of developing a new version of our update server. It’s at the point now where we have a development environment set-up for it, and slaves attempt to submit data to it. Recently we came across an issue where slaves were unable to successfully validate the SSL certificate of the development environment. Specifically, it was raising this error from OpenSSL:

requests.exceptions.SSLError: [Errno 1] _ssl.c:480: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Our client uses the Python Requests library, and gives it a specific certificate to validate the server certificate with. For comparison we tried using openssl’s s_client with the same cert, which was able to validate the certificate just fine given the same inputs. HRM.

I tried tons of things to get it to work – different versions of the Requests library, giving it the root + intermediary cert, different versions of Python – but nothing worked! I couldn’t even reproduce the behaviour with pure openssl no matter what inputs I gave it. I reached out to #security for support and Yvan couldn’t even reproduce the problem on his Mac!

Eventually I reached out to #python-requests, thinking it was a bug in the library. Someone there suggested strace’ing both my python script and openssl. I did that, and found something very interesting: despite being given an explicit certificate bundle, openssl fell back onto the system certificates — my python script didn’t. To verify this, I changed my script to look at the full system root rather than just a specific certificate or bundle. After doing that, everything worked fine.

My belief is that one of PyOpenSSL, urllib3, or Requests doesn’t know how to look at the system certificate store at all on Windows or Linux. On Mac, it seems to fall back just fine on it. One thing is certain: security is hard, especially debugging it.

Big thanks to Jake Maul, Rail, Yvan Boily, Dveditz, and #python-requests for helping me debug this.

Jalapeño Picante Sauce Recipe

I’ve been growing jalapeño peppers this year. My first haul of them came out extremely bland – more like green peppers with almost no spice to them. This happened the last time I tried growing them too, and I nearly gave up hope. After a whole lot of googling I came across multiple suggestions that overwatering can cause bland peppers. I stopped watering my plant immediately and a week later I had spicy jalapeños! In the end I managed to harvest about 25 peppers – more than I ever thought I could get in one season from one plant. With so many peppers to deal with I really had no choice but to make a sauce!

Like my habanero sauce recipe, this jalapeño sauce is largely based on another recipe and rather simple. I didn’t have all the listed ingredients on hand so I had to make some modifications:


  • 10 jalapeño peppers (mine were 1/2 spicy and 1/2 bland), finely chopped
  • 3 hot house tomatoes, chopped
  • 1 large onion, chopped
  • 4 cloves garlic, minced
  • 3/4 cup cider vinegar
  • 2 tbsp oregano
  • salt to taste


  1. Cook tomatoes, onion, garlic, vinegar, and oregano in a saucepan until softened.
  2. Add the jalapeños and cook until they have softened and the mixture has taken on some of the green colour of the peppers.
  3. Using a blender or food processor, puree until smooth. Depending how much moisture was lost during cooking you may need to add a bit of water.
  4. Pour the mixture back into the saucepan over low heat.
  5. Season with salt to taste.

I brought some of this to the office today and folks found it be quite good as a dip for pita chips. I suspect it would work well as a garnish on sandwiches, burritos, etc. as well.

Smaller & faster updates now accessible to more Firefox users

When a user receives an update to Firefox they get either a partial or complete. A complete is nearly identical to its associated installer, and can update any old Firefox version. A partial is a binary diff of a specific old version against a newer one and only compatible with that specific old version. The size difference a partial and a complete can be huge. For example, the complete MAR for 14.0.1, en-US, win32 was 20MB. The partial from 13.0.1 was 7.4MB (even smaller on other platforms, where PGO doesn’t make diffing hard).

Until recently, we’ve only been able to produce partials against a single old version without a lot of extra time consuming and error prone manual work — meaning that a lot of users who could benefit from a partial weren’t receiving them.

With bug 773290 (multiple partial MAR support in release automation) resolved, we can now offer partial updates to many versions without the risk and time cost of doing them by hand. When we shipped 15.0 we offered partial updates to users on 14.0.1, 13.0.1, and 12.0 – which collectively represented just over 75% of our installed userbase. For future releases it’s possible we’ll offer partials to even more previous versions.

I’m sure some of you are asking why we can’t just do partial updates for ALL old releases. There’s a couple of reasons for that. Most importantly, there’s big diminishing returns on partial updates. The 13.0.1 -> 15.0 partial was 12MB, the 12.0 -> 15.0 partial was 14MB, and the complete was 20MB. The further you go back the less you gain from getting a partial. Secondly, computing partial updates is not computationally cheap. We ship 88 locales across 4 platforms — this works out to about 350 partials that need to be calculated. This can be made cheaper through caching and parallelization, but it still ends up adding about 45min to the running time of the release automation for every extra version we want partials for. In turn, this delays QA and other things in the critical path of shipping.

I also want to point out that this work does NOT apply to Nightly or Aurora. We have no plans to offer multiple partial updates on those channels at this time. Due to their relatively low userbase and very high frequency of change (almost every 24h), the cost/benefit just doesn’t work. However, we will be looking at using this on Beta where the userbase is much larger and the rate of change slower (about once a week).

A huge thanks goes out to Rail Aliev and Nick Thomas, who helped work out the design, wrote some parts of it, and provided reviews. We couldn’t have had this ready for 15.0 without their help.

Fresh-from-the-garden habanero hot sauce

WARNING: Do not touch your eyes, nose, mouth, or any other sensitive areas of your body when working with habaneros. You will regret it.

This year I tried to grow some habanero peppers in my garden. I only ended up with a single one from the entire plant, but even a tiny bit packs a punch. Because of that, the only real use for me is to make a sauce. I started by scaling back this recipe which calls for 5 habaneros. The sauce still ended up way too hot for me, so I further modified it to calm it down. The recipe I came up with is as follows:


  • 1 tablespoon olive oil
  • 1/4 cup chopped carrots
  • 1/4 cup chopped onion
  • 2 cloves garlic
  • 1 habanero pepper, seeds removed
  • 1 tbsp water
  • 3 tbsp lime juice
  • 1 tbsp white vinegar
  • 2 tomatoes
  • 1 tbsp sugar
  • salt and pepper to taste


  1. Saute the carrots, onion, and garlic in a saucepan until softened.
  2. Pour them into a blender along with the habanero, water, lime juice, vinegar, and tomatoes. Blend until smooth.
  3. Put the mixture back into the pan and bring to a simmer.
  4. Add the sugar and season with salt and pepper. Stir until combined.
  5. Reduce the mixture to your desired consistency.

The finished sauce should start off sweet and end up firey. If it’s too hot more lime juice or sugar can be added to calm it down.