Monthly Archives: October 2013

How to deal with timezone adjusted “epoch” timestamps in Python

Today I discovered that we have a system that returns “epoch” timestamps, but adjusted for Pacific time. This means that depending on whether daylight savings time is in effect, these timestamps are 7 or 8 hours ahead when interpreted by most tools. These are horribly difficult to deal with as unix timestamps are assumed to be in UTC time. I spent a good deal of time banging my head against Python’s datetime and pytz modules (as well as the wall). With some help from John Hopkins I found a solution:

In the following example we’ll convert the Pacific “epoch” timestamp 1383000394 to a proper epoch timestamp (which is 1382975194).

First, we need a tzinfo object for Pacific time:

>>> import pytz
>>> pacific_time = pytz.timezone("America/Los_Angeles")

Next, we need to get the initial timestamp into a datetime object to work with it. Note that it’s important to use utcfromtimestamp() here otherwise you’ll get a localized datetime object – which will only be useful if the machine you run this on is in Pacific time:

>>> from datetime import datetime
>>> dt = datetime.utcfromtimestamp(1383000394)
>>> dt
datetime.datetime(2013, 10, 28, 22, 46, 34)

It gets a little weird from here. We need to subtract the Pacific offset from the datetime object in order to get it into an actual UTC time. To do that we can force the datetime object in UTC time and then use its built-in astimezone() method to do the conversion. I think this still leaves a 7 or 8 hour window whenever DST starts or ends where this conversion is an hour off – but it’s good enough for my usage:

>>> dt = dt.replace(tzinfo=pytz.utc)
>>> dt
datetime.datetime(2013, 10, 28, 22, 46, 34, tzinfo=)
>>> dt = dt.astimezone(pacific_time)
>>> dt
datetime.datetime(2013, 10, 28, 15, 46, 34, tzinfo=)

Now we have a datetime object with the correct time, but claiming to be in Pacific. We can fix that by replacing the tzinfo again:

>>> dt = dt.replace(tzinfo=pytz.utc)
>>> dt
datetime.datetime(2013, 10, 28, 15, 46, 34, tzinfo=)

The only thing left to do now is convert to epoch time!

>>> import calendar
>>> calendar.timegm(dt.utctimetuple())
1382975194

Voila, the timestamp we were looking for!

Much credit to John Hopkins for his code that taught me how to use datetime.replace() and astimezone(). No credit at all goes to Python’s datetime module, which is sorely in need of an overhaul.

Summit Takeaways

For me, events like the Mozilla Summit are much more about getting to know new people and building rapport. Even given my focus on that there’s lots of interesting and random things that made an impact on me over the past week:

  • Webmaker and Appmaker could be incredibly important. Someone in Santa Clara made the analogy of these tools being like simple tools like hammers, screwdrivers, etc. that let anyone create things from scratch – whether it’s for just themselves, a small group of people, or the wider world.
  • Matt Thompson made a great point about “Reinvest[ing] in mentorship” in his recent blog post. My originally involved with Mozilla probably wouldn’t have gone anywhere if it wasn’t for people mentoring me. I’ve done my share of teaching and helping within my own group at Mozilla, but Matt’s post made me realize that I’ve never actually paid this forward elsewhere.
  • I had a conversation with someone about a request for input I got from someone on a complicated topic. I’ve grown hesitant to reply to these in recent times because I’m afraid of wrongly committing myself or others to work. The person I spoke with reinforced with me that I shouldn’t ever be afraid to comment on technical things because of this. This seems like a simple thing upon reflection, but it really adjusted my perspective.
  • All the face to face time – both with the people I talk with daily and with those I barely know – is very useful. I got new perspective on people I’ve known for a long time, built rapport, and was able to thank a lot of people for things they’ve helped me with over the years. I also learned a lot of things I wouldn’t have otherwise by talking with people I’ve never met. For this reason alone I hope we can find ways to have larger Mozilla events more often (that is, larger than a team’s work week but smaller than a full Summit).

New AUS is live!

As previously announced, Nightly users of Firefox and Fennec were switched to the new update server yesterday. We’ve been doing manual testing, watching logs, and keeping an eye metrics since then and as far as we can tell everything is functioning as expected and users are being updated. If you are on Nightly and experience any problems updating, please stop by #releng or file a bug.

While I’m on the one making this announcement, I’m far from the only person who’s worked on it. Balrog, as we’ve been calling internally, is the the brainchild of Nick Thomas, who made the first commit back in December 2010. The two of us have been working on it off and on since then with lots of help from numerous other people:
* Erick Dransch and Peter Bengtsson created most of the administrative UI and API.
* Brandon Burton, Chris Turra, and Sheeri Cabral set-up all of the infrastructure that hosts Balrog.
* Rail Aliiev wrote the client code that our build machines use to send data to Balrog.
* Too many people to name in RelEng have brainstormed, sanity checked, reviewed code, and helped other ways.

There’s still more work to do on Balrog, but this is a huge step for us, and validates all the work we’ve already done on it. Expect more posts from myself and others as we continue to make progress.