Launching Xbox, Part 1 - The War Room

Monday, 7 December 2009
by Laurie Denness
filed under About Us and Tips and Tricks
Comments: 16

As many of you noticed, a few weeks ago we launched Last.fm on Xbox LIVE in the US and UK. It probably goes without saying that this project was a big operation for us, taking up a large part of the team’s time over the last few months. Now that the dust has settled, we thought we’d write a short series of blog posts about how we prepared for the launch and some of the tech changes we made to ensure that it all went smoothly.

0 Hour: Monitoring.

First up, let me introduce myself. My name is Laurie and I’ve been a Sysadmin here at Last.fm for almost two and a half years now. As well as doing the usual sysadmin tasks (turning things off and on again) I also look after our monitoring systems, including a healthy helping of Cacti, a truck of Nagios and a bucket-load of Ganglia. Some say I see mountains in graphs. Others say my graphs are infact whales. But however you look at it, I’m a strong believer in “if it moves, graph it”.

To help with our day-to-day monitoring we use four overhead screens in our operations room, with a frontend for Cacti (CactiView) and Nagios (Naglite2) that I put together. This works great for our small room, but we wanted something altogether more impressive — and more importantly, useful — for the Xbox launch.

At Last.HQ we’re big fans of impressive launches. Not a week goes by without us watching some kind of launch, be it the Large Hadron Collider, or one of the numerous NASA space launches.

We put a plan into action late on Monday evening (the night before launch), and it quickly turned into a “How many monitors can you fit into a room” game. In the end though, being able to see as many metrics as possible became useful.

So, ladies and gentlemen…

Welcome to the war room

Every spare 24” monitor in the office, two projectors, a few PCs and an awesome projector clock for a true “war room” style display (and to indicate food time).

Put it together and this is what you get:


Coupled with a quickly thrown together Last.fm style Nasa logo (courtesy our favourite designer), we were done. And this is where we spent 22 hours on the day of the launch, staring at the graphs, maps, alerts, twitter feeds.. you name it, we had it.

It was pretty exciting to sit and watch the graphs climb higher and higher, and watch the twists and turns as entire areas of the world woke up, went to work, came back from work (or school) and went to sleep. We had conference calls with Microsoft to make sure everything was running smoothly and share the latest exciting stats. (Half a million new users signed up to Last.fm through their Xbox consoles in the first 24 hours!)

As well as the more conventional style graphs, we also had some fun putting together some live numbers to keep up to speed on things in a more real time fashion. This was a simple combination of a shell script full of wizardry to get the raw number, then piped through the unix tools “figlet” (which makes “bubble art” from standard text) and “cowsay” (produces an ASCII version of a cow with a speech bubble saying whatever you please).

Looking after Last.fm on a daily basis is a fun task with plenty of interesting challenges. But when you’ve spent weeks of 12-hour days and working all weekend, it really pays to sit back in a room with all your co-workers (and good friends!) and watch people enjoy it. Your feedback has been overwhelming, and where would we have been without Twitter to tell us what you thought in real time?

Coming Next Time

We had to make several architectural changes to our systems to support this launch, from improved caching layers to modifying the layout of our entire network. Watch this space soon for the story of how SSDs saved Xbox…

Comments

  1. Medic
    7 December, 16:59

    The “Gone A While” timer on the Last.fm app has to go. What’s the point of streaming music if it stops playing every 20 minutes?

    Medic – 7 December, 16:59
  2. Matthew Ogle
    7 December, 17:05

    @Medic:

    Hmm, every 20 minutes? That doesn’t sound right — the timer should only kick in if you don’t interact with the app for more than an hour.

    The reason for the timer is that we pay for every track we stream, so streaming to an empty room gets expensive. Subscribers have a higher threshold (6 hours), if that helps.

    Matthew Ogle – 7 December, 17:05
  3. Xyz
    7 December, 17:13

    So why is it only released for UK and the US? As a paying subscriber outside those countries, that’s a pretty annoying restriction.

    Of course, I would never use the XBox to listen to music (it’s too noisy), but it sets a bad precedent.

    Xyz – 7 December, 17:13
  4. Matthew Ogle
    7 December, 17:16

    @Xyz:

    We’re looking into rolling it out in more countries next. For a variety of reasons it made the most sense to start with US/UK.

    Matthew Ogle – 7 December, 17:16
  5. Tecfan
    7 December, 17:22

    Xyz: probably licensing… (which Last.fm cannot do anything about – although it is sad to be a non-uk/us subscriber these days)

    as for the blog post: interesting stuff. I’m surprised you actually eat on dishes – not just pizza out of the box! kudos

    Tecfan – 7 December, 17:22
  6. Laurie Denness
    7 December, 17:31

    @Tecfan: I think I ate my breakfast off that plate actually, 18 hours earlier ;)

    Laurie Denness – 7 December, 17:31
  7. Allvaldr
    7 December, 18:44

    Letting my subscription run out this time. Still pissed that even though I pay for Last.fm and for Xbox Live, I get less options available to me than a free account from another country.

    Allvaldr – 7 December, 18:44
  8. Zen
    7 December, 19:32

    Last.fm can’t stream outside those countries for the same reason I can’t listen to Spotify in the us…licensing contracts!

    Zen – 7 December, 19:32
  9. Anon
    7 December, 20:57

    For monitoring, since you’re using Cacti, Nagios, and Ganglia, I’m surprised you aren’t (apparently) using GroundWork Monitor, which integrates those three and other stuff, and rocks pretty hard. Thanks for the great post! Very cool to see the inside story.

    Anon – 7 December, 20:57
  10. Robin
    8 December, 09:00

    Fantastic little article :)

    And thanks for all the pics, you guys are awesome. And grats on the 500,000 new users, its deserved.

    Oh i wish I could work there :D

    Robin – 8 December, 09:00
  11. Polyfragmentiert
    8 December, 12:00

    “turning things off and on again”… a classic “IT CROWD” quote!

    You guys do crazy stuff, nice.

    Polyfragmentiert – 8 December, 12:00
  12. der
    8 December, 16:27

    who cares about this?

    der – 8 December, 16:27
  13. CJ
    9 December, 11:32

    Why is it not possible to stream last.fm on the xbox dashboard or in games =( that would be a killer feature.

    CJ – 9 December, 11:32
  14. Ant718
    9 December, 18:05

    Now that that is all said & done when will you be on the Playstation Network?
    PS3 owners need love too.

    Ant718 – 9 December, 18:05
  15. mischa
    10 December, 06:24

    Reminds me a bit of Nasa’s Hyperwall-2 – Well done last.fm!

    mischa – 10 December, 06:24
  16. kupitkv
    18 January, 10:10

    Now that that is all said & done when will you be on the Playstation Network?
    PS3 owners need love too.

    kupitkv – 18 January, 10:10

Comments are closed for this entry.