Last.fm – the Blog

When we released Tube Tags to our loyal subscribers back in October, we promised to launch more data visualisations features soon.

Since then, we received lots of positive feedback about the infographics created for the newspaper edition of our Best of 2009 feature, including a mention in Fast Company for the “Local vs Global” comparison charts. Many of you asked for personalised versions, so that’s exactly what we’ve done! They’re now available to all subscribers in the VIP zone of our Last.fm ‘Playground.’

Listening Trends allows you to create the same kind of comparisons we did for the “London & New York vs. The Word” graphs, but on a smaller scale for your friends or neighbours. Or, if you’re not into comparisons, you can always get a classic stream graph visualisation of your listening trends over time.

Music Universe shows the artists you listened to the most in 2009. The artists are grouped by tag, depicted as moons orbiting tag planets! Building this visualisation was a particularly long odyssey through circle packing algorithms, font format madness, color tweaks and disapproving cats.

Please send us feedback or join the Playground Group to discuss these features.

And that’s a wrap!

Last week we revealed the final top ten in our Best of 2009 list, with — yup, as some of you guessed on the group shoutbox three weeks ago — Lady GaGa in top place with her album The Fame. With roughly 6 million more scrobbles than #2 artist The Killers had with Day & Age, she definitely earned her spot! Perhaps unsurprisingly, she’s also top of the chart for most unwanted scrobbles thanks to the love-it-or-hate-it “Poker Face.”

We also put up a data download for those of you interested in remixing or visualising the 2009 data. We’d love to hear about your creation too, so don’t forget to post about it in the web services forum when you’re done.

All the scrobbles that’re fit to print

To have a bit more fun with the dataset ourselves, we traded pixels for picas at Last.HQ for a week and created a newspaper edition of the list! In addition to the Top 40, the newspaper includes some local data visualisations for London and New York based on listening in those cities. We were able to make this thanks to the lovely folks over at Newspaper Club, a new London start-up dedicated to helping people print their own newspapers.

If you’re out of our newspaper delivery squad’s range, have no fear: here are downloadable A3 poster versions of the newspaper centrefold visualising month-by-month listening trends for both New York and London. Or, if those cities don’t mean much to you, why not grab the activity page instead? It’s got comics, a music crossword, and our favourite — a crabcore connect-the-dots.

Scrobbling in the name of

Moving from one set of charts to another, the race for UK Christmas #1 has been a hotly debated one this year. For the last 4 years the #1 spot has been taken by the winner of X-Factor, a reality show much like Pop Idol, driven by the infamous Simon Cowell. Bored with the years of “X-Factor monotony”, a husband and wife team set up a Facebook campaign to get Rage Against the Machines’s Killing in the Name to number one. People started buying the single en masse, and thanks to our exciting modern world of downloads and live updating charts on iTunes, Amazon and other retailers, people realised that the competition was actually achievable. Hundreds of thousands of people who had before ignored the UK official charts were suddenly inundating the BBC Radio 1’s website to get the chart scores, and tuned in to the radio to hear the results first.

Much to everyone’s disbelief, Rage managed to pull way out in front of X-Factor winner Joe McElderry, bringing in 50,000 more copies sold than Joe, and totalling 500,000 downloads. The betting industry allegedly lost an estimated £1million, and the single has risen 127 places on our own Top Tracks chart to come in at number 7 for this week.

Last.fm users joined in the campaign:

Pretty cool.

Happy New Year

And that’s all for 2009! Thanks for continuing to support Last.fm seven years on; we’ll be back in 2010 with more. Right now, it’s time for a little Christmas tag radio.

Jingling all the way,
– Team Last.fm

This is the second in a series of posts from the Last.fm engineering team covering the geekier aspects of our recent Last.fm on Xbox LIVE launch. Part one (“The War Room”) is here.

The music streaming architecture at Last.fm is an area of our infrastructure that has been scaling steadily for some time. The final stage of delivering streams to users fetches the raw mp3 data from a MogileFS distributed file system before passing it through our audio streaming software, which handles the actual audio serving. There are two main considerations with this streaming system: physical disk capacity, and raw IO throughput. The number of random IO operations a storage system can support has a big effect on how many users we can serve from it, so this number (IOPS) is a metric we’re very interested in. The disk capacity of the cluster has effectively ceased being a problem with the capacities available from newer SATA drives, so our biggest concern is having enough IO performance across the cluster to serve all our concurrent users. To put some numbers on this, a single 7200rpm SATA drive can produce enough IOPS to serve around 300 concurrent connections.

We’ve been using MogileFS for years at Last.fm, and it’s served us very well. As our content catalogue has grown, so has our userbase. As we’ve added storage to our streaming cluster, we’ve also been adding IO capacity in step with that, since each disk added into the streaming cluster brings with it more IOPS. From the early days, when our streaming machines were relatively small, we’ve moved up to systems built around the Supermicro SC846 chassis. These provide cost effective high-density storage, packing 24 3.5” SATA drives into 4U, and are ideal for growing our MogileFS pool.

Changing our approach

The arrival of Xbox users on the Last.fm scene pushed us to do some re-thinking on our approach to streaming. For the first time, we needed a way to scale up the IO capacity of our MogileFS cluster independently of the storage capacity. Xbox wasn’t going to bring us any more content, but was going to land a lot of new streaming users on our servers. So, enter SSDs…

Testing our first SSD based systems

We’d been looking at SSDs with interest for some time, as IO bottlenecks are common in any infrastructure dealing with large data volumes. We hadn’t deployed them in any live capacity before though, and this was an ideal opportunity to see whether the reality lived up to the marketing! Having looked at a number of SSD specs and read about many of the problems early adopters had encountered, we felt as though we were in a position to make an informed decision. So, earlier this year, we managed to get hold of some test kit to try out. Our test rig was an 8 core system with 2 X5570 CPUs and 12 Gb RAM (a SunFire X4170).

Into this, we put 2 hard disks for the OS, and 4 Intel X25-E SSDs.

We favoured the Intel SSDs because they’ve had fantastic reviews, and they were officially supported in the X4170. The X25-E drives advertise in excess of 35,000 read IOPS, so we were excited to see what it could do, and in testing, we weren’t disappointed. Each single SSD can support around 7000 concurrent listeners, and the serving capacity of the machine topped out at around 30,000 concurrent connections in it’s tested configuration – here it is half way through a test run (wider image here):

Spot which devices are the SSDs… (wider image here)

At that point its network was saturated, which was causing buffering and connection issues, so with 10GigE cards it might have been possible to push this configuration even higher. We tested both the 32Gb versions (which Sun have explicitly qualified with the X4170), and the 64Gb versions (which they haven’t). We ended up opting for the 64Gb versions, as we needed to be able to get enough content onto the SSDs for us to serve a good number of user requests, otherwise all that IO wasn’t going to do us any good. To get these performance figures, we had to tune the Linux scheduler defaults a bit:-

echo noop > /sys/block/sda/queue/scheduler
echo 32 > /sys/block/sda/queue/read_ahead_kb

This is set for each SSD – by default Linux uses scheduler algorithms that are optimised for hard drives, where each seek carries a penalty, so it’s worth reading extra data in while the drive head is in position. There’s basically zero seek penalty on an SSD, so those assumptions fall down.

Going into production

Once we were happy with our test results, we needed to put the new setup into production. Doing this involved some interesting changes to our systems. We extended MogileFS to understand the concept of “hot” nodes – storage nodes that are treated preferentially when servicing requests for files. We also implemented a “hot class” – when a file is put into this class, MogileFS will replicate it onto our SSD based nodes. This allows us to continually move our most popular content onto SSDs, effectively using them as a faster layer built on top of our main disk based storage pool.

We also needed to change the way MogileFS treats disk load. By default, it looks at the percentage utilisation figure from iostat, and tries to send requests to the most lightly-loaded disk with the requested content. This is another assumption that breaks down when you use SSDs, as they do not suffer from the same performance degradation under load that a hard drive does; a 95% utilised SSD can still respond many times faster than a 10% utilised hard drive. So, we extended the statistics that MogileFS retrieves from iostat to also include the wait time (await) and the service time (svctm) figures, so that we have better information about device performance.

Once those changes had been made, we were ready to go live. We used the same hardware as our final test configuration (SunFire X4170 with Intel X25-E SSDs), and we are now serving over 50% of our streaming from these machines, which have less than 10% of our total storage capacity. The graph below shows when we initially put these machines live.

You can see the SSD machines starting to take up load on the right of the graph – this was with a relatively small amount of initial seed content, so the offload from the main cluster was much smaller than we’ve since seen after filling the SSDs with even more popular tracks.

Conclusions

We all had great fun with this project, and built a new layer into our streaming infrastructure that will make it easy to scale upwards. We’ll be feeding our MogileFS patches back to the community, so that other MogileFS users can make use of them where appropriate and improve them further. Finally, thanks go to all the people who put effort into making this possible – all of crew at Last.HQ, particularly Jonty for all his work on extending MogileFS, and Laurie and Adrian for lots of work testing the streaming setup. Also thanks to Andy Williams and Nick Morgan at Sun Microsystems for getting us an evaluation system and answering lots of questions, and to Gareth Tucker and David Byrne at Intel for their help in getting us the SSDs in time.

As many of you noticed, a few weeks ago we launched Last.fm on Xbox LIVE in the US and UK. It probably goes without saying that this project was a big operation for us, taking up a large part of the team’s time over the last few months. Now that the dust has settled, we thought we’d write a short series of blog posts about how we prepared for the launch and some of the tech changes we made to ensure that it all went smoothly.

0 Hour: Monitoring.

First up, let me introduce myself. My name is Laurie and I’ve been a Sysadmin here at Last.fm for almost two and a half years now. As well as doing the usual sysadmin tasks (turning things off and on again) I also look after our monitoring systems, including a healthy helping of Cacti, a truck of Nagios and a bucket-load of Ganglia. Some say I see mountains in graphs. Others say my graphs are infact whales. But however you look at it, I’m a strong believer in “if it moves, graph it”.

To help with our day-to-day monitoring we use four overhead screens in our operations room, with a frontend for Cacti (CactiView) and Nagios (Naglite2) that I put together. This works great for our small room, but we wanted something altogether more impressive — and more importantly, useful — for the Xbox launch.

At Last.HQ we’re big fans of impressive launches. Not a week goes by without us watching some kind of launch, be it the Large Hadron Collider, or one of the numerous NASA space launches.

We put a plan into action late on Monday evening (the night before launch), and it quickly turned into a “How many monitors can you fit into a room” game. In the end though, being able to see as many metrics as possible became useful.

So, ladies and gentlemen…

Welcome to the war room

Every spare 24” monitor in the office, two projectors, a few PCs and an awesome projector clock for a true “war room” style display (and to indicate food time).

Put it together and this is what you get:

Coupled with a quickly thrown together Last.fm style Nasa logo (courtesy our favourite designer), we were done. And this is where we spent 22 hours on the day of the launch, staring at the graphs, maps, alerts, twitter feeds.. you name it, we had it.

It was pretty exciting to sit and watch the graphs climb higher and higher, and watch the twists and turns as entire areas of the world woke up, went to work, came back from work (or school) and went to sleep. We had conference calls with Microsoft to make sure everything was running smoothly and share the latest exciting stats. (Half a million new users signed up to Last.fm through their Xbox consoles in the first 24 hours!)

As well as the more conventional style graphs, we also had some fun putting together some live numbers to keep up to speed on things in a more real time fashion. This was a simple combination of a shell script full of wizardry to get the raw number, then piped through the unix tools “figlet” (which makes “bubble art” from standard text) and “cowsay” (produces an ASCII version of a cow with a speech bubble saying whatever you please).

Looking after Last.fm on a daily basis is a fun task with plenty of interesting challenges. But when you’ve spent weeks of 12-hour days and working all weekend, it really pays to sit back in a room with all your co-workers (and good friends!) and watch people enjoy it. Your feedback has been overwhelming, and where would we have been without Twitter to tell us what you thought in real time?

Coming Next Time

We had to make several architectural changes to our systems to support this launch, from improved caching layers to modifying the layout of our entire network. Watch this space soon for the story of how SSDs saved Xbox…

It’s that time of year again… when the data & catalogue peeps, developers and design folk work together to create our annual Best of 2009 web featurette of what you’ve been listening to the most in 2009.

This year, we did things a little differently: to avoid last year’s Coldplay track list overdose, we decided to go for one big artist list (instead of artists/albums/tracks), and expanded the list to the Top 40. Once we finished compiling the list, we decided it was just too good to spill all at once, so we’re publishing it in weekly parts, along with some other juicy features. This week it’s the MJ Special; for next week we’re cooking up something fun with tag data. Also, don’t miss the Best of 2009 Group — we want to know who’s on your personal list too!

We compiled the Top 40 by looking at scrobbles for albums released between 1st October 2008 and 16th November 2009. We took out live albums, greatest hits collections, EP’s and singles before pulling them together into the lean, mean format you’ll see in this year’s Top 40.

We added some extra data this year, notably the month-by-month scrobbles (so you can see how you fit in with the global artist listens) as well as top tracks, event images and attendance data.

So check out part 1 of Best of 2009 with the artists that made it into #40 to #21!

9 December Update:
Ooops! An eagle-eyed commenter spotted some discrepancies with the “listener” numbers we were displaying in the Top 40, which we tracked down to a bug in our data compilation. We’ve now updated the list to show the corrected data for “listeners”, which represents the total number of people on Last.fm who listened to at least one track by that artist in 2009. (Since the Best of 2009 is ordered by album scrobbles – which weren’t affected by the bug – the order of the list hasn’t changed.) Our apologies to those affected by this error.

In others news, artists #20 to #10 are now live, as well as the Top 1000 by tag.

Earlier this year we were approached by Sony to help develop an interactive idea that would appeal to users of Last.fm. Sony wanted to match their passion for sound quality with our passion for music. There were a lot of interesting ideas and the one that captured our imagination centred around being able to create an ultimate imaginary festival line-up and compete with your friends on who can build the best. So after months of hard work organizing a staggering amount of data, we are happy to announce Sony’s Fantasy Festival.

How it works:

You have €1,000,000 to spend on a 15 artist line up for your Fantasy Festival. Pick your all time favourite artists or pick artists you think are up and coming. If your line up ranks highest in *Buzz points you could win some pretty awesome prizes (more about the prizes here). Each week *Buzz points are determined by what artists are getting the most online buzz based on data from Last.fm, Yahoo!, Twitter and more. The competition will run for six months starting 18th November and *Buzz points scores will start rolling in on the 25th of November.

You can check out my line up here.

Happy booking!

We’re thrilled to announce that the Last.fm Playground now has a brand new VIP zone. In this subscriber-only area we’ll showcase some of our fanciest ideas, visualizations, or plain weird projects as a treat to our loyal subscribers! Here’s an overview of the new stuff we’re releasing today.

Tube Tags

Which genre were you into last summer? How have your listening habits changed over time? Is there a correlation between the music you listen to and important events in your life? We’ve built a unique visualization, the Tube Tags map, to help you answer all these questions at glance, and to marvel at all the twists and turns your music taste has taken through the passage of time. Here’s some details from mine (and some of our intermediary sketches):

Each line is a tag, moving north or south depending on how much you listened to music described by this tag. Your most popular artists for each tag are also shown. Of course, the longer you’ve been scrobbling, the better it looks! I’m really proud that I’ve been scrobbling so regularly over the past years — and that I’ve left this trail of data that allows me to revisit today, through music, past moments of my life.

While only subscribers can currently generate a Tube Tags map from their listening history, the map is visible to anybody — we thought you might want to share it with your friends. If you prefer printing this as a poster, we recommend Diginate’s online poster printing.

At Last.fm, we enjoy being mad scientists, playing with data and infographics — stay tuned for more in the visualization department!

New toys

The new VIP zone also contains a few other toys: Image Chart creates a collage of your top artists’ images, History Chart summarizes your listening activity for your top artists in a neat visualization, and Artist Connections is a musical equivalent to the Six Degrees of Kevin Bacon game; try it out to check if there’s a chain of similar artists linking Paris Hilton to Metallica. Finally, we also added a new World Chart demonstration that shows you in which countries a given artist has been listened to most often.

We’d love to hear more from you about these features – please send us feedback or join the Playground Group to discuss them with others.

Update: We have added an option to let you render the image for your entire listening history (you’ve been warned: some pdf viewers might choke on the large image size, and there might be meatballs lurking between all those lines…). Just add &full to the download url: http://playground.last.fm/demo/tagstube/map?user=flaneur&full)

UPDATE: This is now complete. Thanks for your patience.

We’ll be performing essential maintenance on our backend systems for around 9 hours on the weekend of the 19th-20th of September. The Last.fm website will be available while this is happening, but users will be logged out and the radio will be unavailable. This won’t result in any visible changes, but will help improve the performance of the site.

Most users won’t notice any problems with scrobbling, although some users may not be able to scrobble until the downtime is over. In any case, your client will save everything so scrobbles won’t be lost.

This downtime will also affect the Last.fm desktop and mobile clients as well as third-party applications using our API.

We rarely need to perform maintenance which affects our users; most of our infrastructure is designed so all maintenance can be performed with no effect on the site itself. However, one part we can’t touch without affecting the site is our global database, and this is due for a software upgrade. We’re planning to start this on Saturday morning (our quietest day), London time.

While the maintenance is happening, you can check our status page and twitter account for the latest news.

For the last six months, I have been a part of Last.fm’s data team while writing my master thesis together with Per – and lived to tell the tale.

Presently, I’m sitting in a quiet room where the heat makes more noise than do the people in it. See, music is a big thing here and the hackers on my team mostly prefer to code with their ears hugged by earphones. It wasn’t always this hot, though.

Back in January, Per and I landed in a chilly London where the people wore hats and greeted one another in charming ways. We were on a mission: write a data store that serves as back-end to an in-house visualization tool that renders graphs out of interesting numbers: scrobbles per king of pop, subscription signups broken down by country, that sort of thing.

For someone who digs distributed systems and Big Data, Last.fm is heaven. There is a sizable Hadoop cluster and many more terabytes of data than one can comfortably fathom. At the end of our tenure – and this is the best part – we were to release the code as open source. Sure enough, we had landed our dream gig.

The offices have got that typical Shoreditch media/tech post-startup vibe going: there are copies of The Economist and Wired in the foyer, a flipped-over skateboard lies next to the umbrella rack. The company occupies a whole floor which is split into two sides: the bizniz lot occupies one end (this is where the fax machine is) and the dev teams & ops (nerf guns at the ready) rule the other end. Working hours are generally between ten and seven, so coming in at eight o’clock sharp on your first day is definitely advised against. Not that anyone would.

As those who went before me have noted, Last.fm is a rather awesome place to work. There are some seriously brilliant heads here. Not only is the staff enthusiastic and contagiously dedicated, they also take a large pride in their work and that’s key to producing quality stuff. With passion it’s true.

Per and I were given an insane amount of freedom in implementing our data store. While we did get all the help we needed, both from our closest collaborators as well as anyone else around the office who we harassed with questions, all decisions regarding the project and its execution where ours to make. At the end of the day, we were ourselves fully responsible for our own fortunes and I think it is only from that kind of freedom and trust that truly brilliant things can come. And how did we fare then? Quite well, thank you. Zohmg is out there, and although it may not change the world just this year it might make the life of a data analyst or two a tad easier.

All in all it has been a killer internship experience: I got to present at HUGUK, became involved with HBase and met people who have instilled inspiration in me that will last a long time. I have realized the benefits of working alongside incredibly passionate fellows who are committed to perfect their trade. It will be hard to settle for anything less in the future.

After two years running Last.fm within CBS we feel the time is right to begin the process of handing over the reins. This is the latest stage in a long journey for us founders, which began in a living room in East London in 2002, and took us to the headquarters of one of the biggest media companies in the world.

It’s been a privilege working with the incredible team here in our London office, and we’re extremely proud of what we’ve achieved together. Last.fm’s users have more than doubled in the last 12 months (we are now at an all-time high of 37.3M monthly unique visitors), and we’re confident the site will continue to go from strength to strength. Being a part of CBS, and the recently formed CBSi music group, continues to open up many opportunities for Last.fm. Recent product releases such as the new visual radio, and the Last.fm on XBox announcement, are an indication of how much more Last.fm will achieve.

A huge “Thank You!” has to be said to all of you in front of your computers. With your contribution, enthusiasm and scrobbles you have helped to make Last.fm into what it is today: the best place for music online. Big up yourself for that, as we say here in East London.

That’s all folks, we are going to miss you!

Felix, RJ and Martin