Last.fm – the Blog

Last weekend we decided to take a break from counting your scrobbles and spend our time playing a bit of football instead. It was a chance to swap our football table for a football pitch and take on a few other music related entities at the 6th annual Big Scary Monsters 5-a-side tournament. I was suffering from a hurty knee, but went along to inspire the team Coach Taylor style. “CLEAR EYES, FULL HEARTS!”

Top L>R: Ben Spittle, Michael Horan, Michael Coffey, Sven Over, Paul Blunden

Bottom L>R: Dan Sleath, Matt Clark, Dom Amodio, Nick Calafato

There were 24 teams competing, first in a group then a knockout stage. Last year we went out in the group stage, but felt we’d had a tough group and were eager to prove we could do better this time. However, not even me shouting “man on”, “down the line”, or “well played, that’s liquid” could stop us losing our first game to a fantastic Abeano 7-0. Not a great start, but we followed it with a 2-2 draw against Fanzine about Rocking and then two wins against The Xcerts and Hassle, both 2-1, mainly thanks to the amazing Dan “The Cat” Sleath in goal. Our final group game was against our old friends Drowned in Sound. We’d lost to them at last year’s tournament and were outplayed in a friendly we’d arranged in the meantime, but neither team could break the deadlock and the game ended goalless.

Coach Coffey was an inspiration to his team.

This all meant we were through to the knockout stage where in round 1 we were up against Punktastic, which we’d heard were “pretty handy”. Both teams were tired, but another two late goals from Last.fm meant we were through to play Tall Ships in the quarter-finals. It was another tough game, but a clean sheet and a last minute goal amazingly put us through to the semi-finals. Something none of us really believed was possible at the start of the day.

Last.fc warming up on Wembley’s doorstep.

We were on a high, dreaming of an open top bus tour of the “Shoreditch Triangle” upon our return, but next up were last year’s champions, Old Blue Last. It was clear straight away that these guys had played football before and not even our star goalkeeper could stop the onslaught. The dream was over, but we felt the 4-0 scoreline was respectable against a team of such quality.

Finally, Old Blue Last beat Abeano 2-0 in a rather exciting and closely fought final. We took solace in the fact these were the only teams we’d lost against, claimed third place having beaten the other semi-final loser, and went home looking forward to next year after a great time was had by all.

We work hard at Last.fm to give you and millions of users the services you have learned to love and to come up with new ideas. To keep the spirit high it’s sometimes helpful to break out of the zone for a moment, get some distance between you and the problem you try to crack, have a short break and then come back to your desk with fresh ideas. Many of us at Last.fm like to enjoy a game of table football to fill those moments.

Now our football table has seen better times. The pitch is anything but level, and the last time we moved it to a new place we almost broke it. Twice. So it was time for something different.

Back in April three of us (DavW, marekventur and I) took part in the London Realtime hack day. We managed to impress the jury with YouChoose, a collaborative YouTube playlisting web site. We won a prize and walked away with a brand new iPad… between the three of us.

Rather than complain we sold the iPad to another member of staff and decided to put it towards a new football table. I added my half of the Spotify award of last year’s London Music Hackday to the stock, but it still wasn’t quite enough for a good table.

It would be a shamelessly hubristic act to believe that we would keep winning prizes, but we gave it a try. Two weeks after London Realtime we went to the Buckinghamshire countryside and took part at Game Hack. The three of us were joined by tdhooper. Within 24 hours of hacking and very little sleep, we created a little browser game and we won the prize for the best HTML5 browser game, awarded by Mozilla and Turbulenz.

That gave us a good budget for a shiny new table, funded solely with money we won at hackdays.

And now it is here. We ordered it from Kicker Klaus, a German mail order shop that specialises in all things table football. The Vector III weighs 125kg and we got it in Last.fm red, with red and black players, just as you would expect. I would like to tell you more about it, but I have been challenged to a game, and then it’s back to work: the database doesn’t code itself.

Last week I attended The Great Escape which is a fantastic festival and conference in Brighton. In a previous life I built the Great Escape website for the Mama group and I’ve been to the festival for the last two years. I love it.

If you have never been to TGE, then I’d highly recommend you check it out next year. Spread out over 30 venues it is a great place to listen to new and upcoming bands as well as some more established artists. The vibe of the festival is chilled and when the sun shines, there’s no better place to be than Brighton.

On Friday morning I attended a conference about New Music Radio and listened intently about how curated radio is changing with an interesting panel of podcasters, radio entrepreneurs and DJs, one of which kindly let me crash at his family’s house (thanks Darren!) for the duration my stay.

When the debate was opened up for questions from the audience Duncan Geere from Wired asked how the panel see services like Last.fm. The response from Matt Young who runs Song By Toad was interesting and although I know he has a vested interest as a ‘radio presenter’, what he said was simply not correct. And I quote:
“I really think Last.fm and Pandora are fucking pointless. Whenever I log in to Last.fm, it just plays me 20 songs I already know. There’s no way to listen to anything new”.

Well Mr Young.. have you not checked out your personalised recommendations? Have you missed the new release recommendations, have you checked out Discover? Ever tried Your Recommended radio? browsed music by tag or tried multi-tag radio? There are SO many ways to find new music via Last.fm that the potential to discover tracks and artists you like but haven’t yet listened to is enormous. Better yet, we can offer a choice to account for the needs of the people you seem to have overlooked… people who WANT to listen to music they know and like.

The point was well made by Mr Trick – which thankfully added a reality check to the statements from others on that panel – and that is… not everyone has a desire to have a DJ ‘throw them a curve-ball’. They actually want to listen to, and be recommended, music that we know they will like.

The fact that new, upcoming and independent artists have uploaded almost 4 million of their tracks to be discovered in and amongst the entire catalogue of almost 15 million streamable tracks and the fact that our Last.fm users have scrobbled and added rich data for more than 50 million artists should tell you that the scope to discover new music with Last.fm is huge.

It is interesting to note that the poll running on Wired’s site currently shows that 70% of people (at time of writing) are listening to less traditional radio because of Last.fm.

So, Matt Young, please come on over to Last.fm anytime and join us for a techmosis session, perhaps one Friday. We’d love to show you how good Last.fm really is and why millions of people all around the world know for a fact that what we are doing here is far from “fucking pointless”.

About 6 weeks ago I started a short internship at Last.fm. For my project I wanted to explore Last.fm’s data to learn how listening preferences vary with the listener’s age and gender. Apart from the science, the most important thing I found is that you can make awesome plots with this information.

I started by making a chart to show what kind of music you “should” be listening to if you really want to fit in with the most common artists in your age range and gender:

The sizes of the artists’ names indicate how popular they are, while their position shows the gender mix and average age of their listeners. Based on the positions of the larger names, it’s already obvious which age category is most common amongst Last.fm users.

So, you can now use this plot to decide which music you might want to listen to. For example, if you are a healthy young male in your early twenties, you probably should listen to bands such as Iron Maiden and Metallica. Gorillaz and Radiohead might just be acceptable. If you get older you can then switch to artists like Neil Young and Genesis. It’s all quite obvious really.

Of course, when I realized what nice plots I could make, I tried it on several other types of data as well. Tags for example:

You can use it in the same way as the previous plot. Apparently females like using band names as tags (Super junior, McFly), while males prefer finding lots of ways to say the same thing (metal, jazz). Most importantly we have just used science to prove that men don’t listen to much k-pop.

Obviously music is the most important data that’s available at Last.fm, but there are some other profile items that can be interesting too. The words used in the ‘About Me’ section on users’ profile pages might even lead to the most interesting plot of them all:

There are actually so many fun facts about this plot that it’s just best to check it out yourself. The most obvious one is which hobbies you “should” have depending on your gender. Or you can find out at what age you should retire.

I used all of this to create a fun new playground demo that enables all Last.fm users to compare themselves with their friends. This is the plot for the data and recommendations team for example:

We’ve even thought of those of you who like to print their visualisations as a poster by providing a bigger PDF version that has more artist names on it.

Hopefully you’ll enjoy this demo as much as we did. In any case, we’d love you all to let us know what you think.

A bit less than a year ago we launched the VIP zone on our Playground, with the promise that we would keep adding fancy visualizations to it as a special treat for our loyal subscribers. We already delivered on this promise with the personalised Listening Trends and Music Universe visualisations, and today we’re delivering some more.

This time around we got inspired by the WOMRAD 2010 paper Rocking around the clock eight days a week: An exploration of temporal patterns of music listening. By applying some nifty circular statistics formulas, we managed to create an interesting new visualisation that shows at what times of the day a given Last.fm subscriber has been listening to music over a certain time period. Here’s an example:

In this case we’re looking at Norman‘s listening behaviour for the past 90 days. Red and green represent weekdays and weekends, respectively, and the longer the hand the more the listening was focused around the time to which it points. Generally speaking, Norman seems to listen to music at later times of the day in weekends than on weekdays, and his listening seems to be less restricted to certain hours in the weekend. It’s also quite clear that he tends to listen to music from 10AM to 7PM on weekdays, which isn’t that much of a surprise since those are our working hours here at Last.fm. He accidentally left his radio playing overnight a few times though, as indicated by the smaller red bars from 8PM until 9AM.

Our beloved LAST.HQ‘s listening clock for the same time period is a more extreme example:

Since we use this account for the reception radio in our offices — which plays pretty much 24/7 — the listening is spread out across all times of the day, leading to two hands that are extremely tiny and cute.

We very much hope you’ll enjoy playing around with this new visualisation, and that some of you will point to particularly interesting listening clocks or discuss potential improvements in our Playground forums. Meanwhile, we’ll start working on the next one!

And that’s a wrap!

Last week we revealed the final top ten in our Best of 2009 list, with — yup, as some of you guessed on the group shoutbox three weeks ago — Lady GaGa in top place with her album The Fame. With roughly 6 million more scrobbles than #2 artist The Killers had with Day & Age, she definitely earned her spot! Perhaps unsurprisingly, she’s also top of the chart for most unwanted scrobbles thanks to the love-it-or-hate-it “Poker Face.”

We also put up a data download for those of you interested in remixing or visualising the 2009 data. We’d love to hear about your creation too, so don’t forget to post about it in the web services forum when you’re done.

All the scrobbles that’re fit to print

To have a bit more fun with the dataset ourselves, we traded pixels for picas at Last.HQ for a week and created a newspaper edition of the list! In addition to the Top 40, the newspaper includes some local data visualisations for London and New York based on listening in those cities. We were able to make this thanks to the lovely folks over at Newspaper Club, a new London start-up dedicated to helping people print their own newspapers.

If you’re out of our newspaper delivery squad’s range, have no fear: here are downloadable A3 poster versions of the newspaper centrefold visualising month-by-month listening trends for both New York and London. Or, if those cities don’t mean much to you, why not grab the activity page instead? It’s got comics, a music crossword, and our favourite — a crabcore connect-the-dots.

Scrobbling in the name of

Moving from one set of charts to another, the race for UK Christmas #1 has been a hotly debated one this year. For the last 4 years the #1 spot has been taken by the winner of X-Factor, a reality show much like Pop Idol, driven by the infamous Simon Cowell. Bored with the years of “X-Factor monotony”, a husband and wife team set up a Facebook campaign to get Rage Against the Machines’s Killing in the Name to number one. People started buying the single en masse, and thanks to our exciting modern world of downloads and live updating charts on iTunes, Amazon and other retailers, people realised that the competition was actually achievable. Hundreds of thousands of people who had before ignored the UK official charts were suddenly inundating the BBC Radio 1’s website to get the chart scores, and tuned in to the radio to hear the results first.

Much to everyone’s disbelief, Rage managed to pull way out in front of X-Factor winner Joe McElderry, bringing in 50,000 more copies sold than Joe, and totalling 500,000 downloads. The betting industry allegedly lost an estimated £1million, and the single has risen 127 places on our own Top Tracks chart to come in at number 7 for this week.

Last.fm users joined in the campaign:

Pretty cool.

Happy New Year

And that’s all for 2009! Thanks for continuing to support Last.fm seven years on; we’ll be back in 2010 with more. Right now, it’s time for a little Christmas tag radio.

Jingling all the way,
– Team Last.fm

We’re thrilled to announce that the Last.fm Playground now has a brand new VIP zone. In this subscriber-only area we’ll showcase some of our fanciest ideas, visualizations, or plain weird projects as a treat to our loyal subscribers! Here’s an overview of the new stuff we’re releasing today.

Tube Tags

Which genre were you into last summer? How have your listening habits changed over time? Is there a correlation between the music you listen to and important events in your life? We’ve built a unique visualization, the Tube Tags map, to help you answer all these questions at glance, and to marvel at all the twists and turns your music taste has taken through the passage of time. Here’s some details from mine (and some of our intermediary sketches):

Each line is a tag, moving north or south depending on how much you listened to music described by this tag. Your most popular artists for each tag are also shown. Of course, the longer you’ve been scrobbling, the better it looks! I’m really proud that I’ve been scrobbling so regularly over the past years — and that I’ve left this trail of data that allows me to revisit today, through music, past moments of my life.

While only subscribers can currently generate a Tube Tags map from their listening history, the map is visible to anybody — we thought you might want to share it with your friends. If you prefer printing this as a poster, we recommend Diginate’s online poster printing.

At Last.fm, we enjoy being mad scientists, playing with data and infographics — stay tuned for more in the visualization department!

New toys

The new VIP zone also contains a few other toys: Image Chart creates a collage of your top artists’ images, History Chart summarizes your listening activity for your top artists in a neat visualization, and Artist Connections is a musical equivalent to the Six Degrees of Kevin Bacon game; try it out to check if there’s a chain of similar artists linking Paris Hilton to Metallica. Finally, we also added a new World Chart demonstration that shows you in which countries a given artist has been listened to most often.

We’d love to hear more from you about these features – please send us feedback or join the Playground Group to discuss them with others.

Update: We have added an option to let you render the image for your entire listening history (you’ve been warned: some pdf viewers might choke on the large image size, and there might be meatballs lurking between all those lines…). Just add &full to the download url: http://playground.last.fm/demo/tagstube/map?user=flaneur&full)

One night at the pub we discussed whether one could replace Hadoop (a massive and comprehensive implementation of Mapreduce) with a single bash script, an awk command, sort, and a sprinkling of netcat. This turned into a weekend project dubbed bashreduce.

To be fair, Hadoop probably does a few more things than bashreduce. But we’ve managed to cover a few key concepts in our script:

• Task coordination (kind of! sort of!)
• Mapping/Partitioning
• Reducing
• Merging
• Distributed file system (sort of! if you squint just right)

More than just a toy project, bashreduce lets us address a common scenario around these parts: we have a few analysis machines lying around, and we have data from various systems that are not in Hadoop. Rather than go through the rigmarole of sending it to our Hadoop cluster and writing yet another one-off Java or Dumbo program, we instead fire off a one-liner bashreduce using tools we already know in our reducer: sort, awk, grep, join, and so on.

I think it’s a neat idea! If you think it’s a neat idea, and you look at this gnarly bash code and think of ways to improve it, to make it more useful or more elegant, you would enjoy working for us. We’re looking for a clever C++ developer to help us tackle data mining and scale problems. My favorite line in the job posting is Interested in – we do all those things save one, which you can probably guess.

We’ve collected a few of our developer’s blogs here as well – more fodder for those of you interested in what we do.

There are plenty of great groups on Last.fm. However, some of them can be a bit hard to find. We thought one way to make it easier to find groups would be to make it easier to find similar groups. What do you think?

Here are some of my personal favourites and their similar groups:

• Bossa Nova: they have a great group radio, if you are into that kind of music
• Addicted to Last.fm: the title says it all
• Converse: this special sponsored group has a really cool feature you won’t find elsewhere on the site
• The Special Interest Tag Radio Collective: some of the best tag radio stations I know of are listed there
• People with no social lives that listen to more music than is healthy who …: a wonderful example of how a long group name totally makes sense
• Last.fm Book Club: a great example that there is sometimes more to life than just music
• …

Btw, what are your favourite groups?

Last.fm groups are a great way to find people that share common interests not necessarily related to music, such as ear-chopped-off painter fans, car enthusiasts, and people that clearly like meticulous descriptions of their behaviour.

Group aficionados have probably already noticed that it’s possible to listen to group radio, but unfortunately this has sometimes been a disappointing experience. The reason is simple: the playlist was generated by averaging the tastes of the group members, and this can result in predictable charts.

We have recently been experimenting with ways to emphasize what makes groups unique. By considering the average listener’s behaviour, we can show that members of the Saxophonists group listen to far more Sonny Rollins than others. Even though they listen to The Beatles a lot, it’s less special, because everyone listens to them a lot!

So, next time you tune into group radio, expect a much better experience!

Here’s a new list of artists you’re likely to listen to on these group radio stations:

Soundtrack geeks

• Hans Zimmer
• John Williams

• Danny Elfman

People with no social lives, etc..

• Motion City Soundtrack
• Cobra Starship

• Hellogoodbye

Switzerland

• Züri West
• Die Ärzte

• Patent Ochsner