Last.fm – the Blog

Earlier this week we released our Best of 2011 charts. 2011 saw you spend over 71 thousand years listening to music and scrobble more than 11 billion tracks. We’ve been churning through all of this data to find out what truly defined 2011.

New for this year is the discoveries chart. We went back to the beginning of time (well, to 2003) and checked every one of your 61 billion scrobbles to work out which artists were first scrobbled in 2011.

We’ve also broken these charts down by country and tag. Whatever you’re interested in, from experimental music in Mexico, the latest innovations in Finnish pop, or just what’s Big in Japan, you now have a means to browse them.

Following on from last year we are providing you with a data download. Musicbrainz IDs are now included in this data (where we have them) as part of our continued collaboration with Musicbrainz.

Producing the ‘Best of’ Charts is a very different process to our usual weekly charts. What follows is an overview of the process. In particular I’ll explain how we determined the new albums and discoveries of 2011, and how we turned these into the charts you see on the site.

New Albums

Our top artists are calculated based on albums released in 2011. One issue with albums is that they are typically released many times in many locations. To get around this we used a new version of the Musicbrainz database to find track listings for albums that were first released in 2011.

Of course, that isn’t the end of the story. Our library doesn’t always match up with Musicbrainz. Such issues need to be handled when we align album information from Musicbrainz with our own scrobble data. It’s one of the reasons we’re improving our Musicbrainz ID coverage .

New Discoveries

We label an artist as a new discovery if they were first scrobbled in 2011. As I mentioned previously, this can only be decided by checking through all of the scrobbles we have ever received.

This task is complicated by misspelled artist names, collaborations, and remixes. A nice example is Britney Spears’ collaboration with Sabi. Britney is certainly not a new discovery, even though this incorrectly-titled artist was first scrobbled in 2011. We avoid this by mapping artist names to their correct versions, before sorting through their scrobbles.

Our Human Computer

Our final step was to send the charts to our secret weapon: the music team. They pored through thousands of the top artists of 2011, matching them against their own databases and removing/adding artists that were incorrect or missing.

Data Download

This year we have two data downloads: the first – like last year’s – contains the top artists and albums of 2011; the second contains only the top artists, because they do not all have associated albums. In the data you’ll find all of the artists and albums from Best of 2011, along with play and listener counts, top tags, and image links.

In both cases we have added Musicbrainz IDs to the data. You can use these on our own API, BBC Music, and The Guardian. Use the data as you please; we look forward to seeing what you come up with!

Every year when Best of rolls around, we look at the chart to see if our data could have predicted who’d make it big. While there are a few in there we saw coming * cough * Adele * cough * the reality is that every year things get harder and harder to foresee.

That’s one of the reasons we launched our New Discoveries chart; to show off just how diverse your year in music really is.

Sure, it’s full of credible indie acts; Purity Ring, Death Grips and Work Drugs all did fairly well, while Wugazi – an album of mash-ups between Wu Tang Clan and Fugazi – made it to 13th place after getting huge buzz over the summer.

Someone we might have expected big things from was former Oasis frontman Noel Gallagher. He made it to number three on the New Discoveries chart, but only to 69 on the overall UK chart. That’s not quite as high as we might have expected. Similarly, Gaslight Anthem side project The Horrible Crowes made it to number 12 on the New Discoveries chart, largely off the back of Gaslight Anthem fans trying it out.

Further down the list GLaDOS makes an appearance. The Aperture Science Psychoacoustics Laboratory made it to number 7 on the chart after Valve released several albums worth of material from Portal 2. Soundtracks often jump to the top of the Hype Chart after hardcore fans flock to new releases, and while none of the artists on Drive were eligible for the New Discoveries chart they all got a huge boost when that came out.

Up until the last minute it looked as if the New Discoveries chart would be topped by none other than Rebecca Black. The “Friday” singer was number one on the chart right up until December, but while her video has collected some 17.5 million views on YouTube Last.fm’s music community only played the song 320,000 times between them.

Our first New Discoveries list is actually topped by Youth Lagoon, the project of Boise, ID native Trevor Powers. His dream-like album shot up the Hype Chart in autumn, and appeared to become a fixture throughout the winter for many listeners. He also creeps into the US overall top chart at 100.

For a taster of what these artists have to offer, listen to our New Discoveries playlist on the recently launched Discover app.

In case you missed it yesterday then our design team played with an early cut of our New Discoveries chart to create this neat little poster as a bit of a bonus. Don’t forget that you can also filter the chart to find the New Discoveries that best reflect your tastes using the Country and Tag options.

Here’s to another unpredictable year in music!

Best of 2011 is a reflection of the year in music, highlighting the most popular and hottest new artists all based on the tracks you’ve been scrobbling.

This year’s ‘Top Artists’ chart was compiled by looking at scrobbles for albums released between 1st January and 31st December 2011. As in previous years, we aren’t counting live albums, greatest hits collections, EP’s and singles. You might not be all that surprised when you see who’s sat at number one, but dig a little deeper using our lovely new Country and Tag filtering options to find the No. 1 which suits you!

Another new feature for 2011 we’re really excited about is our ‘Top New Discoveries’ chart. This was compiled by looking at the number of listeners for artists who had their first scrobble between 1st December 2010 and 31st December 2011. Discovering new music is core to the Last.fm experience; so we wanted to highlight the artists who caught your attention this year and who you should keep an eye on during 2012. Again, use the filtering options to personalise your view.

Additionally, we took a look at the Year In Music to see what our data had to say about 2011. We hope you’re as fascinated as we were by the impact of music news on your scrobbles.

For developers, we have provided the chart data as TSV and XML files. Download and start hacking, we’d love to hear what you come up with.

Finally, as a little easter egg, we’ve created a commemorative poster of this year’s New Discoveries chart. The eagle-eyed amongst you will notice that it’s slightly different to what you see online; we made this before taking all of December’s data into account. You can download the poster here.

[Pour la version française, lisez plus bas!]

Last.fm has a mission: to find out what music you already listen to, so that we can recommend new music you will really like. So every time you listen to a track, we want to know – be it on Spotify, your smartphone, your iPod, your Xbox, iTunes or anything else that you play music on. We do this by letting all these websites, phones and devices “scrobble”. Scrobbling means that with your permission, we are sent a message every time you listen to a track.

There are already over 600 websites, phones, applications and hardware devices that scrobble, but we won’t rest until every single music player in the world talks to us! This is because we don’t want to miss any of the listening that makes your taste unique, and because the more scrobbles we get from all around the world, the better our recommendation system gets.

This is why we are very happy to announce that Deezer are now scrobbling. Their 20 million users around the world can now get our personalized music recommendations based on the tracks they listen to on Deezer, and by doing so participate in the great Last.fm project.

Setting up scrobbling on Deezer is easy: just click on your avatar on the top right, select “My account”, and then go to the “Alerts & Sharing” tab. Voilà!

We are really looking forward to working more closely with Deezer – we’ll keep you posted with any updates.

Happy scrobbling!

Last.fm a une mission: découvrir ce que vous écoutez déjà pour pouvoir vous recommander de nouvelles musiques que vous adorerez. A chaque fois que vous écoutez un morceau, nous voulons donc le savoir – que ce soit sur Spotify, votre mobile, votre iPod, votre Xbox, iTunes ou par n’importe quel autre moyen. Nous atteignons notre objectif grâce à ce que nous appelons le “scrobbling”: nous permettons à tous les sites web, mobiles et autre machines de prévenir Last.fm – avec votre permission! – à chaque fois que vous écoutez quelque chose.

Plus de 600 sites webs, téléphones, applications et appareils musicaux scrobblent déjà, mais nous ne serons satisfaits que quand tous les lecteurs media du monde nous parleront! Il y a deux raisons à cela: nous ne voulons rater aucun des titres que vous écoutez et qui forment votre profil musical; et notre système de recommandation musicale s’améliore à chaque fois que nous recevons un nouveau scrobble.

Voila pourquoi nous sommes très heureux de vous annoncer que Deezer vous permet désormais de scrobbler! Leurs 20 millions d’utilisateurs à travers le monde peuvent maintenant recevoir nos recommandations personnalisées basées sur les morceaux écoutés sur Deezer – participant ainsi au grand projet Last.fm.

Configurer le scrobbling sur Deezer est très facile: clickez sur votre avatar en haut à droite, choisissez “Mon compte” et ouvrez “Notifications & partage”. Voilà!

Nous avons hâte de collaborer de façon encore plus proche avec Deezer – on promet de vous garder informés.

A bientôt et bonnes découvertes musicales!

“Let a thousand flowers bloom”

Mao Zedong never said that. It was actually a hundred flowers.

As you take a look at our new Last.fm Discover app, you will find we don’t have a thousand tags either. It’s close to 2 million tags that have been created over the years by our members to describe the music they love.

It’s these user-generated tags that power our HTML5 Discover web app. We launched it yesterday and it’s available around the world, without restrictions and without advertising interruptions.

Last.fm Discover is a personalised music player that introduces you to bands from around the world by letting you browse through musical styles that you may already know or want to learn more about. You won’t find the latest X-factor winner or the latest plastic boyband manufactured by evil scientists in a lab somewhere. Some of them are quite rough around the edges – make use of the ban button when you come across something unlistenable – but you’ll also reach for the love button as you discover diamonds in this amazing library of tracks.

This diversity and eclectic view of the musical landscape is what Last.fm is all about. We don’t live in a cookie cutter world of hackneyed generic labels for music. Most music sites include the standard dozen genres (pop, rock, urban, etc.) and that doesn’t adequately describe the diversity of music out there. Our tags system encourages the weird and the wonderful, the micro-communities and new scenes that are springing up as fast as new, independent bands are formed.

Last.fm Discover celebrates this, and features more than 500,000 artists and labels who have uploaded their music to reach the ears of the Last.fm community. For many of these bands, the main desire is to be heard and enjoyed. They don’t have a piracy problem, they have an obscurity problem. Supporting independents has always been core to Last.fm. At least 20% of the music played on our streaming radio services already comes from these independent artists (and they are all paid via our Artist Royalty Program).

We hope you enjoy discovering new music via our new app, and that you support the independents who make it. They, more than any shiny new gadget, venture-capital supported music service or giant record label, are the future of music.

Our Charts section has been a bit neglected of late. We’d all got a bit fed up of seeing Coldplay, Radiohead and Adele lead the ‘Top Artists’ chart week after week, especially when we could see our Hype Charts and internal data was telling a far more compelling story. So we decided to do something about it.

This week we have launched the first in a series of improvements to our charts section to make them more relevant, giving you a more dynamic picture of what is popular from week to week.

What’s live now?

The most important set of charts is now our Hype Charts. The Hype Charts are core to what we do at Last.fm – drawing attention to upcoming artists – so it was an easy decision to make these more prominent.

We’re also emphasising how much things change in our weekly charts by making it easy to go back and view them by a weekly pull-down menu.

Each chart now has its own page, and we’ve added buttons to each entry so you can quickly add artists to your library, love them, buy their music or add tags.

What’s next?

Every year at this time, most music sites give you a run down on the best acts of the year. We’re also going to have a Best of 2011 feature, but we have pushed it back to January this year in order to include a full year of data. While everyone else’s lists are pretty similar, we think you’ll be surprised by the story that Last.fm’s data is telling about 2011.

We hope you enjoy these changes and we look forward to hearing your feedback.

In the beginning was the word, and the word was Last.fm. Then I joined for three months…

In February 2011 I joined Last.fm’s data team as an intern. Since I’m studying part-time through the Open University I was able to start a flexible internship here. The data team work on many of the back end systems that Last.fm relies on, such as the software for the streaming servers, and the services that manage the songs in the music catalogue.

One of Last.fm’s many streaming servers.

I spent my first weeks writing unit tests, trying to increase the line coverage on a few small projects to over 70%. It was great to see real Java code in action as opposed to looking at examples in exercise books – this was a chance to read lots of it. Finding ways to test it meant that I really had to understand it too. Thanks to team lead Adrian’s enthusiasm for testing, I’m left with a real appreciation of unit testing and how it allows you to confidently change code and know that it still does what it was originally designed and built to do.

With the help of the other members on the team I have been able to improve my coding style, naming conventions and object-interaction design as well as briefly dabbling with other things such as Hive, JDBC, Spring and concurrency; but not at the same time (snigger!)

Great Expectations

Last.fm operates a distributed storage structure named Hadoop which consists of over 60 nodes, each with terabytes of data storage and gigabytes of memory. It is constantly being upgraded and new storage space added – this is necessary since it stores over 300GB of extra data every single day. I was invited to help with a hard-disk upgrade on 10 of the nodes, fulfilling a lifelong dream of mine to visit a data centre. I took a trip to London’s luxurious docklands with 80TB of 2TB hard-disks in the back of a taxi.

Hadoop cluster with 10 nodes offline, hard-disks unclipped and waiting to be removed.

I had many expectations of what it would be like at the data centre, and was eagerly anticipating the millions of pounds worth of equipment I might see. However, the reality of the environment in the data centre was upsetting to my nervous system. For several hours I endured a droning in my ears akin to a jet engine, whilst simultaneously being frozen when walking down one aisle (the cold aisle where the servers take in cold air that is pumped though the floor), and then being boiled whilst walking down the next (where the servers kick out all the hot air). I now know why I was advised to wear “hot pants and a warm jacket”. A lot of walking back a forth between aisles to self regulate my own body temperature was required!

The Hadoop cluster’s old hard-disks awaiting removal from their caddies… and hundreds of screws that keep them in place! These hard-disks were recycled by adding extra capacity to the streaming servers.

After this shock to the senses I was left with a noticeable impression of how much time and effort was put into fail-over equipment and practices to ensure that Last.fm’s systems stay running come rain or shine or diesel fumes. All hardware is planned and installed with redundancy, and file-systems are designed so that important data is replicated. For example, the streaming servers’ tracks are replicated at least three times and most systems are mirrored in all three of Last.fm’s data centres.

What you can expect to get from an internship at Last.fm

- An environment where people care about doing things right; the first time around.

- A place where redundancy matters and where people worry about things going wrong; even when they are going right.

- An agile development process with sprint planning.

- Guidance on naming conventions, coding style and object-interaction.

- A purely open-source development environment – the only experience with Microsoft I had was the logo on my keyboard.

- A relaxed atmosphere where people are free to wear and say whatever they want where meetings spontaneously burst out anywhere and at any time around you.

- Opportunities to dip your toes into new technologies such as Hadoop.

- A chance to be surrounded by decades of Java experience, where people can advise you on how to do things right – and kindly point you to an API page when you try and get them to do it for you :).

- The ability to work with software development tools such as: Subversion, Maven, JUnit and Cobertura.

- An insight into building and developing code that is robust, dependable and
only released when it has gone through multiple stages of thorough testing.

One small step for a member of the data team, one giant leap for an intern.

During my time at Last.fm I was fortunate enough to observe a real programming team in its day to day activities and also had a chance to improve my own programming skills by solving real problems.

Internships can be a good way to increase job prospects and I’m grateful to be able to put the name ‘Last.fm’ on my CV. If I had instead worked at ‘QualTekXYZmobile’ then perhaps potential employers won’t have heard of it – or worse think I’ve invented it. Who knows… maybe your interviewer was listening to Last.fm before you walked into your interview!

The Music Hack Day didn't just see us arrive with a lot enthusiasm, but also with a brand new API extension that exposes audio features, similar song playlists and Spotify URIs. And we won prizes!

Photos by Thomas Bonte

All awesomeness hype aside, the Hack Day really was a nice experience, and even the 3 hour marathon that was Sunday's demo session was a joy to watch because of the great quality of the hacks. It was my first hack day, and I was truly impressed (see Wired's and Insider's take on it). So what did we do?

My oh my, an API!

You may have noticed from my previous blog posts (Anatomy of the UK Charts, Parts 1, 2, 3, 4 and 5) that we have put quite a lot of effort into finding a mix of well-tested and newly developed audio features that capture distinct attributes of audio recordings, such as energy, harmonic creativity and smoothness. Just to be totally clear: no Last.fm tags and no Last.fm scrobble magic are involved, only pure audio features, retrieved directly from the original recordings.

We calculated 21 of these features on 2 Million of our most scrobbled recordings and Mark built a neat, very fast service to host them. Since Friday this service has been publicly accessible through our outward-facing Last.fm API, thanks to Duncan's API magic. You can either ask for certain feature ranges and retrieve a list of songs that satisfy them, or you can retrieve the audio features themselves by providing the track's artist and title. Of course, bringing even the shiniest of APIs doesn't qualify as a hack...

Driver's Seat — steer your music playlisting!

Since I'd been very impressed with Spotify's new app integration I persuaded Sven to help me build a hack that nicely exposes how good our new API is at audio feature playlisting. And because it puts you in control of steering your music we called it Driver's Seat (screenshot). Below you see a video of the resulting Driver's Seat Spotify app in action.

According to your preferences you select a preset, or adjust feature sliders and hit "Go get playlist!" and the app will fire a http request to the Last.fm API that looks like this

http://ws.audioscrobbler.com/2.0/?method=track.findbyaudiofeatures&filter[]=bpm:80:91...

The result is a list of tracks that we then get the Spotify URI of using another brand new API of ours that loves requests such as this:

http://ws.audioscrobbler.com/2.0/?method=track.getPlaylinks&artist[]=radiohead&track[]=creep...

We really liked our hack because it allows music discovery to be uninhibited by artist genre or history — it just gives you the kind of music you request. The Spotify team liked it so much that they gave us their hack prize, which we share with a hack called CTRL — two of the 18 Spotify hacks.

PitchFork Effect

Sven and I weren't the only ones hacking away though. Alex produced some intriguing visualisations of how PitchFork reviews influence Last.fm listening stats... and received one of the two prizes from MusicMetric. Marek also made a cute little virtual album store as an antidote to the all too modern iTunes and Amazon stores. And Coffey re-worked a previous hack of his to scrobbling tracks at gigs you go to: it uses the set lists available through setlist.fm's API—find the hack here.

Every now and again two things come together that make perfect sense, sometimes by design and sometimes just by natural affinity: gin and tonic, cheese and tomato, Jagger and Richards, Lennon and McCartney… two things that get better when they’re together.

Today Spotify announced that they’re opening their platform up to apps, and Last.fm is going to be one of their launch partners.

Introducing Last.fm for Spotify: personalised recommendations based on the music you play.

Track: “Calm Down” by Canterbury

Ever since Spotify launched, I’ve heard many people suggest that Last.fm and Spotify each do different but essential things they want from a music service. Spotify integrated Last.fm scrobbling into their service almost from day one, but scrobbling on its own really didn’t connect the two services meaningfully enough.

I remember saying the exact same thing long before coming to work at Last.fm. “Wouldn’t it be cool if you could somehow use Last.fm’s recommendations to decide what to play on Spotify?” Some of our community even used our APIs to build homebrew versions of it.

So when, earlier this summer, Spotify came to us to talk about their new platform and asked us “what would you do if..?”, it took us just a few seconds to agree that this was a massive opportunity to create something that we’d want to use ourselves, every day. It was a chance to take the scrobbles we were getting from Spotify users and then pay them back, with interest, in the form of great recommendations.

So what does Last.fm for Spotify do?

You can:
• Sign in to Last.fm, or start a new profile (new users have a very easy setup process)
• The overview page displays the listener’s recent top albums…
• …some recommendations
• …their recent tracks
• …and some of their loved tracks
• The Albums and Recommended tabs in the app will offer a deeper insight into those lists
• The app will allow users to build playlists of recommendations, top albums, recent tracks and loved tracks with just one click
• It also powers a Now Playing screen, which features scrobbling stats about artists
• …and users can launch a Similar Tracks playlist from the song currently playing in that tab
• And yes, it scrobbles too

Spotify users without a Last.fm account will be able to use the Now Playing view and Similar Artist playlists, but the personalised experiences are only available to new or existing Last.fm users. As ever, the more you use it, the better those personalised recommendations get.

The one-click playlists really are effortless, and you can play them on any Spotify app (mobile, etc.)

If you want to try this out for yourself, Spotify have a preview version you can download now. It’s going to be going out to the wider world very very soon. You can send us feedback in the Last.fm for Spotify forum as soon as you start using it.

We’re really excited to see this in the wild. We’re big fans of Spotify at Last.HQ, and a lot of work has gone into making this a great experience for old and new users.

Enjoy!

Many moons ago Last.fm set up a collaboration with MusicBrainz, the open-source music metadata database and community. MusicBrainz use special keys known as MusicBrainz Identifiers (MBIDs) to uniquely identify artists, labels, songs and many other music-related entities. One of the main ideas behind them is that anyone can use these keys to identify and cross-reference musical entities, even if they come from different sources.

For example, an album review site could publish the MBID for an album being reviewed and someone could take this MBID and look up the album on MusicBrainz (or some other site supporting MBID lookups) to find out more information about it.

That was then

The plan at the time was to regularly synchronise MBIDs with the artists, albums and tracks in Last.fm’s catalogue. The MBIDs (or should those be MBIDz?) could then be used to query for these entities using our API, or to link across different services using the MusicBrainz ids as an external mashup key. Last.fm API calls also return MusicBrainz ids where applicable (see artist.getInfo for an example).

This continued successfully for a while until a few shake ups and changes over here resulted in us temporarily dropping the ball on this one and the regular updates stopped (we did however keep the historical data).

MusicBrainz members hanging out in Last.fm’s ball pit back in the day (photo by Mayhem).

This is now

The good news is that we reconnected with MusicBrainz while we were visiting San Francisco for this year’s Hadoop Summit and are recommitting ourselves to being good citizens of the MusicBrainz community.

Our first step is to once again synchronise Last.fm artists, albums and tracks with their relatives over at MusicBrainz using their Live Data Feed which means we’ll be updating these once an hour. We’ve already started doing this and have more information on the technical details over in our development discussion forum.

To the future and beyond

Once we are in synch again we will start looking at other ways we can more actively work together with MusicBrainz and also with others like The Guardian and the BBC who use MusicBrainz Identifiers too. We also hope to leverage MusicBrainz’s experience with handling artist and album disambiguation to improve how we model music on Last.fm. We currently have the pleasure of hosting one of their developers (OCharles) at our London offices a couple of days a week to work on these and other related issues.

So, if you have any questions, comments or requests for features please ask them over on the forum, catch you there.