Scrobble with Deezer

Friday, 6 January 2012
by Gilda Maurice
filed under Announcements and Stuff Other People Made
Comments: 10

[Pour la version française, lisez plus bas!]

Last.fm has a mission: to find out what music you already listen to, so that we can recommend new music you will really like. So every time you listen to a track, we want to know – be it on Spotify, your smartphone, your iPod, your Xbox, iTunes or anything else that you play music on. We do this by letting all these websites, phones and devices “scrobble”. Scrobbling means that with your permission, we are sent a message every time you listen to a track.

There are already over 600 websites, phones, applications and hardware devices that scrobble, but we won’t rest until every single music player in the world talks to us! This is because we don’t want to miss any of the listening that makes your taste unique, and because the more scrobbles we get from all around the world, the better our recommendation system gets.

This is why we are very happy to announce that Deezer are now scrobbling. Their 20 million users around the world can now get our personalized music recommendations based on the tracks they listen to on Deezer, and by doing so participate in the great Last.fm project.

Setting up scrobbling on Deezer is easy – just click here and enter your Last.fm username.
Otherwise follow these instructions: when you’re logged into Deezer, click on your avatar on the top right, select “My account”, and then go to the “Alerts & Sharing” tab. Voilà!

We are really looking forward to working more closely with Deezer – we’ll keep you posted with any updates.

Happy scrobbling!

Last.fm a une mission: découvrir ce que vous écoutez déjà pour pouvoir vous recommander de nouvelles musiques que vous adorerez. A chaque fois que vous écoutez un morceau, nous voulons donc le savoir – que ce soit sur Spotify, votre mobile, votre iPod, votre Xbox, iTunes ou par n’importe quel autre moyen. Nous atteignons notre objectif grâce à ce que nous appelons le “scrobbling”: nous permettons à tous les sites web, mobiles et autre machines de prévenir Last.fm – avec votre permission! – à chaque fois que vous écoutez quelque chose.

Plus de 600 sites webs, téléphones, applications et appareils musicaux scrobblent déjà, mais nous ne serons satisfaits que quand tous les lecteurs media du monde nous parleront! Il y a deux raisons à cela: nous ne voulons rater aucun des titres que vous écoutez et qui forment votre profil musical; et notre système de recommandation musicale s’améliore à chaque fois que nous recevons un nouveau scrobble.

Voila pourquoi nous sommes très heureux de vous annoncer que Deezer vous permet désormais de scrobbler! Leurs 20 millions d’utilisateurs à travers le monde peuvent maintenant recevoir nos recommandations personnalisées basées sur les morceaux écoutés sur Deezer – participant ainsi au grand projet Last.fm.

Configurer le scrobbling sur Deezer est très facile: clickez sur votre avatar en haut à droite, choisissez “Mon compte” et ouvrez “Notifications & partage”. Voilà!

Nous avons hâte de collaborer de façon encore plus proche avec Deezer – on promet de vous garder informés.

A bientôt et bonnes découvertes musicales!

Announcing Last.fm Discover

Thursday, 15 December 2011
by Matthew Hawn
filed under Announcements
Comments: 15

“Let a thousand flowers bloom”

Mao Zedong never said that. It was actually a hundred flowers.

As you take a look at our new Last.fm Discover app, you will find we don’t have a thousand tags either. It’s close to 2 million tags that have been created over the years by our members to describe the music they love.

It’s these user-generated tags that power our HTML5 Discover web app. We launched it yesterday and it’s available around the world, without restrictions and without advertising interruptions.

Last.fm Discover is a personalised music player that introduces you to bands from around the world by letting you browse through musical styles that you may already know or want to learn more about. You won’t find the latest X-factor winner or the latest plastic boyband manufactured by evil scientists in a lab somewhere. Some of them are quite rough around the edges – make use of the ban button when you come across something unlistenable – but you’ll also reach for the love button as you discover diamonds in this amazing library of tracks.

This diversity and eclectic view of the musical landscape is what Last.fm is all about. We don’t live in a cookie cutter world of hackneyed generic labels for music. Most music sites include the standard dozen genres (pop, rock, urban, etc.) and that doesn’t adequately describe the diversity of music out there. Our tags system encourages the weird and the wonderful, the micro-communities and new scenes that are springing up as fast as new, independent bands are formed.

Last.fm Discover celebrates this, and features more than 500,000 artists and labels who have uploaded their music to reach the ears of the Last.fm community. For many of these bands, the main desire is to be heard and enjoyed. They don’t have a piracy problem, they have an obscurity problem. Supporting independents has always been core to Last.fm. At least 20% of the music played on our streaming radio services already comes from these independent artists (and they are all paid via our Artist Royalty Program).

We hope you enjoy discovering new music via our new app, and that you support the independents who make it. They, more than any shiny new gadget, venture-capital supported music service or giant record label, are the future of music.

New Charts section!

Wednesday, 14 December 2011
by Sarah Ransome
filed under Announcements and Trends and Data
Comments: 17

Our Charts section has been a bit neglected of late. We’d all got a bit fed up of seeing Coldplay, Radiohead and Adele lead the ‘Top Artists’ chart week after week, especially when we could see our Hype Charts and internal data was telling a far more compelling story. So we decided to do something about it.

This week we have launched the first in a series of improvements to our charts section to make them more relevant, giving you a more dynamic picture of what is popular from week to week.

What’s live now?


The most important set of charts is now our Hype Charts. The Hype Charts are core to what we do at Last.fm – drawing attention to upcoming artists – so it was an easy decision to make these more prominent.

We’re also emphasising how much things change in our weekly charts by making it easy to go back and view them by a weekly pull-down menu.

Each chart now has its own page, and we’ve added buttons to each entry so you can quickly add artists to your library, love them, buy their music or add tags.

What’s next?

Every year at this time, most music sites give you a run down on the best acts of the year. We’re also going to have a Best of 2011 feature, but we have pushed it back to January this year in order to include a full year of data. While everyone else’s lists are pretty similar, we think you’ll be surprised by the story that Last.fm’s data is telling about 2011.

We hope you enjoy these changes and we look forward to hearing your feedback.

The Adventures of a Data Team Intern

Monday, 12 December 2011
by Ashley Diamond
filed under About Us
Comments: 0

In the beginning was the word, and the word was Last.fm. Then I joined for three months…

In February 2011 I joined Last.fm’s data team as an intern. Since I’m studying part-time through the Open University I was able to start a flexible internship here. The data team work on many of the back end systems that Last.fm relies on, such as the software for the streaming servers, and the services that manage the songs in the music catalogue.

One of Last.fm’s many streaming servers.

I spent my first weeks writing unit tests, trying to increase the line coverage on a few small projects to over 70%. It was great to see real Java code in action as opposed to looking at examples in exercise books – this was a chance to read lots of it. Finding ways to test it meant that I really had to understand it too. Thanks to team lead Adrian’s enthusiasm for testing, I’m left with a real appreciation of unit testing and how it allows you to confidently change code and know that it still does what it was originally designed and built to do.

With the help of the other members on the team I have been able to improve my coding style, naming conventions and object-interaction design as well as briefly dabbling with other things such as Hive, JDBC, Spring and concurrency; but not at the same time (snigger!)

Great Expectations

Last.fm operates a distributed storage structure named Hadoop which consists of over 60 nodes, each with terabytes of data storage and gigabytes of memory. It is constantly being upgraded and new storage space added – this is necessary since it stores over 300GB of extra data every single day. I was invited to help with a hard-disk upgrade on 10 of the nodes, fulfilling a lifelong dream of mine to visit a data centre. I took a trip to London’s luxurious docklands with 80TB of 2TB hard-disks in the back of a taxi.

Hadoop cluster with 10 nodes offline, hard-disks unclipped and waiting to be removed.

I had many expectations of what it would be like at the data centre, and was eagerly anticipating the millions of pounds worth of equipment I might see. However, the reality of the environment in the data centre was upsetting to my nervous system. For several hours I endured a droning in my ears akin to a jet engine, whilst simultaneously being frozen when walking down one aisle (the cold aisle where the servers take in cold air that is pumped though the floor), and then being boiled whilst walking down the next (where the servers kick out all the hot air). I now know why I was advised to wear “hot pants and a warm jacket”. A lot of walking back a forth between aisles to self regulate my own body temperature was required!

The Hadoop cluster’s old hard-disks awaiting removal from their caddies… and hundreds of screws that keep them in place! These hard-disks were recycled by adding extra capacity to the streaming servers.

After this shock to the senses I was left with a noticeable impression of how much time and effort was put into fail-over equipment and practices to ensure that Last.fm’s systems stay running come rain or shine or diesel fumes. All hardware is planned and installed with redundancy, and file-systems are designed so that important data is replicated. For example, the streaming servers’ tracks are replicated at least three times and most systems are mirrored in all three of Last.fm’s data centres.

What you can expect to get from an internship at Last.fm

- An environment where people care about doing things right; the first time around.

- A place where redundancy matters and where people worry about things going wrong; even when they are going right.

- An agile development process with sprint planning.

- Guidance on naming conventions, coding style and object-interaction.

- A purely open-source development environment – the only experience with Microsoft I had was the logo on my keyboard.

- A relaxed atmosphere where people are free to wear and say whatever they want where meetings spontaneously burst out anywhere and at any time around you.

- Opportunities to dip your toes into new technologies such as Hadoop.

- A chance to be surrounded by decades of Java experience, where people can advise you on how to do things right – and kindly point you to an API page when you try and get them to do it for you :).

- The ability to work with software development tools such as: Subversion, Maven, JUnit and Cobertura.

- An insight into building and developing code that is robust, dependable and
only released when it has gone through multiple stages of thorough testing.

One small step for a member of the data team, one giant leap for an intern.

During my time at Last.fm I was fortunate enough to observe a real programming team in its day to day activities and also had a chance to improve my own programming skills by solving real problems.

Internships can be a good way to increase job prospects and I’m grateful to be able to put the name ‘Last.fm’ on my CV. If I had instead worked at ‘QualTekXYZmobile’ then perhaps potential employers won’t have heard of it – or worse think I’ve invented it. Who knows… maybe your interviewer was listening to Last.fm before you walked into your interview!

London Music Hack Day — our audio API put us in the driver's seat

Tuesday, 6 December 2011
by Matthias Mauch
filed under Trends and Data
Comments: 10

The Music Hack Day didn't just see us arrive with a lot enthusiasm, but also with a brand new API extension that exposes audio features, similar song playlists and Spotify URIs. And we won prizes!


Photos by Thomas Bonte

All awesomeness hype aside, the Hack Day really was a nice experience, and even the 3 hour marathon that was Sunday's demo session was a joy to watch because of the great quality of the hacks. It was my first hack day, and I was truly impressed (see Wired's and Insider's take on it). So what did we do?

My oh my, an API!

You may have noticed from my previous blog posts (Anatomy of the UK Charts, Parts 1, 2, 3, 4 and 5) that we have put quite a lot of effort into finding a mix of well-tested and newly developed audio features that capture distinct attributes of audio recordings, such as energy, harmonic creativity and smoothness. Just to be totally clear: no Last.fm tags and no Last.fm scrobble magic are involved, only pure audio features, retrieved directly from the original recordings.

We calculated 21 of these features on 2 Million of our most scrobbled recordings and Mark built a neat, very fast service to host them. Since Friday this service has been publicly accessible through our outward-facing Last.fm API, thanks to Duncan's API magic. You can either ask for certain feature ranges and retrieve a list of songs that satisfy them, or you can retrieve the audio features themselves by providing the track's artist and title. Of course, bringing even the shiniest of APIs doesn't qualify as a hack...

Driver's Seat — steer your music playlisting!

Since I'd been very impressed with Spotify's new app integration I persuaded Sven to help me build a hack that nicely exposes how good our new API is at audio feature playlisting. And because it puts you in control of steering your music we called it Driver's Seat (screenshot). Below you see a video of the resulting Driver's Seat Spotify app in action.

According to your preferences you select a preset, or adjust feature sliders and hit "Go get playlist!" and the app will fire a http request to the Last.fm API that looks like this

http://ws.audioscrobbler.com/2.0/?method=track.findbyaudiofeatures&filter[]=bpm:80:91...

The result is a list of tracks that we then get the Spotify URI of using another brand new API of ours that loves requests such as this:

http://ws.audioscrobbler.com/2.0/?method=track.getPlaylinks&artist[]=radiohead&track[]=creep...

We really liked our hack because it allows music discovery to be uninhibited by artist genre or history — it just gives you the kind of music you request. The Spotify team liked it so much that they gave us their hack prize, which we share with a hack called CTRL — two of the 18 Spotify hacks.

PitchFork Effect

Sven and I weren't the only ones hacking away though. Alex produced some intriguing visualisations of how PitchFork reviews influence Last.fm listening stats... and received one of the two prizes from MusicMetric. Marek also made a cute little virtual album store as an antidote to the all too modern iTunes and Amazon stores. And Coffey re-worked a previous hack of his to scrobbling tracks at gigs you go to: it uses the set lists available through setlist.fm's API—find the hack here.

Last.fm for Spotify

Wednesday, 30 November 2011
by Steve Whilton
filed under Announcements
Comments: 28

Every now and again two things come together that make perfect sense, sometimes by design and sometimes just by natural affinity: gin and tonic, cheese and tomato, Jagger and Richards, Lennon and McCartney… two things that get better when they’re together.

Today Spotify announced that they’re opening their platform up to apps, and Last.fm is going to be one of their launch partners.

Introducing Last.fm for Spotify: personalised recommendations based on the music you play.


Track: “Calm Down” by Canterbury

Ever since Spotify launched, I’ve heard many people suggest that Last.fm and Spotify each do different but essential things they want from a music service. Spotify integrated Last.fm scrobbling into their service almost from day one, but scrobbling on its own really didn’t connect the two services meaningfully enough.

I remember saying the exact same thing long before coming to work at Last.fm. “Wouldn’t it be cool if you could somehow use Last.fm’s recommendations to decide what to play on Spotify?” Some of our community even used our APIs to build homebrew versions of it.

So when, earlier this summer, Spotify came to us to talk about their new platform and asked us “what would you do if..?”, it took us just a few seconds to agree that this was a massive opportunity to create something that we’d want to use ourselves, every day. It was a chance to take the scrobbles we were getting from Spotify users and then pay them back, with interest, in the form of great recommendations.

So what does Last.fm for Spotify do?

You can:
• Sign in to Last.fm, or start a new profile (new users have a very easy setup process)
• The overview page displays the listener’s recent top albums…
• …some recommendations
• …their recent tracks
• …and some of their loved tracks
• The Albums and Recommended tabs in the app will offer a deeper insight into those lists
• The app will allow users to build playlists of recommendations, top albums, recent tracks and loved tracks with just one click
• It also powers a Now Playing screen, which features scrobbling stats about artists
• …and users can launch a Similar Tracks playlist from the song currently playing in that tab
• And yes, it scrobbles too

Spotify users without a Last.fm account will be able to use the Now Playing view and Similar Artist playlists, but the personalised experiences are only available to new or existing Last.fm users. As ever, the more you use it, the better those personalised recommendations get.

The one-click playlists really are effortless, and you can play them on any Spotify app (mobile, etc.)

If you want to try this out for yourself, Spotify have a preview version you can download now. It’s going to be going out to the wider world very very soon. You can send us feedback in the Last.fm for Spotify forum as soon as you start using it.

We’re really excited to see this in the wild. We’re big fans of Spotify at Last.HQ, and a lot of work has gone into making this a great experience for old and new users.

Enjoy!

The Brainz are Back in Town

Thursday, 24 November 2011
by Adrian Woodhead
filed under Announcements and Stuff Other People Made
Comments: 18

Many moons ago Last.fm set up a collaboration with MusicBrainz, the open-source music metadata database and community. MusicBrainz use special keys known as MusicBrainz Identifiers (MBIDs) to uniquely identify artists, labels, songs and many other music-related entities. One of the main ideas behind them is that anyone can use these keys to identify and cross-reference musical entities, even if they come from different sources.

For example, an album review site could publish the MBID for an album being reviewed and someone could take this MBID and look up the album on MusicBrainz (or some other site supporting MBID lookups) to find out more information about it.

That was then

The plan at the time was to regularly synchronise MBIDs with the artists, albums and tracks in Last.fm’s catalogue. The MBIDs (or should those be MBIDz?) could then be used to query for these entities using our API, or to link across different services using the MusicBrainz ids as an external mashup key. Last.fm API calls also return MusicBrainz ids where applicable (see artist.getInfo for an example).

This continued successfully for a while until a few shake ups and changes over here resulted in us temporarily dropping the ball on this one and the regular updates stopped (we did however keep the historical data).

MusicBrainz members hanging out in Last.fm’s ball pit back in the day (photo by Mayhem).

This is now

The good news is that we reconnected with MusicBrainz while we were visiting San Francisco for this year’s Hadoop Summit and are recommitting ourselves to being good citizens of the MusicBrainz community.

Our first step is to once again synchronise Last.fm artists, albums and tracks with their relatives over at MusicBrainz using their Live Data Feed which means we’ll be updating these once an hour. We’ve already started doing this and have more information on the technical details over in our development discussion forum.

To the future and beyond

Once we are in synch again we will start looking at other ways we can more actively work together with MusicBrainz and also with others like The Guardian and the BBC who use MusicBrainz Identifiers too. We also hope to leverage MusicBrainz’s experience with handling artist and album disambiguation to improve how we model music on Last.fm. We currently have the pleasure of hosting one of their developers (OCharles) at our London offices a couple of days a week to work on these and other related issues.

So, if you have any questions, comments or requests for features please ask them over on the forum, catch you there.

Mo Data Mo Problems

Tuesday, 8 November 2011
by Stefan Sperber
filed under Trends and Data
Comments: 8

For the past six months, I’ve been interning with Last.fm’s data team and conducting research for my bachelor’s dissertation. My interest lay in working with Big Data: learning more about data analysis, processing massive amounts of data and solving problems that arise from working with data sets that are too large to be stored on one computer. And what better way to do this than to leave the cosy academic world of my university, move abroad and to write my final assignment on a new field in a foreign city, right?

Let me give you a personal account of the work involved for my research on Efficient Record Linkage with MapReduce in Very Large Data Sets.

Y U NO START WITH EXAMPLE?

Say for some reason you are given two sheets of paper with customer data. The first one contains the customers’ names and addresses while the second one lists their names and the results of a recent survey on their favourite colour. Your task is to connect all customers with their favourite colour.

You start with the first line on the one sheet, look at the name and try to find what this customer answered as their favourite colour on the other sheet. You continue doing this until you reach the end of the list.

I came for the view and the free biscuits.

While doing this customer record matching you notice a few things. Some customers have listed their favourite colour more than once, sometimes it is the same colour but sometimes it is not. The address data also is far from perfect; there are customers with similar names that all live at the same address. Conversely, one customer with a very unusual name seems to own several houses in the same street. These problem cases demand that you decide whether the same person is meant or not. If so, you pick one address and colour and write down each connection just once.

Now imagine this matching thing becomes a regular task that you have to complete at the end of each week. Oh, and the size of your customer data has also magically increased; it does not fit on two sheets of paper anymore but suddenly takes up billions of them. In this case it is time to invite your friends over to help you with your matching task.

You will have to agree on how you handle duplicate customer records and also distribute the work equally. Then, how do you even determine all the matches in these vast amounts of paper? I mean, hanging out with friends is excellent but going through many pages looking for all occurrences of just one customer quickly becomes dull, as most comparisons will be made unnecessarily. You will have to think of a way to minimise work.

“Unnecessary” quotes

This example illustrates some of the issues we face daily at Last.fm. We often have to integrate data that we received from our partners into our own music catalogue.

For example, in order to provide those sweet links to Spotify, Hype Machine, Amazon and iTunes on track pages we have to find corresponding entries that relate to similar artists, tracks, or albums in two or more data sets. Generally, this task is known as record linkage, which is a very active research field.

Links, links, links.

The specific question posed for my dissertation was: “What approaches are there that we can use to improve our data matching tasks at Last.fm?” My findings and conclusions will be used in the future to do this.

I first compiled a list of promising and interesting techniques and evaluated them in a small scale. These included approaches for pre-grouping entries that share a certain similarity in an efficient manner to later minimise the number of comparisons that need to be made (for example, an inverted index and a spatial index) and several metrics that can state how similar two entries are (for instance, metrics introduced by Levenshtein and Dice, but also approaches that first map strings to vectors and then measure the enclosed angle). This allowed me to come up with three combinations of techniques that performed best for our kind of data.

Still, the problem of scale remained, as working with large files and data sets introduces another layer of complexity to the initial problem of matching data. In recent years, MapReduce has become the number one choice for working with Big Data. One reason for its success is that MapReduce makes it very convenient to distribute data processing over a number of computers. Instead of having one computer doing all computations one after the other, many computers can work on small tasks at the same time, and the combined efforts generate a final result. The most commonly used implementation of MapReduce is Hadoop.

Valient Thorr!!!

MapReduce removes a lot of work for the programmer (for example, writing code that distributes work, collects results and reacts to failures), however it also demands that a problem must be expressible within the constraints that MapReduce introduces. These make it necessary to investigate if MapReduce really is the right tool for a given task. For example, techniques that worked well with small amounts of data might suddenly not perform as before when the size of the data is scaled to a certain size.

For the adaptations of the previously identified combinations to MapReduce, I switched to the Cascading framework. As mentioned, when you develop a program for MapReduce you will have to “think” in its programming model, which can sometimes be a painful and slow process. Cascading, however, abstracts the underlying MapReduce model using workflows and allows one to write very complex distributed programs in shorter time. We have been using Cascading extensively in the data team and we love it.

In brief, my findings were that you shouldn’t rely on MapReduce alone for data matching, as the record linkage process is difficult to map in whole to the MapReduce model. For example, the biggest performance bottleneck was the sorting and distributing of entries prior to making the comparisons — the step that is supposed to speed up the matching by pre-grouping entries with a certain similarity. I concluded that it is better to introduce another system for storing intermediary results (for instance, a distributed key-value store like memcached) or to evaluate other approaches that I didn’t have time to cover in depth.

Gain experience points, spend them all on coffee.

Last.fm is a dedicated bunch of people and it was great to learn from them how to tackle a problem properly. This environment quickly drew me in and motivated me. I remember sitting at my desk during my first afternoon and thinking: “This is exactly what I have been looking for.” I was trusted with steering my research in the right direction. I was in total control of my decisions, and could freely experiment and make mistakes (on one occasion one of my experiments on the Hadoop cluster went berserk and managed to hog terabytes of storage space), and everyone on my team supported me as much as possible; they were always approachable, no matter how busy they were.

Not sure if everyone shaking or drank too much coffee.

I enjoyed every day although it was, of course, much more work than I had expected; I was fine-tuning my dissertation and polishing my paragraphs right up until delivering it to the printers. Then, on Friday about one month ago I finally submitted it to my university.

What else do I take with me from six exciting months in London? I went to lots of great gigs and consumed vast quantities of excellent coffee. I was introduced to many people and ideas, had countless interesting conversations and got a good introduction to the local tech scene.

I must also have made a valuable contribution to my team. That is why my stay in the data team has been extended for a couple of months (“at least”, to quote a colleague). After all, there’s just so much more to learn.

Last.fm shares the Label Love

Friday, 16 September 2011
by Stefan Baumschlager
filed under Announcements
Comments: 8

Luckily the rioting in the UK last month didn’t hit Last.HQ, but some of the labels nearest and dearest to us were hit badly by the PIAS warehouse fire, destroying the stock of hundreds of independent labels – big and small.

Heading Last.fm’s Music Team means looking after all our relationships with record labels on a daily basis, and naturally I’ve got a lot of affection for them not to mention their skill for finding, building and releasing new artists into the world.

Every day Last.fm plays its part in that as our recommendation engine promotes new artists and tracks to listeners around the world. Usually this is based on almost half a billion tracks the Last.fm catalogue knows about, but we’ve put together something a little special for the labels affected by the PIAS fire.

Label Love provided us with a list of all the labels hit by the fire, and we’ve compiled a special recommendation tool that brings you recommendations based only on the artists affected.

Last.fm users now can enter their username at our Label Love site to find out about more artists they can support. Every recommendation comes with links to purchase music and gig tickets, making it easy for all of us to show a little support.

So share the love and spread the word, and follow @_label_love_ to find out about more ways you can join in.

Farewell Summer!

Monday, 5 September 2011
by Katy Adair
filed under Announcements
Comments: 6

It’s been an amazing summer, but sadly the festival season is over and it’s time to head home. The Last.fm Presents team wrapped up in style at SW4, and you can check out our photos on Flickr now.

Don’t worry though, we’ll be back on the road soon enough, with our Live Fridays just around the corner and much more. Keep up to date with what’s coming up over on the Last.fm Presents page and on twitter – @LastfmPresents.

See you next year!