Library and streaming services outage

Monday, 18 July 2011
by cms
filed under Announcements and About Us
Comments: 291

Since 04:00 GMT on Sunday morning, the primary Radb service has been exhibiting intermittent problems meeting acceptable service levels. This means that libraries, scrobble counts and the services associated with them (stats, radio stations etc) appear to be broken when you use Last.fm.

We’re working very hard to fix this as soon as possible, and we’ve had engineers on it throughout yesterday and last night, but we wanted to keep you posted here with what’s happened and what we know so far.


UPDATE (19/07/11 11:29):
We’ve re-enabled access to your library and chart service after analysing this morning’s traffic in a little more depth.

As we continue working on the fixing the underlying faults it may be that we have to switch them off again for short periods of time. In the meantime keep checking the status page for information about the service.

Thanks again for your patience.


All of our services here at Last fm are predicated around keeping track of your accumulated music plays – your scrobbles – and using statistical methods on the historical data to build awesome things.

Because scrobbles power everything dynamic and wonderful on the site, we need fast realtime access to the scrobbles and associated summary data ( library, charts, neighbourhoods, etc. ). And with milions of people scrobbling, and the size of the historical scrobbles set, doing this fast in realtime, while updating the same sets with new plays as fast as can be managed is a significant challenge. We have a custom, in memory database service that we designed and implemented to address these needs. It is called Radb. It is usually pretty awesome at answering these sorts of questions in a predictably constant time.

But since 4am on Sunday Radb has been failing. The reasons for this are unclear. Failing over to the redundant service providers helps a little, but not enough. The reasons for this are also unclear. We have engineers working flat out to diagnose this problem.

To remove as much stress as we can from the database service layer, we have additionally disabled most of our radio, library and recommendation services, temporarily. We have also stopped accepting new scrobbles for the meantime, for a similar reason. Scrobbling clients will detect this state and will cache your scrobbles until the submission service is reactivated, so no need to panic about scrobbles being lost.

Keep an eye on the status page, and the appropriate forum posts. We’ll update with more information and estimates as we have them.

Find your friends on Last.fm

Wednesday, 13 July 2011
by davids
filed under Announcements
Comments: 43

Today we launched a feature to help you find your friends on Last.fm. It makes finding friends from Facebook or contacts from your email address book super easy. We think a more social Last.fm is a more fun Last.fm, where you can see what your friends are listening to, send them recommendations, compare tastes and more.

You can find friends from Facebook, Gmail and Yahoo! Mail. We’ve already had lots of feedback from people that would like Twitter and Hotmail added to that list and we’ll be looking at them (and others) in the future. There’s also a new feature that will suggest users you might know, based on the friends of your friends.

(If you’re a long time user then you might remember we had a friend finding tool some years ago. It had a habit of sending a few too many emails, sometimes in the wrong language and was a bit rubbish.)

Managing privacy has been a priority for us with friend finding; during early testing we unearthed a few secret staff accounts (packed with guilty pleasure scrobbling) and I even found one of my parent’s secret test accounts (presumably trying to see what this online music website hoo-hah was all about). In both of these cases the user wasn’t expecting to be found, and that’s the kind of thing we’re hoping to avoid.

It’s difficult to strike a balance between privacy and making a useful friend finder, but we think we’ve found middle ground.


  1. There’s a new privacy setting where you can opt out of appearing in results at all.

  2. By default your account will appear in results and a friend request can be sent, but we won’t show your username or profile; basically other people will only know that you have an account, not what your account is.

If you then choose to become friends after getting a friend request it’s then that they’ll get to see your account. Speaking of friend requests…

We’ve also given friend request emails some lovin’; they’ll tell you a bit more – like your musical compatibility or which friends you have in common – to help you decide if you want to be friends.

I’m also happy we got this new feature in front of some users to get their feedback and incorporate it into our work at an early stage. Your feedback is invaluable to us and helps us build a better Last.fm for everyone. This is hopefully going to be the start of more regular beta versions of new features, so watch this space.

Announcing Festival Apps!

Tuesday, 5 July 2011
by Michael Horan
filed under Announcements
Comments: 8

UPDATE 09/08/11: The app is now live in stores around the world.

Music lovers have always flocked to festivals. When else would you be willing to travel to the middle of nowhere at the peak of the summer sun to brave mud, dirty toilets and ridiculously expensive drinks?

And while we all know about some of the monsters of Festivals – Coachella, Glastonbury, Roskilde, SXSW and Fuji Rock – there are thousands of others that could be perfect for you.

A little over a year ago, we announced the launch of Last.fm Festivals, your personalized guide to festivals around the world. This guide allows you to search for the festivals you want to see and discover festivals based on your scrobbling history.

I am pleased to announce we have ported this functionality to new iPhone and Android applications. While there are several applications that use Last.fm’s APIs to provide you with similar functionality, the official Last.fm Festivals Application harnesses the full power of our recommendation engine to show you the festivals you should be going to. Drawing from Last.fm’s database of almost 2 million events, the Festivals App curates this down to approximately 4,500 upcoming festivals worldwide.

The Festivals App also tells you the festivals your friends are attending, and gives you a virtual passport with which you can explore the world in search of more festivals, all with the tap of a finger.

Have friends that aren’t scrobbling? While the Last.fm App is most useful for active scrobblers, unregistered users can search for festivals by name or artist, find festivals closest to their GPS location, or by date. Everyone can learn more about artists playing at any festival in the world; complete with similar artists and bios.

The initial release is only available to users in the US. Sorry! The app is now live around the world – keep an eye on the App’s group page for more information.

Now download the app and get on the road!

Oops, we got our numbers wrong. I’ve updated the total number of events we have in our database from 130k to 2 million (1,956,870, to be exact).

Last.fm Music Manager powers Mp3.com's library of 1M promotional downloads

Friday, 1 July 2011
by Matthew
filed under Announcements and Stuff Other People Made
Comments: 4


Ok, quick internet history lesson: Way back in 1997, there once was a site called Mp3.com. It was famous as one of the first sites to encourage the free flow of music and took its name from what had been an obscure digital music file format, one that we all know and love today. It spread faster than a celebrity sex tape, building an audience that was massive and showed the world that digital files were a fantastic way to discover new music. It was also one of the first big internet music IPOs, raising a stupendous amount of money.

Not surprisingly, that money and audience made it infamous with the big music labels who fought hard to shut it down, creating a familiar yet vicious circle that has continued for more than a decade: technology makes content distribution fast and nearly frictionless; content rights holders throw grit and lawyers at the new products based on it until there is so much slowing it down that it eventually collapses under its own weight. We saw it happen again last week as Turntable.fm shut out ex-US users due to licensing constraints.

But sometimes evolution steps in and changes the pattern a little bit.

Right around the same time some of us were packing our Wellies and heading out to Glastonbury, our parent company CBS Interactive Music group relaunched Mp3.com with a snazzy new interface and a whole load of great new content. We’re happy to say that it’s Last.fm’s Music Manager that powers Mp3.com’s library of music and that Last.fm’s artist wiki pages power a lot of the information on these new bands. The new Mp3.com is editorially driven, drawing on a small staff of writers and the Last.fm Music team to comb through our library to introduce you to great tracks from great artists you might not have discovered yet, as well working with the labels to get new tracks from superstars and great indie bands who are already well-known.

But one of my favorite things about Last.fm is that almost a quarter of the 13M tracks available in our streaming radio system are from unsigned or independent artists who aren’t affiliated with big labels and who own the rights to their music outright. These tracks are uploaded by new bands to Last.fm’s Music Manager and this puts them into Last.fm’s radio database. Bands can also choose to make those tracks into full-length promotional tracks that live on their artist pages and/or make them available for free download. This is, by far, the bulk of the music that’s available on Mp3.com and on Last.fm. That’s true to the original spirit of Mp3.com which did more to launch new bands than almost anyone else back in the late 1990s.

The new Mp3.com will have three features that we think are worth paying attention to:

Free MP3 of the Day
This is a daily feature that will offer free downloads of songs from the big name artists and fast rising indies. Most of CBS Radio’s US music stations will be promoting this on-air and so the featured tracks will be divided up into four of the most popular US radio formats: country, pop, rock and urban.

On the Scene
On the Scene is a more international feature, showcasing a city or area with musical significance with free songs from bands who come from that scene. Expect us to highlight both historical and brand-new scenes that catch our attention.

Label of the Week
The name already gives away the concept behind this one. Don’t let anyone tell you anything different: Not all labels are dying dinosaurs. The smart ones are just evolving and we’re here to help you find the fast moving mammals who we think will avoid the tar pits of doomed business models of the old-school music industry.

The last two weekly features above are written by the mighty Last.fm Music team. Stefan, Nick, and Helen here in the UK want to hear from you. Help us feature your bands, your scenes, your labels. We’re committed to helping people discover vibrant new music and not just hyping the biggest and most popular bands.

We have big plans for Music Manager this summer. We’re adding better rights support for bands who have more complicated rights deals. We’re making it easier for bands to track the downloads and the impact of the tracks they distribute through us and through Mp3.com. And we have some secret products and events coming later this year that will expand the promotion opportunities for indie bands coming up fast on our Hype Charts.

We also have some fun partnerships in the works to improve the tools that indie artists want to use to distribute their music to a wider audience. Stay tuned and if you are an indie artist who wants to reach more people, check out Music Manager and visit us on the label forums to talk about what you want from Music Manager.

The Power of Sound

Wednesday, 29 June 2011
by chrissie
filed under Announcements and About Us
Comments: 2

A live music experience is such a powerful one, with each person taking moments from a gig that they will enjoy and remember for a long time to come. Some of us go back to see a favourite band or artist time and time again and somehow there is something special and different about each performance.

In the commercial team, which is where I am based, we really enjoy the challenge of pulling some of that spirit into work for relevant and interesting brands. A perfect opportunity to do this arrived with HP who are looking to explore the Power of Sound to help promote their new range of Envy laptops and the sound quality from the incorporated Beats Technology.

So, we’re blending the power of the Hype Charts and our expertise in the live arena to pull together some really special events over the new few months, and we want you to join us.

First up we’ve been on the road chatting to artists at our summer festival shows, asking them about The Power of Sound. The interviews from Liverpool Sound City can be seen here and it features artists as varied as Frank Turner, Akala and Willy Nelson. Each has their own take on the concept, and it’s great to hear each artist talk about it in their own unique and diverse way. I can’t choose a favourite from this batch but I am still amazed that the Dutch Uncles reference Bon Iver, J Dilla, Frank Zappa and Biggie Smalls all in one video!

The interviews from Get Loaded will be ready in the next few days and we’ll be at Sonisphere and SW4 amongst others for more. Keep up to date on new interviews in the radio player (UK only for these I’m afraid!) on our website and on HPUK’s Facebook page.

Next up we’ve got three pop up acoustic sets, all set up with the help of Black Cab Sessions. Our first was with Slow Club in Soho Square, and they were great. They powered on through the rain to sing a couple of songs, including a new one from forthcoming album Paradise. We’ve got pictures up on our Flickr page, and you can find footage from the set and an exclusive interview here.

All this is working towards a main gig at the end of summer… but we’re keeping details about that one secret for now. What I can say is that a team of passionate people are working on getting a great line-up as I type, and we are all set to make it a fantastic event.

As a little bonus, all of the artists that are taking part in the project will help curate an HP Power of Sound Custom Radio station, which will be ready to launch in a few days. If you want to give some input into the content of the station again please head to HP’s FB page.

And last but not least we are delighted to let you all know that we are releasing a Last.fm app for the HP Touchpad which we are sure will get a whole lot more people scrobbling!

Huge thanks to everyone involved in pulling these off. Make sure you keep an eye on our Twitter page for info about the live sets throughout the next few months, and for footage from the sessions!

Wishing you a great summer.

Don't mix networks and exhaust fumes

Wednesday, 25 May 2011
by mike
filed under Announcements
Comments: 13

Last.fm is all about bringing you music you love, and a big part of that is the technical infrastructure required to make it all happen. We take site and service reliability very seriously, which is why we get upset when things go wrong.

And this week things went wrong. It has been a particularly busy one for us, as we’ve had to deal with a large router failing on us which has unfortunately had user visible impact in a few places. As such, we want to share some of the back story, so you can understand what happened and how we’re working on making things better.

What the Ops team do

Last.fm began life as a small start-up some years ago now, and as is normal for technology based start-ups, reliability at a large scale wasn’t a big concern at the beginning. Start-ups focus on making things work, and worries about uptime and reliability come later. Reliability also costs money, and adds complexity to systems – it’s easy to inadvertently make a “highly-available” configuration less reliable than a single server. We do a lot today to deliver service reliability that wasn’t part of our earlier architectures, and we can survive many problems with no externally visible signs.

Today, we run Last.fm from multiple separate datacentres, and we build resiliency and failover into all our new systems from the outset. We’re working hard on retrofitting this same level of reliability to all our older systems, though we still have some way to go before everything is where we’d like it to be.

The biggest problem we engineer for is the complete failure of an entire site. That’s a level of problem that we don’t expect to happen often, but we do plan for it and there are many aspects that need to be considered. It’s also the problem we effectively encountered this week, and for the most part everything went according to plan.

What happened

The system that failed was a large core router, which provides our cross site connectivity, and half of our internet connectivity. Its failure effectively isolated all the equipment in that datacentre, and caused us a lot of trouble. The system in question is equipped with fully redundant supervisor modules to prevent this sort of problem, but – for reasons that did not become apparent until later – the redundancy also failed.

We initially saw problems with this system a week ago, and carried out both a component swap out and reload of the software, which we thought had resolved the problem. When it failed again, our hardware service partners concluded we must be looking at a backplane fault, and shipped us a new chassis.

The backplane in this sort of system is essentially just a passive circuit board, so faults of this kind are most unusual. It wasn’t until we removed the old chassis that we discovered a large amount of grime covering its intake vents, which is not what you expect in a data centre with large air filtration and cooling systems.

It turns out that some of the air intake for this hosting facility is pulled in fresh off the roof, and the adjacent building houses the exhaust stack from the diesel generators used as back up power. In a suitably ironic fashion, the diesel exhaust was being pulled into the air conditioning system, depositing fine particulates on surfaces, including our hardware.

Where you have equipment with large fan assemblies, this problem is made worse, and the deposits can cause electrical problems, leading otherwise highly reliable equipment to mysteriously fail. The datacentre we use has only recently become aware of this problem, and is taking steps to resolve it, but in the meantime we’ve had to deal with the effects.

What problems did users see?

During these problems, users may have seen a couple of issues. The first and most visible of these will have been radio failures. Our radio infrastructure is cross site, but currently needs a careful manual failover process of some elements, so you may have been without radio for a period of time. Web site traffic and API traffic fails over automatically, so most people won’t have seen any issues with this. Some users will have though, as the cross site failover process is DNS based – this means it’s not instant, and ISPs that don’t correctly handle DNS timeouts can cause extended problems. This kind of thing seems to be most common amongst mobile providers.

Some of you will be concerned about your scrobbles – no scrobbles were lost during these issues. Client caching should ensure that any that didn’t make it to our servers will have been queued and resubmitted.

We’re sorry for any problems you may have seen while we worked on this behind the scenes. We’re constantly working on making the service better, and making these incidents a thing of the past. Thanks for listening!

Introducing Facebook Connect to Last.fm

Thursday, 19 May 2011
by steve
filed under Announcements
Comments: 26

Last.fm has always been about recommending you new music, but that’s pretty hard to do for anyone new to the service. We need to know what music you’re into before we can recommend more music to you – right?

We call this the “cold start problem”. Up until now we have asked you to tell us what artists you like in order to kickstart your profile, but from today new users can use Facebook Connect to populate your Last.fm profile with any artists you like on Facebook. Neat, huh?

It means we can start delivering you new music recommendations straight away, and give you a better experience of Last.fm from the moment you create your account.

You also have the option of using your Facebook profile pic, name, birthdate and other info to help complete your Last.fm profile. All of this information is editable, allowing you to select only the details you want to bring across to Last.fm, and you even have the option to trash all of it and start from scratch with just one click.

Actually, we’ve made a few changes to social features recently. We’ve renamed the Facebook ‘Like’ button to ‘Recommend’ (no change to functionality though) and also added the new Facebook ‘Send’ button. The Send button allows you to recommend an event, track or artist to specific friends or groups on Facebook as a message to their inbox rather than publishing to everyone via your feed, pretty useful for music you really don’t want your friends to miss out on but don’t want to spam everyone else with.

There’s more coming soon too, including the ‘Friend Finder’ – helping you find and add your friends from other platforms – which we’ll be launching in the next few weeks.

Please don’t forget to give us feedback about features like these on the forums.

Lend us your ears again - Audio Flowers and musical complexity

Wednesday, 18 May 2011
by matthias
filed under Announcements
Comments: 6

Edit: we’ve updated the image of the Audio Flower to give you a better idea of what it visualises.

We’ve been thrilled with the all support we’ve been getting from users who are helping us rate the tempo of music tracks in our Speedo experiment, thanks! Now we’d like to ask you to help us with another fun music experiment for a new project called Audio Flowers.

We are currently doing some research into new techniques to measure structural change (or “complexity”) in rhythm, harmony and timbre directly from mp3 files. The measurements we take from a song are then summarised to produce a little image: an Audio Flower like the one below.

Here’s a description of what the ‘flower’ above is showing us.

We can tell straight away that Mr. Roboto by Styx has strong long term changes: the end of the red rhythm petal is quite thick – as are ends of the harmony and timbre petals. In fact, this suggests that the song is organised in distinctive parts.
Let’s have a closer look at the red rhythm petal then: towards the middle of the petal, its main, opaque part is much thinner then at the tip. This indicates that for most of the song there is little mid-term rhythmic change. However, you can see from the translucent part that there must be some sections of the song with atypically many rhythmic changes … find out if you agree here (link to YouTube).

Find more examples here.

We think this visualisation could be used by other people who’d like to find out about the complexity of music, and that’s why we want to publish the technique at an international scientific conference later this year.

But here’s the problem: although we’re quite happy with our signal processing magic (which automatically guesses the amount of structural change going on in the music), we don’t know yet if human beings like yourself feel the same way as the computers… and we need to know that in order to convince the scientists to publish our results.

The new Audio Flowers demo is similar to the Speedo demo; just listen to short excerpts of tracks on screen and answer a couple of simple questions.

We really hope you can help us again by listening and voting! If you like the Audio Flowers, please tell us, and we’ll see if we can make more of them available for your favourite songs.

Last.fm starts the summer early

Wednesday, 4 May 2011
by helen
filed under About Us and Announcements
Comments: 3

Since 1995, Camden Crawl has established itself as the May Day Bank Holiday weekend’s hottest ticket, and even though it had competition from the Royal Wedding this year it was a great way of kicking our live series Last.fm Presents into gear for the summer season.

The baroque setting of Koko was the venue for our own stage, treating Camden to a cocktail of seven UK acts rising the Hype Charts. First up was Dinosaur Pile-Up, a band touched by the hand of grunge, whose track “My Rock ‘n’ Roll” proved a mission statement for the night. Lethal Bizzle is a star of Last.fm’s grime tag, and he embraced the spirit of Camden Crawl with a stage dive ahead of indie rockers Mazes, who’ve featured heavily in our Hype Chart over the past month.

British Sea Power transformed the Last.fm Presents stage into an ode to nature: foliage sprung up as footage of sea birds played. Epic favourites such as “Waving Flags” gained the biggest reception, before the sets took a turn for the electronic with the last two acts.

Simian Mobile Disco proved that knob-twiddling needn’t be a static affair, and the crowd agreed – four levels of Koko got down to “Audacity of Huge” and “Hustler”– while Hudson Mohawke excelled as last performer of the evening, his mix of r ‘n’ b vocals and groundshaking bass making for a warped electro trip into the night.

If you’re feeling like you missed out on a brilliant night, well, you sort of did. No fear though, Last.fm Presents have a packed festival season ahead this summer.

If you are heading down to (deep breath) The Great Escape, ATP, Liverpool Sound City, Get Loaded, Sonisphere, Rock Werchter, Truck, Field Day, Underage, Summer Sundae or SW4 then keep an eye out for our LFM lobbyists who’ll be ready to shower you with Last.fm goodies, including our tag stickers.

It’s going to be a great summer!

Last.fm: now supporting tea breaks

Wednesday, 13 April 2011
by Dane
filed under Announcements and Code
Comments: 57

Hey, look! It’s a pause button! I know right?!

Pause is a feature that users and staff alike have been requesting for quite some time, so one dreary Thursday evening I decided ‘Enough is enough! We NEED pause and we need it now!’

That’s almost true anyway. It’s been a big challenge to implement, and we’ve spent a little while testing this feature — writing some supporting infrastructure and making sure the feature works well across our different players.

That’s right, the Android and iPhone client also now come with pause too. We’re working on a new version of the desktop client now, and that will come with pause too.

I’ve been getting a ton of use out of it and I hope you guys do too! The specs are now available for partner players (like XBox and Windows Phone 7) to support the feature, and we’ll be updating the FAQ as and when they’ve implemented it.

Drum Roll – It’s time for the main feature!

But… but but but, we’re also releasing something new and exciting into the wild today.

Whenever you tune into a radio station on Last.fm we build a playlist of tracks based on various criteria: for Recommended Radio we’re looking at music that you might like based on what you’ve been listening to recently; for Friends Radio we’re looking at what your friends have listened to recently… and so on and so forth.

Up until now we haven’t surfaced why a particular song is being played to you, but that’s about to change with a little feature that puts some info text in the top left of the player.

When you’re listening to Similar Artist Radio or your Library Radio we’ll show you some information about the track being played (the song selection is kinda obvious — it’s in your library, or it’s similar to the artist you typed in).

Things start to get a little more interesting when you’re tuned to Friends, Neighbours, Recommended or Mix radio. You’ll see information about which artists or users fed into the song selection. If you click the “more” link you’ll scroll down to where there’s a little more detailed information; maybe it’s a few of your friends or a few artists that inspired the selection.

(By the way, if you’re using the Festive cheer or Bah! Humbug! radio settings then you’ll get a reduced amount of information. If you want to experience the magic you’ll have to turn them off for now, sorry!)

Hope you enjoy them! Remember, you can always offer feedback about features like these on the forums, and if you want to join the team who made them just head to the Jobs page.