Possible Last.fm Downtime [confirmed]

Friday, 18 April 2008
by rj
Comments: 527

Our data center is losing power tonight for 8 hours, and we are in the process of shutting down all non-essential systems (and some semi-essential ones) to try and keep the site alive on a bare minimum of servers.

Please bear with us over the weekend – things might break in interesting and mysterious ways, and we’ll be trying to hold things together while DC/AC inverters explode all around us because we’re drawing too much power :)

Your scrobbles will be safe, but this blog post might self destruct in 19 hours….

Update 1
As usual, you can check status.last.fm for more information

Update 2
After painstakingly moving machines over to the backup power supply, the backup power supply flipped out (apparently it won’t happen again).

Things are going to be really broken / slow for a while whilst we bring a couple of hundred servers back online again, and wait for 300GB of memcache to fill up. What a day…

Update 3
The power outage is over now, and we’re slowly powering everything back up. It’ll be a day or three before things are completely back to normal though.