Our data center is losing power tonight for 8 hours, and we are in the process of shutting down all non-essential systems (and some semi-essential ones) to try and keep the site alive on a bare minimum of servers.
Please bear with us over the weekend – things might break in interesting and mysterious ways, and we’ll be trying to hold things together while DC/AC inverters explode all around us because we’re drawing too much power :)
Your scrobbles will be safe, but this blog post might self destruct in 19 hours….
As usual, you can check status.last.fm for more information
After painstakingly moving machines over to the backup power supply, the backup power supply flipped out (apparently it won’t happen again).
Things are going to be really broken / slow for a while whilst we bring a couple of hundred servers back online again, and wait for 300GB of memcache to fill up. What a day…
The power outage is over now, and we’re slowly powering everything back up. It’ll be a day or three before things are completely back to normal though.