Last.fm is burning!

[Vox Imperatoris]

http://www.last.fm

Failure in some cooling devices at the Last.fm datacenter has caused temperatures to soar from 20-30°C to 40-50°C. This is not a good thing. It's unclear whether accounts will be damaged by this, but they're offering updates on Twitter. All links to Last.fm redirect to that same page.

[Zabigail]

Wow! I hope they did recent backups! Or, would this also take out any backups?!

[TheFlame]

Perhaps now wasn't the best time to try and scrobble all the music I've been playing on my iPhone for the past 2 weeks.

[Baldwin]

So, what is last.fm?

[Vox Imperatoris]

So, what is last.fm?

It's a website that has mostly user-maintained profiles for pretty much all music artists you can think of, along with an internet "radio" service that also lets you listen to music it recommends for you. You can make an account for free, and it keeps track of all the music you listen to (both on the site, and on iTunes/Windows Media Player, if you download their "scrobbler"). So, what people are worried about is that this will take out the, in my case, nearly 17,000 "plays" I have reported to that site.

TheFlame, the Last.fm Scrobbler will store all of your plays in a cache if it can't connect to the server when you try to scrobble.

Also, 40-50°C is the ambient temperature in the room. Inside the servers (which aren't actually owned and operated by Last.fm, but by the Level 3 server company), I imagine it is much hotter.

ETA: The latest Twitter update repeats what I said about the client caching the scrobbles and says none should be lost. So, I'm guessing that the profiles are going to be okay.

[Vox Imperatoris]

We're slow, but we're back. Please bear with us for the next couple of hours.. We're missing a lot of capacity still.

That was fast! Profiles are fine.

[CatInASuit]

Air Con failure - the bane of any datacentre.

It's always fun when that happens

[Vox Imperatoris]

Air Con failure - the bane of any datacentre.

It's always fun when that happens

You'd think it would be designed so that one chiller failing wouldn't cause all the others to fail, as well.

[WhyNot]

Air Con failure - the bane of any datacentre.

It's always fun when that happens

You'd think it would be designed so that one chiller failing wouldn't cause all the others to fail, as well.

So...it's like a string of old Christmas tree lights? Brilliant.

[What Exit?]

Air Con failure - the bane of any datacentre.

It's always fun when that happens

When I first started at my current job we had a sprinkler head in the data center and only a single AC unit and haphazard UPS setup. Since then, I got the sprinklers out and a Halogen like system, dual ACs, UPS and by boss got religion on 24x7 and redundancy and we now have a backup diesel that kicks on automatically and will run the servers and one AC unit.

Data Centers can be surprisingly poorly thought out.

[What Exit?]

Without details, there can be two different issues with the cooling. I use to do HVAC at a hospital.

The Data center at the Hospital had separate power for the Computers and AC units. You can lose AC without losing servers. Also servers usually have UPS and other backup power sources and AC usually does not.

Our AC units were chill water anyway. If the chill water is not being made, no cooling for the computer room. They eventually built a new Computer Room with two dedicated Freon units with separate power sources.

[Vox Imperatoris]

Air Con failure - the bane of any datacentre.

It's always fun when that happens

You'd think it would be designed so that one chiller failing wouldn't cause all the others to fail, as well.

So...it's like a string of old Christmas tree lights? Brilliant.

Not literally in series like that, but the others couldn't handle the heat after one failed, causing them to break and the whole thing to keep getting hotter.