OVH fire

Don’t know if you’ve seen this…

https://www.theregister.com/2021/03/10/ovh_strasbourg_fire/

Really big bad fire took out one data centre and damaged another. Looks like the other two are okay (one safe, one saved by fire crews).

My email last synchronised at 11am (so doesn’t appear to be affected), but the server has now vanished. As has, ironically, their “work notifications” server at http://travaux.ovh.com/

I suspect they’ve shut the entire site down (including the okay centre) until they can figure out what the hell to do next.

https://www.ovh.com/fr/news/presse/cpl1785.incendie-notre-site-strasbourg

https://www.youtube.com/watch?v=mIrdC5pEk1s

https://www.youtube.com/watch?v=v0Uf0KCRhas

So, don’t email me on my heyrick.eu account, it was there. You know, if you had the urge to email me for some reason. ;-)

Instead, ponder why such a large data centre didn’t have adequate fire prevention methods. Don’t these places have systems to pump out CO2 before, well, before it ends up as, well, this:

PS: Expect a number of French websites to be iffy right now.

Sorry you’ve got a problem Rick: hope it gets fixed soon. Re the fire, given the current state of UK/EU relations they’ll probably blame Lord Frost….

I may or may not have lost some (all?) of my heyrick.eu emails. It’s not mission critical, just an annoyance as there are some things I’ve not had time to reply to.

On the other hand, I rather imagine there are a few wishing they’d worn brown trousers today (or a nappy), having been sold on all the myriad benefits of The Cloud but…not so much the idea of maintaining backups.

Data centre SBG2 is gone. Completely. Anything there has gone.

And… Um…

https://mobile.twitter.com/playrust/status/1369611688539009025?

Probably here some more stories like that in the coming days. If you do stuff “in the cloud”, take this as a hint to buy a USB harddisc and keep a local copy of data. You know, if your entire business depends upon it………it is important.

Luckily something like 140 firefighters did a good job of getting this under control, and nobody at ovh was killed, so it’ll be a job for forensics, not undertakers.

For those with backups, it’ll just take time. One of the gouv.fr portals (hosted there) is already back at a different site. OVH have a bunch of sites around Europe.

For those without backups, have a little sympathy. Yeah, it’s stupid, especially as it might mean a complete loss of an entire business, livelihood, or years of work…but nobody thinks it’ll happen to them until it does. In this Covid working-from-home world, we depend upon data and its integrity more than ever.

When was your last backup performed and verified?

At least the fire suppression kicked in :|

It’s not CO2 that used. For large data centres, its either water based or inert gas depending on the situation and requirements. Gas is quicker release, but needs pressure equalisation otherwise the doors blow out! Water is slower, as the pressure has to build before the nozzles “pop” but doesn’t require large bottles and can be mains fed.

I’d be interested to see the outcome of the investigation.

Gas is quicker release, but needs pressure equalisation otherwise the doors blow out!

In the case of two server “rooms” and two main comms rooms the answer is FM200

One of those rooms is an old LINAC “bunker” with approximately 4 foot of concrete all round (walls, floor, ceiling) and a sealed steel entry door (after the standard 60-min corridor door)

Oh, and there’s a two-foot square hole with fan and baffle system.

Wifi-calling is handy in there…

Whoo-ee.

“Around 3.6 million websites across 464,000 distinct domains were taken offline”

https://news.netcraft.com/archives/2021/03/10/ovh-fire.html

http://travaux.ovh.net/ is back up. Lots of red.

Though, my email seems to be syncing again and a test email sent to a different account was received. Impressive work, given that “email bundled with a cheap domain name” is unlikely to be on anybody’s list of priorities.

There’s a lot of techies who are going to earn their paycheques in the coming week.
http://travaux.ovh.net/?do=details&id=49484

One useful thing to come of this, I’ve just discovered lichess.

I’m not a good player, I dabble. But holy hell, some of the stuff on the “TV” is insane.
https://lichess.org/TV

One side has 30 seconds, one side has 45 (I think this depends upon rankings? some games are a more sedate ~4 minutes). When they get going, it’s more than enough to clear the board and wipe out an opponent.

Thank god for replay, the speed of this will make your head spin.

https://lichess.org/6bF7TfNR

For anyone with no non-cloud backup that looks like a lot of cloud data heading for the clouds

Hopefully this will be a wake up call for a lot of IT and risk managers.

You’re joking, right? That sort of thing always gets overriden by the bean counters…

That sort of thing always gets overridden by the bean counters…

We have a partially effective system. I refer to it as “selective rendering of factual information”
Plus, our departmental #2 regularly ties the bean counters in knots (and knows their budget code)

The Covid wind of change is interesting too – a query today, from said #2, is “what improvements to your rest area(s) would you like” and this is coming from the top.

So, at least people can have their nervous breakdown in a nicer room. Maybe.

That sort of thing always gets overriden by the bean counters…

Risk is invariably a driving factor in large business, so the trick is to get issues onto the risk register. Once on the register, risk owners become liable and cost becomes a secondary factor.

From the security point of view, it’s a shame it only took out one of their cheap as chips data centres, which are a favourite for botnet use.

Costin Raiu, director of the Global Research and Analysis Team at Kaspersky pointed out what might be seen as a silver lining. Some of the activities hosted in any cloud provider are not entirely beneficial, and some malware operators were using the service: “Out of the 140 known C2 [command and control] servers we are tracking at OVH that are used by APT and sophisticated crime groups, approximately 64% are still online. The affected 36% include several APTs: Charming Kitten, APT39, Bahamut and OceanLotus.”

It could also mean botnet herders have better disaster recovery procedures than your average budget cloud user.

For some stupid reason I expected that cloud service providers backed up their systems. Oh, and kept the backups in a different building. Seems one if not both of those assumptions was wrong in this case.

Never treat cloud as a backup, unless it is the backup of what you are running and storing locally. If your data is in the cloud, you must still back it up – to another provider, or locally – or lose it when these events happen.

My early assumption when cloud services were first offered was that they were resilient – if a location failed, your data was still accessible on a mirrored site somewhere else – but that doesn’t seem to be the case, see how many times Google/Microsoft/Amazon cloud facilities go completely offline due to a configuration or security certificate error, etc. My only cloud use as a business is for yet another backup location for important data – and only in Switzerland.

For some stupid reason I expected that cloud service providers backed up their systems. Oh, and kept the backups in a different building. Seems one if not both of those assumptions was wrong in this case.

OVH are a provider of bare-metal dedicated servers and virtual private servers (VPS). In both of those cases their customer is renting infrastructure (IAAS), which means it’s up to their customer to handle backups. If they want to high-availability and resilience, it’s up to them to build (in an application-specific way) and pay for it. If you pay for software at different levels of the cloud stack (SAAS) then the provider handles it for you.

Similar kinds of breakage happens when big AWS datacentres drop off the internet which happens occasionally, although none have gone up in smoke.

Plus it’s fine to have backups, but you can only use them when you have servers to restore them to. Not so easy when you lose tens of thousands of machines in one go.

For some stupid reason I expected that cloud service providers backed up their systems. Oh, and kept the backups in a different building. Seems one if not both of those assumptions was wrong in this case.

Probably because you’re getting different services confused.

Cloud storage, of the OneDrive, GoogleDrive, DropBox variety that you or I would be using will almost certainly do that. I’ve not bothered to check anything but OneDrive, and in the first page that I found, MS clearly explained how they ensure that all files written to the service are held in (at least) two geographically distinct locations.

What OVH seem to be offering are low-level services for their customers to build on to. For example, a smaller cloud storage provider might rent OVH systems to run their storage service. This is more like the service offered by the familiar AWS. Here, there are no backups, because that’s the client’s problem. Our hypothetical small cloud storage operator would probably (if they’re competent) be using other providers than OVH, and be distributing the data around.

This is very obvious from The Register’s coverage of this¹. While some (less experienced?) customers are ranting on Twitter about their businesses being lost, those who knew what they were doing were up and running from other data centres within 24 hours. You wouldn’t consider running your desktop computer without backups; what OVH offer is the cloud equivalent.

¹ I had a more “hands on” experience of this when A2 Hosting had a small oops with some Windows servers back in 2019, which never really got resolved as far as I know². The small charity site that I was responsible for was able to restore completely from backups, while on Twitter small businesses were indicating that their entire company data had gone AWOL.

² After a week (the charity involved didn’t consider its web presence mission critical), we took A2 up on their offer of a refund and pointed the DNS to a new hosting company.

Probably because you’re getting different services confused.

A lot of people seem to confuse cloud and hosted. Relocating a server to a physical or virtual equivalent in someelse’s DC is not cloud based.

The distinction I would use is you are “cloud” if you have no idea where the data is, as you’re not pointing at a server. If you rely on named servers, you’re simply using a hosted service.

The former is backed up by the provider, the later is not.

It’s perfectly possible to build a zero-infrastructure environment these days without needing any servers or managed LAN/Firewalls, gold builds etc. Very few companies have gone this route yet as its only really been possible in the past two years.

Looking at it on Google Maps (you’ll see a big open space on street view, back up down the road a little and it’ll appear), it seems to me that there are two issues here.

The first, it looks like the place was built out of recycled shipping containers. That might have led to some interesting design decisions, such as…

…a top down (satellite) view gives the impression that it was four walls with emptiness in between, in other words, a giant chimney.

It looks from a preliminary announcement that the problem may have been a recently serviced (as in, the day before) UPS exploded. There’s not a lot of hope of you have a burning battery in your data centre.
I sure as hell wouldn’t want to be that service technician.

It looks from a preliminary announcement that the problem may have been a recently serviced (as in, the day before) UPS exploded

First rule of culpability – blame someone else first.

First rule of culpability – blame someone else first

Indeed. The UPS battery cabinets in one of our rooms demonstrated the effect of gravity and point loading exceeding load bearing capacity.¹
This of course was NOT the installers responsibility as the floor was clearly weak concrete (actually it was board with construction dust crusted on), and spreader plates are not required on concrete.
Beside this, the floor beams should take the load² as the legs were directly in line with other loading further across the room, so they assumed a continuation.
Building fault? Installer fault? Two sets of finger pointing.

¹ An entertaining 40-degree tilt when a point load (leg with no spreader plate) poked a hole in the under floor.

It might be a building fault (design feature!), but the ultimate liability ought to rest with the installer. The whole duty of care thing necessitates that one verifies that the emplacement is sound before putting a heavy object there.

But, finger pointing is always easier.

It might be a building fault (design feature!)

PFI cost cutting for the weak floor.

Installer, replaced an older UPS on a box plinth¹ (it matches a raised false floor) with a plinth having legs at each corner²

We didn’t care as long as they fixed things.

Since the temporary fix was a UPS in the plant room below and a new switching arrangement between that and the upstairs (not working) power board meant that fitting the replacement upstairs left us with dual UPS we just smiled.

¹ Effectively a load spreader