OVH fire
Pages: 1 2
Rick Murray (539) 13806 posts |
Don’t know if you’ve seen this… https://www.theregister.com/2021/03/10/ovh_strasbourg_fire/ Really big bad fire took out one data centre and damaged another. Looks like the other two are okay (one safe, one saved by fire crews). My email last synchronised at 11am (so doesn’t appear to be affected), but the server has now vanished. As has, ironically, their “work notifications” server at http://travaux.ovh.com/ I suspect they’ve shut the entire site down (including the okay centre) until they can figure out what the hell to do next. https://www.ovh.com/fr/news/presse/cpl1785.incendie-notre-site-strasbourg https://www.youtube.com/watch?v=mIrdC5pEk1s https://www.youtube.com/watch?v=v0Uf0KCRhas So, don’t email me on my heyrick.eu account, it was there. You know, if you had the urge to email me for some reason. ;-) Instead, ponder why such a large data centre didn’t have adequate fire prevention methods. Don’t these places have systems to pump out CO2 before, well, before it ends up as, well, this: PS: Expect a number of French websites to be iffy right now. |
George T. Greenfield (154) 748 posts |
Sorry you’ve got a problem Rick: hope it gets fixed soon. Re the fire, given the current state of UK/EU relations they’ll probably blame Lord Frost…. |
Rick Murray (539) 13806 posts |
I may or may not have lost some (all?) of my heyrick.eu emails. It’s not mission critical, just an annoyance as there are some things I’ve not had time to reply to. On the other hand, I rather imagine there are a few wishing they’d worn brown trousers today (or a nappy), having been sold on all the myriad benefits of The Cloud but…not so much the idea of maintaining backups. Data centre SBG2 is gone. Completely. Anything there has gone. And… Um… https://mobile.twitter.com/playrust/status/1369611688539009025? Probably here some more stories like that in the coming days. If you do stuff “in the cloud”, take this as a hint to buy a USB harddisc and keep a local copy of data. You know, if your entire business depends upon it………it is important. |
Rick Murray (539) 13806 posts |
Luckily something like 140 firefighters did a good job of getting this under control, and nobody at ovh was killed, so it’ll be a job for forensics, not undertakers. For those with backups, it’ll just take time. One of the gouv.fr portals (hosted there) is already back at a different site. OVH have a bunch of sites around Europe. For those without backups, have a little sympathy. Yeah, it’s stupid, especially as it might mean a complete loss of an entire business, livelihood, or years of work…but nobody thinks it’ll happen to them until it does. In this Covid working-from-home world, we depend upon data and its integrity more than ever. When was your last backup performed and verified? |
Jon Abbott (1421) 2641 posts |
At least the fire suppression kicked in :| It’s not CO2 that used. For large data centres, its either water based or inert gas depending on the situation and requirements. Gas is quicker release, but needs pressure equalisation otherwise the doors blow out! Water is slower, as the pressure has to build before the nozzles “pop” but doesn’t require large bottles and can be mains fed. I’d be interested to see the outcome of the investigation. |
Steve Pampling (1551) 8155 posts |
In the case of two server “rooms” and two main comms rooms the answer is FM200 One of those rooms is an old LINAC “bunker” with approximately 4 foot of concrete all round (walls, floor, ceiling) and a sealed steel entry door (after the standard 60-min corridor door) Oh, and there’s a two-foot square hole with fan and baffle system. Wifi-calling is handy in there… |
Rick Murray (539) 13806 posts |
Whoo-ee. “Around 3.6 million websites across 464,000 distinct domains were taken offline” |
Rick Murray (539) 13806 posts |
http://travaux.ovh.net/ is back up. Lots of red. Though, my email seems to be syncing again and a test email sent to a different account was received. Impressive work, given that “email bundled with a cheap domain name” is unlikely to be on anybody’s list of priorities. There’s a lot of techies who are going to earn their paycheques in the coming week. |
Rick Murray (539) 13806 posts |
One useful thing to come of this, I’ve just discovered lichess. I’m not a good player, I dabble. But holy hell, some of the stuff on the “TV” is insane. One side has 30 seconds, one side has 45 (I think this depends upon rankings? some games are a more sedate ~4 minutes). When they get going, it’s more than enough to clear the board and wipe out an opponent. Thank god for replay, the speed of this will make your head spin. |
Steve Pampling (1551) 8155 posts |
For anyone with no non-cloud backup that looks like a lot of cloud data heading for the clouds |
Jon Abbott (1421) 2641 posts |
Hopefully this will be a wake up call for a lot of IT and risk managers. |
Rick Murray (539) 13806 posts |
You’re joking, right? That sort of thing always gets overriden by the bean counters… |
Steve Pampling (1551) 8155 posts |
We have a partially effective system. I refer to it as “selective rendering of factual information” The Covid wind of change is interesting too – a query today, from said #2, is “what improvements to your rest area(s) would you like” and this is coming from the top. So, at least people can have their nervous breakdown in a nicer room. Maybe. |
Jon Abbott (1421) 2641 posts |
Risk is invariably a driving factor in large business, so the trick is to get issues onto the risk register. Once on the register, risk owners become liable and cost becomes a secondary factor. |
David J. Ruck (33) 1629 posts |
From the security point of view, it’s a shame it only took out one of their cheap as chips data centres, which are a favourite for botnet use.
It could also mean botnet herders have better disaster recovery procedures than your average budget cloud user. |
Alan Adams (2486) 1147 posts |
For some stupid reason I expected that cloud service providers backed up their systems. Oh, and kept the backups in a different building. Seems one if not both of those assumptions was wrong in this case. |
Grahame Parish (436) 480 posts |
Never treat cloud as a backup, unless it is the backup of what you are running and storing locally. If your data is in the cloud, you must still back it up – to another provider, or locally – or lose it when these events happen. My early assumption when cloud services were first offered was that they were resilient – if a location failed, your data was still accessible on a mirrored site somewhere else – but that doesn’t seem to be the case, see how many times Google/Microsoft/Amazon cloud facilities go completely offline due to a configuration or security certificate error, etc. My only cloud use as a business is for yet another backup location for important data – and only in Switzerland. |
Theo Markettos (89) 919 posts |
OVH are a provider of bare-metal dedicated servers and virtual private servers (VPS). In both of those cases their customer is renting infrastructure (IAAS), which means it’s up to their customer to handle backups. If they want to high-availability and resilience, it’s up to them to build (in an application-specific way) and pay for it. If you pay for software at different levels of the cloud stack (SAAS) then the provider handles it for you. Similar kinds of breakage happens when big AWS datacentres drop off the internet which happens occasionally, although none have gone up in smoke. Plus it’s fine to have backups, but you can only use them when you have servers to restore them to. Not so easy when you lose tens of thousands of machines in one go. |
Steve Fryatt (216) 2103 posts |
Probably because you’re getting different services confused. Cloud storage, of the OneDrive, GoogleDrive, DropBox variety that you or I would be using will almost certainly do that. I’ve not bothered to check anything but OneDrive, and in the first page that I found, MS clearly explained how they ensure that all files written to the service are held in (at least) two geographically distinct locations. What OVH seem to be offering are low-level services for their customers to build on to. For example, a smaller cloud storage provider might rent OVH systems to run their storage service. This is more like the service offered by the familiar AWS. Here, there are no backups, because that’s the client’s problem. Our hypothetical small cloud storage operator would probably (if they’re competent) be using other providers than OVH, and be distributing the data around. This is very obvious from The Register’s coverage of this1. While some (less experienced?) customers are ranting on Twitter about their businesses being lost, those who knew what they were doing were up and running from other data centres within 24 hours. You wouldn’t consider running your desktop computer without backups; what OVH offer is the cloud equivalent. 1 I had a more “hands on” experience of this when A2 Hosting had a small oops with some Windows servers back in 2019, which never really got resolved as far as I know2. The small charity site that I was responsible for was able to restore completely from backups, while on Twitter small businesses were indicating that their entire company data had gone AWOL. 2 After a week (the charity involved didn’t consider its web presence mission critical), we took A2 up on their offer of a refund and pointed the DNS to a new hosting company. |
Jon Abbott (1421) 2641 posts |
A lot of people seem to confuse cloud and hosted. Relocating a server to a physical or virtual equivalent in someelse’s DC is not cloud based. The distinction I would use is you are “cloud” if you have no idea where the data is, as you’re not pointing at a server. If you rely on named servers, you’re simply using a hosted service. The former is backed up by the provider, the later is not. It’s perfectly possible to build a zero-infrastructure environment these days without needing any servers or managed LAN/Firewalls, gold builds etc. Very few companies have gone this route yet as its only really been possible in the past two years. |
Rick Murray (539) 13806 posts |
Looking at it on Google Maps (you’ll see a big open space on street view, back up down the road a little and it’ll appear), it seems to me that there are two issues here. The first, it looks like the place was built out of recycled shipping containers. That might have led to some interesting design decisions, such as… …a top down (satellite) view gives the impression that it was four walls with emptiness in between, in other words, a giant chimney. It looks from a preliminary announcement that the problem may have been a recently serviced (as in, the day before) UPS exploded. There’s not a lot of hope of you have a burning battery in your data centre. |
Jon Abbott (1421) 2641 posts |
First rule of culpability – blame someone else first. |
Steve Pampling (1551) 8155 posts |
Indeed. The UPS battery cabinets in one of our rooms demonstrated the effect of gravity and point loading exceeding load bearing capacity.1 1 An entertaining 40-degree tilt when a point load (leg with no spreader plate) poked a hole in the under floor. 2 The floor beams didn’t extend into that corner due to a build feature being designed in and then omitted. We were supposed to have a goods lift allowing delivery of servers etc direct to the room, so they left an area of floor ready for cutting out to fit the lift. |
Rick Murray (539) 13806 posts |
It might be a building fault (design feature!), but the ultimate liability ought to rest with the installer. The whole duty of care thing necessitates that one verifies that the emplacement is sound before putting a heavy object there. But, finger pointing is always easier. |
Steve Pampling (1551) 8155 posts |
PFI cost cutting for the weak floor. Installer, replaced an older UPS on a box plinth1 (it matches a raised false floor) with a plinth having legs at each corner2 We didn’t care as long as they fixed things. Since the temporary fix was a UPS in the plant room below and a new switching arrangement between that and the upstairs (not working) power board meant that fitting the replacement upstairs left us with dual UPS we just smiled. 1 Effectively a load spreader 2 A brilliant point load… |
Pages: 1 2