More at:
Europe's large web hosting provider knocked offline following fire
Several websites and online services have been knocked offline following OVH issue
www.techradar.com
Upgrade if old
-
Ever wanted an RSS feed of all your favorite gaming news sites? Go check out our new Gaming Headlines feed! Read more about it here.
-
We have made minor adjustments to how the search bar works on ResetEra. You can read about the changes here.
OVH, one of the world's largest Cloud providers, suffers major fire in Europe; one data center completely destroyed
- Thread starter DiipuSurotu
- Start date
More at:
www.techradar.com
Upgrade if old
That's kind of embarrassing tbh. Protection against fire is one thing that should have very high priority for data centers, yet they lost a whole building? What happened, did a plane drop on the building? At least nobody got hurt it seems.
The game Rust apparently lost a lot of data due to this:
www.eurogamer.net
I always assumed being in the Cloud also meant having the type of failover systems that would seamlessly spin up new servers somewhere else.
Rust developer warns "large amount of data" lost in devastating server blaze
UPDATE 1.30pm UK: Rust developer Facepunch has released a fresh update on the game's EU servers following last night's …
www.eurogamer.net
I always assumed being in the Cloud also meant having the type of failover systems that would seamlessly spin up new servers somewhere else.
Agreed, I gave been in multiple tier 3 data centers. The fire suppression systems are crazy.
I don't understand how something so large was able to happen.
Just because the building was destroyed doesn't mean there was no suppression system in place. Sometimes fires just rage hard and there's definitely been many corporate or industrial buildings destroyed by fires.
it sounds like multiple clusters were destroyed at once. If that includes backups, then yeah it's gone.The game Rust apparently lost a lot of data due to this:
Rust developer warns "large amount of data" lost in devastating server blaze
UPDATE 1.30pm UK: Rust developer Facepunch has released a fresh update on the game's EU servers following last night's …
I always assumed being in the Cloud also meant having the type of failover systems that would seamlessly spin up new servers somewhere else.
Fire destroys OVHCloud's SBG2 data center in Strasbourg
SBG1 also badly damaged, SBG3 and SBG4 are safe but won't restart today
www.datacenterdynamics.com
More details in here
Yeah, a lot of questions there. Data centres usually have world class grid based fire suppression - so for it to get this out of hand, either external influence or a lot of the BMS failed
You usually have to pay extra for those services, and design them into your setup.
I remember working for a company, where a DC we used lost power for nearly a day, despite all the redundancy.
The CTO/Founder is providing updates on twitter:
Always figured that most cloud providers had enough redundancy so that if one site was lost, the data would still be available elsewhere.
edit:
edit:
Oh yeah, I should probably have known that.
There are also different levels of redundancy, each set up differently and with different levels of cost. Some providers allow you to simply make your data redundant within the DC to protect against a server failing or a drive going kaput. You can also make your data redundant redundant at an availability zone level (so within the same region but at another DC at least x miles away) or you can do something like set up regional redundancy.
These services always cost money - you're basically paying the cloud provider to provide the infrastructure and technicians on your behalf, but you pay them instead of managing the process and costs yourself.
And if you set up redundancy, it doesn't mean the data is always instantaneously copied to all locations. A lot do it on a cadence, so in the event of a flood or fire, it's still likely you lose *some* data if you're collecting it real time.
Sounds like the whole building was in a catastrophic state by the time the first fire response units arrived. Someone used the wrong materials if that is the case. Reminds me of that story in the UK of the Apartment fire going up the entire building in minutes due to the exterior building materials.
They announced their IPO like two days ago, worst possible timing for them.
Add me to the pile of surprised people because every data center I've seen always had a ton of fire protections and warning systems.
I always assumed geographical redundancy was built in all their hosting services, apparently it's not.
Anyway, I assume this is gonna make a lot of businesses very wary when choosing cloud hosting providers, considering this one is well regarded.
Add me to the pile of surprised people because every data center I've seen always had a ton of fire protections and warning systems.
I always assumed geographical redundancy was built in all their hosting services, apparently it's not.
Anyway, I assume this is gonna make a lot of businesses very wary when choosing cloud hosting providers, considering this one is well regarded.
OP
OP
This would be huge. You don't just walk into a data center if you don't work there.
So I'm guessing this cloud provider only has one location? Otherwise I don't understand how they could lose all this data. Was there no data replication setup at all at a different location?The game Rust apparently lost a lot of data due to this:
Rust developer warns "large amount of data" lost in devastating server blaze
UPDATE 1.30pm UK: Rust developer Facepunch has released a fresh update on the game's EU servers following last night's …
I always assumed being in the Cloud also meant having the type of failover systems that would seamlessly spin up new servers somewhere else.
Came for thissecurity camera caught the culprit apparently :
A cloud provider just means you have the option of (usually worldwide) replication. Its on the customers to opt for it, but it usually doubles/triples/etc costs depending on how "cloudy" you want it.
Lots of companies treat cloud providers as just colocation hosts.
True. It is an option for the client to opt in or out of... I just find it bizarre that a fully online game didn't setup any data replication.
But this isn't like a normal office or skyscraper. The systems and precautions that are/should be in place for a data center, especially one of this magnitude, are crazy. We're talking all the way down to the cabling itself being jacketed in fire retardant materials, redundant independent cooling systems, crazy inert gas systems that not only displace oxygen but literally absorb heat, etc. Data centers do not fuck around when it comes to fire, so for them to not only lose a whole building but also have damage to others is shocking to me.
They either skimped on protection, had some major defect in something critical like power line insulation, had multiple redundant system failures, or there was foul play.
"Cloud" just means you aren't personally/physically managing the servers and that you're sharing server HW with other people, "virtually". It doesn't necessarily mean all the physical locations of those servers are geographically spread out, although they usually areThe game Rust apparently lost a lot of data due to this:
Rust developer warns "large amount of data" lost in devastating server blaze
UPDATE 1.30pm UK: Rust developer Facepunch has released a fresh update on the game's EU servers following last night's …
I always assumed being in the Cloud also meant having the type of failover systems that would seamlessly spin up new servers somewhere else.
I just started cloud development this past year and this was one of the biggest misconceptions I had about it going in. I always assumed basic redundancy was a given but you really do need to architect your product around it in order to get those benefits and have it be "seamless".
Rust is big but I don't think it's THAT big. Plus what are the odds almost an entire data center catches fire?
Was doing some reading on HN and I totally miss understood the levels they use. I was assumed your basic ass fire system but learned of the oxygen one you mentioned.
Always assumed every cloud storage has an equal backup elsewhere in a different country. One backup, sure, but equal. Now I'm second guessing moving everything to Google Drive...
Thing is, a datacenter is just a datacenter. It's up to the customers using their services to have backups and redundancy.
A game like Rust? Maybe just in the one building. Maybe they did have backups,but it was on a server in another room.
Someone like google? They have their stuff redundant across the globe
Yeah, data centers have really specific needs as a normal fire suppression system would cause as much or more damage as the fire would. You need retardants that disperse cleanly with no residue, as they are going to get in and on absolutely everything. There's also strict protocols on foreign materials in a data center (anything fibrous and flammable is a no go, including packaging materials for gear), requirements for detectors in the room itself along with ones above drop ceilings and below raised floors, etc. Of course this varies depending on the place, the scale, and so on, but I would have imagined a company like OVH would be erring on the paranoid side of things given the size of their operation.
I wonder if this will make our organisation's Directors more keen or less keen to throw all our on-prem stuff at the cloud. Probably both.
dude I was right there with you. It's a steep learning curve and there are no true turn-key solutions. Data centres give you all the options you could dream of, but you need to design everything around it.
Its hard work, but accepting nothing is a given makes you a much better systems designer.
Google Drive is a consumer end-user product, not a cloud hosting provider. Not saying you're 100% guaranteed to avoid problems like this, but its not the same as B2B services.
Depends on the offering. If you host an application in the cloud, the provider will give you options on redundancy (or a lack thereof).The game Rust apparently lost a lot of data due to this:
Rust developer warns "large amount of data" lost in devastating server blaze
UPDATE 1.30pm UK: Rust developer Facepunch has released a fresh update on the game's EU servers following last night's …
I always assumed being in the Cloud also meant having the type of failover systems that would seamlessly spin up new servers somewhere else.
The game Rust apparently lost a lot of data due to this:
Rust developer warns "large amount of data" lost in devastating server blaze
UPDATE 1.30pm UK: Rust developer Facepunch has released a fresh update on the game's EU servers following last night's …
I always assumed being in the Cloud also meant having the type of failover systems that would seamlessly spin up new servers somewhere else.
Yeah, a data storage company with no offsite backups is highly sus.
Again, this is likely something they offer but Rust did not utilize.
