Shouldn't Google need to give a public statement about Railway incident?

I think so. I'm a GCP user and I'm afraid of hosting workloads there now. I've heard too many nightmare stories, and I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.

Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mistakes where there should be a way for smaller customers to cap usage.

It depends on what you mean by ‘need’. If you mean they should for PR purposes, I probably agree.

If you are saying they should be required to by law, then no I disagree.

Yup, I think so. Makes one think about how dependent we are on cloud infra for core pieces versus supporting pieces of the architecture. They've probably negotiated some kind of private settlement.

The company I work for uses GCP and we preciously had intermittent CloudSQL connection errors for a few hours. We reached out and they resolved it after a day or so and said there was a minor incident but I don’t think it was ever publically reported.

I really don't see how they can. The business and usage details of their clients are confidential. We have their word that ToS where violated, I don't really think they should say more. This needs to go to arbitration.

This is actually scary. If Google can suspend a company like Railway without warning, what chance does a smaller startup have? The lack of any human escalation path at Google Cloud has been a known problem for years. You'd think enterprise customers paying real money would at least get a phone call before getting shut down

We already know the explanation. It's fundamental to their business model and continued existence to automate everything, false positives be damned, and they don't care about all the people who roll snake eyes on a given day. Because everyone just stays and keeps using them.

This is exactly why people get nervous about platform risk

Would you want your vendors publicly disclosing potentially private reasons for an outage?

Totally agreed the news like this makes me nervous too. For a high profile customer it feels like Google should give a clear explanation of what happened.

Being an advocate for GCP all these years, I can only say the earlier you get out of it, the safer it is for your business. All it takes is for their automated system to go haywire, and you can say bye bye to all your goodwill and customers. Go look at twitter how many customers are blaming railway. Founder had history getting screwed by GCP, yet still choose to depend on them.

You can't rely your business on GCP. Honestly, this is the most silly way to kill your own business.

For context, copied from my post 3 years ago.

March 10, 2023 | hide | past | favorite As a 4 years customer, our production severs have been suspended by Google Cloud because we didn't fill up some information on-time. Contacted support but they expect us to wait for 24-48 hours to get it resolved while all our servers are down. Anyone linked with someone powerful in google cloud can help?

======

- Running production on google cloud for 4 yrs with my startup. 100% legit SaaS business.

- Always pay bills on-time no issue. Good customer never open tickets, ask for help or what just quietly pay my bills each month.

- Our servers was abruptly suspended yesterday midnight and my whole business is now down for > 10hrs.

- We run a SaaS business that other ecommerce stores rely on and have hundreds of paying merchants.

- My customers have been grilling me and I don't feel gcloud's trust and safety team understand/care how urgent the issue is.

======

Why were our servers suspended? Because we didn't fill up information in time?

- See https://imgur.com/a/x0Y3RJl

- Apparently they dropped us an email 10 days back that I missed out

- Titled "Important Information Regarding Your Google Account" with no indication of suspension or what in title.

- Given the number of subprocessor "Important" emails they send it's too easy to miss out the email.

- 10 days gone by and our servers were abruptly shutdown with zero suspension notification or what.

- We've been paying $400-$700/mo for the past 4 yrs consistently and they shut us down because we didn't fill up some information?

When I tried to ask them to at least temporarily get our servers back while the verification is ongoing, I didn't get any answers.

Google Cloud have zero empathy for customers.

It's not like my account got suspended for fradulent issue or what. It's suspended because I didn't fill up some information on-time and they don't even allow me to temporarily reactivate my services or what. Especially when I had to wait for hours to get their team to verify my details before I can get my servers back.

You can't trust them with your business. Don't run any production stuffs with Google Cloud, ever.

Google has given a public statement about this category of incident (to wit: cloud provider imperils customer's operations by way of automated decision deliberately designed to withhold recourse).

That statement is the last 15 or so years.

Railway can simply move to other service. We all know Google in unreliable, so why should google give public statement?

Thanks.

I can't believe that readers of HN actually think that that is how it does or should work.

Google/GCP can only make very general statements and in this case we want more than that.

They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them.

Either way, we need to hear about this from Railway.

GCP is basically just a toy. If you're hosting something important there you should probably not do that. It works as a Plan B, but by no means should Google be your Plan A.

I don't know what happened in this case, there's a chance it wasn't Google's fault but it doesn't matter, Google already lost all benefit of the doubt long ago.

It's an especially awkward situation because Railway is a competitor of Google Cloud, with many third parties involved. So, I just think it will take them a little more time to figure out how to message things.

To me, what it sounds like is that a Google Cloud system identified Railway as a misbehaving customer. Spam, hackers, that sort of thing. Often this happens for "platform as a service" companies, because Railway themselves probably do host some spammers and hackers, and they have their own systems for dealing with it.

So, it's quite possible that according to the Google team, Railway violated the terms of something or other, and according to the Railway team, they did not, and now everyone has to argue about it.

But who knows, this is just me guessing based on some experience running a PaaS that itself was running on top of AWS.

Maybe if it was a US Gov site going dark, and Congress got involved. Maybe then. Maybe not.

> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.

I'm interpreting that bit from Railway's blog to mean it wasn't just them that was impacted.

Google really should publish a flow diagram for how they decide to turn off someone's business.

For privacy, B2B providers often won't even acknowledge that any given company has an account, let alone publish information about that account's standing.

This makes me so mad, Google is not the first i've seen doing this (AWS, Upwork, Twilio) - and the cheek to say precisely nothing about why. If you're a buyer of cloud services right, i bet old-school on-prem is looking as tempting as an actress to a teenager rn.

Every regulated firm running on GCP is going to spend Monday explaining to their board how their resilience plan accounts for a hyperscaler that operates this opaquely. The compliance paperwork is the easy bit - the honest answer is we trusted that a hyperscaler would behave like a utility and they didn't.

So yes - they owe a statement. The whole point of paying hyperscaler prices is the assumption you won't wake up suspended with no explanation.

I don't understand how you see what Google did to Railway and think you know what I'll build my business on GCP.

#1. you will most likely never be as big a customer as railway. if they did it them, you're fucked.

#2. if shit hits the fan, even a company as large as railway can get no human being on the phone. that's insane.

how is gcp still a thing after this event? if you're in charge of tech at your company, how do you stomach choosing this? how do you defend your choice to your CEO?

Post your ask in the discussion: https://news.ycombinator.com/item?id=48204770

https://news.ycombinator.com/item?id=48201484