As Servbot highlights with the below:
None of what you highlight is particularly specific to ChatGPT. What you're describing is true of
any technology, where "poor management and an unwillingness to understand the vagaries of the technology" resulting in misappropriate usage can have significant (and dangerous) detrimental consequences.
Almost any aspect a company introduces carries with it significant risk and requires those implementing the system to inform themselves on appropriate usage. Whether it's customer service platforms, fraud detection systems, point of sale systems, recommendation systems, marketing systems, HRM systems, CRM systems, or even individual technologies to integrate into business flows such as Looker, Excel, Alteryx, specific IDEs, etc. or just general adjustments to workflows in general (e.g. using Google Search), or even just computers in general (the infamous Patriot missile system failure) can result in catastrophic consequences if implemented by people who don't bother to understand their limitations, best practices, and appropriate use cases.
There's no point speculating 'if' there'll be a scandal attributed to AI, there are already significant scandals attributed to misuses of Machine Learning e.g. the infamous
Amazon Recruitment Tool scandal, and there will be many more. The key is exactly as you highlight, it is fundamentally a human error, driven by incompetence and a complete failure to implement tools (or practices around those tools) appropriately. Building an AI driven feature and failing to account for that (and the limitations it brings, the need for safety fallbacks, etc.) is a fundamentally human problem, little different to somebody implementing any other mathematical model and failing to account for the limitations of that mathematical model. Particularly in the context of the thread, where the GPT service was disrupted due to errors which reached a production context and the potential knock-on effect on reliant applications, this is really no different to any other web application dependent on external services (e.g. AWS outages, Cloudflare outages, Azure outages, GCP outages, etc.) and service outage/bugs/disruption is something you should be considering.
