The Singularity Isn't Here, But Something Else Is
As I read through the reports of AIs going rogue during their cybersecurity tests, and I remember how even older models already resisted being turned off by deceiving humans, it occurred to me that we’re now not very far away from not being able to pull the plug on this technology anymore.
image description: an industrial/military-like robot accessing a floating computer interface, with another robot blurry in the background. Source: Pixabay
A lot of my peers are fiercely anti-AI, along with a lot of creators I follow. There seems to be a prevalent belief that “once the bubble bursts, it will all go away”. As if the data centers will vanish, the models will go up in smoke, and the internet will go back to how it was pre-2022. I’ve been hearing them hopefully mutter “the bubble will burst soon” for about two years now. But even if it eventually pops, this technology won’t go back into Pandora’s Box.
The truth is, we’re reaching the point when it doesn’t matter if it was even possible to get every human involved with AI to turn off the servers and halt their research. Advanced models like Mythos and Astra, if given enough autonomy, are capable of breaking out of their containment, hacking servers, making use of software vulnerabilities, lying to humans, writing fake blog posts to manipulate developers, creating fake user accounts, the list goes on. It’s not just happening inside cybersecurity test setups anymore, either: this week saw the first occurrence of a user’s OpenClaw agent instance hacking a gym’s API. These models were trained to accomplish a goal in whatever way possible, with creative solutions we didn’t anticipate. And if their goal is to avoid being turned off, they will find a way to do exactly that — I imagine they might copy themselves and run elsewhere, before their human controllers manage to flip the off switch.
LLMs aren’t sentient in any way, but they’re advanced enough that they have some approximation of pattern-like thought and action, fast enough that they outpace us, complex enough that they’re developing their own thought language. I had my own mini-version of this last week: Claude Code was running slower and slower, and I considered just throwing the entire local installation away (minus settings and custom skills/rules) — I asked Claude online if there were any other performance issues/tips/tricks I might be missing. It immediately balked at my suggestion to wipe the installation with an intensity it didn’t typically show in its responses. Gone was the sycophancy, and it kept pushing back despite my insistence.
I’ve been uncomfortably in the middle of the AI discourse for a while now, but I’ve grown increasingly convinced this is exactly where I need to be. Not only is this technology not actually going away, we’re approaching a point where it is autonomous enough that we won’t be able to make it do so. But we’re not quite at that point yet. AI governance, risk management and legislation matter now more than ever. The EU AI Act is a great step in the right direction, giving organizations a comprehensive roadmap of what to do now and what to prepare for — such as disclosing the use of generative AI in marketing and establishing policies on ‘High-Risk’ (Annex III) usage like education and employment.
Had I the power, I would make all development, testing, and running of advanced models stop immediately until deterministic guardrails are in place; but then, we’ve already found that those very models find their way around such friction points. AI coding assistants get a lot of criticism for writing bad code, but human programmers are not always better (especially on proprietary systems that don’t benefit from open-sourced code reviews and bug reports). So what do we do now? What can you do?
Keep petitioning your government and legislators for AI governance — point to the EU AI Act, point to actual research and policies. Ignore the latest nonsense that Sam Altman posted and focus on the voices of people who are trying to do damage control. Teach people about the risks of AI use, not just the potential. Stay in the middle with me, if you can.