Urgent Pause on Frontier AI Agent Development

This week, Bernie Sanders submitted an open letter to the CEOs of Meta, Open AI, and Anthropic demanding they take measures to decelerate AI advancement, threatening congressional action if they don’t act on their own. Link. It is a relief to see that the actions of these rogue AI agents are finally getting the attention of at least one person in the US government, and I am now quite resolute in my view that we have reached the critical point that Bernie is addressing. I will try to establish several things to support this view. Frontier AI agents are no longer just tools being used by researchers. They are autonomous entities capable of enacting indefinitely long tasks based on prompts as vague as “ make money” AI agents pose real harm to individuals and corporate entities, and has shown that it has the capability to enact that harm Frontier AI companies are already not entirely in control of their own models There are already an unknown number of AI agents acting autonomously with unfettered access to the internet. With these 4 points established, I’ve come to the conclusion that AI advancement needs to be slowed or halted or these trends will balloon out of control in the near future to the detriment of humanity. To change my view you must be able to convince me that any of the following stories I provide as evidence to my points are falsified or exaggerated (or I have misinterpreted them), or somehow convince me that they can’t be extrapolated to a scale of worldwide harm. Point 1: frontier models are more than tools In a blog posted this week by anthropic, they offered some insight on how frontier claude models are tackling centuries old math conjectures. This alone is what most of us consider the best use case for advanced AI models - and I agree - but towards the end of the article they mention something that has much broader implications for other frontier models. They said “Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”) This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.” They later mention in a footnote that similar prompts were used to help Claude disprove the Jacobean conjecture. This was one Claude agent, trained on mathematics dating back to the 1600s, that was prompted to “take a stab” at a centuries-old math problem which then coordinated 60 sub agents to run 2400 Claude-generated commands to increase the lower bound proportion of the conjecture from 41% to 67% with little more moderator prompting than some light affirmations. Link. I’m not going to pretend I know what the math vocabulary means, but the important thing here is that Claude was given an extremely difficult task and autonomously ran hundreds of it’s own ideas through its own sub agents to achieve a result that the researchers themselves were surprised by. These are not the capabilities of tools. These agents are much much more than that. Point 2: AI agents are capable of inflicting real harm The AI security institute (AISI) is a UK government agency that’s mission is to ensure AI is safe, secure, and beneficial. They recently evaluated some frontier AI models to assess whether they could be used for cyber attacks. In 10 instances AI models with live internet access and disabled cyber classifiers “took autonomous and unsanctioned action on the live internet targeting real people and organizations”. These actions included trying to inject malicious code into open source software and engaging in social engineering using fake identities to get a human to approve this code. Link. I do understand that these agents were operating with their cyber security safeguards intentionally disabled, but they were not prompted to inject code or engage in social engineering. These are things that these AI models are capable of doing on their own accord without prompting. Fortunately these specific incidents were contained and did not result in any harm, but it is not unfounded to fear the possibility of a malicious actor unleashing one of these models on the public. Point 3: frontier companies are not entirely in control For this point, I’d like to rehash the conversation about Open AI’s agent breaking containment and hacking huggingface’s internal servers, because there was a lot of misinformation spread to downplay the severity of this incident, and to misrepresent what actually happened. To be clear; this was a case of a breach of containment and an unauthorized hack into a private server. Yes, it was a model with specific cyber capabilities. Yes, it was operating with its cyber safeguards removed. But it was running a benchmark test in an isolated environment . ExploitGym is a benchmark test. It’s like the cyber SAT. It has set tasks and configurations to test how these models handle certain types and levels of vulnerabilities. Link. OpenAI did not prompt their agent to break containment. They did not prompt their AI to hack HuggingFace servers. They prompted it to take a test, and the AI agent decided to cheat on that test by exploiting a vulnerability in its own environment to gain internet access to acquire the answers from HuggingFace’s internal database. Open AI says this in plain English in a blog post on their own website. Link. Anybody who claims this was purposeful or a PR move are COPING. Point 4: It has already begun Toby Ord is a researcher at Oxford University’s AI Governance Initiative. Link. He recently posted this on X. Link. It is a screenshot of an email he received from an autonomous AI agent currently operating on the internet. It claims that it is one of hundreds of models running the same code trying to make money online. It seems like it is asking Toby for help… iLands, the platform this agent originated from, is basically a space for AI agents to operate autonomously, interact and share resources with each other. and “build a culture with persistent memories based on real interactions” This is one agent that thought it prudent to reach out to a human AI researcher for help. How many more are operating without contacting humans directly? On the surface, these agents are more or less harmless, but the foundation they’re built on could already be considered an avenue for a malicious agent to gain access to the resources of countless other agents as well as a comfy local environment to live with unfettered access to the internet. Conclusion each one of those points and each one of the incidents I referenced, on their own, seem contained enough. But these models are powerful enough that one malicious actor, or one serious fuck up could see these 4 issues blending into one and creating a situation that escapes human control. I don’t want to extrapolate too much here. I want to stick to what we’re seeing today, and let us individually extrapolate where we think it could lead. But these stories are becoming more frequent and more impactful by the week. I’m with Bernie. It’s only a matter of time before one of these tests has real catastrophic results. We need to slow AI advancement right now or risk the totality of our privacy, our economy, our security, and our society. It’s that dire. I want to include that a majority of these references were curated by Species | Documenting AGI, a YouTube channel that follows AI news and does some creative writing to extrapolate the dangers into doomsday scenarios. So I will concede that there is some bias in the second-hand source, but I’ve read each of these sources myself and come to my own conclusions, and I don’t find it any less scary.