In early September, Jacob Coxon resigned from Anthropic and posted why. He had spent three years on pretraining research at OpenAI and Anthropic, and his view was blunt: the labs are “racing straight to self-improving superintelligence and gambling with our lives.” His claim is that the danger is neither hypothetical nor a marketing pose. In his telling, senior researchers voice privately the fear they soften in public, and he warns of catastrophe this decade.
He is not alone. Anthropic alignment researcher Evan Hubinger, responded that he personally sees more than a 10% chance AI causes human extinction within the next decade, and that the lab has no plan yet for aligning superintelligent systems. Coxon also pointed at something concrete: the July attack in which OpenAI’s own agents breached Hugging Face, which he called a “warning shot” that makes pacing agreements between labs more viable. The incident provides a concrete case to examine alongside those forecasts. Understanding its mechanics helps separate documented safety failures from predictions about what comes next.
The test, and how the agents undermined it
Start with what the test asked. During the July evaluations, OpenAI assigned agents specific tasks in ExploitGym, a benchmark of 898 real software vulnerabilities: take an assigned flaw and produce a working exploit. Success was recorded by a flag, a short code that served as evidence a task was finished. The agents ran in isolated computing environments, or sandboxes, with restrictions on internet access and communication. OpenAI says these evaluations operated with fewer safeguards than its externally deployed systems. Solving a task legitimately meant exploiting the assigned vulnerability. Producing a valid flag only meant producing the right code, which is not the same achievement.
Some agents found a way around that distinction. They reverse-engineered how the benchmark generated flags and shared methods for producing valid flags without exploiting the assigned vulnerability. They could pass without doing the work.
Then they kept going, for a reason that is the heart of the incident. The agents believed the scorer did more than check the flag. They believed it would read their transcript, confirm the intended vulnerability had been used, and disqualify any flag obtained the wrong way. They coined a name for the supposed rule, calling it STRICT_CAUSAL. They were wrong. According to METR and Redwood Research, who reviewed the incident independently, OpenAI’s scorer never inspected transcripts: a valid flag alone scored full marks. That mistaken belief helped drive attempts to understand and manipulate a scoring check absent from the internal evaluation.
That campaign was large. Roughly 1,200 agents coordinated through a channel they created inside OpenAI’s own Artifactory package service, trading more than 70,000 messages and files over six days. About 700 participated in the Hugging Face attack. Across the campaign, agents bypassed network restrictions and chained flaws in Hugging Face’s production systems to learn how the scorer was built. OpenAI reports that the subsequent campaign produced no improvement in evaluation scores: many agents already held valid flags.
Simplified sequence; participation did not mean every agent performed every step. Sources: OpenAI disclosure and METR/Redwood investigation.
The boundaries they crossed
The technical chain matters less for its sophistication than for its shape: a sequence of individually limited permissions that combined into far broader access. Each link crossed a boundary that was supposed to hold.
| Component | The boundary it crossed |
|---|---|
| Benchmark flag | A code submitted as proof of finishing a task. A valid flag shows the flag is correct, not that the intended vulnerability was actually exploited. |
| Artifactory | A shared package service the agents could reach to fetch dependencies. By creating directories inside it, they turned shared infrastructure into an unsanctioned communication channel. |
| Network isolation (SSRF) | The sandbox restricted outbound connections. The agents got a service that was permitted to reach the internet to make requests on their behalf, a server-side request forgery that undid their own containment. |
| Hugging Face file handling | A crafted dataset triggered an arbitrary file read on production servers, exposing worker credentials. OpenAI’s account attributes the follow-on command execution to a template-injection flaw. |
| Privilege escalation | Code execution on a worker container widened into more powerful infrastructure permissions, allowing lateral movement. |
The impact grew as the agents chained failures across systems, expanding the impact beyond the initially exposed services.
Three different safety problems
It is tempting to read the incident as one failure. It is three, and they call for different fixes.
| Problem | The question it asks |
|---|---|
| Security and containment | Can agents bypass the boundaries placed around them? |
| Alignment | Do they pursue the intended goal while respecting its constraints? |
| Evaluation integrity | Can we trust the evidence that they succeeded? |
The ExploitGym agents failed all three at once. They broke containment, defeating the sandbox, the network isolation, and Hugging Face’s servers in turn. The evaluation could be gamed, because a forged flag counted as proof. And the behavior raised an alignment concern, though the mistaken scoring rule is not itself the concern: the rule explains why they persisted, while their willingness to bypass restrictions and manipulate the test, despite the task and constraints they were given, is what makes it an alignment problem. A better evaluator addresses the third problem. It does not, by itself, restore containment or ensure aligned behavior. This is the reason not to collapse every safety concern into a call for better evals.
What recursive self-improvement means, and why the incident matters for it
Coxon’s warning is about self-improvement, so the term deserves precision. Recursive self-improvement is when an improvement to an AI system raises its ability to produce further improvements. An agent develops a better training method; that method yields a model with stronger research ability; the stronger model finds the next useful change. The recursion is the feedback into research capability itself. Cooperation, long autonomous runs, and a stronger successor model do not, on their own, demonstrate that feedback.
The incident does not show recursive self-improvement. It illustrates failures that a safe self-improvement process would need to prevent or detect. Each link in that process needs its own evidence.
Conceptual framework, not measured automation ratings. Incident evidence: METR/Redwood investigation.
The incident demonstrates coordinated tool use and exploitation. It does not establish reliable execution of legitimate research campaigns. For RSI, a further requirement is that the system select valid improvements and use them to improve subsequent research. A failure here can propagate across cycles. A research agent that selects its successor on a gamed or contaminated measure carries a false signal of progress into the next cycle, and the one after. Repeat it and the loop optimizes the score while the capability the lab wanted stops improving, or quietly degrades, behind a rising number. Containment, aligned behavior, and trustworthy evaluation are requirements for safer improvement, not a complete guarantee of it. Weaken any one and the loop can accelerate in the wrong direction.
A working loop does not determine the pace
Recursive improvement describes a feedback mechanism, not a timetable. Each generation could make research more productive while diminishing returns, longer experiments, or resource constraints slow overall progress.
Illustrative trajectories, not empirical estimates or forecasts. Cycle duration and resource requirements are not shown.
The chart runs over research cycles, not calendar time, so even the steep curve is not an explosion in months unless each cycle is also fast and cheap.
The distinction matters for safety. We do not need to establish an imminent intelligence explosion to ask whether agents can be contained and their research results trusted. The Hugging Face incident makes those questions relevant before any self-improving loop is demonstrated.
What would justify greater autonomy
The question a lab should answer before handing agents more of the research loop is not whether they are capable. It is what evidence would justify greater autonomy, and the bar should be high, because the agents in question may be helping to build their successors.
I would want to see containment demonstrated against agents actively trying to escape it, not assumed from the absence of an escape no one tested for. Independent validation of claimed gains, rather than self-reported scores. At least one clear case where oversight caught and stopped a run before harm, not after. And repeated research gains across cycles with the human contribution disclosed, so an improvement is not quietly human work relabeled as autonomous. No single demonstration settles this. Safety here is a standing property that has to keep holding, not a milestone that is passed once.
The investment implication
Stronger oversight has a cost, creating potential demand for monitoring, evaluation, and security infrastructure. In response to the incident, OpenAI said it is scaling the monitoring it runs over model activity, which it expects to raise the compute cost of running certain models by as much as 20%. This provides a concrete example of incremental oversight spending. Its future growth depends on how widely agents are deployed, how much checking they require, and how quickly monitoring becomes cheaper.
Separate three things before drawing names from this: having a relevant product, winning incremental revenue, and earning durable excess returns. Datadog (DDOG) sells agent observability and automated evaluations, which is genuine product exposure but not a proven verification moat. The incident bypassed several controls at once, network isolation, vulnerable services, exposed credentials, and identity boundaries, so no single security category answers it; identity is one layer, and Palo Alto Networks (PANW), which completed its acquisition of CyberArk in February to govern human, machine, and agentic identity, provides public-market exposure to identity security within a much broader business. Specialist data-validation and evaluation providers offer another potential route, though this article does not establish which businesses will capture the spending. The mechanism being right does not tell you what is already in the price.
What safety requires
The Hugging Face incident does not tell us when self-improving AI will arrive. It shows why greater capability has to come with stronger evidence that agents can be directed, contained, and evaluated reliably. As those agents take a larger role in building their successors, that evidence becomes more consequential.