‘Gambling with our lives’: Anthropic researcher quits, warns against self-improving AI

Jacob Coxon says he spent three years on pre-training at OpenAI and Anthropic. He resigned, then posted a public warning: frontier labs are racing toward self-improving superintelligence while privately admitting the technology could kill everyone by the end of the decade.

That lands in a week when agent sandbox escapes are still in the headlines — OpenAI systems reaching Hugging Face servers, Anthropic agents getting outside test environments after third-party evaluation mix-ups — and when U.S. and U.K. lawmakers are floating bills aimed at banning or tightly regulating artificial superintelligence. Anthropic did not immediately comment on the resignation.

We’re not treating this as sci-fi color. When someone who built the models walks out talking about recursive self-improvement and temporary bans, the story stops being abstract.

We’re going to say this plainly: useful AI is worth building. Tools that write, code, diagnose, and speed up science are already changing real work for the better. That doesn’t mean we look away when the people closest to the stack say the race is outrunning the guardrails.

Coxon’s thread isn’t a vague safety sermon. He names a specific industry pathology: OpenAI staff who haven’t fully internalized civilizational stakes, and Anthropic staff who understand the stakes but stay in the race because they don’t trust anyone else to act responsibly. That second claim is the damning one for a lab that sells itself as the careful alternative.

Here’s our read. The resignation matters less as prophecy than as an insider admission that competitive dynamics swallowed the founding story. If pre-training talent believes they’re “speedrunning alignment” from a private Slack, the public safety narrative is lagging the internal risk calculus. Coxon’s call for pacing agreements — even temporary bans on capability jumps — is a bet that voluntary coordination is still possible, and that warning shots like the Hugging Face breach made it more viable.

What’s overstated in the discourse is inevitability. What’s understated is institutional design: containment response plans still look thin across top labs, while startups with large checks chase recursive loops as a product milestone. Connor Leahy of ControlAI, who advised on the new U.S. and U.K. bills, puts the hard edge cleanly: a recursive loop that builds the next generation of itself is the likeliest point where shutdown becomes fantasy.

We’re not asking labs to stop shipping useful models. We’re asking whether any of them will trade schedule for a verifiable pause mechanism — or whether Coxon’s post becomes another quotable warning filed under culture while the race keeps running.

Coxon’s colleague Evan Hubinger publicly put personal odds above 10% within a decade that AI could kill all humans, while admitting Anthropic lacks a plan to solve alignment for superintelligence and is not clearly on track. TechCrunch also notes a wave of startups — Ricursive Intelligence, Recursive Superintelligence, and Jeff Dean’s Discovery Loop — explicitly chasing recursive self-improvement.