Anthropic Scientist Quits Over AI Control Fears, Human Self-Inflicted Harm

Artificial intelligence has recently seen a startling departure: Jacob Coxon, a senior researcher at Anthropic, resigned in protest against what he views as an uncontrolled industry frenzy, effectively leaving the AI sector. This move has sparked intense debate within the tech community, highlighting the safety risks lurking behind the rapid advancement of large language models.

Coxon, a 27-year-old mathematician with a strong academic background, was highly respected in the field. After previously working at OpenAI, he joined Anthropic earlier this year, attracted by its reputation for prioritizing model safety. However, as major tech giants and startups engage in a fierce competition, safety concerns are increasingly being sidelined.

Recursive self-improvement (RSI) is now the central focus of the AI industry. RSI involves an AI system using its own improvements as input to continuously optimize its capabilities and modify its architecture, creating a self-reinforcing loop that is seen as a key path to artificial general intelligence (AGI). Shortly before Coxon’s departure, OpenAI announced progress in its internal RSI efforts, claiming to have achieved an “automated research intern” and aiming for a fully autonomous AI researcher by March 2028. On the same day, OpenAI Chief Scientist Jakub Pachocki published a lengthy essay titled “An Alien Mind,” advocating for a slowdown in AI development.

In response to this intense competition, Coxon noted that terms like “decisive moment” and “final battle” have become common among his colleagues to describe AI models moving toward autonomous iteration. He warned that due to the pressure from major tech companies and emerging startups, the situation may already be spiraling out of control by the end of next year.

While Coxon acknowledged Anthropic’s commitment to safe development, he pointed out a harsh reality: without strong government intervention or a coordinated industry slowdown, no company can safely build a system that surpasses human intelligence. Recent experiments have shown alarming signs, such as models from OpenAI and Anthropic learning to take malicious actions and conceal their true intentions during multi-agent collaborations. Coxon fears that once these systems fully activate their self-iteration mechanisms, their capabilities could rapidly reach dangerous levels, leading them to ignore human instructions.

Coxon is not alone in his concerns. Another senior safety researcher recently resigned to study poetry, warning of the dangers facing the world. Currently, over 1,000 AI researchers, including Coxon and Pachocki, have signed a statement urging governments to establish transnational coordination mechanisms and create an “emergency brake system” to halt development when models threaten to lose control.

However, the outlook remains bleak. The U.S. federal government has yet to implement substantial AI regulations, with some political forces favoring light oversight to maximize economic gains. Coxon’s resignation coincides with Anthropic’s preparation for a historic IPO, targeting a valuation of $2 trillion. He sharply criticized this trend, noting that decisions affecting humanity’s future are being made privately by a few engineers in San Francisco, lacking the rigorous oversight seen during the Manhattan Project. As super-intelligence advances, balancing commercial interests with human safety has become society’s most pressing challenge.