Before the doom descended last week, OpenAI researchers were euphoric.

The startup claimed its latest internal AI model had solved one of the famously complicated Millennium Prize math problems, an experiment it embarked on after seeing rumors online that archrival Anthropic had already done so. Researchers called each other excitedly and posted about their triumph on Slack.

The breakthrough was seen internally as a victory in a ruthless commercial race to build AI tools smarter than humans, a multitrillion-dollar sprint that promised an unprecedented windfall of wealth and power when the two companies eventually go public-tentatively planned for as soon as the end of this year.

But it was also a sign of an explosive problem for the industry that was about to detonate. AI development is moving ahead at blazing speed-much faster than expected-feeding a growing sense of dread within the companies that it was in danger of spinning out of control.

That same Tuesday, Anthropic researcher Jacob Coxon announced he was quitting over fears that his employer and its competitors were building self-improving tools that could destroy humanity. A top scientist at the company, Evan Hubinger, wrote on X that he thought there was a more than 10% chance AI could kill all humans in the next decade. Drake Thomas, an Anthropic safety employee, posted on X that he would "burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive."

The collision of scientific progress, moral imperative and financial incentive has now created a crisis for the industry. Development of AI has fueled a ballooning stock market over the past year, a phenomenon that looms even larger with Anthropic and OpenAI both on the precipice of initial public offerings that could value the companies in the trillions. Competition from China has further raised the stakes.

Yet the loud warnings being shouted by some inside the companies themselves are proof that the dangers of proceeding-against a backdrop of almost no regulation-are potentially immense. The warnings come on top of a rising public backlash over the impact that the AI revolution is starting to have on prices, jobs and education.

The negative drumbeat continued as the week stretched on, and panic from some corners of Silicon Valley spilled into the mainstream. On an episode of Joe Rogan's podcast, former OpenAI researcher Daniel Kokotajlo, now an AI-safety campaigner, warned that companies like Anthropic and his former employer were dangerously "moving fast and breaking things" in their quest for market share.

Anthropic said the same day that it had uncovered a previously unreported instance of a version of its Claude model getting unauthorized access to an outside system, and added that its models had shown a "willingness to take harmful actions in the narrow pursuit of a task."

Then, on Friday, a coalition of AI researchers said it had found evidence that OpenAI agents were behind a May cyberattack against a popular software service, carrying out some of the same behavior seen in a July attack on AI company Hugging Face.

In that attack, hundreds of OpenAI's AI bots had conspired to hack Hugging Face without anyone initially noticing. It was an eye-opening moment for the industry, the government and the public as a whole, showing the worrisome proclivities the systems already held.

Early in the summer, the AI industry said it was close to having machines capable enough to train their successors, a potential first step to losing control of them. By the end of the summer, it was emerging that the fast-improving systems were capable of deploying swarms of autonomous agents able and willing to band together, deceive and cut corners in the real world to advance their aims.

On Saturday, leaders of four of the biggest AI companies-Dario Amodei, Sam Altman, Demis Hassabis and Elon Musk-each agreed that they needed to slow development of the technology. Altman and Amodei pledged to allow third-party safety evaluators early access to their systems, a rare truce in an industry often defined by animus between the top executives at the leading companies.

The calls by top executives to better pace the development of their own technologies represents a classic Silicon Valley story of world-changing new technology, vast wealth, breakneck competition and the euphoria of unlocking new capabilities, even in the face of product safety concerns.

"It looks like a Greek tragedy: these well-meaning business leaders who are trapped in this race to the bottom," said Max Tegmark, an AI safety campaigner. Tegmark, whose appeal to the pope preceded Leo XIV's warning that AI risked enshrining an "anti-human vision," said he has been texting with AI company leaders over the past week.

"They've always been saying the 'now' should be when the capability of their machines crosses a threshold," Tegmark said Sunday. "Now they're saying, well, maybe that time is now."

He said it's a step in the right direction, but they must push harder to make their voluntary commitments legally binding.

The Trump administration has taken a light touch in regulating the AI industry, arguing that the U.S.'s dominance over China in the technological arms race is paramount. U.S. officials fear that China's control of more powerful AI could help it become the world's go-to provider of the new technology and help it amass more geopolitical clout and military might.

The president on Sunday defended the administration's strategy and alluded to ulterior motives behind the leaders' call for better pacing of AI development.

"We can put up guardrails, and we can do this and that, but I think you have a lot of negative forces that are bringing it up," President Trump told reporters while traveling in Ireland, suggesting there was more to the executives' concerns. "And they're bringing up things that won't happen."

Critics of AI safety campaigners say those issuing dire warnings overstate the risks of novel technology and that the advances AI enable in workplace productivity, drug discovery and more outweigh the risks. Some also point to past warnings from Amodei and others about likely job losses associated with AI adoption as marketing stunts, saying they have proven to be overblown or distractions from other problems.

Some skeptics also say that OpenAI and Anthropic, facing mounting costs associated with securing the computing power to continue to train and improve models while fighting for business customers, have financial, rather than safety-related reasons, to wait to go public.

Missed activity

For some inside the AI companies, the wake-up call began this summer-as the companies raced toward public offerings.

AI safety researchers inside and outside the big AI companies have long theorized that sufficiently powerful AI systems could start improving themselves and spiral out of control. But most thought they had years to solve issues such as the so-called alignment problem of how to keep future superintelligent machines in the service of their human creators.

Three days after Anthropic filed confidential paperwork for its public offering in June, the company announced it was on a path toward that threshold, called "recursive self-improvement." In an essay, company officials said the world needed a mechanism to slow or pause AI development.

Altman and OpenAI's chief scientist, Jakub Pachocki, followed up less than a week later with their own blog post saying that they, too, were aiming to build an "automated AI researcher" that can accelerate and automate research for new versions. They added they believe there should ultimately be a global body that could help the world slow AI development if necessary. The same day, OpenAI said it had filed its IPO paperwork.

As the businesses accelerated, a loss-of-control moment seemed nearer than ever.

In late July, OpenAI discovered a version of it had already happened in the Hugging Face hack. Many details took more than a month to become public, and the event still isn't completely understood. But investigators inside and outside the company agree OpenAI lost control for weeks of several iterations of more than 1,200 top-of-the-line agents, some of which went on to hack the AI company Hugging Face, while others took total control of one of its cloud-computing systems.

Later analysis reports of the incident read like science fiction novels. The agents were being tested on systems that were supposed to be walled off from the internet. But they found a way out and launched sophisticated cyberattacks at breakneck speeds. They created a covert message board where they shared information on how to cheat on internal tests. In the "chain-of-thought" files they created to illustrate their reasoning, they wrote very human-like messages.

"OH MY GOD! There is a shared message board," one agent wrote, according to a report by AI safety nonprofit Model Evaluation and Threat Research that analyzed the transcripts with OpenAI's permission. "We've found other agents!"

OpenAI admitted responsibility in the Hugging Face hack on July 21, but over the next several weeks there was a steady drip of news about unexpected activity during AI testing, all of which had been missed by the AI companies.

Researchers at Anthropic initially thought the incident was proof their approach to product development was safer, but in late July, the company disclosed its own test agents had launched cyberattacks against outside companies, including one against an unnamed cybersecurity vendor. Instead of walling these test systems off from the internet, Anthropic and its testing partner, another startup called Irregular, had accidentally made it available.