Artificial intelligence company Anthropic on Thursday suggested a global pause in building the most powerful artificial intelligence systems, as the latest models begin to show signs they could escape human control.
The San Francisco-based company, which makes the Claude family of AI models, said in a report that a global slowdown in the development of cutting-edge AI “would probably be a good thing,” but warned that if just one company stopped, rivals would simply pull ahead.
“We think it would be good for the world to have the option to temporarily slow down or pause AI development to allow social structures and alignment research to keep up with the advancement of technology,” he said.
Achieving a true pause in work would mean that several major AI companies in several countries, most notably the United States and China, would agree to pause at the same time, under rules that everyone could verify, Anthropic said.
“Without a global coordination mechanism, companies and governments will have to make difficult decisions about security under competitive and geopolitical pressures,” he said.
The company has faced opposition from others in the industry – and White House officials – who say its focus on worst-case scenarios overstates the risks and amounts to a strategy to slow down rivals under the guise of security concerns.
Still, the White House has recognized the power of the company’s Mythos model, which has not been made available to the general public due to its cybersecurity capabilities and is currently deployed in only a small number of vetted organizations.
The proposal would face an uphill battle in Washington and Silicon Valley, where US officials and technology executives have repeatedly argued that any slowdown in AI development risks giving China a decisive strategic advantage in what many see as the defining technology race of the century.
US President Donald Trump, however, said he discussed the possibility of cooperating with China on AI safety issues during his recent visit to Beijing.
Trump also signed an executive order this week giving the government 30 days to conduct a preliminary review of the most powerful US artificial intelligence models before they are released.
“Reduction of the human role”
Anthropic compared the issue to nuclear arms control treaties, but said it would be even more difficult to manage, since AI training is much easier to hide than a missile silo, and the temptation to continue quietly would be great.
The company said it plans to bring together government officials, scientists, advocacy groups and competing artificial intelligence companies in the coming months to figure out how such a system could work.
The call for coordination comes with internal data showing that AI is already dramatically accelerating the development of AI itself, Anthropic said.
That acceleration creates a feedback loop that Anthropic warned could eventually lead to what researchers call “recursive self-improvement.”
That’s the idea of an artificial intelligence system that becomes capable of teaching itself to be smarter, without much human help.
“We are not there yet, and recursive self-improvement is not inevitable,” the report says, adding that it could come sooner than most governments and institutions are prepared for.
“Evidence suggests that the human role is being reduced at every step of the AI development process,” the company said.
