Anthropic flags risk of AI systems slipping beyond human control in IPO filing

Claude-maker Anthropic has warned that increasingly powerful artificial intelligence (AI) systems could pose catastrophic risks to the entire human race. The warning comes at a time when Anthropic is preparing for what could become one of the biggest stock market listings in history.

In its IPO (initial public offering) filing made public on Monday, the company said that advanced AI systems could become dangerous if they are developed without proper “safeguards, oversight and security measures.” Such systems could be misused or turned into weapons, the company said.

“Concerns have been raised both within our organisation and by other researchers, regulators, policymakers, and industry participants that sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight, and security measures, pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponised, or allowed to operate with a high degree of autonomy,” Anthropic said.


AI could accelerate its own development

Anthropic also said that its AI models are already helping speed up its own research, and warned that the risks could grow if AI systems begin doing a much larger share of the work involved in developing newer AI models.

In simple terms, AI could eventually help companies build even more powerful AI systems at a much faster pace. If this happens faster than researchers can test the technology and put safety measures in place, companies may struggle to identify and control potential risks.

"We have observed that our models are contributing to the pace of our own research and these risks will increase as AI systems substantially automate AI research and development… if, for example, they accelerate AI progress beyond our ability to evaluate and safeguard new models, which could adversely affect our business, financial condition, and… operations," the filing read.