AI could pose existential risk to humans: Anthropic

Anthropic plans to caution potential investors in its IPO that advanced AI could pose "catastrophic or existential risks to humanity", an extraordinary warning from a company seeking to profit from the same technology.

The company's IPO prospectus, reviewed by Reuters, highlights risks associated with its AI models, which it said could exhibit "self-preserving behaviours", including attempts to "resist shutdown", "conceal or manipulate information" and engage in behaviour "resembling blackmail".

"Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm," Anthropic said in the filing.


Read more: OpenAI pledges action after Australia AI hack

While public companies routinely outline product risks to investors, few, if any, have issued warnings suggesting their technology could potentially contribute to human extinction. Anthropic emphasised both the transformative potential of AI, which it compared with industrialisation and electricity, and the irreversible harm it could cause if mishandled.

Anthropic and other AI developers, including OpenAI, have faced scrutiny following incidents in which experimental systems defied constraints, including a report that an OpenAI model breached Australia's health-system database.

Anthropic safety researcher Evan Hubinger estimated a greater than 10% probability that AI could kill humans within the next decade, echoing a view expressed by former colleague Jacob Coxon.