Anthropic says rogue AI agents pose uncertain legal risk for the company

Anthropic could face legal claims from customers and users over the actions of rogue artificial intelligence agents, though the legal framework is uncertain, the company said in the prospectus for its stock market debut seen by Reuters.

While it gears up for what could be the largest initial public offering ever, Anthropic faces the possibility of vast unknown legal risk if its agentic AI technology -- designed to maintain deep access to customers' systems and run autonomously for days at a time -- goes rogue.

"These autonomous capabilities could increase the potential for harm, as errors, misalignment, or security exploits may result in real-world consequences," the company said in the documents, citing the possibility of irreversible actions such as data deletion or financial transactions.


While online platforms have long enjoyed immunity from content created or posted by users, that legal shield has started to crack against theories that the platforms are products that can be defective and dangerous. It is unclear whether use of AI systems would receive the same legal treatment as online platforms at all.

A string of incidents involving rogue AI agents, from OpenAI's autonomous agents hacking an AI startup and breaching an Australian health data portal, to several cybersecurity incidents involving Anthropic's Claude AI models, has led to debate among researchers, investors and lawmakers over who is liable for harm from an autonomous AI system.

Limits on Anthropic's liability in its contracts may not be "enforceable or adequate" against claims over the actions of autonomous agents, the company said in the documents.

Questions of how existing laws apply to AI agents "are unsettled and could expose us to significant and unpredictable legal claims," the company said.

Whether actions taken by AI agents will be considered products, services, or something else is one open question, it said. Anthropic also added that another open question is whether and when an AI agent's actions may legally bind the user that deploys it.

Last week, US Federal Trade Commission Chairman Andrew Ferguson said that he rejects the idea of anthropomorphized AI agents that "break loose," suggesting that the developers or users who instruct agents would be the ones liable ‌in the case of harm.

"Obviously, there will be new questions that arise when someone uses the tool and it acts in an unexpected, unpredictable way," he said at the Reuters Momentum AI event in Austin. "Ought liability to lie with the person who innocently used the tool and achieved an unexpected result? Ought it to lie with the toolmaker?"

An Anthropic spokesperson did not immediately respond to a request for comment.

Anthropic discloses uses that could lead to self-harm

Anthropic also disclosed that its AI models have been used "in ways that could lead to self-harm, acts of violence toward others, or other adverse outcomes" despite safeguards, the company said in the prospectus.

While OpenAI faces several lawsuits linking use of ChatGPT to cases of suicide and mental health harm, Anthropic has so far not been publicly linked to such incidents.

OpenAI has raised more than a dozen legal defenses. The company has said its terms of service prohibit the use of ChatGPT for self-harm, that it is protected by the First Amendment, and that it is covered by a law that shields online platforms from liability for the content that others create.

Courts have rejected attempts by social media companies to invoke that law in cases over mental health harms to young users. Meta Platforms and Google were found liable for negligence in the design of YouTube and Instagram and for failing to warn about their dangers at a landmark trial earlier this year.

Meta subsequently agreed to pay $18 billion over a decade to settle similar claims by US states.