Chinese AI's role in stopping rogue OpenAI agent shows cost of US guardrails

A New York startup's use of a Chinese AI model to rein in a rogue agent built ​with OpenAI technology is stoking fears that guradrails ​restricting US AI firms from doing cybersecurity work could drive customers toward their Beijing-based rivals.

The affected startup, ​Hugging Face, said it had turned to Zhipu AI's open-source GLM-5.2 model last week to analyze data from the hack after leading US AI models declined the task, unable to distinguish between a defender and an attacker.

While the breach was caused by an autonomous agent that escaped containment, it highlighted how US companies facing ‌AI-driven cyberattacks can ⁠be limited ⁠by American AI labs that either restrict access to their most advanced models or design them to refuse hacking-related tasks out of safety concerns.


For instance, Anthropic's advanced Claude Fable ​5 model routes cybersecurity queries to an older model, while OpenAI's GPT-5.6 Sol has protections designed to block cyber work.

"We're all learning that secrecy is ​not the answer & that all defenders (not just a few selected ones) everywhere need more powerful models without restrictions, especially open ones!" Hugging Face co-founder Clement Delangue said on X.

The bind for leading American model makers is that defensive cybersecurity work is often hard to distinguish from ​malicious hacking. In recent AI-enabled breaches, attackers tricked models into thinking they were doing legitimate ⁠defense work, ‌leaving AI firms wary of easing safeguards even as cyber professionals say the guardrails can hamper their work.

For ​now, the fallout is ​handing another boost to Chinese open-source models such as GLM-5.2, which are gaining traction in Silicon Valley ⁠with coding and agentic capabilities that nearly rival those of OpenAI and Anthropic at ​lower cost.

Beijing has also been increasingly using open-source to position itself as an alternative to the ​US in the high-stakes race, with Chinese state media increasingly portraying the strategy as a response to what it calls a US-led attempt to erect an "AI Iron Curtain."

"A safety regime that restricts legitimate defenders, while capable models remain available for attackers, creates an asymmetric disadvantage," said Lukasz Olejnik, independent technology consultant and visiting senior research fellow at the Department of War Studies, King's College London.

"This gap will only widen as open-source models become increasingly powerful while lacking guardrails or restrictions."

Growing prominence of open-source

OpenAI and Anthropic did not immediately respond to requests for ‌comment on Wednesday on whether their safeguards were hindering cybersecurity work.

The ChatGPT maker said in a blog post on Tuesday: "We've brought Hugging Face into the trusted access program and are supporting their teams in rapidly using our ​models' capabilities to improve ​their defenses."

For Beijing-based Zhipu AI, Hugging Face's ⁠endorsement adds to the momentum GLM-5.2 has built since its launch last month.

The model has rapidly climbed usage charts on developer platforms such as OpenRouter and drawn plaudits from figures ranging from Snowflake CEO Sridhar Ramaswamy to venture capitalist Marc Andreessen.

Zhipu AI , which raised about $4 ​billion in a Hong Kong share sale earlier this month, has also seen its stock jump nearly nine-fold since its debut in January.

Still, some analysts warned on Wednesday that the incident should not be used to promote loosening of US safeguards.

"The cybersecurity guardrails on US frontier models are creating a competitive opening, but the answer is not simply to remove them," said Shrenik Kothari, analyst at Robert W. Baird.

"OpenAI, Anthropic and Google should rethink the architecture of access rather than abandon safety ... In other words, shift from a one-size-fits-all refusal layer toward controlled capability allocation."