Pushing back against AI 'doomsday narratives', Nvidia creates platform to keep rogue agents in check

The initial wonder of agentic artificial intelligence (AI) tools cutting down the time taken for complex tasks has recently given way to fear of which protocols they are breaking and which systems they are hacking.

To that end, Nvidia has created a set of software tools called the Nvidia Open Agent Safety Platform. The company said a platform such as this would have prevented the hack of Hugging Face.

The safety debate


Numerous instances have been reported by AI leaders Anthropic, OpenAI and Google's Gemini team of their agents going rogue and breaking through the firewalls of government and private systems to accomplish assigned tasks. This has led to a roaring debate within the AI community about the safety of the technology, and whether top companies should slow down.

Anthropic CEO Dario Amodei, in his essay "We Must Pace the Frontier", proposed that the fast-growing technology be developed at a slower pace so that humans remain in control. He wrote that pacing does not mean stopping progress, but rather taking adequate time to align and safeguard models before releasing more powerful systems. In a rare moment of agreement, OpenAI CEO Sam Altman and Elon Musk backed the idea.

Others stayed firmly on the opposite side, saying the answer lies in guardrails, not a slowdown. One of them was Nvidia's Jensen Huang, who publicly criticised Amodei's and others' position that a coordinated AI slowdown was required, calling extinction warnings "doomsday narratives" with a "0% chance" of happening.

"Over the past few weeks, these reports [of agentic hacking] have led to a serious debate about the pace of agent development. We believe we need to increase the pace of AI safety research and engineering in collaboration with frontier labs and the broader community," Nvidia said in a blog post announcing the new platform.

What are the tools?

One tool, released Monday, is called OpenShell. It uses hardware features on Nvidia's central processor chips to contain agents. Nvidia said it is also working with Arm Holdings and Intel to ensure the system works on their central processors too.

Another tool, called Sentry, works in tandem with OpenShell and uses a separate Nvidia chip to cut off a rogue agent if it tries to escape its container on a central processor.

How does it work?

The Nvidia tools use mathematical formulas to detect when agents are trying to use workarounds, such as spawning several "sub-agents" to circumvent efforts to block the main agent, Ali Golshan, senior director of AI software at Nvidia, told Reuters.

OpenShell runs each agent in a sandbox and turns the operator's instructions into a verifiable policy. Operators define which files, networks, tools, processes and credentials an agent can access. OpenShell checks those limits before the agent runs and enforces them as it works.

The Open Agent Safety Platform is optimised to run on systems based on Nvidia's Vera CPU and BlueField DPU, and is also compatible with other hardware.