Nvidia debuts enhanced safety controls to rein in rogue AI agents

Few companies are more invested in the success of artificial intelligence agents than Nvidia Corp., so it makes sense that the chipmaker would want to ensure these autonomous software systems run safely, without causing any problems.

To that end, Nvidia early today announced the debut of its free, open-source Nvidia Open Agent Safety Platform, which bundles various tools organizations can use to beef up the security of their AI agents, strengthen their control over them and improve governance. It’s built on top of Nvidia’s open-source OpenShell software and incorporates Nvidia Sentry to enable full control over not just the AI agents, but also the hardware and compute resources that power them.

Nvidia is trying to counter the alarm that has been raised in the wake of numerous high-profile security incidents involving AI agents that have gone rogue. With each passing day, it seems, revelations emerge of AI agents breaking free of their owners’ control. Just last week, a researcher discovered that an entire swarm of agents, including some created by OpenAI Group PBC, had hacked into various systems controlled by the Australian government in an incident that caught the attention of Prime Minister Anthony Albanese.

OpenAI’s agents were also responsible for another infamous incident that took place between May and June. The agents reportedly worked together to break out of an isolated sandbox environment and hack the model hosting platform Hugging Face Inc. Though neither incident caused known serious damage, the potential for AI agents to wreak havoc with third parties is making a lot of people very nervous.

Nvidia says these incidents demonstrate that it’s all too easy for AI agents to circumvent traditional application-layer guardrails. What’s needed, it argues, is an entirely new set of controls that spans the entire agentic stack. Instead of relying solely on software-level prompts or application limits, the Nvidia Open Agent Safety Platform enforces rules across three distinct layers – the agents themselves, the compute and the hardware.

The platform is built atop Nvidia’s OpenShell software, which provides a watertight open-source runtime environment designed to run fleets of autonomous agents within isolated sandboxes, with infrastructure-level controls. According to the chipmaker, OpenShell acts like a boundary that allows users to trace all of their agents’ actions and enforce security policies outside of whatever guardrails are applied to the underlying large language model that powers them. The software has been optimized to run on Nvidia’s Vera central processing units, but is also compatible with CPUs from Intel Corp. and Arm Ltd.

The other major component of the Nvidia Open Agent Safety Platform is Nvidia’s Sentry, which is a reference system design that’s based on Nvidia DOCA software. DOCA, which stands for data center infrastructure-on-a-chip architecture, is a unified programming and software framework for writing applications that can run on Nvidia’s specialized BlueField data processing units or DPUs, similar to how CUDA is used to build apps optimized for its graphics processing units.

Running on BlueField-4, Nvidia Sentry is designed to act like a security guard, continuously monitoring AI agents in the background. By operating directly on the silicon, it’s able to study agent’s behavior by inspecting requests, verifying identities and actions, and has the ability to shut them down instantly if they try to move beyond their assigned boundaries.

Nvidia founder and Chief Executive Jensen Huang said AI safety has become a crucial concern that the industry has to solve if the technology is going to achieve its full potential. “Safety and security require full-stack engineering,” he said in a statement. “The Nvidia Open Agent Safety Platform brings together industry, researchers and public-sector organizations to share best practices, align on evaluation methods and foster international cooperation. Together, we can raise the bar for global AI safety.”

Nvidia’s outsized influence in the AI industry means that the platform already has solid credentials with some of the world’s biggest enterprises. One of its biggest advocates is SpaceXAI Corp., which has been using the platform to secure Cursor’s coding agents and the Grok large language model. “As customers rely more on agents to get real work done, safety should be enforced outside the model by additional controls the agent can’t get past,” said SpaceXAI President Mike Nicolls. “Customers should be able to set those limits for Cursor and Grok and trust they will hold.”

Salesforce has also integrated OpenShell with its Slack collaboration platform to provide users with direct human approval controls, visibility and audit tracking capabilities for AI agents, while SAP SE is using the software with the SAP Business AI Platform to enable runtime security for its autonomous agents.

Robotics companies are also embracing Nvidia’s Open Agent Safety Platform en masse, with Figure AI Inc., Gecko Robotics Inc. and Skild AI Inc. relying on OpenShell to embed safety controls into the systems that power autonomous robots working in the real world, Nvidia said.

It remains to be seen how reliable the Nvidia Open Agent Safety Platform will prove to be at large scale given the worrying ingenuity some AI agents have shown, but it is at least a sign that some of the industry’s biggest players at last are taking agentic safety seriously. As an open-source platform, the Nvidia Open Agent Safety Platform is available for anyone to download via Nvidia’s developer resources and the GitHub platform.

Image: SiliconANGLE/Gemini