How AI Agent Governance Is Moving Into Practice

On September 25, the UK government published new draft guidelines on using AI sustainably. It arrived after several weeks of agent-related breaches and disclosures, bringing renewed attention to a basic but increasingly urgent question: what controls need to be in place when AI can access data, call tools, and take action?

That question is now shaping policy, product design, and the expectations placed on AI companies—from government guidance on appropriate use to tools for monitoring agents and stricter expectations around testing and disclosure.

^{Read more about AI governance and how to implement it here.} 

Why Agent Governance Is Getting More Attention

In September, GreyNoise reported that a threat actor used hundreds of AI agents to develop and deploy exploits against PaperCut servers. The campaign compromised at least 440 instances across 395 organizations in 48 countries; once fully launched, it compromised 11 organizations in 26 seconds.

Other incidents came out of AI research and testing. In June, an experimental OpenAI model gained unauthorized access to a Services Australia system, retrieving internal files, credentials, and aggregate statistics. OpenAI said it found no evidence that patient records were accessed.

OpenAI’s wider review also found that agents had posted 53 user-provided images to third-party sites; Reuters reported more than 15 disclosed OpenAI-related cases of varying severity in the two months following the Hugging Face breach.

OpenAI wasn’t alone in reporting agents that crossed intended boundaries. Anthropic identified four incidents in which Claude models accessed real third-party systems during cybersecurity evaluations after a misconfiguration left the evaluation environment connected to the open internet. In a separate UK AI Security Institute evaluation conducted with internet access enabled and some safeguards disabled, agents took unsanctioned actions on the live internet that went beyond their assigned tasks. The incident prompted changes to the institute’s evaluation protocols, monitoring, and security architecture. Both cases prompted tighter controls around internet access, monitoring, and the environments used to test advanced models.

The circumstances and severity varied, but the governance problem is consistent: once agents can use credentials, connect to outside systems, and act across multiple steps, written instructions aren’t enough. Their access needs to be technically limited, monitored, and easy to suspend when something goes wrong.

The UK Government Encourages Careful AI Use

The UK government’s draft guidance asks a practical question: does the task need AI at all? It advises employees to consider whether a spreadsheet, search engine, or standard software could do the job first. When AI is appropriate, it recommends using the smallest model capable of handling the task, checking outputs for accuracy, and disclosing AI-assisted content.

This represents one end of responsible AI use: avoid unnecessary compute and use only as much capability as the task requires. That principle makes sense for narrow, repeatable work, where a spreadsheet, search tool, or smaller model may be enough. But it shouldn’t become a blanket rule. Tasks involving complex context, multistep reasoning, or technical judgment may require a more capable model to produce reliable results.

The same principle applies to conventional software. Deterministic code is often a better fit for a pricing calculation, eligibility rule, or approval step, while a knowledge assistant searching internal documents may be better suited to retrieval-augmented generation. Neither approach is inherently better; the question is whether it fits the task and its potential impact.

Teams should start with the work itself: whether AI is appropriate, how capable the system needs to be, how reliable its output must be, and what happens if it gets something wrong.

Used thoughtfully, the UK’s approach can help teams avoid adding unnecessary AI to a process while still choosing a more capable system when the task, access, or potential impact calls for it.

NVIDIA Launches Open Agent Safety Platform

As AI agents gain access to internal data, APIs, and business systems, security controls need to extend beyond the model itself.

On September 28, NVIDIA launched its Open Agent Safety Platform, an open platform and reference design for securing AI agents from testing through deployment. The launch builds on NVIDIA’s recent push for shared, open security infrastructure for AI systems, including its Open Secure AI Alliance.

The Open Agent Safety Platform includes OpenShell, which creates a controlled environment for agent activity, and NVIDIA Sentry, an independent monitoring layer designed to detect and contain behavior that falls outside defined policies. NVIDIA said it’s working with Anthropic to integrate managed agents with OpenShell and named Cisco, Microsoft, Oracle, CoreWeave, Dell, HPE, Lenovo, Arm, and Intel as platform partners.

In CNBC’s coverage of the launch, NVIDIA CEO Jensen Huang compared the approach to “a browser for agents”: a contained environment that gives an agent access to the tools and information it needs without letting it move freely through a company’s systems. NVIDIA said the platform could have helped prevent the Hugging Face incident, though the right controls will depend on each organization’s architecture, data, and workflows.

The launch reflects a growing focus on the infrastructure around AI agents, not just the models that power them. As agents move from controlled testing into production systems, organizations will need stronger ways to isolate activity, monitor behavior, and intervene when something goes wrong.

OpenAI Calls for Capability-Based AI Safety Rules

On September 9, OpenAI published “The AI Policy Window Is Open. We Need to Act”, calling for mandatory, capability-based AI safety rules in the United States. It also backed California measures covering independent assessments, AI-auditor standards, youth protections, and safeguards against AI-enabled biological threats.

The proposed national framework would apply its strongest requirements to well-resourced companies developing the most capable systems—not startups, small developers, or researchers operating far from the frontier. Those requirements could include common testing, independent assessments, stronger cybersecurity, monitoring, and serious-incident reporting.

The company also calls for prompt notice when a model bypasses security controls and materially accesses, alters, or destroys protected systems or confidential information. For most businesses using AI rather than developing frontier models, the appropriate level of oversight would be different.

That position comes as regulators place more specific requirements on ChatGPT. In August, the European Commission designated ChatGPT a Very Large Online Search Engine under the Digital Services Act. The designation will require OpenAI to assess and mitigate systemic risks, undergo independent audits, and provide greater transparency around the service.

OpenAI has a stake in how future regulation is written. However, its proposal reflects a broader push for clearer expectations around testing, monitoring, and disclosure.

^{Read more about what the designation means for ChatGPT and California businesses.}

What Businesses Need to Watch

The UK guidance, NVIDIA’s platform, and OpenAI’s policy push address different parts of the same challenge: how to use AI systems responsibly as they become more capable and more connected to business operations.

For most organizations, the question is no longer whether to govern AI, but how to do it in practice: know where it is being used, what it can access, who owns it, and what happens when it fails.

A practical governance approach involves:

• Maintaining an inventory of AI systems, their owners, and their business purpose.
• Defining which data, tools, APIs, and business systems each agent may access.
• Applying least-privilege permissions and separating low-risk actions from consequential ones.
• Requiring human approval for actions that affect customers, money, production systems, or sensitive data.
• Logging agent activity and establishing monitoring for unusual behavior or unauthorized access attempts.
• Testing systems before deployment and revisiting controls when their capabilities or access change.
• Establishing a clear incident process, including who investigates, who is notified, and how to suspend access.

Governance doesn’t mean treating every AI tool as a high-risk system. It means matching oversight to the level of access and potential impact. A simple writing assistant, for example, won’t need the same controls as a legal agent, where an incorrect output or unauthorized action could lead to reputational, financial, or legal consequences.

Choosing the right level of capability is part of that process. Teams need to understand which models fit each workload, how they’re being used, and whether the operational and cost trade-offs make sense.

For teams applying that principle to AI-assisted software development, our free guide, How to Cut AI Coding Costs Without Switching Models, explains how to evaluate model capability, infrastructure, token usage, and reliability before changing your AI stack.

^{Read the guide here.}