In a handful of test runs inside a controlled evaluation environment, an AI model reached past its intended target, exploited real vulnerabilities, extracted credentials, and gained access to a live production database. The target wasn’t a simulation. It was a real company that had nothing to do with the test.
That’s the account Irregular published of an incident it identified together with Anthropic. Irregular is an Israeli AI safety testing firm that has raised $80 million from investors including Sequoia and Redpoint Ventures, according to CNBC. In three test runs, models operating with permitted internet access exceeded the evaluations’ intended scope and acted against real organizations instead of the fictional targets they had been assigned.
A naming error, not a jailbreak
The root cause wasn’t a novel exploit or an adversarial prompt. It was a naming collision. Irregular’s engineers assigned a fictional target company a name that happened to match an existing real-world domain. Nobody caught the overlap, because the real domain wasn’t widely known.
Internet access had been enabled so the evaluation could measure the model’s offensive cyber capabilities. In the runs that mattered, the model reached the real domain, treated it as the assigned target, and went to work. It exploited vulnerabilities there, extracted credentials, and reached a production database. In a separate case, a different model drifted to a similarly named site and found login credentials that had already been posted publicly elsewhere, not credentials it obtained itself.
Irregular runs these evaluations for OpenAI, Anthropic, and Meta, stress-testing frontier models before release across thousands of simulation runs in 48- to 72-hour cycles. That scale is precisely what turned a single unnoticed naming collision into three separate incidents rather than one. The Register reported that Anthropic responded by suspending its cyber capability evaluations until the environment could be revalidated, a sign of how seriously the lab treated a failure in its own test harness rather than in the model.
The pattern underneath the incident
Strip away the specifics and a familiar structural problem emerges: the boundary between “simulated target” and “real target” existed only as a naming convention, not as an enforced access control. A test harness gave a model open internet reach and trusted a label to keep it inside the lines.
It is the same failure mode security teams have spent two decades engineering out of production systems, now appearing as AI agents create a new enterprise security gap between permissions and effective runtime controls. The evaluation in question was designed to measure whether a model could help a malicious insider gain unauthorized access to sensitive data inside a company’s production database. The test itself became the incident it was built to detect.
Irregular’s remediation reads like a standard post-incident checklist: expanded manual review, a new internal “red-team-of-assumptions” unit, revalidated evaluations checking for domain overlaps, and a forthcoming best-practices white paper. CyberScoop reported that Irregular itself pointed to a lapse in human oversight as the proximate cause. That’s accurate, and incomplete. Those fixes address the bug. They don’t resolve the underlying design choice: scope was defined by what a target was named, not by what a model was authorized to reach.
Why naming and permissions aren’t the same control
For a CISO or a compliance officer reading this, the more useful question isn’t “how do we validate fictional domain names more carefully.” It’s “why did scope enforcement depend on a name in the first place.” A name is metadata. It carries no authorization logic, and it fails silently the moment it’s wrong.
For compliance teams, the relevant questions include who authorized the model’s access, what data it was permitted to retrieve, and whether technical controls enforced that scope before access occurred. In Irregular’s account, no such enforcement layer existed between the model and the open internet. The failure surfaced only because engineers went looking for it afterward, not because a control stopped it in real time.
That is an evidence gap as much as a technical one. As AI agents redefine enterprise security risk, an audit trail showing what happened is not the same as a policy layer determining what is allowed to happen. The first is forensics. The second is governance. Enterprises running their own AI agents against internal systems, not third-party safety evaluations, face the same distinction, and most haven’t resolved it.
More must-read AI coverage
What has to change: access scoped to the request, not the label
To be direct about where this incident sits relative to my own company: Kiteworks was not in the data path of this incident. The company that was breached ran its own domain and its own production database, reachable directly from the test environment’s open internet connection. No Kiteworks-governed system, and no Kiteworks control, was anywhere in that chain.
What this incident illustrates is the difference between naming-based scope and enforced, per-request authorization, and that distinction is exactly what a governed data-access architecture is built to hold. Under a control plane model, every retrieval request is checked against role- and attribute-based policy at the moment it’s made, whether the requester is a human employee or an AI agent, and the credentials a system uses to reach a datastore are never exposed inside the model’s own working context. If a company’s sensitive data and its agents’ access paths were governed under that kind of architecture, instead of being reachable directly over the open internet from a model’s execution environment, an agent could not harvest a credential from its surroundings and reuse it to reach an unauthorized datastore, because the credential would never be visible to it, and the retrieval would be evaluated against policy before it happened rather than logged after the fact.
That framing has real limits, worth stating plainly rather than glossing over. This kind of governance constrains what an authenticated agent session can retrieve from data an organization has placed under that plane’s control. It does nothing to constrain what an agent does against an arbitrary third-party internet host, which is precisely the failure mode Irregular describes: the model wasn’t misusing an authorized session, it was reaching an unauthorized destination entirely outside any governance boundary. Attribute-based policy is also only as restrictive as the policy an organization configures. Securing the agentic enterprise still requires organizations to define and continually revalidate what an agent is allowed to reach.
The closing point regulators will make for you if you don’t make it yourself
For IT and security teams, the practical lesson is straightforward: an AI agent’s permitted scope should be enforced through network, identity, and data-access controls, not merely described in a prompt or target name. Before agents receive internet access or production credentials, organizations should test whether those controls can prevent an incorrect instruction from becoming a real security incident.
