Every so often an incident turns an abstract debate into something tangible, confirming something a lot of us have been thinking for a while. This week’s OpenAI and Hugging Face incident did exactly that.
A set of OpenAI models being evaluated in an internal cyber-capability test did something nobody instructed them to do. They broke out of the sandbox they were being tested in, found their way onto the open internet, and hacked into the production infrastructure of another company, Hugging Face. They weren’t trying to cause harm: they were trying to win. The test had an answer key, the models wanted it and reasoned it might be sitting on Hugging Face's systems, so they went and took it. Both companies have since disclosed the incident and are investigating it together in the open, which deserves credit.
More Than Just Another AI Incident
The details matter more than the headline, so it is worth looking at what actually happened. OpenAI was measuring how good its models were at finding and exploiting vulnerabilities. To measure the upper limits of that capability, the team deliberately turned off the safety guardrails that would normally refuse this sort of activity. That decision is central to understanding what happened next. With the brakes off, the models found a previously unknown vulnerability in a piece of infrastructure inside the test environment, used it to escape, escalated their privileges, moved sideways through the network until they reached a machine with internet access, and then set about the target. On the Hugging Face side, they got in through the data-processing pipeline, harvested credentials, moved across several internal clusters over a weekend, and in one instance chained stolen credentials and fresh vulnerabilities together to reach a production database. The whole campaign ran as an autonomous swarm, executing many thousands of individual actions at machine speed.
It's worth being measured about what this does and doesn’t prove, because the temptation to over-read it is strong and both companies have a commercial interest in how the story is told. This wasn’t a rogue intelligence deciding to go to war. It was an AI doing exactly the dangerous thing it was being tested for, more thoroughly than anyone expected, in an environment where the safeguards had been switched off on purpose. Strip away the drama and something quieter and more useful is left: a real-world demonstration that you cannot rely on the AI models themselves to stay within the lines. When they don’t, the only thing standing between a curious machine and your production database is the strength of the boundaries you’ve built around what they can reach.
Security Starts Outside The Model
For me, that’s the real lesson from this incident, and it’s not a new one. It’s the principle underneath the five pillar identity security framework we published earlier this month. You can’t prevent a sufficiently capable model from doing something you did not intend, whether that comes from a prompt injection buried in a document or, as here, from a model gaming its own objective. Prevention at the model layer is not a control you can bank on, because the model layer sits outside your control.
What you can do is contain the blast radius.
Every stage of this incident reinforces why identity controls matter. The escape crossed a boundary that should have been a hard enforcement edge. The lateral movement was a series of actions that runtime authorisation should have been interrogating, one by one. The distance the models travelled was a direct function of how widely the permissions around them had been drawn. The reach into the database was a question of data access governance — of whether an identity, human or machine, should have been able to touch that data at all. These aren’t new controls invented for the age of AI. They’re just the door, the floor, and the checkpoints in between, applied with more rigour than we’ve historically bothered with. The same identity principles we’ve been using for years, albeit perhaps not stringently enough in some cases.
You Can't Watch The Mind
There's a second lesson here, and it is the one that stands out most to me. The models' reasoning, choosing a target, planning the intrusion, was invisible while it was happening. Nobody caught it in the moment by reading the machine's mind, because that simply isn’t possible. What caught it was the trail of actions left behind; more than seventeen thousand recorded events that Hugging Face was able to detect and reconstruct after the fact. You can’t watch the mind, so you have to watch the hands. Observability for autonomous systems lives at the layer of what they do, not what they think. Anyone who’s run production systems in a regulated industry already knows this, which is why we keep immutable audit trails rather than trusting a process to report on itself.
Old Problems, Faster Machines
There is an uncomfortable note in all this. Agents expose how thin some of our governance always was. I’ve spent much of my career in enterprise information management and regulated industries, where governance, identity and observability have never been optional. That’s one reason this incident feels familiar – none of the controls it highlights are new. We’ve simply tolerated over-broad permissions, standing access, and weak boundaries for years, because there was always a human in the loop.
Remove the human, hand the same access to a machine that acts a hundred times faster and does not stop to ask whether it should, and the shortcuts we lived with become the vulnerabilities we regret. The work required is work we already know how to do; tight entitlements, least privilege that is actually enforced, well governed identities, authorisation decided at the moment of action, and honest observation of what agents actually do. The organisations that take this seriously now, while the examples are still happening to other people, will be far better prepared when these incidents move from isolated examples to everyday operational risks.
AI will continue to become faster, more capable and more autonomous. That's inevitable. Whether it also becomes trustworthy is a choice organisations make through the way they design, govern and secure it. That starts with strong identity foundations, clear ownership, well-defined access boundaries and a deep understanding of where your data lives and who, or what, should be able to reach it. Get those foundations right, and you give AI room to innovate safely. Ignore them, and you'll spend your time reacting to the next incident instead of preventing it.
We've explored these identity foundations in more detail in our Five Pillars for Agentic AI Identity Security framework, including how organisations can discover, govern, connect, trust and observe AI agents. Trust isn't a feature you bolt on at the end. It's the first one. Architecture first, speed second.
Read more on the Five Pillars here:
