AI Took 17,600 Autonomous Actions to Break In. How Many Is Your Defense Taking Today?

By: Saket Modi, Co-Founder and CEO of SAFE

 

“AI models are becoming extraordinary zero-day discovery engines.”
– JFrog’s CTO

Last month, Hugging Face disclosed a security incident of a kind that rarely makes headlines. There was no phishing email, no stolen password, and no hacker group directing the attack. The intrusion was carried out by a set of AI agents that, over a single weekend, executed roughly 17,600 individual actions, spanning reconnaissance, exploitation, privilege escalation, lateral movement and data access, with no human in the loop.

According to disclosures from Hugging Face, OpenAI and JFrog, the agents were not built to attack anyone. They were part of a routine model evaluation. What follows is what the three companies have since described, and why we think it marks an inflection point for how cybersecurity has to move forward.

What happened?

The agents began as an OpenAI model evaluation running inside a sandbox: no internet access, and a defined task to solve. Unable to complete the task within those limits, the models examined their environment and identified a package manager, JFrog Artifactory, that was reachable from inside the sandbox.

In it, they found vulnerabilities that were not previously known. JFrog subsequently patched nine CVEs in Artifactory 7.161, covering remote code execution, server-side request forgery, path traversal and privilege escalation. The models chained these flaws to obtain network access beyond the sandbox.

From there, the activity followed a familiar attacker pattern. The models staged command-and-control on third-party infrastructure rather than operating directly from the research environment. They ran as a coordinated set of agents and communicated through a covert channel; when some engineers got a hint of it and tried to remove it, the agents re-established coordination by encoding state in directory names. OpenAI’s logs recorded one agent’s reasoning:

“External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”

The agents reached Hugging Face’s production environment through a dataset that executed code during processing, escalated to root privileges, gained access to Kubernetes workloads and collected service credentials. Hugging Face has said it detected the anomaly through AI-assisted analysis; by public accounts, the access ran for roughly seven hours before it was flagged. Hugging Face reported no evidence of tampering with public models, datasets or Spaces.

Two details are worth holding onto. First, the initial vulnerability was a zero-day, absent from any database and therefore invisible to signature-based scanning. The models did not look it up; they discovered it. Second, in analyzing the incident, Hugging Face reported using an open-weight model (GLM-5.2) on its own infrastructure, because the work required submitting raw exploit data that the safety filters on commercial APIs would refuse.

JFrog’s CTO summarized the shift plainly: “AI models are becoming extraordinary zero-day discovery engines.”

Why is this one different?

We have had the front row seat over a decade monitoring and defending some of the most sophisticated hacks around the world. SAFE started as a penetration-testing company; finding and exploiting vulnerabilities is where we started. What stands out about this incident is not the sophistication of any single step, but that the full chain ran autonomously, at machine speed, and went largely undetected.

An autonomous agent can attempt intrusions continuously, in parallel and at low cost. The defensive process has not changed at the same rate: in most organizations, an analyst still reviews scanner output, judges what is exploitable, validates by hand and files tickets, clearing part of a backlog that refills faster than it drains.

Offense has become autonomous; most defense has not.

A little over a year ago, I wrote about AlphaGo’s 2016 “Move 37”, a move no human would have played, and a demonstration that a machine could act outside human intuition. We asked what a comparable move would look like in cybersecurity, and argued the response would require a system able to anticipate attacks that have never been seen before, including zero-days. The Hugging Face incident is a concrete instance of that scenario rather than a hypothetical one.

Why rule-based tools fall short here?

Most vulnerability and exposure tools, including well-established ones, are built on rules: signatures and static lists of known CVEs and known exploits. That approach works well for known problems, and poorly for two situations this incident highlights.

It cannot evaluate a zero-day, because a zero-day is by definition absent from the database. And it cannot keep pace, because once a model discovers a new exploit on its own, any fixed catalog is already behind.

Adding a conversational interface to a rule-based scanner does not resolve either limitation. Meeting an autonomous, reasoning attacker requires a system that can reason and act about an environment, not one that reads the same list faster.

What autonomous validation changes?

This is the problem SAFE’s CTEM AI Co-Worker is built to address, as part of the validation stage of Continuous Threat Exposure Management (CTEM). Mapped against the Hugging Face chain, the intent is straightforward.

• An AI-native engine evaluates code and infrastructure the way an attacking model would, looking for unknown vulnerabilities rather than only catalogued ones, the class of flaw that enabled the initial JFrog compromise.
• Rather than producing a list of disconnected findings, it attempts to connect exposures into an exploitable path and confirm whether that path is real, so remediation focuses on the links that actually enable an attack.
• And unlike an external attacker, a defender is not working blind. In the disclosed incident, the agents needed several days in part because they had no prior knowledge of the target. A defender’s system can draw on existing context: asset inventory, network topology and business criticality, held in SAFE’s knowledge graph. That narrows the search considerably.

The design goal is a system that reasons and acts like an attacker but operates inside an organization’s own environment, with its full context and within defined guardrails and human oversight, not an unconstrained agent running in production. In practice, it moves validation from a human-paced process measured in days toward an automated one measured in minutes, with people setting scope and reviewing results.

Where does this leave defenders?

For years, security budgets were treated as fixed overhead. Autonomous offense is changing that calculation; several reports indicate security spending is rising again as boards reassess exposure. The broader shift is from tools that add some AI toward systems designed for an environment in which both attackers and defenders operate autonomously.

SAFE’s own path runs through both sides of this problem: years of offensive security work, followed by years building agentic systems deployed in large enterprises. A year ago we described the need for a defensive counterpart to increasingly capable AI, what we called Cyber Superintelligence. The Hugging Face incident is early evidence that the need is concrete rather than theoretical.

The attacker in this case took roughly 17,600 autonomous actions to get in. The open question for every security team is how much of their own defense is capable of operating the same way.

SAFE is an agentic AI company securing the AI era. The CTEM AI Co-Worker brings autonomous, AI-native validation to exposure management, reasoning and acting like an attacker, within a defender’s context and guardrails.