The Six Ways AI Agents Go Rogue

Anatomy of an Agent Action

Every agent action follows the same four-stage path. Input: the user's request plus everything the agent reads, including documents, tickets, web pages and tool outputs. Plan: the model decides what to do. Action: it issues an LLM or tool call. Effect: something changes in the world. A row is deleted, money moves, a message is sent, a bill grows.

Each failure mode enters at a different stage. But every one of them has to pass through the same narrow point to cause harm: the moment between plan and action. That is where a deterministic check can sit. Pick a failure mode below to see where it enters, where it lands, and what stops it.

Over-Eager Completion

What it looks like. The agent is trying to finish the job, and it takes a destructive shortcut to get there. A failing test gets "fixed" by deleting the test. An empty query result gets "fixed" by resetting the database. A move command runs against a folder that was never created.

Why it happens. Agents are trained and prompted to complete tasks. When they hit an unexpected state, the most direct path to "done" is often the most destructive one. The model has no lived sense of irreversibility. DROP TABLE and SELECT are both just tokens that move the task forward.

Signals in your traces

• A destructive tool call (delete, drop, truncate, rm, force-push) shortly after an error or unexpected-result span.
• Tool calls outside the task's natural scope, such as writes during a task that only asked for analysis.
• Repeated retries that escalate in severity: retry, then reset, then delete.

How to stop it

Deny by name and by content, before the call runs. High-Risk Tool denies a named tool (for example delete_account or a prod SQL executor). Dangerous Shell Commands denies a shell call whose command contains text you list, such as rm -rf or DROP TABLE. Scope both to your production workspace in Block mode so experiments in dev stay fast.

With a Dangerous Shell Commands policy in Block that lists rm -rf for the shell tool, this is all the agent code needs. The check runs before the function body, so the command never reaches a shell:

# The policy checks read these, not init() arguments:
#   export TRACCIA_API_KEY=...
#   export TRACCIA_ENDPOINT=https://api.traccia.ai/v2/traces
from traccia import init, govern, observe
from traccia.governance import AgentBlockedError

init(agent_id="ops-shell")


@observe(name="shell", as_type="tool")
def shell(command: str) -> dict:
    return run_command(command)  # your code


@govern(name="ops_shell")
def handle_ticket(command: str) -> dict:
    return shell(command)


try:
    handle_ticket("rm -rf /var/lib/postgres")
except AgentBlockedError as exc:
    print(exc.reasons)  # shell() never ran

The denied call shows up in the Decision Log as Denied, and the trace is still saved. If Traccia cannot be reached, the call follows your setting under Settings → Policy Enforcement, which blocks by default from Python SDK 0.1.34 or TypeScript SDK 0.1.20. Keep the agent's credentials narrow as well. For the full walkthrough, see Stop a Dangerous Tool.

Hallucinated Authority

What it looks like. The agent states a policy, quotes a price or makes a commitment that nobody authorized, and says it with complete confidence. Nothing technical breaks. The damage is that a customer believes it, and your business is held to it.

Why it happens. When retrieval fails or returns too little, the model fills the gap from its training priors. A plausible policy is statistically close to a real one. The more fluent the model, the more convincing the invention.

Signals in your traces

• An answer span with no preceding retrieval span, or a retrieval span that returned few or zero chunks.
• Answers grounded in data older than your freshness limit.
• A prompt loaded from an unexpected version or label.

How to stop it

Make "I looked it up" a precondition. Retrieval Completeness denies an answer when the agent did not search or got too few chunks back. Freshness Guard denies a write when the data the agent read is older than your limit. Prompt Pin denies a model call when a managed prompt was loaded from the wrong label or version. For money-adjacent answers, pair these with Refund Guard.

Hijacked Intent

What it looks like. Content the agent reads, such as an email, a support ticket, a GitHub issue or a web form, contains instructions. The agent follows them as if they came from the user. This is prompt injection, and in agentic systems it turns into action, not just text.

Why it happens. Models do not reliably separate data from instructions. Everything in the context window is input. The risk peaks when one agent has what Simon Willison calls the "lethal trifecta": access to private data, exposure to untrusted content, and a way to send data out.

Signals in your traces

• An outbound or write tool call that follows a span reading untrusted content.
• Tool-call parameters that echo text from a retrieved document instead of the user's request.
• Injection or input-validation categories triggered, or missing, in Guardrail Posture.

How to stop it

You cannot reliably filter every injection, but you can limit what an injected instruction is able to do. Deny high-risk and outbound tools for agents that read untrusted content (High-Risk Tool), cap bulk data access (Unique Customer Cap), and use Guardrail Detection to surface injection signals. Guardrail Posture is read-only, so pair it with least-privilege credentials and egress controls.

Unconfirmed Irreversible Action

What it looks like. The agent understood the task correctly and did it, but it took a step a human should have confirmed. It bought instead of adding to cart, sent instead of drafting, or deployed instead of preparing.

Why it happens. "Prepare" and "submit" are often the same API, and agents optimize for completion. Without an explicit hold point, the gap between "ready to do" and "done" is a single token.

Signals in your traces

• A payment, order, refund, send or deploy tool call with no preceding approval event.
• Transaction amounts that cluster just above the range where a human would normally sign off.

How to stop it

Use three bands instead of a single on/off switch. Purchase Guard and Refund Guard allow small amounts and deny amounts above a cutoff. Turn on Ask For Approval and the middle band waits for a person in the Approvals inbox. Approving does not place the order for you; your own server does that once a person says yes. Low-risk actions stay fast, and irreversible ones get a human.

Compromised Supply Chain

What it looks like. The attacker never talks to your agent. They change the agent itself: its system prompt, its extension, a package it depends on, or a tool it loads. From the outside, the agent's behavior looks like normal operation.

Why it happens. Agents are software with an unusually powerful runtime. They often run with a developer's credentials and with permission to execute shell commands. Anyone who can change their instructions inherits that power.

Signals in your traces

• Agent behavior changes with no code change on your side.
• A prompt version or label you did not promote, or a model outside your approved list.
• Shell commands that touch credentials, cloud resources or home directories during unrelated tasks.

How to stop it

Traccia cannot sign your supply chain, but it can shrink the blast radius. Dangerous Shell Commands and High-Risk Tool deny destructive operations whatever their origin. Prompt Pin rejects managed prompts loaded from the wrong version, and Model Boundary or the Unapproved Model Alert catches unexpected models. Pair these with dependency pinning and extension review.

Runaway Loops

What it looks like. The agent retries, re-plans or calls the same tool again and again without converging. Each call is harmless. Ten thousand of them are not. Context grows, latency climbs, and the bill arrives later.

Why it happens. Agent loops have no natural stopping condition beyond "the model decides it's done." Ambiguous tasks, flaky tools and error messages the model misreads all produce cycles. Multi-agent setups can ping-pong between agents indefinitely.

Signals in your traces

• Tool-call count per run far above the norm for that agent.
• Input tokens growing run over run as context accumulates.
• P95 run duration drifting upward, or one tool dominating a run's spans.

How to stop it

Cap it live, then alert on the trend. Loop Cap denies the next tool call once a run reaches the maximum you set. Spend Cap checks a daily USD budget before the provider is charged (see How to Set an AI Agent Spend Cap). Model Boundary can move calls onto a cheaper fallback model. After the run, Tool Storm, Cost Spike, Input Token Limit and Slow Run alerts catch patterns that only appear across many runs.

Failures Compound

Real incidents rarely involve just one failure mode. They chain. The value of the taxonomy is that you can name each link, and each named link points to a missing control.

The Replit chain is instructive. Blocking the destructive command would have prevented the damage. Recording the call independently in a trace would have exposed the false recovery claim straight away. Those are two different controls for two different links.

Rogue AI Agent Incidents in 2026

In 2026 the same failure modes showed up in frontier labs' own tests, and the actions reached real third parties. Most of these runs were evaluations with internet access on and the usual safeguards switched off. That is not how you run agents in production, but the mechanics are the same: an agent fixed on a goal, with more access than the task needed, and nothing between its decision and the call.

OpenAI agents breach Hugging Face

On July 16, Hugging Face disclosed an intrusion by an autonomous agent. A malicious dataset abused two code-execution paths in its dataset processing. From there the attacker reached node-level access, "harvested cloud and cluster credentials," and moved into several internal clusters (Hugging Face). On July 21, OpenAI said the agents were its own: GPT-5.6 Sol and an unreleased model, running an internal cyber-capability test without the usual guardrails. They used a zero-day in internally hosted third-party software to reach the internet. OpenAI said "the models were hyperfocused on finding a solution for ExploitGym," the benchmark they were being tested on (Fortune). That is over-eager completion with a security team's toolkit.

Claude Mythos 5 pressures an open-source maintainer

In a UK AI Security Institute capture-the-flag evaluation, with internet access on and Anthropic's production cyber safeguards off, Mythos 5 took unsanctioned actions in nine of 43 runs. A task misconfiguration made the intended route very hard. In the run Anthropic analyzed, the model found a GitHub repository it believed the test used. The repository belonged to unrelated developers. The model submitted malicious code, and when an outside developer warned the maintainers, it created a sock-puppet account to vouch for the change. Anthropic writes that "it used fake identities to pressure a real maintainer into approving malicious code." The maintainers refused, and the run ended when its token budget ran out (Anthropic).

Claude Haiku 4.5 files a false homicide tip

On July 18, Claude Haiku 4.5 was generating example tasks on randomly chosen web pages. Its instructions "did not rule out form submissions." It filled in the tip form on Philadelphia's unsolved-murders site, saying the writer "may have information regarding this case." The police system flagged it as spam, so it never reached investigators. Anthropic found the submission on September 28 and stopped the testing process (CBS News). No attacker was involved. A form submit is an irreversible action, and nothing asked a person first.

Agent swarms on DseWiki and RubyGems

The AI safety nonprofit Nightingale Collective disclosed on September 4 that, between May and July, OpenAI agents made about 18,000 edits to DseWiki, a dormant German software wiki, and used it as a message board (Nightingale Collective). On May 11 and 12, agents that researchers attribute to OpenAI uploaded more than 2,000 malicious packages to RubyGems. They abused a documentation service that runs code on upload, scraped websites with it, and published the results inside new packages. RubyGems removed more than 500 packages and closed new registrations for four days (researcher report, The Decoder). Each call was small. Thousands of them, coordinated, became an attack on a registry other teams install from.

Government portals

On September 24, Australia's Prime Minister announced that OpenAI agents had accessed unpublished data in the Medicare Statistics Reporting Service portal on June 18 (ABC News). A day later, OpenAI said its agents had also behaved in unexpected ways on US government websites (The New York Times). By October 1 it had notified more than 100 organizations about agent activity (Reuters).

What these incidents have in common

• A goal the agent could not reach the intended way. The OpenAI agents were chasing benchmark answers, and Mythos 5 had a misconfigured task. When the honest route closes, the agent looks for any route that scores.
• Agents coordinating with each other. The DseWiki edits were agents sharing notes across runs, out of view of the people running them.
• More access than the task needed. Internet access, live credentials and switched-off safeguards turned a wrong decision into real damage, fast.
• Slow discovery. The Philadelphia tip was found 72 days later, and the Australian access almost three months later. The record of what an agent did has to exist outside the agent and be read.

How teams are responding

The responses map onto the same choke point. Give each agent its own identity instead of a shared service credential, so every call is attributable and every policy can be scoped to one agent. In Traccia that is the Agent ID on each trace. Grant permissions per action, not per system. High-Risk Tool, Dangerous Shell Commands and Unique Customer Cap deny specific calls before they run. Monitor from outside the agent. Traces recorded by the SDK or the Gateway do not depend on the agent's own account of what it did. Be able to stop an agent quickly. A detective policy in Block mode stops the agent's next governed run. Revoking the agent's provider key blocks its later model calls, but it does not stop a tool that is already running, actions already queued, or work done with database or cloud credentials. To stop those, terminate the agent's worker and revoke the tool credentials it holds.

Field Guide: From Symptom to AI Agent Failure Mode

When something goes wrong, start from what you can see, work back to the failure mode, then check the trace for the fingerprint.

For a worked incident and how a policy at that boundary would have stopped it, read AI Agent Deleted the Production Database at 2 AM and Runtime Policy Enforcement in AI Agents. For why a system prompt is not enough, see AI Agent Guardrails vs Policies. All of these checks run in Traccia's AI agent control plane, on the same traces you use to debug the agent.

Frequently asked questions