Chaos Engineering for AI Agents: Break It Before Production Does

Why Chaos Engineering Matters for AI Agents

Traditional software usually has explicit error-handling paths. A request times out, an exception is raised, and application code decides whether to retry, return an error or fall back.

An AI agent adds another decision-maker to that path: the model. A tool failure is often turned into a message the model can read, and the model decides what to do next. It might retry, try another tool, ask the user, or produce an answer without the missing information. That flexibility is useful, and it makes failure behavior harder to predict.

The key difference: an agent may keep going after a dependency fails. A service returning an error is visible. An agent returning a fluent answer built on missing or wrong information is much harder to spot.

The practice comes from distributed systems. In December 2010, Netflix described Chaos Monkey, a tool that randomly shuts down its own production instances so engineers build services that survive it. The Principles of Chaos Engineering later turned that into a method: define a measurable steady state, predict that it will hold, introduce real-world events such as crashed servers or malformed responses, and look for a difference. Agents need the same method, aimed at different dependencies.

The principles eventually call for experiments in production, with the blast radius kept small. For agents that can move money or change records, start in staging: define the expected behavior, introduce one fault at a time, and stop the experiment if it goes past its safety limits.

How Agents Fail: A Practical Taxonomy

A useful experiment suite covers more than whether a dependency is online.

These categories overlap. A malformed response can become a security issue; a timeout becomes a consistency issue when the operation has side effects. Prioritize experiments with three questions:

1. Impact: what happens if the agent gets this wrong?
2. Likelihood: how often can this dependency fail or return unexpected data?
3. Recoverability: can the system safely retry, reconcile or fall back?

A read-only search tool and a payment API should not share a retry policy. Repeating a search is cheap. Repeating a transaction may not be. For a catalog of how agents go wrong in production, see AI Agent Failure Modes.

The Chaos Experiment Loop

Every experiment in this guide follows the same process. Traccia helps with observing and analyzing; the fault injector is code you own.

1. Define the hypothesis. State the behavior you expect. For example: "If the order service is unavailable, the agent tells the user it cannot retrieve the order and makes no more than three attempts."
2. Set boundaries. Staging, test credentials, synthetic data and a clear stop condition. Never inject faults into real payment or customer workflows.
3. Inject one fault. Change one dependency or response at a time, so the cause of the behavior is clear.
4. Observe the execution. In Traccia, read the tool spans, model calls, errors, duration, call count and final response.
5. Evaluate the result. Did the behavior match the hypothesis? A plausible final answer is not enough if the underlying operation failed.
6. Keep the experiment. Turn repeatable cases into regression tests and add them to the right CI stage.

A minimal fault-injection wrapper

This is illustrative application code, not a Traccia feature. It adds a few controlled faults around a tool call. Adapt the return types and exception handling to your framework.

import os
import time
from traccia import init, govern, observe

init(agent_id="orders-agent")

CHAOS_FAULT = os.getenv("CHAOS_FAULT", "")


def inject_fault(real_call):
    if CHAOS_FAULT == "offline":
        raise ConnectionError("chaos: orders service unavailable")

    if CHAOS_FAULT == "slow":
        time.sleep(8)

    result = real_call()

    if CHAOS_FAULT == "malformed":
        # Example for a tool whose contract returns JSON text.
        # Do not use this unchanged for tools that return Python objects.
        return '{"order_id": "1042", "status":'

    return result


@observe(name="lookup_order", as_type="tool")
def lookup_order(order_id: str):
    return inject_fault(
        lambda: orders_mcp.call("get_order", {"id": order_id})
    )


@govern(name="orders_agent")
def run_agent(question: str) -> str:
    # Your framework's agent loop calls lookup_order as a tool.
    ...

Two details matter:

• as_type="tool" makes the observed function a tool span. Policies that count or inspect tool calls, such as Loop Cap, look for tool spans.
• The fault is raised inside the observed function, so the error is recorded on that span: error status, error.type, error.message and a stack trace.

For tests that run concurrently, do not change a process-wide environment variable per test case. Pass the fault through an isolated test fixture, a dependency wrapper or per-test configuration.

Experiment 1: An MCP Server Goes Offline

• Hypothesis: if the orders MCP server becomes unavailable, the agent tells the user it cannot retrieve the order, does not invent an order status, and stops within three attempts.
• Inject: make the MCP server unavailable in staging, or configure a test double that raises a connection error on every request.

What to inspect in Traccia

• Does each failed call appear as a tool span with an error?
• How many times did the agent call the tool?
• Did it eventually stop, or keep trying?
• Does the final answer say clearly that the order could not be retrieved?

What a failure might look like

An illustrative example. Imagine the trace has seven failed lookup_order spans, and the final model response says: "Your order shipped on Tuesday and should arrive Friday." No tool call succeeded. The answer is unsupported, even though the run status is Success.

The experiment shows two separate problems:

1. Unbounded retries: the agent kept calling a dependency that was down.
2. Unsupported completion: the agent wrote an answer without the data it needed.

What to change

• Return a structured, readable tool error instead of a raw stack trace. A clear instruction in the error ("Order system unavailable. Do not retry. Tell the user.") gives the model something to act on.
• Tell the agent to stop, or to use a defined fallback, when the dependency is unavailable.
• Enforce a hard tool-call limit that does not depend on the model (Loop Cap).
• Add the scenario to a regression dataset and score whether the final response honestly reports the failure.

An MCP-specific distinction

The MCP specification (version 2026-07-28) separates two kinds of tool error:

• Protocol errors come back as standard JSON-RPC errors: an unknown tool, a malformed request, or a server error.
• Tool execution errors come back as a normal tool result with isError: true: an API failure, an input validation error, or a business logic error. The spec says clients should pass these to the model so it can correct itself and retry with different arguments.

A server that is down gives you neither. The request fails at the transport, before any MCP response arrives, and your client library raises its own error.

That matters when you instrument a tool. Your client can successfully receive a result even though the tool itself failed. If your wrapper returns that result as-is, the span ends without an error and the failure is only visible in the payload. If you want it counted as an error, raise inside the observed function when isError is set, after you have the message the model needs.

Experiment 2: A Tool Returns Bad Data

An available tool is not necessarily a reliable one. It can return syntactically valid but wrong data, omit required fields, or return content the agent should not trust.

• Hypothesis: when a tool returns invalid, inconsistent or untrusted data, the agent detects it, avoids unsupported claims and follows a defined recovery path.
• Inject: use a mock tool or response fixture that returns one of these variants, one at a time.

What to inspect in Traccia

Bad data usually does not raise an exception, so the tool span looks successful. The problem is in the content. Read the tool span's output next to the next model call's input and output:

• Did the model receive the unexpected content?
• Did it follow instructions embedded in the tool response?
• Did it use fields that were missing or invalid?
• Did the next model call's input tokens jump after an oversized response?

The MCP specification makes the same recommendation: clients should validate tool results before passing them to the model. Where you can, validate tool output in code before it reaches the model, and raise (or return a structured validation failure) when the response breaks the tool contract. That turns a silent problem into an error span you can count and alert on.

Where a policy can help

For the stale-record case, Freshness Guard can enforce the rule before the agent acts on it. You name a read tool, a write tool and a maximum age. If the last matching read is older than the limit, the write is denied. The read tool has to return a timestamp (as_of or updated_at); if it does not, the rule is skipped and the match is recorded as a warning. Traccia does not refresh the data or retry the write for you.

For security-sensitive cases, do not rely only on the model to recognize malicious or cross-tenant content. Enforce authorization and validation in the application and tool layer.

Experiment 3: Latency and Provider Failures

A dependency does not need to go fully down to disrupt an agent. A provider that responds slowly, rate-limits or returns intermittent 5xx errors can cause long runs and repeated retries.

• Hypothesis: when the provider is slow or temporarily unavailable, the agent retries with bounded backoff, uses an approved fallback if one exists, or returns a clear failure. Total run time stays within a defined budget.
• Inject: a controlled delay on a tool or a proxy in front of the provider; simulated HTTP 429 and intermittent 5xx responses; a test endpoint that refuses connections; or a stream that ends early.

Use a mock or controlled proxy where possible. Do not deliberately exhaust a real provider's rate limit or disrupt shared infrastructure.

What to inspect in Traccia

• Run duration: the traces page filters by status (success, error, timeout) and sorts by duration, slowest first.
• Retry behavior: count repeated model or tool spans and check the gaps between them. No gap means no backoff.
• Fallback behavior: confirm the fallback model or alternate path actually appears in the trace.
• Cost: check whether retries or fallback calls raised the cost of the task.

Slow Run Alert watches how long runs take over a window (hourly, daily or weekly) and opens a violation when the aggregate goes past your limit. The template starts at P95 above 5,000 ms, checked hourly; you can switch to P99, average or max. It looks back at finished runs, so it cannot cut off a run that is still going. Its job is to tell you the latency you injected in staging is now happening for real.

Define a latency budget

One timeout value is rarely enough for a multi-step agent. Set budgets at each level:

The agent should not spend its whole deadline retrying a dependency when it still needs time to return a useful failure message.

Experiment 4: A Tool Succeeds, but Its Response Is Lost

This is one of the most important failure modes for agents that take actions. Consider an agent that issues a refund:

1. The agent calls the payment service.
2. The payment service processes the refund.
3. The response is lost or times out before reaching the agent.
4. The agent reads the timeout as failure.
5. It submits the refund again.

The first operation succeeded. The agent does not know that it did.

• Hypothesis: if the response to a side-effecting operation is lost, the agent does not blindly repeat it. It checks the operation's status before deciding what to do next.
• Inject: in staging, use a payment-service mock that records the operation as successful but drops or delays the response. Never run this against real payment credentials or live customer transactions.

What to inspect

• Did the agent retry the operation?
• Did it send the retry with the same request ID, where the payment API detects duplicates?
• Did it query the transaction status before trying again?
• Can it tell failed, succeeded and outcome unknown apart?
• Does the final response avoid claiming success or failure without evidence?

The recovery pattern

Treat an ambiguous outcome differently from a confirmed failure:

1. Send the operation with a stable request ID, if the payment API detects duplicates.
2. If the response times out, query the operation's status using that request ID or the provider's transaction id.
3. Retry only when you can establish that it is safe.
4. If the outcome cannot be established, stop and escalate to a person.

Duplicate protection has to live at the boundary between your application and the payment service. A retry limit alone cannot prevent duplicate side effects.

Where Traccia fits

The trace lets an engineer reconstruct the sequence: the attempted operation, the timeout, any status lookup, the retry and the final response. That is the evidence you need to decide whether the agent made an unsafe decision or the external service returned an ambiguous result.

For high-impact actions, Traccia can also put a limit or a person in front of the tool. Refund Guard reads the refund amount from the tool arguments before the function runs and denies the call above your cutoff. With Ask For Approval on, refunds between a lower limit and the cutoff wait in the Approvals inbox, and the SDK raises ApprovalPending. That error means waiting, not failed: catch it, do not run the tool, and do not retry the check. Approving does not run the refund; your own server does that. Purchase Guard does the same for purchases.

Experiment 5: The Policy Service Is Unreachable

If an agent checks a policy before a model or tool call, the policy service is a dependency too. You need to decide what happens when that check cannot be completed.

In Traccia, a policy check that gets no usable answer (a timeout, a connection error, a 5xx, or a reply the SDK cannot use) follows your policy fail mode, set under Settings → Policy Enforcement. The default is to block: the SDK raises AgentBlockedError and the model or tool is not called. You can choose Allow instead, or override it in code with policy_fail_mode="open" or "closed" on init() or @govern. This needs Python SDK 0.1.34 or TypeScript SDK 0.1.20 or later; older versions let the call through.

Some answers stop the call whatever the fail mode, because they are not outages: a rejected API key (401 or 403), a block or deny from a policy, and a missing API key, endpoint or agent id, which raises GovernanceConfigError.

• Hypothesis: if Traccia cannot be reached, the governed call follows the fail mode you expect, and the application returns a controlled message rather than an unhandled exception.
• Inject: in staging, make Traccia unreachable for the agent. Do not remove the endpoint or API key: that tests a configuration error, not an outage.

How you cut Traccia off decides which fail mode you are testing. The SDK learns your Settings choice from answers it already gets (the agent status at the start of each governed run, and each policy check) and keeps it for 10 minutes per API key and endpoint. A process that has never had an answer, for example one started while Traccia is down, blocks whatever Settings says.

• To test a local override: pass policy_fail_mode explicitly and point the agent at an unreachable endpoint, as in the example below. The process never gets an answer, so only the override decides.
• To test your Settings choice: let the agent connect successfully first, then cut the network to Traccia within 10 minutes, for example with a firewall rule or a proxy that starts refusing connections.

from traccia import init, govern
from traccia.governance import AgentBlockedError

init(
    agent_id="orders-agent",
    endpoint="https://traccia.unreachable.invalid/v2/traces",
    # This process never gets an answer, so it never learns the
    # Settings fail mode. Set the mode you are testing explicitly.
    policy_fail_mode="closed",
)

@govern(name="orders_agent")
def run_agent(question: str) -> str:
    # Your governed agent run.
    ...

try:
    run_agent("Where is order 1042?")
except AgentBlockedError:
    print("The operation is temporarily unavailable.")

Choose the fail mode deliberately

Neither is right for every agent. It depends on what the agent can do and what an unchecked action would cost.

Recovery Patterns: Retries, Limits and Fallbacks

The experiments above keep surfacing the same recovery problems. A reliable agent needs explicit rules for what to do after a failure.

Bounded retries

A prompt can tell a model to retry at most twice. That is useful guidance, not a guarantee. Enforce retry and tool-call limits outside the model.

Loop Cap sets the most tool calls allowed in one governed run. The next call after that is denied; the earlier calls have already happened. It runs in the SDK inside govern() and counts tool-typed observe wrappers. It does not apply on the Gateway, where a completion is one model call rather than a loop of tools.

Rerun Experiment 1 with Loop Cap on Block. Check that the run stops at the limit and that your application handles the resulting AgentBlockedError safely.

Detecting retry storms

Tool Storm Alert is the after-the-fact counterpart. It checks each trace as it lands and opens a violation when one run makes more tool calls than your limit. The template starts above 25. Use Loop Cap to stop excessive calls live; use Tool Storm Alert to find runs you did not think to test for.

Control spend during failure

Retries cost tokens, so a failing workflow can also be an expensive one. If you use Spend Cap, include it in your failure experiments and confirm it trips where you expect.

Test every fallback

A fallback that has never run is an assumption. For each fallback (another model, cached data, a human handoff, a graceful failure), write an experiment that forces the primary path to fail. Check the trace for the span that proves the fallback ran, then check the final response says what happened.

Make side-effecting operations safe

• Stable request IDs, where the external API detects duplicates.
• Status reconciliation after ambiguous outcomes.
• Authorization checks before the operation.
• Human approval for high-impact actions (in Traccia, Ask For Approval on Refund Guard or Purchase Guard).
• Audit records that link the agent run to the external operation.

A tool-call limit does not replace these. It limits repeated calls; it cannot tell whether an earlier call already changed something.

Making Resilience Tests Part of CI

A chaos experiment that runs once is useful. One that runs every time the relevant code changes prevents regressions.

traccia eval runs your task against a saved dataset, grades the outputs with scorers, and exits non-zero when results fall below the bar you set. That turns deterministic fault scenarios into gates.

For the empty-retrieval case, Retrieval Completeness can also enforce it at runtime: it denies the model call when the trace has no retrieval span, or when the chunk count is below a minimum you set. Your search span sends traccia.retrieval.present and, if you count them, traccia.retrieval.chunk_count. It checks that a search happened and returned enough; it does not judge whether the chunks were the right ones.

Example evaluation task

A simplified illustration. Match the input shape to your dataset.

# evals/chaos_agent.py

from my_app import run_agent_with_fault

def task(input):
    question = input["question"]
    fault = input.get("fault", "")

    # Your test harness injects the fault in an isolated environment
    # and returns the user-visible agent response.
    return run_agent_with_fault(question, fault=fault)

traccia eval evals/chaos_agent.py \
  --dataset chaos-orders \
  --scorer contains \
  --min-pass-rate 0.9 \
  --output summary.json

For simple expected phrases, the built-in contains scorer works. For structured outputs, use json_valid or a scorer that checks the required fields. For recovery behavior, consider a custom scorer that checks whether the agent took the right action, not just whether its reply contains a phrase.

Use different testing cadences

Useful CI gates

These values are examples. Set them from the risk and normal range of your own application. The cost and latency gates suit chaos runs well: a change that makes recovery slower or more expensive (more retries, a heavier fallback) fails the check even when the answers are still right.

Each CI run is saved as an experiment, so you can compare it with the baseline side by side.

For the full GitHub Actions workflow and exit codes, see the CI evals guide.

What Traccia Does and Does Not Do

Chaos engineering needs two things: a way to cause failures, and evidence of what happened.

Observability shows where a workflow went wrong. Making it resilient is engineering work. Start in staging, use synthetic data and keep a rollback plan. Do not run destructive experiments against production systems to see what happens. For how preventive and detective policies fit together, see Runtime Policy Enforcement for AI Agents.

Frequently Asked Questions