What Is AI Agent Observability? The Complete Guide to Seeing What Your Agents Do

What Is AI Agent Observability?

The word comes from control theory: a system is observable when you can infer its internal state from its outputs. For software, it means you can answer a question you did not plan for in advance, from data you already collected.

LLM observability usually means watching single model calls: the prompt, the response, the latency, the token count. That is necessary, but an agent is not one call. It plans, picks tools, reads their output, changes course, and sometimes hands off to another agent. AI agent observability follows the whole run, so you can see not only what each call returned but why the next step happened.

Why Traditional Monitoring Falls Short for Agents

Application performance monitoring was built for deterministic services: the same request takes the same path, and a failure shows up as an error code or a slow endpoint. Agents break both assumptions.

The Signals Worth Capturing

Good agent observability is not "log everything." It is a small set of signals, attached to the right step, that let you answer real questions later. Here is what to capture, and the attribute Traccia's SDK uses for it.

Alongside spans, aggregate metrics answer fleet-wide questions such as token spend per model per day, without reading individual traces.

Anatomy of an Agent Trace

A trace is one run. A span is one step inside it, with a start time, a duration, a parent, and attributes. Nesting is what turns a pile of events into a story: the tool call and the model call sit under the agent step that made them.

With that tree, the questions from the opening story have answers. The duplicate tool call is two sibling spans. The oversized prompt is a token count on one model span. The cost of the run is the sum of the cost on its model spans.

Why OpenTelemetry Matters for Agent Observability

OpenTelemetry (OTel) is the open standard for traces, metrics, and logs, supported by every major observability backend. Building agent observability on it has three practical benefits.

How Traccia Observes AI Agents

Traccia's observability starts in your process, with the open-source traccia-py SDK. It is built on the OpenTelemetry SDK, so everything below produces standard spans and metrics. Here is what it does, step by step.

1. Initialize once, and model calls are traced

Install the SDK and call init() before your agent runs.

pip install traccia openai

from openai import OpenAI
from traccia import init

init(api_key="YOUR_TRACCIA_KEY", agent_id="support-agent", env="production")

client = OpenAI()
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Where is order ORD-1?"}],
)

That chat completion is now a span named llm.openai.chat.completions carrying the model, the prompt, the completion and its finish reason, prompt and completion token counts from the provider's usage data, and the cost in USD. With patching on, which is the default, the SDK instruments:

• OpenAI chat completions made with the synchronous Python client
• Anthropic Messages API calls, with input and output tokens, cost, and the stop reason
• Google Gemini calls through the google-genai Interactions API
• Outbound HTTP made with requests: each call gets a client span, and a W3C traceparent header carries the trace to the service you call

Framework integrations cover the agent layer:

• OpenAI Agents SDK: registered automatically when the package is installed; records agent, generation, tool, handoff, and guardrail spans with model and token usage
• CrewAI: registered automatically; records crew kickoff, task, agent, and LLM call spans
• LangChain: add Traccia's callback handler to trace LLM and chat-model runs

2. Turn your own functions into spans with @observe

Auto-instrumentation sees the model. It cannot see your agent loop or your tools. The @observe decorator makes any function a span, and nesting follows your call stack.

from openai import OpenAI
from traccia import init, observe

init(api_key="YOUR_TRACCIA_KEY", agent_id="support-agent")
client = OpenAI()

@observe(name="lookup_order", as_type="tool")
def lookup_order(order_id: str) -> dict:
    return {"order_id": order_id, "status": "shipped"}

@observe(name="support_agent")
def run_agent(question: str) -> str:
    order = lookup_order("ORD-1")
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"{question} {order}"}],
    )
    return resp.choices[0].message.content

This produces the trace in the diagram above. For each decorated function, the SDK records:

• The function's arguments as attributes, and its return value as result
• The step type from as_type: "tool", "llm", or "guardrail"
• Any exception, with error.type, error.message, and a stack trace, and the span marked as an error
• Both regular and async functions

3. Give every run an identity

Every span carries agent.id, agent.name, and env, so you can filter one agent in one environment. Set them in init() for a single-agent service, or per run with run_identity when one process serves several agents.

from traccia import init, runtime_config

# Each request becomes its own trace instead of joining one process-wide trace.
init(api_key="YOUR_TRACCIA_KEY", auto_start_trace=False)

with runtime_config.run_identity(
    agent_id="billing-agent", agent_name="Billing Agent", env="production"
):
    run_agent("Why was I charged twice?")

4. Cost on every model call

The SDK prices each model span from a bundled snapshot covering more than 2,000 models, and writes the result to llm.cost.usd. It also records where the price came from (llm.pricing.source, llm.pricing.model_key, llm.pricing.snapshot_version) and where the token counts came from: the provider's own usage data first, then a local tokenizer, then an estimate, each labeled in llm.usage.source.

You can override prices for negotiated rates, and traccia pricing refresh pulls a newer snapshot. On the platform, Traccia recomputes cost against its own up-to-date catalog, so dashboards stay current even when an agent runs an older SDK.

5. Metrics for the whole fleet

Traces explain one run. Metrics show trends across all of them. The SDK records OpenTelemetry metrics at the moment each model call completes:

Each data point carries the provider, the model, and the agent, so you can break spend down by any of them.

6. Sample traces without losing the totals

At high volume you may not want every trace. init(sample_rate=0.1) keeps one in ten traces, decided once at the root, so a kept trace is always complete and a dropped one leaves no orphan spans.

Sampling does not touch the numbers that matter for cost. Token, cost, and latency metrics are recorded inline, at the moment each model call completes, and never pass through the trace sampler. Keep 10% of traces for debugging and your spend totals still count 100% of calls.

7. Keep sensitive data out of telemetry

Prompts and tool arguments can contain personal data. Two controls run inside your process, before anything is exported:

from traccia import init, observe

init(api_key="YOUR_TRACCIA_KEY", redact_pii=True)

@observe(skip_args=["password"], skip_result=True)
def login(user: str, password: str) -> str:
    ...

• redact_pii=True masks email addresses, US phone numbers, and SSN-like patterns in prompt, completion, and message attributes, and marks the span with governance.redaction_applied. It is pattern-based, not a classifier.
• skip_args and skip_result keep chosen arguments and return values out of the span entirely. Use them for anything redaction's patterns would not catch.

8. Send it anywhere

By default the SDK exports to Traccia over OTLP/HTTP. Point it at your own collector instead, or print spans to the console while you develop. The SDK does not require a Traccia account.

from traccia import init

# Send spans and metrics to your own OpenTelemetry collector (Jaeger, Tempo, ...)
init(
    endpoint="http://localhost:4318/v1/traces",
    metrics_endpoint="http://localhost:4318/v1/metrics",
    agent_id="support-agent",
    env="dev",
)

# Or keep everything on your machine while you develop:
# init(use_otlp=False, enable_console_exporter=True, enable_metrics=False)

9. Guardrail signals in the same trace

Mark your own checks with @observe(as_type="guardrail"); a check that returns True sets guardrail.triggered. The SDK also picks up the provider's own safety signals, such as an OpenAI completion that ends with a content_filter finish reason, and flags tool calls that fail with permission errors. It collects these findings per trace and summarizes which guardrail categories it saw and which it would expect to see given what the agent did.

From Observability to Control

Observability answers what happened. It cannot stop the next run from repeating it. For that you need governance: rules checked before an action executes. The two are not separate systems; governance runs on the same trace. When a call runs under Traccia's @govern(), the policy decision is written onto the span it applies to, as traccia.policy.decision_id and traccia.policy.effect. The evidence of what was allowed or blocked sits next to the evidence of what happened, and every span also carries a governance.integrity_hash for audit evidence.

That is the idea behind an AI agent control plane: one layer that observes agents, evaluates them, governs what they may do, and proves it afterwards. For the difference between watching and stopping, see AI Observability vs AI Governance.

An AI Agent Observability Checklist

Frequently Asked Questions