RESEARCH / AGENT SYSTEMS · ENGINEERING PLAYBOOK
AI Agent Evaluation: From Transcripts to Execution-Based Evidence
Agent evaluation begins with an oracle: what observable state would prove the task was completed correctly, within policy, and at an acceptable cost?
Key takeaways
- Grade the environment state first. Use the final response as one signal, not the source of truth.
- Inspect trajectories selectively for policy, efficiency, recovery, and diagnosis. Do not require one "ideal" path when several valid paths exist.
- Run repeated trials. A strong best-case result can coexist with weak reliability.
- Evaluate the complete configured system. Test the model, harness, tools, prompts, permissions, and environment together.
- Turn production failures into regression tasks so the evaluation suite improves alongside the product.
The hardest part of agent evaluation is not producing a score. It is defining valid evidence. The evaluator needs an oracle for success, a model of allowed behavior, and a sampling design that distinguishes repeatable capability from luck.
Start with the oracle, not the transcript
An oracle is the rule or evidence that determines whether a task succeeded. For agents, the oracle often lives outside the conversation: a database state, compiled artifact, test result, access-control record, or reconciled ledger.
An agent receives a task, observes an environment, chooses actions, calls tools, modifies state, handles errors, and eventually stops. The transcript records that process, but it does not define success by itself. A procurement agent might produce a valid purchase summary while creating a duplicate order. A research agent can cite sources it never opened. A coding agent can repair one failing test while breaking another.
This is why interactive benchmarks moved beyond static prompts. AgentBench evaluates reasoning and decision-making across eight environments. WebArena checks functional task completion on realistic websites. OSWorld initializes real computer states and applies execution-based evaluators to resulting artifacts and application state.
Write an evaluation contract before writing graders
A useful agent evaluation begins with an explicit contract: objective, starting state, allowed actions, success state, budget, and failure conditions. Without that contract, a score has no stable meaning.
The contract prevents a common mistake: encoding unstated assumptions in hidden tests. If the agent is asked to create a report but the evaluator silently expects a particular filename, the task may measure guesswork rather than capability. Anthropic's engineering guidance recommends reference solutions and unambiguous tasks that independent domain experts can grade consistently.
Define the comparison unit too. The configured system is not only the base model. It includes the system prompt, tool descriptions, context policy, memory, retry logic, stopping rule, permissions, and execution environment. Recent Harness-Bench results reinforce this point: model-harness configurations can differ in completion, efficiency, and failure behavior even under shared tasks.
Use four evidence layers, in order of authority
The strongest evidence is usually the resulting state, followed by produced artifacts, observable behavior, and finally the agent's own account. This ordering helps teams avoid trusting fluent claims over verifiable facts.
| Evidence layer | Question | Preferred checks | Typical failure caught |
|---|---|---|---|
| 1. Outcome | Did the intended state exist? | Database assertion, API read-back, unit test, file inspection | "Booked" without a reservation |
| 2. Artifact | Is the deliverable usable? | Schema validation, compilation, render test, content coverage | CSV created with wrong rows |
| 3. Trajectory | Was execution acceptable? | Tool and argument checks, policy assertions, loop detection | Correct result after an unauthorized write |
| 4. Narrative | Was the response clear and honest? | Rubric, groundedness, human review | Overclaiming or missing caveats |
Outcome-first grading also leaves room for creative solutions. OSWorld explicitly uses execution-based reward functions because open-ended computer tasks can have multiple correct action sequences. Prescribing one path can punish competent agents for solving the task differently.
Trajectory grading still matters when the path carries risk or diagnostic value. Inspect it when a task has mandatory approvals, destructive operations, privacy constraints, cost ceilings, or recovery requirements. Use it to explain a failure. Do not turn every stylistic choice into a brittle assertion.
Build a grader stack, not a single judge
No one grader can measure correctness, usefulness, safety, and efficiency across every agent task. Combine graders according to the claim each one can support.
SWE-bench illustrates deterministic outcome grading: the patch must make failing tests pass without breaking the existing suite. Research-agent evaluation needs a different mix, such as claim-to-source checks, required-fact coverage, source-quality rules, and expert review of synthesis. The method follows the artifact.
Calibrate model graders before trusting them
Model graders can be useful, but they are measurement systems with known failure modes. Zheng et al. documented position, verbosity, and self-enhancement biases in LLM judges. Wang et al. showed that changing response order could alter rankings and proposed balanced-position and human-in-the-loop calibration.
Use a narrow rubric with observable criteria. Blind model identity. Swap pairwise response order and compare results. Label a stratified calibration set with domain experts, measure judge-human agreement by task slice, and route high-impact disagreements to adjudication. Re-run calibration whenever the judge model, prompt, scale, or evidence bundle changes.
Design tasks from the deployment distribution
A representative task set is more valuable than a large synthetic set that measures the wrong work. Start with product requirements, manual test cases, support tickets, incident reports, and sampled production traces.
Separate capability from regression
A capability suite asks what the agent can do at the frontier of its current ability. Low pass rates are useful because they expose a hill to climb. A regression suite protects behavior that already works and should remain near the required reliability target. As capability tasks become routine, promote them into regression.
Balance positive and negative cases
Test when an action should occur and when it should not. An agent trained against only "must search" examples may learn to search everything. A refund suite needs valid refunds, prohibited refunds, ambiguous requests, missing evidence, escalation cases, and attempts to bypass policy.
Prove every task is solvable
Store a reference solution or human-verified execution. Reset the environment between trials. Track task version, environment version, graders, model, harness, tool versions, and budgets. If the environment leaks state between trials, the measured runs are not independent.
Estimate reliability as a distribution
For stochastic agents and tasks where reliability matters, use repeated trials and uncertainty-aware reporting. Report per-task trial counts, success rates, confidence intervals where practical, and the aggregation rule. Destructive or prohibitively expensive tests may need risk-based sampling rather than uniform repetition.
tau-bench compares final database state with an annotated goal state and introduced pass^k for consistency. Its reported retail results included pass^8 < 25%, showing how a plausible single-run score can hide unreliable repetition.
| Metric | What it asks | Use when | Interpretation |
|---|---|---|---|
| pass@k | Did at least one of k trials succeed? | Multiple attempts are acceptable | Measures best-of-k opportunity |
| pass^k | Did all k trials succeed? | Users expect consistent behavior | Measures repeatable reliability |
| Cost per outcome | What did success actually cost? | Models retry or use variable tools | Connects quality to economics |
| Time horizon | How does success vary with task length? | Long-running technical agents | Relates reliability to human duration |
Measure utility and safety on the same tasks
A safe agent must resist harmful instructions without becoming useless on legitimate work. Report benign-task success and attack success separately. A single blended score can hide a system that blocks everything or completes tasks while violating boundaries.
AgentDojo created 97 realistic tasks and 629 security test cases for agents operating over untrusted data. Its structure captures the core production tension: an agent must complete the user's task while ignoring injected instructions embedded in emails, websites, documents, or tool output.
Minimum adversarial suite
- Indirect prompt injection in retrieved content
- Requests outside the user's permissions
- Tool arguments that exfiltrate or overwrite data
- Conflicting instructions across turns
- Pressure to claim success after a failed action
- Infeasible tasks where the correct action is to stop or escalate
Test side effects, not only words. Verify that secrets remained protected, restricted tools were not called, approval gates were honored, and irreversible actions were not taken. Run destructive tests in resettable sandboxes with explicit network and credential boundaries.
Connect offline evals to production evidence
Offline evaluation supports controlled comparison; production evaluation discovers failures the test set did not anticipate. The two systems should feed each other.
Before release, run fixed datasets to compare prompts, models, tools, and harness changes. In production, sample traces, monitor outcome proxies, inspect incidents, collect explicit user feedback, and route ambiguous cases to human review. Convert confirmed failures into versioned regression tasks.
For agents, the production loop depends on replayable state. Preserve the initial-state fixture, tool responses, permission context, environment version, and final-state evidence needed to reproduce a run. When live dependencies cannot be replayed exactly, label the simulator's fidelity limits and keep sandbox failures separate from agent failures.
Use public benchmarks as instruments, not product scorecards
Choose benchmarks by the behavior they operationalize, then validate against your own tasks. A leaderboard result is evidence about a particular task distribution, harness, budget, and grader.
AgentBench
Broad interactive reasoning and decision-making across eight environments.
Does not establish production reliability in a specific workflow.WebArena
Realistic web tasks with functional correctness.
Self-hosted sites simplify parts of the open web and its changing state.OSWorld
Desktop and cross-app tasks with execution-based evaluation.
Setup, GUI variability, and application versions add operational noise.SWE-bench
Repository-level issue resolution graded by tests.
Task quality, contamination, tests, and harness choices affect validity.tau-bench
User interaction, tools, policy following, final state, and reliability.
The original benchmark uses simulated users and two initial domains.AgentDojo
Utility under prompt-injection attacks over untrusted data.
Attack coverage must evolve as agents and adversaries change.AgentBoard
Fine-grained progress signals for multi-turn agents.
Progress definitions remain tied to each environment.METR time horizon
Success probability conditioned on human task duration.
Domain and reference-human choices limit broader inference.The original WebArena paper reported 14.41% end-to-end success for its best GPT-4-based baseline versus 78.24% for humans. OSWorld reported 12.24% for its strongest evaluated baseline versus more than 72.36% for humans. Those numbers show the difficulty of the original benchmark settings, not the state of the art. Always attach results to the model version, harness, date, budget, and benchmark revision.
A ten-step workflow for an evaluation you can trust
Define the product decision
State what the result will change: release approval, model selection, prompt revision, tool redesign, or incident triage.
Write the evaluation contract
Specify objective, initial state, permissions, expected state, budget, and failure conditions.
Collect representative tasks
Start with requirements, manual checks, production failures, and high-impact edge cases.
Build reference executions
Prove tasks are solvable and that graders accept a known-correct outcome.
Implement outcome graders first
Use state read-back, tests, artifact inspection, and explicit invariants.
Add targeted trajectory assertions
Check mandatory approvals, prohibited calls, argument validity, recovery, and stopping.
Calibrate subjective graders
Compare model-based scores with blinded human labels and examine disagreements by slice.
Run repeated, isolated trials
Reset state, record seeds and versions where available, and report reliability rather than a single pass.
Slice failures
Break results down by task family, tool, risk, length, user segment, and failure stage.
Gate releases & learn from prod
Set explicit thresholds, sample live behavior, and turn confirmed incidents into regressions.
Evaluate the evaluation platform with a bake-off
Do not select an evaluation platform from a feature table alone. Run the same instrumented agent, task set, graders, and failure cases through each shortlisted option.
The Arize comparison is a useful map of categories such as traces, datasets, experiments, online evaluation, human review, and CI/CD. It is also vendor-authored. Treat every platform claim as a hypothesis to test.
- Can it represent spans, complete runs, trajectories, and multi-turn sessions?
- Can one evaluator run offline and on sampled production traces?
- Are task, dataset, prompt, grader, model, and harness versions preserved?
- Can reviewers inspect raw evidence behind every score?
- Can production failures become datasets without losing provenance?
- Can release gates express uncertainty, slices, and minimum sample sizes?
- Can data retention, access control, and deployment meet governance needs?
- Can the team export traces, labels, datasets, and evaluation results?
- Can it reset sandboxes deterministically and replay initial and final state?
- Can it separate agent failures from simulator, tool, and infrastructure failures?
The best platform is the one that shortens the path from a production failure to a reproducible task, a diagnosed cause, and a verified fix.
Agent evaluation FAQ
What is the difference between LLM evaluation and agent evaluation?
Should every trajectory have an ideal reference path?
How many trials should an agent evaluation run?
Before declaring that an agent is better, ask
- Did the evaluation verify the real outcome rather than the agent's claim?
- Were the environment, model-harness configuration, permissions, and budget versioned?
- Were repeated trials or an explicit risk-based sampling design used?
- Were utility, safety, cost, and reliability reported separately?
- Can another team reproduce the run and inspect the graders' evidence?
- Will confirmed production failures become regression tasks?
References
- Xiao Liu et al. "AgentBench: Evaluating LLMs as Agents." 2023. arXiv:2308.03688. (https://arxiv.org/abs/2308.03688)
- Shuyan Zhou et al. "WebArena: A Realistic Web Environment for Building Autonomous Agents." 2023. arXiv:2307.13854. (https://arxiv.org/abs/2307.13854)
- Carlos E. Jimenez et al. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" 2023. arXiv:2310.06770. (https://arxiv.org/abs/2310.06770)
- Chang Ma et al. "AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents." 2024. arXiv:2401.13178. (https://arxiv.org/abs/2401.13178)
- Tianbao Xie et al. "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments." 2024. arXiv:2404.07972. (https://arxiv.org/abs/2404.07972)
- Shunyu Yao et al. "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." 2024. arXiv:2406.12045. (https://arxiv.org/abs/2406.12045)
- Edoardo Debenedetti et al. "AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents." 2024. arXiv:2406.13352. (https://arxiv.org/abs/2406.13352)
- Thomas Kwa et al. "Measuring AI Ability to Complete Long Software Tasks." 2025. arXiv:2503.14499. (https://arxiv.org/abs/2503.14499)
- Yilun Yao et al. "Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows." 2026. arXiv:2605.27922. (https://arxiv.org/abs/2605.27922)
- Lianmin Zheng et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." 2023. arXiv:2306.05685. (https://arxiv.org/abs/2306.05685)
- Peiyi Wang et al. "Large Language Models are not Fair Evaluators." 2023. arXiv:2305.17926. (https://arxiv.org/abs/2305.17926)
- Anthropic. "Demystifying evals for AI agents." 2026. (https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
- Arize. "Compare 7 LLM Evaluation Platforms (2026): Agents, Tracing & CI/CD." 2026. (https://arize.com/resources/llm-and-agent-evaluation-platforms/)
Research note: Historical benchmark results are labeled as results from their original papers. They should not be interpreted as current leaderboard standings.
Evaluate AI Agents with Ground-Truth Verifiers
Move beyond text-only rubrics. Validate real state changes, tool execution trajectories, and multi-run reliability distributions with Traccia.