Compare Two Experiments

Platform

Pick a baseline and a candidate on the same dataset, then read score, cost, and latency deltas before you promote.

Compare tells you whether a prompt or model change actually helped. Both runs must use the same dataset. You can create those runs in Prompt Playground or from code with evaluate(). For the full compare surface, see Experiments.

Step 1: Save Two Runs

  • Run the current production prompt on a golden dataset (Playground Run On A Dataset, or evaluate() with prompt= / prompt:).
  • Change the prompt, model, or task, and run again on that same dataset.

Step 2: Open Compare

  1. Go to Evaluate → Experiments (or the dataset Experiments tab).
  2. Select exactly two runs, then click Compare. The older run is the baseline by default; swap anytime on the compare page.
  3. From an experiment detail page, use Compare with… instead.

Multi-Panel Runs

If either run used more than one Prompt Playground panel, choose which panels to pair. Switching panels stays on the same page and refreshes the metrics for that pair.

Step 3: Read The Summary

Start with the overall grade: Improvement, Regression, Tradeoff, or Tie. Then check pass-rate delta, cost delta, and mean latency delta. Aggregates are for the selected panel pair only.

Filter the per-row table to Regressions, Improvements, Cost Up, or Any Fail. Expand a row to see both outputs side by side.

Next Steps

Experiments

Saved runs, Open Trace, and evidence to attach on promote.

Run Experiments From Code

Call evaluate() and open the saved run in the app.

Prompt Playground

Run On A Dataset from the UI, then compare the saved experiments.

Evaluate in the SDK

Data modes, scorers, and the evaluate() reference.

© 2026 Traccia.