Run Experiments From Code

Install the SDK, point it at a dataset, call evaluate(), and open the saved run under Evaluate → Experiments.

This walkthrough saves a graded experiment from Python or TypeScript. The same runs appear next to Prompt Playground dataset runs. For signatures, scorers, and local-only mode, see Evaluate in the SDK.

Step 1: Install The SDK

bash
pip install traccia

Step 2: Set Your Workspace Key

Create a workspace API key in the app under Settings → API Keys. Export it, or pass it to init.

bash
export TRACCIA_API_KEY=tr_…

Step 3: Call evaluate()

Point data at a platform dataset name (or UUID), pass a task that returns an output, and list scorers. The run is saved as an experiment unless you set persist off.

eval_run.py
python
from traccia import init, evaluate, load_prompt


init()


prompt = load_prompt("support-reply", label="production")


def task(inp):
    return call_model(prompt.compile(**inp))


result = evaluate(
    "support-reply-v3",
    data="support-golden",
    task=task,
    scorers=["exact_match"],
    prompt="support-reply",
)
print(result.summary())
print(result.url)

No Dataset Yet

Pass inline rows instead of a dataset name. With persist on, Traccia still saves a full experiment. Those helper datasets are hidden on Evaluate → Datasets unless you turn on Show SDK-Created.

Step 4: Open The Experiment

  • Follow result.url from the SDK, or open Evaluate → Experiments in the app.
  • Review pass rate, per-row outputs, and Open Trace when a cell has a trace.
  • Compare this run with another on the same dataset. See Compare Two Experiments.

Next Steps

Run Evals In CI

Stop a merge when the eval misses the bar.

Evaluate in the SDK

Data modes, scorers, defaults, and the full evaluate() reference.

Compare Two Experiments

Pick a baseline and a candidate on the same dataset.

Experiments

Open, compare, and attach a run when you promote.

Prompts in the SDK

Load and compile a versioned prompt inside your task.

© 2026 Traccia.