Run Experiments From Code

Both

Install the SDK, point it at a dataset, call evaluate(), and open the saved run under Evaluate → Experiments.

This walkthrough saves a graded experiment from Python or TypeScript. The same runs appear next to Prompt Playground dataset runs. For signatures, scorers, and local-only mode, see Evaluate in the SDK.

Step 1: Install The SDK

bash
pip install traccia

Step 2: Set Your Workspace Key

Create a workspace API key in the app under Settings → API Keys. Export it, or pass it to init.

bash
export TRACCIA_API_KEY=tr_…

Step 3: Call evaluate()

Point data at a platform dataset name (or UUID), pass a task that returns an output, and list scorers. The run is saved as an experiment unless you set persist off.

eval_run.py
python
from traccia import init, evaluate, load_prompt


init()


prompt = load_prompt("support-reply", label="production")


def task(inp):
    return call_model(prompt.compile(**inp))


result = evaluate(
    "support-reply-v3",
    data="support-golden",
    task=task,
    scorers=["exact_match"],
    prompt="support-reply",
)
print(result.summary())
print(result.url)

No Dataset Yet

Pass inline rows instead of a dataset name. With persist on, Traccia still saves a full experiment. Those helper datasets are hidden on Evaluate → Datasets unless you turn on Show SDK-Created.

Step 4: Open The Experiment

  • Follow result.url from the SDK, or open Evaluate → Experiments in the app.
  • Review pass rate, per-row outputs, and Open Trace when a cell has a trace.
  • Compare this run with another on the same dataset. See Compare Two Experiments.

Next Steps

Evaluate in the SDK

Data modes, scorers, defaults, and the full evaluate() reference.

Compare Two Experiments

Pick a baseline and a candidate on the same dataset.

Experiments

Open, compare, and attach a run when you promote.

Prompts in the SDK

Load and compile a versioned prompt inside your task.

© 2026 Traccia.