evaluate() runs a task and scorers over a dataset from Python or TypeScript. Pass a platform dataset or inline rows, a task that produces an output, and the scorers to grade with. By default the run is saved as an experiment you can open, compare, and attach when you promote. Set persist=False / persist: false to keep the loop local with no platform write.

Saved runs appear under Evaluate → Experiments. Prompt Playground Run On A Dataset saves the same kind of experiment, so SDK and Playground runs sit on the same list and can be compared. For a walkthrough, see Run Experiments From Code. To compare two saved runs, see Compare Two Experiments.

Prerequisites

Use a workspace API key from Settings → API Keys (the same key as tracing and load_prompt). Set TRACCIA_API_KEY or pass api_key / apiKey. Platform datasets and scorers live under Datasets and Scorers.

Quick Start

eval_run.py

python

from traccia import init, evaluate, load_prompt

init(api_key="tr_…")

prompt = load_prompt("support-reply", label="production")

def task(inp):
    messages = prompt.compile(**inp)
    return call_model(messages)

result = evaluate(
    "support-reply-v3",
    data="support-golden",       # platform dataset name or id
    task=task,
    scorers=["exact_match"],     # builtins, platform names/ids, or callables
    prompt="support-reply",      # stamps prompt version on the experiment
    max_concurrency=10,
    persist=True,                # default
)
print(result.summary())
print(result.url)

Python is a keyword-only call after the experiment name. TypeScript takes one options object. Both also work as Traccia.evaluate(...) on the TypeScript namespace.

Data Modes

Platform dataset: pass the dataset name or UUID string. Items are fetched over the API (input, expected_output, metadata).
Local rows + persist (default when you pass a list): inline objects { input, expected, metadata }. Traccia creates an ephemeral dataset named sdk-eval/<name>/<short-id> so the experiment appears in the UI like any other run. Those datasets are hidden from Evaluate → Datasets by default. Turn on Show SDK-Created to list them, or open the dataset from the experiment.
Local-only: the same inline rows with persist off. Results stay in process. There is no experiment URL and no ephemeral dataset.

Row Shape

Prefer an explicit input object plus expected (alias expected_output). If input is omitted, the whole dict is treated as the input object. input must be an object, not a string or array. An empty dataset or empty list raises EvaluateError.

The Task Function

The task maps each row to an output. Outputs should be JSON-serializable (string, number, bool, dict/list). Non-serializable values are stringified.

TypeScript always receives the row's input object: task(input). Async functions are awaited.
Python inspects the first parameter name. If it is row, item, example, or case, the full row is passed (id, input, expected_output, metadata). Any other name (including input) receives only the input dict. Async tasks are supported.

The experiment cell label is Task unless you pass prompt= / prompt:, in which case the prompt name is used. That is how SDK runs differ from Playground panels in the experiment table.

Scorers

Mix builtins, platform library scorers, and inline callables in one list. Each scorer runs in its own child span scorer.<name>.

Builtins (in-process): exact_match, contains, json_valid. Use these snake_case ids, not the Playground display names. exact_match and contains compare against the row's expected output (case-insensitive unless a platform scorer config sets case_sensitive). json_valid checks that the output is a JSON object or array.
Platform scorers by name or UUID. If the library type is a builtin, it still runs locally (including that scorer's config). LLM-as-judge and code scorers run on the server. Pass provider_keys / providerKeys for judges (keys such as openai, anthropic, gemini, groq).
Inline callables: Python functions take keyword args input, output, expected, metadata. TypeScript functions take one object with those keys. Return a score dict, a number (pass if ≥ 0.5), or a bool.

A row passes only when every scorer on that row passes. Scorer failures are isolated: a throwing scorer becomes passed: false with reason scorer_error: … and the rest of the run continues.

What Happens By Default

The run is saved as an experiment (persist=True / persist: true). Pass persist=False / persist: false to skip the save.
Up to 10 items run at a time (max_concurrency=10 / maxConcurrency: 10).
Per-item errors are isolated. A throwing task records error on that cell, adds an entry to result.errors, and continues. Configuration and API failures raise EvaluateError.
Progress prints N/M to stderr. Disable with progress=False / progress: false. Optional on_item_complete / onItemComplete is (done, total, row).
If the save fails after scoring, in-memory rows remain and persist_error / persistError is set. url stays empty.

Run Against A Platform Dataset

Fetch a named dataset, run your production prompt, grade with a builtin.

python

from traccia import evaluate, load_prompt

prompt = load_prompt("support-reply", label="production")

def task(inp):
    return call_model(prompt.compile(**inp))

result = evaluate(
    "support-golden-prod",
    data="support-edge-cases-csv",
    task=task,
    scorers=["contains"],
    prompt="support-reply",
)
assert result.url
print(result.aggregates["pass_rate"])

Inline Rows, Saved As An Experiment

No dataset in the UI yet. Inline cases still become a full experiment via an ephemeral sdk-eval/... dataset.

python

from traccia import evaluate

rows = [
    {"input": {"question": "Where is my order?"}, "expected": "tracking"},
    {"input": {"question": "I want a refund."}, "expected": "refund"},
]

result = evaluate(
    "support-inline",
    data=rows,
    task=lambda inp: call_model(inp["question"]),
    scorers=["contains"],
)
print(result.dataset_id)  # ephemeral sdk-eval/support-inline/<id>
print(result.url)

Local-Only (No Experiment)

Iterate in CI or a notebook without writing an experiment. No URL, no ephemeral dataset.

python

from traccia import evaluate

result = evaluate(
    "local-check",
    data=[{"input": {"q": "ping"}, "expected": "pong"}],
    task=lambda inp: "pong" if inp["q"] == "ping" else "nope",
    scorers=["exact_match"],
    persist=False,
)
print(result.url)           # None
print(result.experiment_id) # None
print(result.summary())

Grade With An LLM Judge

Use a scorer you created under Evaluate → Scorers. Pass BYO keys for the judge model. The judge span is stamped as an LLM child (model, tokens, cost) when the score API returns them.

python

from traccia import evaluate, load_prompt
import os

prompt = load_prompt("support-reply", label="production")

result = evaluate(
    "support-helpfulness",
    data="support-golden",
    task=lambda inp: call_model(prompt.compile(**inp)),
    scorers=["Helpfulness"],  # platform scorer name or UUID
    prompt="support-reply",
    provider_keys={"gemini": os.environ["GEMINI_API_KEY"]},
)
print(result.url)

Grade With A Platform Code Scorer

A code scorer in the library is the body of score(output, expected, input). Reference it by name or id. It runs on the server, not in your process.

library_body.py

python

# Body you save on Evaluate → Scorers (type: code)
return expected is not None and str(expected).lower() in str(output or "").lower()

python

result = evaluate(
    "support-code-scorer",
    data="support-golden",
    task=task,
    scorers=["test-custom-code"],
)

Grade With An Inline Function

python

def mentions_refund(*, input, output, expected, metadata):
    text = str(output or "").lower()
    return {"name": "mentions_refund", "passed": "refund" in text, "score": 1.0 if "refund" in text else 0.0}

result = evaluate(
    "callable-scorer",
    data=[{"input": {"question": "I want my money back."}, "expected": None}],
    task=task,
    scorers=[mentions_refund, "json_valid"],
    persist=False,
)

One Row Fails, The Run Continues

One throwing row does not abort the run. Other items still score. The failed cell shows the error (not a blank output mixed into scorer reasons).

python

def task(inp):
    if inp.get("fail"):
        raise RuntimeError("model timeout")
    return "ok"

result = evaluate(
    "isolation-check",
    data=[
        {"input": {"fail": False}, "expected": "ok"},
        {"input": {"fail": True}, "expected": "ok"},
        {"input": {"fail": False}, "expected": "ok"},
    ],
    task=task,
    scorers=["exact_match"],
)
print(len(result.rows))     # 3
print(result.errors)        # one item_id + "model timeout"
print(result.aggregates["error_count"])

Result Object

Python returns EvaluateResult. TypeScript returns the same class. Field names differ on the wrapper; row payloads stay snake_case in both SDKs.

Attribute	Description
`name`	Experiment name you passed.
`rows`	One object per item: item_id, input, expected_output, panels[].
`aggregates`	item_count, panel_count, scorer_count, pass_count, scored_count, pass_rate, error_count, source=evaluate; optional mean_latency_ms and total_cost_usd.
`url`	Experiment URL when persist succeeded. Otherwise empty.
`experiment_id / experimentId`	UUID of the saved experiment.
`dataset_id / datasetId`	Platform or ephemeral dataset UUID.
`errors`	[{ item_id, error }] for throwing tasks.
`persist_error / persistError`	Set if scoring finished but the experiment save failed.
`summary()`	Human-readable pass rate, item count, errors, and URL.

Each panels[0] cell includes output, error, scores, passed, latency_ms, optional cost_usd, trace_id, and label (Task or the prompt name). On the experiment page, Open Trace uses that trace_id.

Experiment Traces

Each item is wrapped in an evaluate.item span. Scorers are children. If the task itself emits an LLM span (for example @observe(as_type="llm")), it nests under the item. Playground batch runs emit the same identity attributes with traccia.eval.source=playground_batch.

evaluate() will call init() if tracing is not already started, so Open Trace links resolve. HTTP calls to /api/v1/eval-runtime/ are not attached as spans.

Attribute	Description
`traccia.experiment.id`	Experiment UUID for this evaluate or Playground batch run.
`traccia.experiment.name`	Human-readable experiment name.
`traccia.eval.source`	`evaluate` or `playground_batch`.
`traccia.dataset.id`	Dataset UUID when known.
`traccia.dataset.item_id`	Dataset item UUID for this cell.

Prompt Metrics Exclude Eval Traffic

Prompt detail Metrics counts live traced LLM calls for that prompt. Traces that carry traccia.eval.source (SDK evaluate and Prompt Playground batch) are excluded so eval Groq/Gemini calls do not inflate production totals.

Python vs TypeScript

Python evaluate() is synchronous (thread pool). TypeScript evaluate() is async and must be awaited.
Wrapper fields: Python experiment_id, dataset_id, persist_error, max_concurrency, provider_keys, on_item_complete. TypeScript uses camelCase for those options and wrapper fields.
Task argument: TypeScript always gets input. Python may get the full row when the first parameter is named row / item / example / case.
Inline scorers: Python keyword arguments; TypeScript a single object.

What This Does Not Cover

evaluate() is offline, in your process (plus server score for judge/code). It does not ship CI threshold gates, online sampled production evals, human labeling, or post-hoc scoring of an existing experiment. Playground dataset runs stay the UI path for interactive compare panels.

Reference

evaluate (Python)

python

def evaluate(
    name: str,
    *,
    data: str | Sequence[dict],
    task: Callable,
    scorers: Sequence[str | Callable] | None = None,
    prompt: str | None = None,
    max_concurrency: int = 10,
    persist: bool = True,
    provider_keys: dict[str, str] | None = None,
    api_key: str | None = None,
    prompt_api_base: str | None = None,
    progress: bool = True,
    on_item_complete: Callable[[int, int, dict], None] | None = None,
) -> EvaluateResult

evaluate (TypeScript)

typescript

async function evaluate(opts: {
  name: string;
  data: string | Array<Record<string, unknown>>;
  task: (input: any) => unknown | Promise<unknown>;
  scorers?: Array<string | ((args: {
    input: unknown;
    output: unknown;
    expected?: unknown;
    metadata?: unknown;
  }) => unknown | Promise<unknown>)>;
  prompt?: string;
  maxConcurrency?: number;   // default 10
  persist?: boolean;         // default true
  providerKeys?: Record<string, string>;
  apiKey?: string;
  promptApiBase?: string;
  progress?: boolean;        // default true
  onItemComplete?: (done: number, total: number, row: Record<string, unknown>) => void;
}): Promise<EvaluateResult>

Errors

Missing name, non-callable task, max_concurrency < 1, bad data type, empty items, and dataset/scorer API failures raise EvaluateError. See Error Reference and the Python / TypeScript API pages.

Next Steps

Run Experiments From Code

Install, pass a dataset, call evaluate(), and open the run in the app.

Compare Two Experiments

Pick a baseline and a candidate on the same dataset.

Experiments

Open, compare, and promote with experiment evidence.

Scorers

Built-ins, LLM-as-judge, and code scorers in the library.

Prompts in the SDK

Load and compile versioned prompts inside your task.

Python SDK API

evaluate() signature, EvaluateResult, and EvaluateError.

TypeScript SDK API

evaluate() options, EvaluateResult, and EvaluateError.

Evaluate in the SDK

Quick Start

Data Modes

The Task Function

Scorers

What Happens By Default

Run Against A Platform Dataset

Inline Rows, Saved As An Experiment

Local-Only (No Experiment)

Grade With An LLM Judge

Grade With A Platform Code Scorer

Grade With An Inline Function

One Row Fails, The Run Continues

Result Object

Experiment Traces

Python vs TypeScript

What This Does Not Cover

Reference

evaluate (Python)

evaluate (TypeScript)

Errors

Next Steps

Run Experiments From Code

Compare Two Experiments

Experiments

Scorers

Prompts in the SDK

Python SDK API

TypeScript SDK API