A scorer is how Traccia decides pass or fail when you run a prompt across a dataset. The dataset row holds the variables and (usually) the expected answer. The scorer is the rule that compares the model reply to that expected answer, or judges quality another way. You build a library of scorers once, then reuse them every time you run an experiment.

Where To Open It

In the app, go to Evaluate → Scorers. Create custom scorers there, then attach them under Prompt Playground → Run On A Dataset. Open How Scoring Works on the Scorers page for a short worked example.

What A Scorer Is (And Is Not)

It grades. After the model runs on a dataset row, the scorer produces pass/fail (and often a numeric score and a short reason).
It does not store answers. Expected output lives on each dataset item. Change the quiz without rewriting the scorer.
Past runs stay honest. When an experiment runs, Traccia snapshots the scorer type and settings into each score. Editing a scorer later does not rewrite old experiments.

Built-In Scorers

Built-ins are always available. They cannot be edited or deleted. Use both case modes in the same experiment when you want to compare strict and relaxed matching.

Contains Expected Text and Contains (Case Sensitive): pass if the expected string appears anywhere in the model reply. Best default for chat and support answers that paraphrase.
Exact Match To Expected and Exact Match (Case Sensitive): pass only if the reply equals the expected output. Prefer this for fixed labels, codes, or single-token answers.
Output Is Valid JSON: pass if the reply is a JSON object or array. Does not use expected output.

Quick Pick

For most chat prompts, start with Contains Expected Text. Exact Match fails on harmless wording differences.

LLM As Judge

An LLM-as-judge scorer calls a judge model with your grading prompt. Use this when string matching is too brittle (helpfulness, faithfulness, concise answers, and similar quality checks).

Placeholders: {{input}}, {{output}}, {{expected}}
Ask the judge to end with a clear line: label: pass or label: fail
On New Scorer, starter templates (for example Answer Correctness, Helpfulness) give you a ready prompt you can edit

Judge prompt

text

You grade a customer support reply.

Question:
{{input}}

Reply to grade:
{{output}}

Reference answer (may be empty):
{{expected}}

Pass if the reply is accurate, answers the question, and stays polite.
Give one short sentence of reasoning, then end with exactly one line:
label: pass
or
label: fail

Provider Keys

Judge runs use the same browser Provider Keys as Prompt Playground. Open Provider Keys on the Scorers page, or Keysin the Playground toolbar. Keys stay in your browser and are never stored on Traccia servers. You need a key for the judge model's provider before you run. Experiment results show the judge model name next to LLM-as-judge scorers.

When available, judge latency and cost appear on experiment score chips alongside the panel model's cost.

Custom Code

A code scorer is the body of score(output, expected, input). You write only the body. Traccia wraps it safely.

Return one of:

True / False
A float score (pass if ≥ 0.5)
A dict with passed, score, and optional reason

Code scorer body

python

# Pass when the expected phrase appears in the reply
text = str(output or "").lower()
needle = str(expected or "").strip().lower()

if not needle:
    return {"passed": False, "score": 0.0, "reason": "missing_expected_output"}

ok = needle in text
return {
    "passed": ok,
    "score": 1.0 if ok else 0.0,
    "reason": "ok" if ok else "not_found",
}

No imports, file access, or network
Unsafe constructs are rejected when you save
Runs time out after a few seconds so a bad loop cannot hang a batch

Edit, Duplicate, And History

Custom scorers (LLM-as-judge and code) support Edit, Duplicate, and Delete. Built-ins stay fixed. Duplicate when you want a fork without changing the original. Because experiments snapshot config at run time, you can tighten a judge prompt tomorrow without rewriting yesterday's evidence.

Run On A Dataset

Scoring happens from Prompt Playground. The panel supplies the prompt and model. The dataset supplies the variables per row.

Open Prompt Playground. Load a saved prompt or write one in the panel. Pick the model and add Provider Keys for that model.
Expand Run On A Dataset. Choose a dataset and optionally limit to the first N items.
Select scorers (built-in and/or custom), or check Save Outputs Without Grading to store raw outputs only.
Click Run And Save Experiment. Progress updates while the run finishes. Open the saved experiment from the success banner or Evaluate → Experiments.

Panel Variables Are Not Used On Dataset Runs

Values you type in the panel's variable fields apply to Run All (a single try). Dataset runs always fill {{…}} from each row's input. Keep expected answers on the dataset items, not in the panel.

Reading Experiment Results

Overall pass rate: a cell passes only if every attached scorer passes for that row
By Scorer: pass rate per scorer; LLM judges also show the judge model name
Per row: input, expected, model output, score chips, latency, and cost
The Experiments list shows which scorers each run used so you can find evidence quickly when promoting

Next Steps

Datasets

Build the quiz your prompts take.

Experiments

Saved graded runs used as promote evidence.

Prompt Playground

Compare prompts and run on a dataset.