Run Evals In CI

Catch quality regressions in review, the same way unit tests catch broken code. On every pull request, CI runs your task against a saved dataset, grades the outputs with your scorers, saves the run as an experiment, and fails the check when the result is below the bar you set.

1. Task. A function in your repository that takes a dataset row's input and returns your app's output.
2. CLI. traccia eval (Python) or traccia-ts eval (TypeScript) runs the task over the dataset, scores it, and saves an experiment.
3. Exit code. 0 means the gate passed. Anything else fails the job. Details land in summary.json.

Quickstart

You need a dataset and a scorer saved in the app, and a workspace API key from Settings → API Keys. A builtin scorer such as exact_match is enough to start.

# evals/support_agent.py
from my_app import answer  # your application code


def task(input):
    # input is the dataset row's input, for example {"question": "..."}
    return answer(input["question"])

export TRACCIA_API_KEY=...

traccia eval evals/support_agent.py \
  --dataset support-golden \
  --scorer exact_match \
  --min-pass-rate 0.9 \
  --output summary.json

echo $?   # 0 = pass, 1 = below the bar

# .github/workflows/evals.yml
name: Evals
on:
  pull_request:

permissions:
  contents: read

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    concurrency:
      group: evals-${{ github.event.pull_request.number }}
      cancel-in-progress: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt   # includes traccia

      - name: Run evals
        env:
          TRACCIA_API_KEY: ${{ secrets.TRACCIA_API_KEY }}
          # Optional: record the commit and pull request on the experiment.
          TRACCIA_GIT_GITHUB_SHA: ${{ github.sha }}
          TRACCIA_GIT_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
          TRACCIA_GIT_HEAD_REF: ${{ github.head_ref }}
          TRACCIA_GIT_BASE_REF: ${{ github.base_ref }}
          TRACCIA_PR_NUMBER: ${{ github.event.pull_request.number }}
          TRACCIA_GIT_REPOSITORY: ${{ github.repository }}
          TRACCIA_GIT_ACTOR: ${{ github.actor }}
        run: |
          export TRACCIA_GIT_CHECKOUT_SHA="$(git rev-parse HEAD)"
          traccia eval evals/support_agent.py \
            --dataset support-golden \
            --scorer exact_match \
            --min-pass-rate 0.9 \
            --output summary.json

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-summary
          path: summary.json

Choose A Gate

Pick what should block a merge. You can combine gates; the check fails if any of them fails.

The bar applies to each scorer on its own. A Jev Decision scorer counts each question as its own scorer, so one failing question fails the bar even when the other questions pass. To record a scorer without gating on it, pass it to --informational as well as --scorer. Platform LLM-judge and Jev scorers read provider keys from the environment (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY or GOOGLE_API_KEY, GROQ_API_KEY, TYPESAFE_API_KEY). TRACCIA_PROVIDER_KEYS can be a JSON object of provider name to key. The CLI forwards whichever of these are set and does not print them.

Compare Against A Baseline

The regression, cost, and latency gates compare the pull request with a baseline experiment. The usual baseline is the experiment you attached when you promoted the prompt that is in production now. Set it up once:

1. Run the CLI on your main branch with --prompt support-reply --prompt-label production (or --prompt-version 7) and no gate flags. This tags the experiment with the prompt version it tested.
2. In the app, open the prompt, choose Promote To Production, and attach that experiment.
3. In the pull request workflow, add --baseline-prompt support-reply and --max-drop.

traccia eval evals/support_agent.py \
  --dataset support-golden \
  --scorer exact_match \
  --scorer faithfulness \
  --prompt support-reply --prompt-label production \
  --baseline-prompt support-reply \
  --max-drop 0.05 \
  --min-pass-rate 0.8 \
  --output summary.json

Every promotion that attaches an experiment moves the baseline forward. Promoting without one clears it, and the next gated run fails with baseline_missing. To compare with one specific run instead, pass --baseline-experiment <id>. If that experiment ran more than one prompt version side by side, add --baseline-panel <n> to pick one (zero-based).

Read The Results

The CLI prints progress and the final label to stderr. --output writes summary.json, which is the stable machine-readable result (schema_version: 1). Use the process exit code to pass or fail the job. Do not parse the label. A regression looks like this (trimmed):

{
  "schema_version": 1,
  "exit_code": 1,
  "code": "regression",
  "label": "Failed",
  "quality_gate": false,
  "failures": [
    { "code": "regression", "message": "exact_match dropped more than allowed" }
  ],
  "gates": { "max_drop": 0.05, "baseline_experiment_id": "8c1d…" },
  "experiment_id": "0f6e2a54…",
  "url": "https://app.traccia.ai/…",
  "dataset_fingerprint": "a41f9c0e…",
  "dataset_verified": true,
  "scorers": [
    {
      "name": "exact_match",
      "expected": 50, "completed": 50, "missing": 0, "errored": 0,
      "passes": 42,
      "pass_rate": 0.84,
      "baseline_pass_rate": 0.9,
      "informational": false
    }
  ]
}

pass_rate is null when any row is missing or a scorer errored, so a partial run never looks like a low score. Open url to see every row, its trace, and its score next to the baseline.

Exit Codes

Problems with the evidence are reported before quality, so a crash is never mistaken for a regression.

The full list of codes is in the CLI reference.

Post Results On The Pull Request

The Traccia Eval Action is optional. It runs your command in the environment you already installed, records the git metadata for you, refuses a stale or inconsistent summary.json, and keeps one comment per job up to date on the pull request. The step still exits with the CLI's exit code.

❌ **Traccia Eval failed** · `regression` (exit 1)

| Scorer | Pass rate | Baseline | Change | Graded |
|---|---|---|---|---|
| `exact_match` | 84.0% | 90.0% | −6.0 pts | 50/50 |

**Failures**
- `regression` exact_match dropped more than allowed

**Gates:** Max drop 5 pts · Baseline `8c1d77e2-5b0`

[Open experiment](…) · [View run](…) · commit `4b7e19c`

permissions:
  contents: read
  pull-requests: write   # lets the action post the comment

# ...same job as the quickstart, with the "Run evals" step replaced by:
      - id: eval
        uses: traccia-ai/traccia-eval-action@v1
        with:
          command: >-
            traccia eval evals/support_agent.py
            --dataset support-golden --scorer exact_match
            --min-pass-rate 0.9 --output summary.json
        env:
          TRACCIA_API_KEY: ${{ secrets.TRACCIA_API_KEY }}

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-summary
          path: ${{ steps.eval.outputs.summary_path }}

Outputs, set even when the step fails: exit_code, code, label, quality_gate, experiment_id, experiment_url, summary_path, and comment_posted. More workflows, including a matrix and a monorepo setup, are in the Action README.

The eval command cannot read github_token. The default token updates the same comment on later pushes. A custom GitHub App token posts a new comment on each run.

Other CI Systems

The CLI works anywhere Python or Node runs. Set TRACCIA_API_KEY, run the command, and let its exit code fail the job. To show the commit and merge request on the experiment, set the TRACCIA_GIT_* variables from your CI's own variables. GitLab:

# .gitlab-ci.yml
evals:
  image: python:3.12
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  variables:
    TRACCIA_GIT_CHECKOUT_SHA: $CI_COMMIT_SHA
    TRACCIA_GIT_HEAD_SHA: $CI_COMMIT_SHA
    TRACCIA_GIT_HEAD_REF: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
    TRACCIA_GIT_BASE_REF: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
    TRACCIA_PR_NUMBER: $CI_MERGE_REQUEST_IID
    TRACCIA_GIT_REPOSITORY: $CI_PROJECT_PATH
    TRACCIA_GIT_ACTOR: $GITLAB_USER_LOGIN
  script:
    - pip install -r requirements.txt
    # TRACCIA_API_KEY is a masked CI/CD variable in project settings.
    - traccia eval evals/support_agent.py --dataset support-golden
        --scorer exact_match --min-pass-rate 0.9 --output summary.json
  artifacts:
    when: always
    paths: [summary.json]

Scorers In Your Repository

For a check that only makes sense in code, define the scorer next to the task and select it with --scorer local:name. The name is the key in the exported scorers map.

def task(input):
    return answer(input["question"])


def mentions_refund(output, expected, **kwargs):
    # Return True/False, a 0-1 score, or {"passed": ..., "score": ...}
    return "refund" in str(output).lower()


scorers = {"mentions_refund": mentions_refund}

Python scorers are called with keyword arguments input, output, expected, and metadata, so accept **kwargs. TypeScript scorers receive one object with the same fields. Local scorers work with --min-pass-rate but not --max-drop.

All Flags

The flags are the same for traccia eval and traccia-ts eval.

Troubleshooting

• The job fails on fork pull requests. GitHub does not give secrets to forks, so the API key is empty. Skip the job with if: github.event.pull_request.head.repo.full_name == github.repository. Do not switch to pull_request_target, which would run untrusted code with your secrets.
• traccia-ts: command not found. Run it through npx --no-install traccia-ts or an npm script, after npm ci.
• A TypeScript task file won't load. Use a .mjs or .js file. Node 22.18 and later can also load .ts files directly.
• baseline_missing on every run. The production version of the prompt has no attached experiment. Promote it again and attach an experiment from step 1 of the baseline setup.
• latency_concurrency_mismatch. The baseline ran at a different concurrency, for example from the Prompt Playground. Use a CLI run as the baseline, or pass --allow-concurrency-mismatch.
• The pass rate is lower than in the app. The gate leaves errored rows out of the pass rate and fails the run instead. Check errored and missing in summary.json.

What A Green Check Guarantees

• Every row in the dataset snapshot ran and was graded exactly once. Missing, duplicate, or errored results fail the check.
• Rows that change while the run is in progress fail the check with dataset_drift.
• Scorers saved in the app are frozen when the run starts, so an edit mid-run does not change the grades.
• Cost and latency gates pass only with a complete measurement on both sides. Unknown cost never counts as zero.
• The experiment was saved, so the link in the summary always opens.