Registry and versions
Scorers are named checks whose behaviour lives in immutable, digested versions that move from draft to review, shadow and, through a governed activation, active.
A scorer turns one case — an output, a trace or an exact commit — into a verdict. The project's registry lists every scorer with its kind, its latest version and its active version. The scorer itself is only a header (name, description, kind); what it does is written in versions, and a version is never edited. Changing a check means writing the next version.
The registry is the Scorers page of a project, with tabs for Scorers, Rubrics and Runners. Reading it needs scorer.read; writing scorers, versions, fixtures and rubrics needs scorer.write; requesting an activation needs config.activate. See roles and access.
Status (25 October 2026). The scorers component is built in the application (registry, versions, rubrics, calibration and runners pages are live in the shell); its requirements EV-019 to EV-028 are not yet marked implemented in the requirements ledger and are being verified now.
Scorers and kinds
A scorer has one of four kinds, fixed when it is created:
| Kind | What a version does | Assurance of its results |
|---|---|---|
deterministic |
runs in-process on the recorded output or trace (deterministic scorers) | independently_observed |
llm_judge |
a model grades one case through Bookbag SSO (LLM judges) | reported |
pairwise_judge |
a model compares a candidate with a baseline (pairwise) | reported |
executable |
a command in a pinned image, run by a runner against an exact commit (executable verification) | platform_verified with the container driver |
Archiving a scorer needs its exact name typed; an archived scorer stays readable, its versions still identify past scores, and it takes no new versions until it is restored.
Versions
A version is created in draft from a body that names the implementation, the score type (binary, ordinal, continuous or categorical), the range and direction, a pass threshold where the score type has one, a time limit, what the case must carry (output, expected, trace, subject, repository), parameters, rules and applicability conditions. Defaults are written into the version, so a later change of a default in the code cannot change what a stored version does.
Every version carries a digest: sha256 over the canonical JSON of its whole configuration, recomputed from the stored rows on every read. A version whose rows no longer match its digest scores nothing and cannot move forward. A new version can start from an earlier one, which copies its fixtures, and carries a change note; the scorer page compares any two versions field by field.
Applicability and required inputs
An applicability condition names a field under example, output, trace or subject and one of equals, not_equals, in, contains, exists, not_exists. A case outside the conditions is skipped, with the condition named. A case that lacks something the version requires is error, never a pass.
Result states
Every result is one of pass, fail, error, timeout, skipped, unsupported, abstained or inconclusive; an executable scorer may first answer pending while a runner holds the job. Only pass satisfies a required check. Each result names the version it came from (scorer_version_id, scorer_version_digest) and carries evidence entries and an assurance. The explanation never quotes the case, so anyone with scorer.read may see it; a judge's rationale and body-derived details are shown only to people with dataset.read_body.
Lifecycle and governed activation
draft → review → shadow → active → retired
Scorer authors move a version between draft, review and shadow, and retire it. Nobody moves a version to active directly. From review or shadow, a release admin requests activation: a governed action (activate_scorer_version, or activate_judge_configuration for judges) bound to the version's digest. When the project is linked to a MadeBook workspace, MadeBook decides; otherwise the standalone separation-of-duties rule applies, and the author cannot approve their own change. EvalBook never authorizes. The version becomes active only when the decision comes back approved for the same digest, re-checked against the rows; in the same transaction the previously active version is retired. Without the governance service the request fails closed and nothing changes.
An activation request is requested, pending_approval, approved, deferred, denied, activated, expired, cancelled or failed; asking again refreshes a deferred or pending decision. A judge's model must be in the organization's SSO catalog at creation and again at activation. See MadeBook.
Fixtures and the sandbox
Each version keeps up to 200 fixtures — cases of at most 256 KB each, with an expected status — and runs them, or up to 50 pasted cases, in a sandbox run. Deterministic versions run in the request; judges run as a job under an AI budget made for the run (20,000 tokens by default); executable versions create one runner job per case.
API
Session routes live under scorers/api/…. A project API key with scope read or evaluate reads GET /api/v1/scorers, GET /api/v1/scorers/show and GET /api/v1/scorers/version (a version by id, or a scorer's active version). See API keys.