Suites and thresholds

A suite is a named set of checks. Each check names a scorer, usually a dataset version and split, an aggregation, a comparator and a threshold, and whether it is required. A suite's content lives in versions. A draft can be edited; once frozen for review, a version's content never changes, every reference in it is pinned to an exact version and digest, and the whole is digested. The suite's gate is its active version, and a version becomes active only through a governed activation.

Status (25 October 2026). The evaluations API exists in the application (suites, evaluations, experiments, reports, gates and the peer endpoints are routed) but the Suites, Experiments and Reports pages are not yet in the shell, and requirements EV-031 to EV-038, EV-045, EV-048 and EV-054 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.

A check

Field	Meaning
check_key	stable id, lowercase letters, digits, _ . -
required	a required check decides the verdict; an optional one is reported only (a frozen version needs at least one check and at least one required check)
scorer_ref, scorer_version	the scorer; empty version means its active version, pinned when the draft is frozen
dataset_ref, dataset_version, split, slice_ref	the examples; empty version means the latest frozen one; split is development, validation or holdout
aggregation	pass_rate (default), mean, min or all_pass over unique examples
comparator, threshold	gte (default), gt, lte, lt, eq; a pass rate is between 0 and 1; all_pass is always 1
min_samples, min_coverage	at least this many scored examples (default 1) and this share of the expected ones (default 1): below either, the check cannot pass
case_limit	at most this many examples (2,000 by default, 10,000 at most)
slices	up to 50 slice criteria, each with its own aggregation, comparator, threshold, min_samples and required; a failed required slice fails the check

A version holds up to 100 checks. A check without a dataset evaluates the subject itself (an executable scorer on a repository change).

A version's other settings

• Decision rule: absolute (each check against its threshold), non_inferiority (against a baseline, with a margin declared before any result is seen) or superiority; alpha is 0.05 by default (between 0 and 0.5); correction is none, holm or benjamini_hochberg across the family of required checks and slices; sequential_plan is none or obrien_fleming with 2–10 looks, and needs a comparative rule.
• Modes the version applies to: offline by default; online, simulation, release_gate (see modes).
• validity_hours of a report (168 by default), repeat_count per example (1–20), timeout_ms of an evaluation (one hour by default, at most seven days) and an optional budget_tokens bound that a request can lower but never raise.

Draft, freeze, shadow, canary

draft → review → shadow → canary → active → retired

A suite has one draft at a time; a new draft starts empty or from any earlier version. Saving a draft replaces its content; saving anything else is 409 VERSION_FROZEN. Freezing (suite.write) resolves every reference: each scorer to an exact version id, digest, kind and direction, each dataset to a frozen version id and digest. It fails closed when the scorers or datasets component cannot answer, when a scorer has no active version, or when a dataset has no frozen version. A check on a holdout split is refused when the dataset version has leakage findings (HOLDOUT_CONTAMINATED); see slices and leakage and versions and splits. The canonical content (evalbook.suite_version.v1) is digested and the version enters review.

A frozen version in review, shadow or canary can be run by naming it explicitly; it never gates. config.activate moves a version to shadow or canary and retires a version that is not the gate.

Governed activation

Asking to activate (config.activate) hands a governed action to the integrations component: activate_suite_version, or change_threshold when the candidate differs from the active version only in thresholds and comparators. The request names the person asking and the version's authors by their SSO identity. When the project is linked to a MadeBook workspace, MadeBook decides; otherwise the standalone separation-of-duties rule applies. EvalBook never authorizes. See MadeBook.

Activation happens only when the approved action's digest equals the version's digest and the rows still hash to it. In one transaction the previous gate is retired, the new version becomes active, open requests are marked activated, still-valid reports issued under the previous version are superseded by policy, and suite.activated is announced to linked peers. A retired version can be made active again the same way.

API

Session routes under evaluations/api/suites/…: list, show, create, update, version, draft, saveDraft, freeze, move, activate, diff (field-level differences between two versions). A project API key reads GET /api/v1/suites. CodeBook reads a project's suites, their active version, required checks and modes through POST /project/api/peer/suites; see CodeBook.