Release gates

An evaluation pins a suite version, a subject digest, a deadline and a budget at admission, runs to a report, and answers CI and linked products with a gate verdict and an exit code.

/

A release gate is an evaluation of an exact subject against a suite's active version, whose report another system decides from: a CI job through the public API, CodeBook for a station, MadeBook as evidence on a pull request. EvalBook measures and reports. It never authorizes: whether work proceeds is the consumer's decision, taken from a report it fetched and verified.

Status (25 October 2026). The evaluations API exists in the application (suites, evaluations, experiments, reports, gates and the peer endpoints are routed) but the Suites, Experiments and Reports pages are not yet in the shell, and requirements EV-031 to EV-038, EV-045, EV-048 and EV-054 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.

Admission

Starting an evaluation pins everything it will use: the suite version (the active one unless a shadow or canary version is named), and through it the exact scorer and dataset versions and the decision rule; the subject's digest; the deadline (the suite version's timeout by default, at most seven days away); and the token budget (never above the suite version's bound). A repository_change subject must carry repository_full_name, commit_sha and base_commit_sha; the subject is digested exactly as sent, so nothing may be null or unknown.

Admission is idempotent per organization, origin and Idempotency-Key: the same key with the same request digest answers the first evaluation; a different body is 409 IDEMPOTENCY_CONFLICT. A callback_url must sit under the caller's address in the peer directory.

Lifecycle and the verdict

queued → running → (cancelling) → completed | failed | cancelled

The status is the job's lifecycle; the verdict is the report's result. A plan job snapshots the pinned examples and writes one case per check, example and repeat; case jobs produce the output under test and score each case once; executable checks answer pending until their runner job is delivered; a deadline job finalizes whatever is still open as timeout. The evaluation stops early on budget_exhausted, sequential_stop, the deadline or a cancellation, and the report's reasons say so. An evaluator failure is an incident and an error result, never a quality verdict. A cancelled evaluation keeps its partial check results and issues no report.

The gate answer

GET /api/v1/evaluations/:id/gate returns { gate, exit_code, reasons } for CI:

`gate`	`exit_code`	When
`pass`	0	every required check passed with complete evidence
`fail`	1	a required check failed
`error`	2	a required check could not be evaluated, or the evaluation failed or was cancelled
`inconclusive`	3	anything else, including a superseded or expired report
`pending`	4	the evaluation has not finished

The public CI API

With a project API key of scope evaluate (experiment.run; see API keys):

POST /api/v1/evaluations          # Idempotency-Key honoured → 202 { evaluation }
GET  /api/v1/evaluations/:id      # status, progress, budget
GET  /api/v1/evaluations/:id/report
GET  /api/v1/evaluations/:id/gate
POST /api/v1/evaluations/:id/outputs   # { outputs, complete } in batches of up to 500
POST /api/v1/evaluations/:id/cancel

{ "suite_id": "…", "mode": "release_gate", "execution_mode": "repository_change",
  "subject": { "kind": "repository_change", "resource_id": "pr:42", "revision": "sha:…",
    "environment_id": "ci", "artifact_set_digest": "sha256:…",
    "repository_full_name": "acme/api", "commit_sha": "<40 hex>", "base_commit_sha": "<40 hex>" },
  "deadline": "2026-10-25T12:00:00Z", "budget": { "tokens": 200000 } }

In existing_outputs mode the outputs can come with the request or in later batches; complete: true seals the set and starts the run. The tenant is the key's project.

Linked products

CodeBook asks for an evaluation of the exact candidate through POST /project/api/peer/evaluations with mode: release_gate, over an active link, signed and keyed; EvalBook answers 202 with the subject digest, the pinned suite reference and the required check ids, sends evaluation.completed (or .failed, .cancelled) to the callback and to every linked product, and CodeBook re-fetches the report by id before it decides. MadeBook reads the current reports about one commit through reports/forSubject and counts a pass only when its assurance is platform_verified. See CodeBook and MadeBook.

release_gate mode has one rule of its own: a result from an unisolated (process-driver) runner is turned into unsupported, so it can never pass a gate (see executable verification). A new gate report for the same subject and suite supersedes the earlier current one; activating a new suite version supersedes by policy the still-valid reports issued under the old one (reports).

Evaluator health

GET evaluations/api/health answers, for the last seven days, the evaluations by status, the queue, error rates per scorer version with their denominators, flaky checks (the same subject and suite version with different results), reports that no longer verify, and the open incidents, which a project admin acknowledges and resolves. Nothing unmeasured is shown as 0. Meta-evaluation is opt-in: a project admin can send a deterministic sample of judge results (a rate of at most 0.5, a daily cap of at most 500) to a human review queue; no judge ever scores another judge.

← PreviousReports Next →Playground