Reports
A report is the immutable, digested verdict of one evaluation; its envelope says whether it is current or superseded, and a reproducibility bundle exports everything it rests on with an explicit list of what is missing.
When an evaluation finishes, EvalBook aggregates every check, writes the check results and issues one report in a single transaction. The report body is canonical JSON with a sha256 digest, stored verbatim and never changed. Supersession, expiry and the page URL live in an envelope beside the body, so the digest never changes after it is issued. A consumer verifies a report by fetching it from EvalBook and recomputing the digest; a callback never carries the verdict it acts on.
Status (25 October 2026). The evaluations API exists in the application (suites, evaluations, experiments, reports, gates and the peer endpoints are routed) but the Suites, Experiments and Reports pages are not yet in the shell, and requirements EV-031 to EV-038, EV-045, EV-048 and EV-054 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.
The body
Schema version 1.0, producer svc_evalbook. The body names the organization, the project, the report and evaluation ids, the subject exactly as it was sent, suite_ref (id, version, digest), dataset_refs, mode, execution_mode, the decision rule (rule, alpha, margin, correction), result, evidence_complete, reasons, checks, coverage over the required checks, issued_at and valid_until (issue time plus the suite version's validity hours).
Each check carries its check_id, required, status, a schema result (pass, fail, error, inconclusive, missing, stale), scorer_ref with digest, assurance, up to 20 evidence_refs (digests of the outputs, traces and runner artifacts scored; beyond 20, one manifest digest), score, explanation, coverage (expected, executed, missing), observations and slices.
Result states
A check's status is one of pass, fail, error, timeout, cancelled, skipped, unsupported, abstained, inconclusive, missing or stale. The report's result is pass, fail, error or inconclusive:
passonly when every required check passed and every required check's evidence is complete: all expected examples executed, each with digested evidence;failwhen a required check failed;errorwhen a required check could not be evaluated (an evaluator failure, never a quality result);inconclusiveotherwise, including a run that passed on what was scored but stopped early.
Optional checks are reported only. The report's assurance is the weakest of its required checks; a check's is the weakest of its cases (development_unisolated, reported, independently_observed, human_attested, platform_verified).
How a check is aggregated
An example counts as executed only when every repeat of it was scored pass or fail; errors, timeouts, skips, abstentions and anything still pending leave it unexecuted. Coverage below the check's min_coverage, or fewer scored examples than min_samples, makes the check not-pass with the status that explains it. An aggregation that has already failed (all_pass with a failure, min below a threshold) is fail whatever is missing. A failed required slice fails the check.
A single-repeat pass rate carries a Wilson score interval; a pass rate over repeats and a mean carry a seeded bootstrap interval over examples (2,000 resamples). Comparative rules use the paired methods described in experiments, with the frozen family correction; a check that passed before correction may fail after it, never the reverse. Every interval records its level and method.
Supersession and the envelope
The envelope is { status: current | superseded, superseded_by, superseded_at, superseded_reason, url }. A later gate report for the same subject digest under the same suite supersedes the earlier current ones (re_evaluation). Activating a new suite version supersedes by policy the still-valid gate reports issued under the version it replaced. A superseded report is still served by id; it no longer gates. report.superseded is announced to linked peers, who re-fetch both.
POST evaluations/api/reports/verify re-hashes a stored report; the evaluator health view flags one that no longer verifies as an incident.
Reproducibility bundles
POST evaluations/api/reports/bundle (report.export) builds a bundle (evalbook.reproducibility_bundle.v1): the report with its envelope, the frozen suite version's canonical content and digest, the pinned scorer versions as the scorers component describes them, the dataset versions with the example snapshots the run used, every check result and every case with its evidence and output digest, and two lists — unavailable (a payload stored at SSO over 256 KB, bodies removed by retention, a scorer version that no longer exists or now hashes differently, cases beyond the first 20,000) and redacted (example bodies and outputs when the exporter lacks dataset.read_body; trace bodies, which stay in Traces under the capture policy). The bundle is stored at SSO, digested and listed under reports/bundles; verify-bundle.mjs in the application recomputes every digest the bundle can prove.
Who reads reports
report.read lists and shows reports (evaluations/api/reports, GET /api/v1/evaluations/:id/report). CodeBook fetches a report through POST /project/api/peer/report and decides from it; MadeBook asks for the current reports about one commit through reports/forSubject and records each as evidence, trusting only platform_verified as observed. See CodeBook, MadeBook and release gates.