Evaluations

Suites and thresholds, experiments, immutable reports, release gates, the playground, and simulation versus replay versus rerun.

• Suites and thresholdsA suite version freezes which scorer and dataset versions are run, each check's threshold and minimum coverage, and the decision rule; only a governed activation makes it the suite's gate.
• ExperimentsAn experiment runs a baseline and up to five candidate variants against the same frozen suite version as separate evaluations and compares them example by example under the decision rule frozen in that version.
• ReportsA report is the immutable, digested verdict of one evaluation; its envelope says whether it is current or superseded, and a reproducibility bundle exports everything it rests on with an explicit list of what is missing.
• Release gatesAn evaluation pins a suite version, a subject digest, a deadline and a budget at admission, runs to a report, and answers CI and linked products with a gate verdict and an exit code.
• PlaygroundA playground run is an isolated experiment of prompt and model variants, run through SSO models under a token budget, that writes nothing outside itself and announces nothing.
• ModesAn evaluation records two things — the purpose it serves (offline, online, simulation, release_gate) and how the output under test is produced (existing outputs, a managed prompt, an HTTP target, a repository change, a trace replay); there is no separate "rerun".