Modes

The evaluation service distinguishes two settings on every evaluation and report. The mode says what the evaluation is for and which modes a suite version applies to. The execution mode says where the output under test comes from — and therefore what EvalBook executes, calls or merely reads, and how much the result can be trusted. The requirement behind this page is named "simulation versus replay versus rerun"; what the application has is the two settings below, and this page describes exactly those.

Status (25 October 2026). The evaluations API exists in the application (suites, evaluations, experiments, reports, gates and the peer endpoints are routed) but the Suites, Experiments and Reports pages are not yet in the shell, and requirements EV-031 to EV-038, EV-045, EV-048 and EV-054 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.

Mode

A suite version lists the modes it applies to (offline by default); an evaluation that names a mode the version does not list is refused with MODE_NOT_APPLICABLE. The mode is recorded on the evaluation and in the report body.

Mode	What the evaluation service does with it
offline	the default for an evaluation or an experiment started by a person or an API key
simulation	accepted for evaluations and experiments and recorded; the service treats it like offline
release_gate	the default for an evaluation CodeBook asks for; a result from an unisolated runner (development_unisolated) is turned into unsupported, so it can never pass
online	a value a suite version may list and an evaluation may carry; the evaluation service gives it no behaviour of its own — scoring live traffic is the review component's online scoring

Peers see a suite's mode as release_gate when its active version lists it, otherwise the first mode listed. See online scoring for what runs against production traffic.

Execution mode

Execution mode	The output under test	What EvalBook executes or calls	Input assurance
existing_outputs	submitted per example with the request or in later batches (or read from a linked trace by trace_id)	nothing	reported
managed_prompt	EvalBook runs a variant's prompt through an SSO model for each example	a model call through Bookbag SSO	independently_observed
http_target	EvalBook posts each example to the customer's endpoint through an HTTP target binding	one outbound HTTP call per example and repeat	independently_observed
repository_change	the subject is an exact commit; checks without a dataset run against it, executable scorers in a runner	nothing in the API or worker; the runner executes	reported, raised to platform_verified only for a runner's container result
trace_replay	recorded traces are scored (1–500 trace_ids); the root span's output is read through the traces component under the capture policy	nothing: a replay executes nothing and calls nothing	reported

The execution mode is chosen at admission (repository_change when the subject is one, existing_outputs otherwise) and recorded on the evaluation, every output row and the report. Outputs may be submitted only in existing_outputs; trace_ids only in trace_replay; http_target needs a binding; managed_prompt is started as an experiment or in the playground. In trace_replay an executable scorer is unsupported ("trace replay executes nothing"), and a trace the capture policy will not release to an evaluation leaves its case unsupported rather than scored on nothing. See redaction and capture.

A scorer's own assurance is never raised by the execution mode and is lowered to the input's when the input is weaker: a judge's verdict over submitted outputs is reported however the judge ran, and platform_verified is possible only for a repository_change evaluation whose executable check a runner executed. See executable verification.

Replay, simulation and running again

• Replay is trace_replay: scoring what was recorded, without producing anything new.
• Simulation is a mode label an evaluation or experiment may carry; producing new outputs is a matter of execution mode (managed_prompt or http_target), not of the label.
• Running again has no mode of its own. A new evaluation of the same subject under the same suite is an ordinary admission: it gets its own cases, outputs and report, and a gate report supersedes the earlier current one for that subject (see reports). Case results are written once; a restarted worker touches only the cases still queued, and model calls reuse their logical request id so SSO charges once. Nothing re-scores a finished case in place.

Where the two settings appear

Both are in the evaluation (mode, execution_mode), in the report body, in the peer answer CodeBook receives, and in the reproducibility bundle. An experiment records its execution mode once and its mode (offline or simulation) for every variant; see experiments and release gates.