Experiments

An experiment compares configurations. Every variant — one baseline and one to five candidates — is run against the same suite version, pinned when the experiment starts, so each sees the same dataset versions, scorer versions and decision rule. Each variant is an evaluation of its own; the experiment then compares candidates with the baseline example by example. Checks that are not comparable and examples that have no counterpart are flagged, never ranked.

Status (25 October 2026). The evaluations API exists in the application (suites, evaluations, experiments, reports, gates and the peer endpoints are routed) but the Suites, Experiments and Reports pages are not yet in the shell, and requirements EV-031 to EV-038, EV-045, EV-048 and EV-054 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.

Starting one

POST evaluations/api/experiments/create (session) or POST /api/v1/experiments (API key) with experiment.run:

• name, suite_id and an optional suite_version (a version in review, shadow, canary or active);
• execution_mode: managed_prompt (default), http_target or existing_outputs; replay and repository changes are evaluated directly, not as experiments;
• variants: 2–6, exactly one with role: baseline, each with a key, a name and its configuration — model, system_prompt, prompt_template (required for a managed prompt; {{input}} or {{input.field}} is the example's input), temperature (0–2), max_tokens (up to 32,000) or http_binding_id;
• repeat_count (1–20), budget_tokens, deadline, seed and mode (offline or simulation).

A managed-prompt experiment calls models and must carry a token budget (its own or the suite version's); the budget is shared equally across the variants. The baseline's evaluation is admitted first so that candidates can name it. Each variant's configuration is digested, and the variant becomes the evaluation's subject. An experiment is running, then completed, cancelled or failed once every variant's evaluation has ended; cancelling it cancels them.

The comparison

GET evaluations/api/compare?baseline=…&candidate=… answers, per check, the baseline's and candidate's aggregate over executed examples and a paired analysis over the examples scored under both:

• a check is comparable only when both sides were scored by the same scorer version digest on the same dataset version digest;
• single-repeat pass/fail checks use the Newcombe paired difference interval and the exact McNemar test;
• anything else uses a paired bootstrap of 2,000 resamples with the recorded seed;
• the family correction frozen in the suite version is applied across the comparable checks' p-values;
• required slices are compared the same way.

The decision rule, alpha and correction are the candidate's suite version's; asking for a comparison under another rule is 409 RULE_FROZEN. Significance is never chosen after the outcome is seen. The answer also carries each evaluation's latency p50 and p95 (measured, from the recorded outputs), billed tokens ("allocated from SSO usage") and budget consumption, marked unavailable when there is nothing to measure; see usage and cost.

Sequential monitoring

When the suite version names obrien_fleming with a comparative rule, the primary required check is monitored as the variants run. Each look (Lan–DeMets O'Brien–Fleming boundaries, one-sided, at the version's alpha) is recorded once with its information fraction, the paired count, the boundary, the statistic and the decision. When every candidate has crossed the efficacy boundary, the remaining cases are not run (stop_reason: sequential_stop) and the reports say how much was covered. The last look is the final analysis.

HTTP targets

An http_target variant posts each example to an endpoint. A project admin creates a binding (evaluations/api/targets/bindings/create) with a name, the endpoint URL (or a target that has one; private, link-local and unresolvable addresses are refused), an optional header name with a credential that is sealed at rest, a timeout (30,000 ms by default, 1,000–300,000), a concurrency of 1–8 (2 by default) and the field of the answer that holds the output (output by default). EvalBook sends { evaluation_id, example_id, repeat_index, input } with an Idempotency-Key that is stable per example and repeat and an X-EvalBook-Evaluation header; the endpoint is expected to deduplicate. Revoking a binding discards its credential.

Reading an experiment

evaluations/api/experiments/show (and GET /api/v1/experiments/:id) lists the variants with their configuration digests, each evaluation's status, result and progress, and the recorded looks. Each variant's report is an ordinary report of kind experiment; experiment reports do not supersede gate reports and are not announced to peers. The playground is an isolated experiment of prompt variants.