Playground

The playground is where a prompt or a model choice is tried against a suite before anyone builds an experiment around it. A playground run is an experiment of kind playground: one to six managed_prompt variants, each run as its own evaluation over the suite version's pinned examples, scored by the suite's checks. It is isolated by construction. Its evaluations are of kind playground, so they issue no gate report, supersede nothing, send no event to a linked product, record no failure signal and are never sampled for meta-evaluation.

Status (25 October 2026). The evaluations API exists in the application (suites, evaluations, experiments, reports, gates and the peer endpoints are routed) but the Suites, Experiments and Reports pages are not yet in the shell, and requirements EV-031 to EV-038, EV-045, EV-048 and EV-054 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.

Starting a run

POST evaluations/api/experiments/create or POST /api/v1/experiments with kind: "playground" and experiment.run. The body is the experiment body (see experiments) with these differences: one variant is enough, the execution mode is always managed_prompt, and the first variant is the baseline when none is marked. The suite version may be in review, shadow, canary or active; a draft cannot be run.

Each variant names:

• prompt_template (required): the user message, with {{input}} for the example's input or {{input.field}} for one of its fields; a value that is not text is written as JSON;
• system_prompt, optional;
• model, optional, as SSO names it; without one SSO's default for the organization answers;
• temperature (0–2) and max_tokens (1,024 by default, up to 32,000).

A playground run calls models, so it must carry a token budget, its own or the suite version's. The budget is shared equally across the variants, each call reserves its worst-case estimate before it is made, and a variant that reaches its share stops with stop_reason: budget_exhausted; its remaining cases are skipped with BUDGET_EXHAUSTED, never scored. Usage is reported as input, output and billed tokens "allocated from SSO usage"; see usage and cost.

What is recorded

For every example and repeat the generated output is stored once, with the model that answered, its latency and its token usage; a retry of the same call reuses its logical request id so SSO charges once. Each case is scored by the check's pinned scorer version exactly as in an evaluation; a judge check reads the generated output, and the capture policy applies to any trace content it is shown. The outputs EvalBook produced carry input assurance independently_observed; the scored results keep the scorer's own assurance and the report's assurance is the weakest involved.

Reading the run (evaluations/api/experiments/show, GET /api/v1/experiments/:id) gives each variant's configuration digest, its evaluation's status and progress, the per-check results with their intervals, and the comparison of each candidate with the baseline under the suite version's frozen decision rule (experiments). Example bodies and outputs are visible only with dataset.read_body.

What it is not

• It is not a gate. A playground evaluation is never release_gate mode and its report never reaches CodeBook or MadeBook.
• It is not an HTTP target or repository run; those are experiments or evaluations in their own execution modes (modes).
• It does not change the suite. A prompt that works is written into a scorer version or an application configuration by a person; nothing is activated from the playground.

The page

The playground has no page in the shell yet; it is reachable through the API described here. The data it needs — a frozen suite version with a dataset check (versions and splits) and scorers that read an output (deterministic scorers, LLM judges) — is set up on the pages that exist.