Curation and synthetic data

Datasets grow from production. A candidate is a working-copy example with review status pending: it carries where it came from, it is never part of a frozen version until a curator approves it, and nothing labels it ground truth on its own. The same holds for everything a model produces — generated examples and suggested expected outputs are labelled and unvalidated until a reviewer's approval, which is their validation.

EvalBook never runs customer code to make any of this. Generation and suggestion are model calls through the organization's AI access at Bookbag SSO, under an explicit token budget.

Status (25 October 2026). The datasets component is implemented and its requirements EV-010 to EV-018 are marked implemented but unverified in the requirements ledger; verification is in progress.

Candidates and the review queue

Datasets → (dataset) → Candidates is the review queue. Each candidate shows its input, its provenance (source kind and reference) and the expected output with its origin. A reviewer approves or rejects it, with a note; a decision can be reopened. Only approved examples are frozen into a version (Versions and splits).

From a trace span

Candidate from a trace span takes a span id. The input is read through the traces component's one body gate, which applies the project's capture policy and the person's trace.read_body; a span whose body is unavailable, redacted to nothing or never captured makes no candidate (SPAN_BODY_UNAVAILABLE). The candidate records the source's privacy facts as metadata — source.capture_mode, source.redacted_paths and the first redacted paths — and the tags from-trace and, when anything was redacted, redacted-source.

The expected output is None — a reviewer writes it, or the observed output kept and labelled observed_output: observed, not verified. It stays labelled until a reviewer approves the candidate, and nothing asks a model to relabel what was observed. See Redaction and capture and the Explorer.

From a review, proposal or incident

Candidate from a review, proposal or incident takes the source and a reference (INC-42), kept as provenance, with the input and an optional expected output typed in. The same path is the interface other components use (createCandidateExample): a review annotation that should become a test case, a proposal's reproduction, an incident. A review's expected output is human; a proposal's or an incident's is source. See Queues and annotation and Proposals.

Synthetic generation

Datasets → (dataset) → Generate starts a generation run:

Setting	Rules
How many	1–200
Token budget	5,000–2,000,000 SSO-billed tokens, declared up front
Into split	development or validation — never the holdout
Model	the organization's default, or one named
Instructions	what the examples should be about
Also suggest expected outputs	labelled ai_suggested, unvalidated

The run works in batches of up to 10, each one model call with purpose dataset_generation. Every call reserves its worst case against the budget first, and the run stops where the budget cannot cover the next batch, saying so. The model is asked for structured output (a submit_examples tool; a JSON answer is accepted) and every item is validated against the dataset schema. Valid items become pending candidates labelled synthetic and unvalidated, with their provenance — generator model, prompt digest, tokens, the AI call; invalid items are recorded as issues. A batch whose outcome is unknown is retried under the same logical request id, so SSO charges it once.

The generator is shown approved development examples as references only. Holdout examples are never shown to it and never generated: a holdout must stay independent of what produced the candidates. The Runs table shows the budget used, the prompt digest, and each batch's attempt, status, tokens billed, items generated, stored and rejected. Spend appears under Usage and cost.

AI-suggested expected outputs

On a candidate's AI suggestion tab a reviewer asks a model for a suggested expected output (budget 5,000–200,000 tokens). The suggestion is a proposal the reviewer decides on; one that is taken is written with origin ai_suggested and stays unvalidated until the candidate is approved. A holdout example's expected output is never suggested (HOLDOUT_INDEPENDENT), and an input too large to be inline is not sent (PAYLOAD_NOT_INLINE).

What counts as ground truth

Each frozen example carries expected_origin, is_synthetic, validation_status and a derived ground_truth: false for synthetic or AI-suggested content until it is validated by approval. Consumers of a version receive these flags, so a suite can say how much of its evidence rests on human labels; see LLM judges.