Datasets and examples

A dataset is a schema and a working copy of examples that curators add, correct, review and remove. Evaluations never run against the working copy: they run against an immutable version frozen from it (Versions and splits). This page is about the working copy and the rules it enforces whichever door an example comes in by.

Status (25 October 2026). The datasets component is implemented and its requirements EV-010 to EV-018 are marked implemented but unverified in the requirements ledger; verification is in progress.

Creating a dataset

Datasets → New dataset asks for a name, a description, a kind, a split policy, whether fields the schema does not declare are allowed, and whether the dataset is protected. The kind — text, structured, conversation, code, trajectory, document or mixed — starts the schema with default fields you can change on the Schema page. A protected dataset changes only through governed change sets (Protected data and governance).

The schema

A schema is a list of up to 200 fields, each with a role and a type:

• roles: input (a key of the input object), expected (a key of the expected-output object), metadata (a scalar metadata key);
• types: string, number, boolean, object, array, file, code_patch; metadata fields are string, number or boolean only;
• constraints: required, minimum and maximum length or value, a pattern (a regular expression; one that nests repetitions is refused), allowed values, and for files a modality with its content types.

A role with no declared fields is unconstrained. A role with fields requires an object, and unknown keys are refused unless the dataset allows extra fields. Validation reports every problem with the field it is about; a row is rejected with diagnostics, never coerced or truncated. Saving the schema writes a new schema revision, and every version records which revision it was frozen under.

An example

Part	Rules
stable_key	1–200 characters, immutable for the example's life; when omitted it is derived from the input's digest (k-…) so a re-sent row finds itself
input, expected	JSON, validated against the schema; stored as canonical JSON with a sha256 digest, inline up to 256 KB and at Bookbag SSO's file store above that; at most 5 MB each
expected_origin	none, human, source, observed_output or ai_suggested — where the expected output came from
split, family_key	development, validation or holdout; examples sharing a family key stay in one split
tags, metadata	up to 50 tags of 120 characters; up to 100 scalar metadata keys
attachments, repository_fixture	up to 20 referenced files with declared modalities; a pinned repository, commit, patch and protected harness paths for code evaluations

Rules every door enforces

• Exact dedup. Two live examples of one dataset never share an input digest (DUPLICATE_INPUT); a removed duplicate is offered for restoring instead.
• Families stay together. Under the manual split policy an example whose family already lives in another split is refused (FAMILY_SPLIT_CONFLICT); under the hash policy the family is the hash basis, so it cannot straddle splits.
• Keys are unique. A second example with an existing stable key is STABLE_KEY_TAKEN; an import or a batch may choose to update instead.
• Limits are refused, never truncated.

Review status

Every example has a review status: pending, approved or rejected, with reopening back to pending. Only approved examples are frozen into a version. A curator's own data may be approved as it is written; anything curated, imported as candidates or generated arrives pending (Curation and synthetic data).

Correction history

Correcting an example keeps every previous value in its history, with the reason given (Why (kept with the history)). The example page's History tab lists each correction, newest first, with the value it replaced. Removing an example takes it out of the working copy; frozen versions keep it, its history stays, and it can be restored.

Who may do what

dataset.read opens structure, counts, digests, tags and provenance; example bodies — payloads, metadata values, history, diffs, candidates' content, exports — need dataset.read_body. Writing needs dataset.write, exporting dataset.export, and protected data dataset.protected.write. The dataset_curator role holds the first four; see Roles and access. Archiving a dataset needs its exact name, checked on the server; an archived dataset refuses writes until it is restored.

The SDK endpoints

With a project API key (Authorization: Bearer evb_…, scopes read and datasets:write), the tenant always the key's project:

GET  /api/v1/datasets                                   list
POST /api/v1/datasets                                   create (Idempotency-Key)
POST /api/v1/datasets/:id/examples                      insert or upsert up to 500 (Idempotency-Key)
GET  /api/v1/datasets/:id/versions                      list versions
POST /api/v1/datasets/:id/versions                      freeze (Idempotency-Key)
GET  /api/v1/datasets/:id/versions/:version/examples    frozen examples, pages of up to 500, with bodies

A batch is { "mode": "insert" | "upsert", "review_status": "approved" | "pending", "examples": [ … ] } and is validated whole: one invalid item refuses the batch with every item's problems by index, so a client never half-applies a batch it believes was rejected. upsert corrects an existing stable key with history kept and reports identical content as unchanged.