Import and export

Imports bring rows into the working copy through the same validation as every other write, after a dry run that reports every problem before anything is written. Exports hand over a frozen version as bytes a recipient can verify: the same version, format and holdout choice always produce the same file and the same digests.

Status (25 October 2026). The datasets component is implemented and its requirements EV-010 to EV-018 are marked implemented but unverified in the requirements ledger; verification is in progress.

Import formats and limits

Datasets → (dataset) → Import rows takes JSON, JSONL or CSV, pasted (up to 5 MB) or from an uploaded file (up to 20 MB). An import holds at most 50,000 rows and 500 CSV columns, and no single payload may exceed 5 MB. Over a limit the import is refused whole, never truncated.

A JSONL row, or an element of a JSON array (or of { "examples": [ … ] }):

{ "stable_key": "faq-0412", "input": { "question": "…" }, "expected": { "answer": "…" },
  "split": "development", "tags": ["billing"], "metadata": { "language": "de" },
  "family_key": "faq-04", "expected_origin": "human" }

A CSV has a header row and every column must be mapped or ignored — no column is dropped unseen. Targets are stable_key, input, expected, input_field:<name>, expected_field:<name>, split, tags (separated by |), family_key, metadata:<key> and ignore; unmapped headers are matched by name (id, expected_output, input.<field>, meta.<key> and so on), and a column that matches nothing is an error. The delimiter is a comma, a semicolon or a tab; fields follow RFC 4180; a BOM is dropped; blank lines and rows with the wrong number of fields are reported with their line.

Import options

• Existing stable key: skip the row, update the example (history kept) or fail.
• New examples are: approved (a curator's own data) or pending (candidates for review).
• Split when a row has none: development, validation or holdout (governed).

Rows that would change a protected dataset or holdout membership are not written; they are collected into one governed change set, named after the person who created the import (Protected data and governance).

Dry run, then apply

An import moves created → validating → validated → applying → completed, or failed or cancelled. The dry run parses every row and validates it against the schema and the working copy as it is now; every problem is a row-level diagnostic (up to 20 per row, then one saying how many more). Nothing is written until you Apply.

Apply re-reads the source — its digest must still match — plans each row again against the current working copy, and writes in batches of 200 rows. Each batch commits together with its checkpoint, so an interrupted apply resumes after the last committed batch and never writes a row twice. The Imports page lists every import with its source, status, rows and what it applied; diagnostics are paged by phase and severity.

The staged source (pasted text or the uploaded file) is a raw_bodies content class: the project's retention purges it after the import finishes, keeping its digest so what was imported stays verifiable.

Exports of a version

Datasets → (dataset) → Export exports a frozen version as JSONL or CSV. The bytes are a pure function of the version, the format and the holdout choice: records in version position order, each the canonical JSON of the frozen example (JCS with NFC), CSV cells unaltered. The manifest is canonical JSON with no timestamps. Both are stored at Bookbag SSO's file store with sha256 digests, and the page shows the content digest beside the Data and Manifest downloads.

The manifest names the dataset and version (number, digest, schema revision and digest, protected), the format and media type, the content digest and byte count, the examples per split with whether each is included, and every limitation — a payload SSO could not return is named, never passed off as an empty value. When every split is included, the sha256 of the records' content_digest values in order reproduces the version digest.

A CSV carries stable_key, split, family_key, input, has_expected, expected, expected_origin, tags, metadata, source_kind, source_ref, is_synthetic, validation_status, attachments, repository_fixture and content_digest. Cells are exported exactly as stored; a cell beginning with =, +, - or @ is data, not a formula, and the manifest says so. Open the file as text.

The holdout in exports

Holdout examples are excluded unless the requester holds dataset.protected.write and asks for them explicitly. Asking without it is refused (HOLDOUT_EXPORT_FORBIDDEN), never silently downgraded. An export that includes the holdout is marked on the page — anyone who receives it can tune against it — and the request and every download are in the audit log.

Capabilities

Importing needs dataset.write. Exports carry bodies, so they need dataset.read_body as well as dataset.export. The SDK's batch endpoint, which follows the same planning rules as an import, is described under Datasets and examples.