Protected data and governance

Some data must not change on one person's say-so: a protected dataset, and the holdout split of any dataset. Any change to either becomes a change set — the proposed rows, their digest, and a governed action asked of whoever governs the project. EvalBook never authorizes: when the project is linked to MadeBook, MadeBook decides; a standalone project applies separation of duties. The change is applied only when an approval for that exact digest arrives, and if governance cannot be reached the change is refused, not written.

Status (25 October 2026). The datasets component is implemented and its requirements EV-010 to EV-018 are marked implemented but unverified in the requirements ledger; verification is in progress.

What is governed

Kind	Triggered by
modify_protected_dataset	any change to a protected dataset: its examples (create, correct, remove, restore, review), its schema, its split policy or strictness, and turning protection off
change_holdout	any change that adds an example to, removes one from, or alters one in a holdout split, in any dataset — approved holdout examples are "members"

These are two of the governed action kinds EvalBook shares with its peers, beside activating suite, scorer and judge versions and changing thresholds; see MadeBook.

Turning protection on and off

Protecting a dataset only tightens, so it is direct: Protected when creating it, or datasets/api/protect later, with dataset.protected.write. Removing protection weakens and is always a change set (set_unprotected), even for a dataset nobody has edited.

A change set

When a governed write is attempted — in the editor, an import, the SDK batch, a curated candidate, a review decision — EvalBook writes the proposal instead of the change: examples with status proposed, schema fields without a revision, the ordered items, and a digest over the dataset, the kind and the items. The set is handed to governance with that digest and the identity of its author and requester; an import stores its creator's identity so a governed change from a background apply names them.

The set moves requested → pending_approval → approved → applied, or ends denied, deferred, expired, cancelled, conflicted, withdrawn or failed. It is applied only by EvalBook's own activation handler, only for an approved action, and only when the action's digest equals the set's digest and the rows still hash to it. An action for a different kind is refused (KIND_MISMATCH); a set whose rows no longer match is marked conflicted and nothing is written. If the governance module is missing or unreachable, the request is refused with GOVERNANCE_UNAVAILABLE — fail closed.

A denial, expiry or cancellation discards the proposals. Rows in an import or an SDK batch that are governed are collected into one change set, and the rest of the batch is written normally.

Where it shows

Datasets → (dataset) → Governance lists change sets with when they were requested, what they change, their state, the governed action and the digest. Opening one shows the decision, the proposed changes field by field, and Withdraw, which only the person who proposed it or a release admin may use; an approval that arrives later is refused. In the examples list, Proposed (governed) filters the rows waiting on a decision. On the Schema page of a protected dataset, saving says that the new schema is sent for a governed approval and applies only when approved.

Holdout independence

The holdout is the evidence an evaluation's claim rests on, so EvalBook keeps anything that could tune against it out of reach:

• no model generates holdout examples or suggests their expected outputs (Curation and synthetic data);
• exporting it needs dataset.protected.write, asked explicitly, and is audited with every download (Import and export);
• evaluations refuse holdout evidence when a leakage scan finds contamination or has not covered the version (Slices and leakage);
• under the hash split policy the holdout cut is deterministic from keys and percentages, and a protected dataset's percentages change only through governance (Versions and splits).

Audit and confirmation

Every write in the datasets component is audited in the same transaction and carries expected_revision, so a stale edit is a 409 rather than a silent overwrite. Archiving a dataset requires its exact name, checked on the server. The capability that reaches protected data, dataset.protected.write, belongs to the release_admin role, separate from dataset_curator; the roles are separated duties, not a ladder. See Roles and access and Security model.