Slices and leakage

A slice is a named subset of a dataset defined by conditions over its examples, so a suite can report on billing questions in German on its own rather than inside an average. A leakage scan finds examples whose inputs are the same or nearly the same across splits, datasets and versions, and says when a holdout is contaminated so evaluations can refuse it.

Status (25 October 2026). The datasets component is implemented and its requirements EV-010 to EV-018 are marked implemented but unverified in the requirements ledger; verification is in progress.

Slice definitions

A definition is match: all | any over up to 50 conditions, each { subject, key?, operator, value | values }:

Subject	Operators	Key
tag	eq, neq, in, not_in, contains, exists, not_exists	—
metadata	all of the above, plus gt, gte, lt, lte	the metadata key
field	all	a path starting input. or expected., such as input.language
split, source_kind, review_status	eq, neq, in, not_in	—
synthetic	eq	—

in and not_in take up to 200 values. A definition may add a deterministic sample: the N examples with the lowest sha256 of a seed plus the stable key, optionally stratified by a metadata key with the largest-remainder rule, so the same seed always picks the same examples.

Editing a slice's definition writes a new, immutable definition number; the slice points at its current one, and an evaluation records exactly which definition and digest it used.

Counts are computed, never stored

The Slices page shows, for the working copy and the latest version, matched of total plus unevaluable: examples a field condition could not be checked on because their payload is stored at Bookbag SSO rather than inline. A count never silently omits them. Counting by payload or metadata values needs dataset.read_body; Preview evaluates a definition before saving it.

Required slices

A slice marked Required in suites must be reported on its own by every suite that uses the dataset, so an aggregate pass cannot hide a failed slice. The evaluations component asks for them (requiredSlices) with their definition number and digest; see Suites and thresholds.

The leakage method

The method is stated once (minhash_lsh_v1) and stored on every scan:

• Exact: two examples whose canonical input JSON has the same sha256 digest.
• Near: the input's text — every string and number, in canonical key order — is normalized (NFKC, lower case, every run of non-letters and non-digits to one space) and cut into character 5-gram shingles. A 64-permutation MinHash signature is banded into 16 bands of 4 rows; inputs sharing any band are candidates, and every candidate is verified with the exact Jaccard similarity of the two shingle sets. Pairs at or above 0.8 are near duplicates.

Signatures and bands are cached per project and input digest — never the text — so a scan only computes inputs it has not seen.

What a scan compares

A version scan compares the version with itself across splits, and with every frozen version of every dataset in the project. The same example carried unchanged into a later version of its own dataset is not leakage; the same stable key in a different split is. A finding is holdout contamination when exactly one side is in a holdout split. Every freeze queues a scan automatically; Leakage → Scan runs one on demand, and a scan of the working copy is advisory.

Findings and coverage

The Leakage page lists scans (subject, status, how many compared, exact and near findings, holdout contamination) and findings, where A is the scanned example and B the overlapping one with the versions that contain it; Holdout contamination only narrows the list.

For evaluations the answer is leakageFindings: contaminated, the findings, and complete. A version is fully covered only when its own scan completed and every version frozen after it in the project has a completed scan (those compared themselves against it). Evaluations refuse holdout evidence when the answer is contaminated or not complete — an unscanned holdout is unknown, not clean.

Fixing a finding

Move or remove the offending example in the working copy and freeze again; a change that touches a holdout member is a governed change set (Protected data and governance). The new version is scanned when it is frozen, and the old version keeps its findings.