Queues and annotation

A review queue is where people label things EvalBook has recorded: a production span, a case result from an evaluation, a dataset example, or two outputs side by side. A queue asks its reviewers one or more score definitions, holds items as references rather than copies, and keeps every answer with its history.

The API is under review/api/…. Other components add items through the enqueueReview interface; a person adds them with review/api/items/enqueue.

Status (25 October 2026). The review API exists in the application (queues, items, score definitions, online policies and alert rules are routed) but the Review, Online scoring and Alerts pages are not yet in the shell, and requirements EV-029, EV-030, EV-039 and EV-040 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.

Score definitions

A score definition (review/api/definitions) is one question with a typed scale: binary (fixed yes / no), ordinal (labelled points with increasing numeric values), categorical (unordered labels), pairwise (fixed a / b / tie) or free_text. A scale has at most 21 points. Once any annotation uses a definition its points can no longer change; labels, guidance and the scorer link can, and every edit is audited. A definition may name the scorer it calibrates; its resolved labels are what humanLabelsFor returns (see calibration).

Queues

A queue (review/api/queues) declares the item kinds it accepts (span, case_result, example, pairwise; a pairwise queue takes nothing else), the definitions it asks for (at most 20), an assignment_mode of round_robin, claim or fixed, reviewers_per_item (1–10), sla_hours (1–2,160), an optional expire_after_hours no shorter than the SLA, blind_mode, an owner who answers for SLA breaches and disagreements, a reviewer pool (at most 200 people, each holding review.annotate), sampling rules and an optional correction dataset. Status is active, paused or archived; an archived queue takes no items.

How items arrive

enqueueReview(ctx, { organizationId, projectId, queueId, items, reason }) takes at most 500 items per call. Each item is a reference, never a body: an application span (trace_id and span_id), an example (dataset_id, version, example_id), a case result (case_ref with its span, its example or both) or a pair of spans.

The reason decides admission: manual, human_check and low_confidence are always admitted, while online_sample, error and disagreement are offers decided by at most 20 sampling rules (random, stratified_attribute with an optional daily cap per value, score_band, error_triggered, disagreement_triggered), the same rules an online scoring policy uses; draws are deterministic, so a resend gets the same answer. The answer lists queued, not_sampled and refused, and the queue keeps the daily denominators offered, admitted, not_sampled, duplicate and refused. Each item records who may not review it: its authors, its producer and, for a human_check or low_confidence item, the person who asked.

Assignment and the item lifecycle

An item moves pending → assigned → in_review → submitted → resolved, with adjudication_needed between submitted and resolved when reviewers disagree (see adjudication and agreement), and skipped or expired as the other ends. An item waiting for adjudication never expires.

A reviewer's hold is an assignment: open → submitted | released. Round-robin and fixed queues are assigned by a background job; in a claim queue reviewers take items themselves (review/api/items/claim), and two reviewers claiming the last slot cannot both win. Eligibility is asked of the guard live: a person who loses review.annotate is passed over, and the queue's sweep releases their open assignments and overdue ones, marks SLA breaches once and tells the owner, and expires items past expires_at. Assignments are announced through the SSO inbox and by email.

Reviewing and correcting

The workspace (review/api/items/workspace) reads content through the owning component under the viewer's own permissions: span bodies need trace.read_body, example bodies need dataset.read_body, and someone without them gets an explicit "not available to you" state (see the trace explorer). In blind mode model and variant identity is hidden, and a pairwise item's sides are shown as left and right in an order drawn per reviewer; the answer is stored as the canonical a / b.

A reviewer submits one answer per definition at once (review/api/items/submit); a value that does not fit its scale is refused. Free-text answers are notes, never labels. A correction needs a reason, keeps every earlier version in the history, can re-make a decision reached by agreement, and is refused once an adjudicator has decided.

A reviewer who thinks the dataset is wrong proposes a change (review/api/items/propose-correction). EvalBook creates a pending candidate through the datasets component, which a curator approves or not; review never rewrites protected ground truth. See curation and synthetic data.

Reading needs review.read, claiming and submitting review.annotate, deciding review.adjudicate (all held by the reviewer role), and creating queues, enqueueing and skipping review.manage (the project admin). See roles and access.