Adjudication and agreement

When a queue asks for more than one review per item, the reviewers may not agree. EvalBook keeps that disagreement explicit until a person who did not review the item decides it. The resolved value of each definition on each item is a label, and labels are what judge calibration and the evaluation that asked for a human check read. Everything here is served under review/api/items/… and review/api/queues/….

Status (25 October 2026). The review API exists in the application (queues, items, score definitions, online policies and alert rules are routed) but the Review, Online scoring and Alerts pages are not yet in the shell, and requirements EV-029, EV-030, EV-039 and EV-040 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.

How an item reaches a decision

When the last of an item's reviews is in, each labelled definition (binary, ordinal, categorical, pairwise) is decided on its own. One reviewer's answer stands as a single_reviewer label; several reviewers giving the same value stand as an agreed label; several giving different values open a disagreement, and the item waits with no label for that definition. Free-text answers are notes; they never disagree and never become labels.

An item with an open disagreement moves to adjudication_needed, the queue owner is told through the SSO inbox, and the item never expires while it waits. A correction made after an item was resolved by agreement is re-decided, and can reopen it.

Adjudication

review/api/items/adjudication returns the adjudication view: the references, every annotation with its version history, earlier decisions, and a content_digest over all of it. review/api/items/adjudicate decides one definition with a value on its scale and a rationale. The call carries the digest the adjudicator saw; if an answer was corrected or another decision made since, it is refused with ITEM_CHANGED and the current digest, so the content a person decides on is the content the decision records.

Who may decide is checked live. The caller holds review.adjudicate; someone who reviewed the item or holds one of its slots is refused (ADJUDICATOR_IS_REVIEWER), as is anyone listed as its author, producer or requester (SEPARATION_OF_DUTIES). A pairwise decision is made on the canonical A and B, never on a reviewer's drawn left and right.

A decision is append-only: a later decision on the same definition supersedes the earlier one, and both stay. When the last open disagreement is decided the item becomes resolved with resolution adjudicated, and a reviewer can no longer correct an answer an adjudicator has decided.

Labels

Each decision writes a ReviewLabel: the definition, the value, the source (agreed, adjudicated, single_reviewer), how many reviewers it rests on, and when it was resolved. A correction or a new adjudication supersedes the row; nothing is edited in place.

humanLabelsFor(ctx, { organizationId, projectId, scorerId, since?, includeSingle? }) returns the current labels of every definition linked to a scorer: by default only agreed and adjudicated ones, with a single reviewer's answer returned only when asked for and marked as the weaker reference it is. The answer carries the definitions, per-source counts, an as_of time and a truncated flag; at most 10,000 labels come back per call. This is the input to judge calibration: the kappa and alpha figures an LLM judge is held against are computed from these labels and their sample counts.

Outcomes back to the evaluation

A resolved case_result item is reported to the evaluation that asked for the human check: a ReviewOutcome row is written in the same transaction, and a delivery job calls the listener the evaluations component registers with { item_id, queue_id, case_ref, evaluation_ref, resolution, resolved_at, labels }. With no listener the outcome is undelivered; a failed listener makes it failed; neither is dropped or pretended delivered, and a person with review.manage can deliver them again. How an evaluation uses a human verdict is described under reports and release gates.

Inter-rater agreement

review/api/queues/agreement computes agreement per definition from the queue's annotations, on request. The basis is each reviewer's first submission by default, the independent judgement before any correction; basis=current uses the answers as corrected. Items reviewed once are counted but not pairable; free-text definitions are not applicable. At most 5,000 items are read, and the answer says when it was cut.

For each definition the answer gives the sample counts (items rated, pairable items, ratings, raters), percent agreement, Fleiss' kappa, and Krippendorff's alpha (nominal, or ordinal for ordinal definitions) with a percentile bootstrap interval over items whose resamples, level, seed and generator are recorded so it can be recomputed. Where a coefficient is undefined the value is null and a note says why. Nothing here is a rating of a person. The judge_disagreement metric of an alert rule is the operational use of these labels against a judge.