Pairwise

A pairwise judge does not score an answer; it says which of two answers is better. Each case carries an output of the form { candidate, baseline }. The judge sees them as A and B in a seeded random order — and, by default, in both orders — and every trial is recorded with the order shown, the position preferred and which answer that was, so the result can be re-derived and an order effect is visible rather than averaged away.

Status (25 October 2026). The scorers component is built in the application (registry, versions, rubrics, calibration and runners pages are live in the shell); its requirements EV-019 to EV-028 are not yet marked implemented in the requirements ledger and are being verified now.

What a pairwise version pins

A pairwise judge shares the judge configuration described in LLM judges: a model from the SSO catalog, a prompt template, an optional rubric version, sampling settings, extra verdict fields and the same fenced prompt and strict reply parsing. The differences:

• the template must show both {{candidate_a}} and {{candidate_b}} (and may use {{input}}, {{expected}} and {{rubric}});
• the score type is continuous on a fixed 0–1 range with no threshold;
• both_orders (default on) asks each repeat in both orders;
• allow_ties (default on) lets the judge answer tie;
• tie_policy says what a tie counts as: inconclusive (default), pass or fail;
• the time limit is 180,000 ms by default.

The reply's fixed fields are preferred (A, B or tie), reasoning_summary, cited_evidence (sources input, expected, rubric, trace, candidate_a, candidate_b) and abstain.

Order and seed

The order of the first comparison comes from a seed. A sandbox run makes one and records it; an evaluation or calibration passes its own; without one the logical request id is used, so the same case asked again under the same id is shown in the same order. With both_orders, each repeat is asked twice with the candidates swapped. The result records the seed, and the evidence lists every trial as pairwise_trial with A=… B=… → position (answer).

The result

• Any unusable reply makes the case error; half or more abstentions make it abstained.
• If, within one repeat, the two orders preferred different answers, the result is inconclusive with order_effect: true: no preference is claimed.
• Otherwise the votes are counted. A clear majority for the candidate is pass with score 1; for the baseline, fail with score 0. An even split, or a majority of ties, is a tie with score 0.5 and the status the tie_policy names.

The votes (candidate, baseline, tie) are part of the result. The rationale is the first usable reasoning summary and is shown only to people with dataset.read_body.

Calibration

A pairwise judge is calibrated on the scale candidate, baseline, tie: a person's label is compared with the judge's answer, and a judge result with an order effect is counted as not scored. The statistics are percent agreement and Cohen's kappa with bootstrap intervals. See calibration.

Where the pair comes from

The scorer expects the two answers on the case itself. In the sandbox, a fixture or a pasted case gives output: { candidate, baseline } directly. A pairwise version is activated through activate_judge_configuration like any judge (see registry and versions). Comparing two configurations of an application across a whole dataset is the job of an experiment, which runs each variant as its own evaluation and compares them example by example with paired statistics; it does not need a pairwise judge to do so.