Calibration
A calibration measures how well one judge configuration agrees with approved human labels, with sample counts, confidence intervals whose method is recorded, and a stale mark when the judge changes.
A judge's verdicts carry no measured agreement with people until it is calibrated. A calibration runs one judge version over the approved human labels the review component holds for that scorer and compares the two on the judge's own scale. Every statistic carries its sample count and a confidence interval with the method that produced it; a statistic that is undefined for the data is shown as undefined with a note, never as 0.
Status (25 October 2026). The scorers component is built in the application (registry, versions, rubrics, calibration and runners pages are live in the shell); its requirements EV-019 to EV-028 are not yet marked implemented in the requirements ledger and are being verified now.
Starting one
Open a judge scorer and choose Calibration. Deterministic and executable scorers are not calibrated; they are checked by their fixtures. Calibrate a version (scorer.write) names a version that is not retired and a budget in SSO-billed tokens (200,000 by default; 1,000 to 5,000,000). The run is queued as a job and the page re-reads it while it runs.
Labels come from the review component (humanLabelsFor): a person's judgement on each item, marked approved, with the case content the judge needs. Without the review component a calibration cannot start (REVIEW_UNAVAILABLE): no labels, no agreement figure. Up to 500 labels are read. See queues and annotation and adjudication and agreement.
What is compared
A label is used only when it is approved, carries its case, and its value is on the judge's scale: pass/fail (or true/false, 1/0) for a binary judge, one of the categories, a number on the ordinal or continuous range, or candidate/baseline/tie for a pairwise judge. Labels that cannot be used are counted and listed as not compared with the reason, never dropped silently.
The judge scores each usable item through SSO under the run's budget. An item the judge abstained on, errored on or could not score is counted and listed too. On a continuous scale the two agree when they are within 10 % of the range; on every other scale they must be equal.
A run ends completed when at least 10 items were paired, and insufficient_data otherwise; the statistics are stored in both cases, but with fewer than 10 items agreement is not stated as established. A run that cannot read its labels is failed with its code.
The statistics
| Scale | Statistics |
|---|---|
| binary, categorical, pairwise | percent agreement, Cohen's kappa |
| ordinal | percent agreement, quadratic-weighted kappa, Krippendorff's alpha (ordinal metric), Spearman's rho |
| continuous | Pearson's r, Spearman's rho, mean absolute difference |
Intervals are at 95 %. Agreement, the kappas, alpha and the mean absolute difference use a seeded percentile bootstrap with 2,000 resamples; the seed and the resample count are recorded so the interval can be recomputed. Pearson's r uses the Fisher z transform; Spearman's rho uses Fisher z with the Fieller–Hartley–Pearson variance, 1.06 / (n − 3). Kappa is undefined when a single category appears throughout, a correlation when one side has no variance, and an interval needs more than three items; each such case is a note on the row.
Reading a run
The run shows how many labels were received, usable, scored, errored, abstained and unsupported, the paired sample count, the statistics table, every disagreement (the person's value, the judge's value and status, the judge's explanation) and the items that were not compared. The judge's own words (its rationale) need dataset.read_body.
Which judge a calibration covers
A calibration measures one judge configuration: the version's material digest (model, prompt template, rubric version digest, output fields, temperature, categories, score type). A version's page shows its calibration state: calibrated, insufficient_data, stale, or uncalibrated with the reason (never calibrated, or calibrated only for another configuration of this scorer).
When a materially different judge version is activated, every earlier calibration of the scorer that measured another configuration is marked stale, with the parts that changed named. Freshness is also computed when a run is read: a run whose configuration is not the active version's is marked as not covering the active judge, and a scorer with no active version says so. Calibrate the active version again after a material change.