LLM judges

An LLM judge asks a model to grade one case against the author's instructions. EvalBook holds no provider key and calls no provider: every judge call goes through Bookbag SSO with the organization, a stable logical request id per case and repeat (so a retry is charged once) and the run's AI budget. A judge's result carries assurance reported — a model's claim about the output.

Status (25 October 2026). The scorers component is built in the application (registry, versions, rubrics, calibration and runners pages are live in the shell); its requirements EV-019 to EV-028 are not yet marked implemented in the requirements ledger and are being verified now.

What a judge version pins

• Model, as provider::model_identifier from the organization's SSO catalog. SSO must allow an exact choice (provider_model_selection); the catalog is checked when the version is created and again at activation, and a model SSO cannot confirm is refused.
• Prompt template, at most 20,000 characters, with placeholders {{input}}, {{output}} (required), {{expected}} and {{rubric}}.
• Rubric version, optional: one exact version with its digest.
• Sampling: temperature 0 (0–2), 800 reply tokens (16–16,000), 1 repeat (up to 10) and a reasoning summary of 1,200 characters (50–4,000) by default.
• Extra verdict fields, up to 30, each string, number, integer, boolean or enum with bounds.
• Score type: binary (verdict pass or fail), categorical (at least two categories, with pass_categories), or ordinal/continuous with a range and a pass threshold. A scored judge without a threshold reports its score as inconclusive.
• A time limit of 120,000 ms by default, at most 600,000 ms.

Rubrics

A rubric is what a judge (or a person) scores against: instructions and 1–50 criteria, each with a key, a title, a weight, whether it is required, and up to 20 anchored levels (a score value and a label). Rubrics are versioned immutably with a digest; a judge version names one exact rubric version. Editing a rubric creates a new rubric version, which needs a new judge version — a material change its calibration does not cover. Rubrics live under Scorers → Rubrics (scorer.write).

How the prompt is built

The case's content — input, output, expected value, trace content — is fenced between markers carrying a random boundary made for that one call, and the system prompt says that only the author's template is instruction and everything fenced is data. Content cannot close a fence whose boundary it cannot know.

Content held in stored spans is read only through the traces component with purpose judge, which applies the project's capture policy. A span the project forbids sending to an external judge makes the case unsupported (CAPTURE_POLICY_FORBIDS); the content is never sent. See redaction and capture.

What a reply must be

Exactly one JSON object with the fixed fields verdict, score, reasoning_summary, cited_evidence (at most 10 items of { source, quote }, quotes of at most 300 characters) and abstain, plus the declared extra fields — no more, no fewer, each of the declared type. A binary verdict's score must be 1 or 0 to match it; a scored judge's score must lie in the range (a whole number on an ordinal scale); a categorical verdict must be one of the categories. Anything else — prose, an extra or missing field, a reply cut off at the token limit — is an error. A malformed judgement is never a pass.

Repeats and the result

With repeat_count above 1 the judge is asked that many times. Any unusable reply makes the case error; half or more abstentions make it abstained. For binary and categorical judges the majority verdict wins and an even split is inconclusive; for scored judges the median score is compared with the threshold. The result reports its variability (agreement, or minimum, maximum, mean and standard deviation) rather than hiding it in an average, and its usage as tokens "allocated from SSO usage".

The explanation states the verdict and how it was reached and is visible to anyone who may see scores. The rationale is the judge's own reasoning summary, which may quote the case, and needs dataset.read_body.

Budgets

A sandbox run, a calibration or an evaluation creates an AI budget, and every call reserves a worst-case estimate before it is made. When the budget is spent, remaining cases are error (sandbox) or skipped (evaluation) with BUDGET_EXHAUSTED. See usage and cost.

Activation and calibration

A judge version is activated through the governed action activate_judge_configuration (see registry and versions). Its material digest — model, prompt, rubric digest, output fields, temperature, categories and score type — is what calibration measures; when a materially different version becomes active, earlier calibrations turn stale and say what changed.