Scorers

The registry and versions, deterministic scorers, executable verification in a runner, model judges and their calibration, pairwise comparison.

• Registry and versionsScorers are named checks whose behaviour lives in immutable, digested versions that move from draft to review, shadow and, through a governed activation, active.
• Deterministic scorersFourteen in-process checks over the recorded output or trace, with no type coercion, bounded pattern matching and explanations that never quote the case.
• Executable verificationExecutable scorers run a pinned command in a sandbox on a separately installed, outbound-only runner against an exact commit; EvalBook draws the verdict from what the runner observed.
• LLM judgesA judge version pins a model from the SSO catalog, a prompt template, an optional rubric version and a strict verdict schema; every call goes through Bookbag SSO and a malformed answer is never a pass.
• CalibrationA calibration measures how well one judge configuration agrees with approved human labels, with sample counts, confidence intervals whose method is recorded, and a stale mark when the judge changes.
• PairwiseA pairwise judge compares a candidate with a baseline in a seeded random order, records every trial, and claims no preference when the order changed the answer.