Deterministic scorers

Deterministic scorers are pure functions that run inside EvalBook on the recorded output or trace. They need no model and no runner, finish within a time limit (2,000 ms by default, at most 60,000 ms), and their results carry assurance independently_observed: EvalBook computed them, but they are only as true as the recorded output they read. The API and worker execute no customer code; a deterministic scorer is EvalBook's own code with the version's parameters.

Status (25 October 2026). The scorers component is built in the application (registry, versions, rubrics, calibration and runners pages are live in the shell); its requirements EV-019 to EV-028 are not yet marked implemented in the requirements ledger and are being verified now.

The implementations

Implementation	What passes	Needs
exact_match	the output equals the expected value, as text (trim, collapse whitespace, letter case and Unicode NFC are options) or as canonical JSON	output, expected
contains / not_contains	the output text contains every (or any) listed text, or the expected value; or none of them	output
regex	the output matches a pattern (match, full_match or no_match)	output
json_schema	the output is valid against a JSON Schema (draft 2020-12 subset; an unsupported keyword is refused when the version is created)	output
numeric_tolerance	the output number is within an absolute or relative tolerance of the expected number	output, expected
required_fields	the output object has each listed field with its declared type (required_field rules)	output
required_artifacts	the case lists each required artifact by kind and name, with a sha256 digest, in the output, the subject or trace links; it checks what the case reports and fetches nothing	output
edit_similarity	normalized Levenshtein similarity (0–1) meets the version's pass threshold	output, expected
tool_call_match	the named tool was called with arguments that satisfy every tool_param rule (equals, not_equals, matches, one_of, min, max, present, absent, type); occurrence is any, all or exactly_one; other tools may be forbidden	trace
trajectory	every required_step happened, in order and without error, no forbidden_tool was called, and each declared handoff was recorded	trace
retrieval_relevance	precision, recall or F1 at k of the retrieved document ids against the example's labelled relevant ids meets the threshold	expected, trace
citation_support	every document id the answer cites was retrieved for it	output, trace
secret_scan	none of the chosen credential detectors match the output	output

Rules every implementation follows

• Nothing is converted. An output of the wrong type fails ("the output is a number, not a string; nothing is converted"). An expected value or a parameter of the wrong type is an error: the dataset or the scorer is wrong, not the candidate. Neither is ever a pass.
• The expected value is the example's expected (see versions and splits) or an explicit parameter; a field of either can be picked with output_path and expected_path. A missing expected value is an error.
• Explanations carry no payload text. They say what kind of value was seen, how long it was, where two texts first differ and which rule failed. Body-derived detail goes in details, shown only to people with dataset.read_body.
• A metric that is undefined for the case (nothing retrieved, no relevant id labelled) is inconclusive, never 0.

Regular expressions

Patterns are checked for catastrophic backtracking when the version is created and refused if they can blow up. Matching runs under a time limit (100 ms by default, at most 2,000 ms) and answers timeout when it is stopped. An output longer than max_input_chars (100,000 by default) is an error, not a match.

Traces and tool calls

tool_call_match and trajectory read the case's trace spans in start order. For a stored trace, tool arguments are not in the span list: the scorer fetches them through the traces component with purpose scoring, which applies the project's capture policy. A span whose arguments the policy did not keep makes the case unsupported (ARGUMENTS_UNAVAILABLE), never "no arguments"; without the traces component the case is unsupported too. See redaction and capture.

retrieval_relevance and citation_support work on document ids only. They never fetch a document, so a scorer cannot read something its caller may not see, and every result says where its ids and its labels came from.

Secret scan

The detectors are AWS access key ids, GitHub tokens, PEM private keys, Slack tokens, OpenAI and Anthropic keys, Google API keys, Stripe secret keys, Bookbag tokens (evb_, evr_, bbk_) and JSON Web Tokens. A finding names the detector, the position and the length, never the value. A clean result says that none of the detectors that ran matched; it does not claim the output holds no secret.

Deterministic versions are checked with fixtures and sandbox runs, not with calibration.