Online scoring

An online scoring policy scores a sample of the completed application traces of one environment with exact scorer versions, on a schedule. It is how a deterministic scorer or an LLM judge is run against production traffic rather than against a dataset. Every policy keeps its coverage, so a rate on the Alerts side always has a denominator.

The policy API is under review/api/online/…; the Online scoring page is being finished.

Status (25 October 2026). The review API exists in the application (queues, items, score definitions, online policies and alert rules are routed) but the Review, Online scoring and Alerts pages are not yet in the shell, and requirements EV-029, EV-030, EV-039 and EV-040 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.

A policy

A policy (review/api/online/policies, with create, update, enable and disable) names one environment_key of the project (see environments and targets) and pins the scorer versions it runs by id and digest, read through the scorers component at save time.

Setting	Default	Range
scorers	—	1–10 pinned versions
sample_rate_ppm	100,000 (10 %)	0–1,000,000; used when no sampling rule is declared
daily_score_limit	1,000	1–1,000,000 attempts per UTC day
daily_token_budget	none	required when a judge is among the scorers
max_concurrency	2	1–16
batch_size	100	1–500 traces per run
run_interval_seconds	300	60–86,400
filters	none	at most 20 span-attribute filters: equals, not_equals, exists, not_exists, contains
slice_keys	none	at most 10 attributes kept on each score for alert slicing

Only deterministic and llm_judge scorer kinds run online: a pairwise judge needs two outputs and an executable scorer needs a runner per case. A policy with a judge uses the judge workload class, so judge calls never share a job lane with deterministic scoring. Sampling rules are the same five kinds as a review queue. A policy may name a span queue and a review_rate_ppm: that share of scored results is offered for human review with reason online_sample. version_attribute names the span attribute that labels each score's version; without it the model name is used.

Online scoring runs no customer code. Deterministic scorers are EvalBook's own implementations and judges go through Bookbag SSO. Executable scorers, which run customer code only in the separately installed, outbound-only runner, are not offered online. See the scorer registry.

States

A policy starts disabled. Enabling makes it active and arms its run chain. From active it can move to paused_budget (today's token budget or score limit is spent; it resumes at the next UTC midnight), paused_capacity (the AI gateway refused for capacity; exponential backoff of at most one hour) or blocked (a component it needs is not available on this server; it fails closed until a person enables it again). Disabling works from any state.

A run

Each run is a job (review.online.run or review.online.run.judge) that schedules the next run first, keyed by its time slot, so a failed run never ends the chain. It reads completed traces after the cursor through the traces interface with purpose application, which never returns evaluation, investigation or EvalBook-internal traces, and filters every span to application purpose again, so no judge judges a judge. It applies the environment and attribute filters, samples per trace, and scores each sampled trace with each pinned version under a logical request id of policy, trace and version, which is unique: a re-run never scores the same trace with the same version twice. Then it writes the counters and the cursor and moves the policy's state if a limit was hit.

Coverage per UTC day records seen → filtered → eligible → sampled | not_sampled for traces and scored, errored and skipped for scores, with skips by reason: budget_exhausted, daily_limit, capacity, already_scored, payload_* (the body was not captured, was purged, or is not available under the capture policy) and scorer_*. The score limit spends on attempts, including errors.

Results and budgets

review/api/online/scores lists a policy's scores, newest first: the trace, the scorer version and digest, status, score, explanation, assurance, the version label, which rule sampled it and the review item it was offered to. show returns 14 days of coverage, recent runs and today's budget state, labelled source: measured.

A judge policy gets an AI budget per policy per UTC day of daily_token_budget tokens. Every judge call reserves against it before it is made and settles from the usage SSO returns; EvalBook holds no provider key. When the budget is spent the policy pauses. The cost appears under usage and cost.

Who may do what

online.manage (the project admin) creates, changes, enables and disables policies; review.read sees policies, coverage and scores. Scores feed alert rules and, when offered for review, human labels and judge calibration.