Alerts

An alert rule measures one metric over a window of online scores or evaluation results and compares it with a baseline. It fires only when the window has enough samples, and every evaluation of a rule is recorded whether it fired or not, so "why did it not alert?" always has an answer. An alert names what regressed, against what, for which version and slice, on how many samples, and who answers for it.

The alert API is under review/api/alerts/…; the Alerts page is being finished. Alerts on impact contracts are a separate thing, described under impact contracts.

Status (25 October 2026). The review API exists in the application (queues, items, score definitions, online policies and alert rules are routed) but the Review, Online scoring and Alerts pages are not yet in the shell, and requirements EV-029, EV-030, EV-039 and EV-040 are not marked implemented in the requirements ledger. This page describes the contract the component is built to.

A rule

A rule (review/api/alerts/rules, with create, update, enable and disable) has:

Field	Values
source	online (the scores of one online scoring policy) or evaluation (case results, through the evaluations interface)
metric	pass_rate, mean_score, error_rate, judge_disagreement (online only)
slice_key, slice_value	optional; the key must be one of the policy's slice keys
window_minutes	default 60, 5–43,200
evaluate_every_minutes	default 15, 1–1,440
baseline_kind	fixed (with baseline_value), previous_window, pinned_report (with baseline_report_ref)
comparator, threshold	drops_below or rises_above the baseline by more than the margin threshold
min_samples	default 30
cooldown_minutes	default 60, 0–10,080
severity	info, warning, critical
owner_user_id	the next responsible person; must hold review.read on the project

An online rule names its policy and may name a scorer; an evaluation rule may name a suite. A pinned report must be a report of this project that carries the metric (see reports). Status is active or disabled. Writing rules needs alert.manage.

Evaluations

Each evaluation measures the window ending at the last slot boundary and records one of fired, ok, insufficient_data (fewer than min_samples in the window or in a windowed baseline; a small sample never fires), baseline_unavailable (the pinned report could not be read or lacks the metric) or blocked (the source of the metric is not available on this server; the rule fails closed). One evaluation per rule per window is kept, so a retried job changes nothing. review/api/alerts/rules/evaluate evaluates a rule now. Each evaluation stores the current value and sample count, the baseline value and its sample count, and the baseline reference.

Events

A firing opens an event (review/api/alerts/events): the metric and value, the baseline and where it came from, the margin, sample counts, the window, the affected version (the most frequent version labels among the scored traces), the slice, the scoring coverage behind it (eligible, sampled, scored from the policy's daily counters; source: unavailable for an evaluation rule) and the owner.

Repeats are deduplicated: while an event is open or acknowledged, another firing of the same rule and slice updates it (fire_count, latest values) instead of opening a second, and an ok evaluation stamps cleared_at without closing it. The owner is notified through the SSO inbox and by email on the first firing, and again only after cooldown_minutes.

Events move open → acknowledged → resolved. Resolving needs a note and retires the deduplication key, so the next firing is a new incident. Events are the incident record and are not purged by retention; evaluations, online scores and run history older than the project's results cutoff are.

Failure signals

Opening an event records a failure signal with the improvement component through recordFailureSignal with kind alert, a signature over the rule, slice and metric, the affected version, and the event as evidence. This runs in a job of its own, so an unavailable improvement component never blocks an alert; the event shows signal_state as pending, recorded (with the cluster it joined) or unavailable, and an unavailable signal is tried again. See failure clusters.

Judge drift

judge_disagreement is measured on online scores that people reviewed: for each binary definition linked to the same scorer, the share of resolved human labels that disagree with the judge's pass or fail in the window. It needs a policy that offers scores to a review queue, and it is the alerting side of judge calibration and LLM judges.

Notification delivery

Every notification EvalBook owes a person (review_assigned, sla_breach, adjudication_needed, alert_fired) is a row with a unique key per fact, person and channel, so it is sent at most once by construction; nothing secret travels, only a title and a link of public ids. A send that may have reached SSO without answering is recorded as failed with OUTCOME_UNKNOWN and left for a person: review/api/notifications lists the history and retry sends a failed one again.