Sampling and backpressure

Three things limit what a project stores. Sampling is the project's own choice: which traces to keep, per environment, decided per trace. The plan quota is the organization's daily span allowance. Backpressure is an operational guard on spans per minute. Each refusal is counted, so the denominators beside every list stay true, and a refused batch is never half-stored.

Status (25 October 2026). The traces component is built in the application (ingestion, explorer, settings, exports and deletions pages are live in the shell); its requirements EV-002 to EV-009 are not yet marked implemented in the requirements ledger and are being verified now. Nothing on this page is marked verified.

Sampling is decided per trace

A trace's spans may arrive in several batches, so the decision is made for the trace id, not the span, and the same id gets the same answer every time. In order:

1. An EvalBook-internal span (its own evaluation and judge spans) is always kept.
2. A trace EvalBook already holds keeps every later span, however it was first kept.
3. The first rule that matches sets the rate; otherwise the policy's rate.
4. The trace is kept when a deterministic hash of its id falls under the rate.
5. A trace that was not kept, under a policy with Always keep traces with an error, is kept as kept_error if a span of it in this batch is an error. Spans of that trace from earlier batches were already sampled out; the trace page says so.

Rates are integer parts per million in the API and a percentage in the app. With no policy for an environment, and no * policy for every environment, every trace is kept.

Rules

A policy holds up to 50 rules, each with a rate and an optional label. A rule matches when any span of the trace in the batch satisfies it:

match_kind	Matches
attribute	an attribute key equals a value (span or resource attributes)
root_name	the root span's name
span_kind	the trace has a span of that kind

Setting a policy

Traces → Settings → Sampling lists the policies by environment with their rules. Only a project admin (project.admin) edits them. A new rate applies to traces that arrive after it is saved; nothing already stored changes. Removing a policy means its traffic is kept in full again unless the * policy covers it. Every write is audited and carries expected_revision.

The same through the API: GET traces/api/settings, POST traces/api/settings/sampling with environment_key, rate_ppm, keep_errors and rules, and POST traces/api/settings/sampling/delete.

The decision you will see

Each trace records how it was kept — kept_rate, kept_rule (with the rule's label), kept_error, kept_always — and the rate in force; the trace page shows it as Kept by. Dropped traces are not stored, but they are counted: dropped_sampling_traces and dropped_sampling_spans per hour, and dropped_by_sampling on the batch's receipt.

The plan quota

The organization's plan sets trace_spans_per_day, counted per UTC day across its projects; a plan with no such limit has none. A batch that would cross it is refused whole with 429 QUOTA_EXCEEDED and a Retry-After reaching the end of the day, and counted as refused_quota_batches and refused_quota_spans. If Bookbag SSO cannot say which plan applies, the batch is allowed and the page marks the plan as unchecked. Traces → Ingestion shows the plan, spans per day, the allowance used and today so far.

Backpressure

Independently of the plan, one organization may send at most a configured number of spans per minute per API process — 120,000 by default; operators set EVALBOOK_INGEST_SPANS_PER_MINUTE. Over it is 429 INGEST_RATE_LIMITED with a Retry-After to the end of the minute. It is an operational guard against one tenant flooding the ingestion path, not a commercial limit. Two concurrent batches can each pass the daily check; the overshoot is bounded by one batch per concurrent request.

The counters

Per organization, project, environment and UTC hour EvalBook keeps: received_spans, accepted_spans, rejected_spans, duplicate_spans, dropped_sampling_spans, dropped_sampling_traces, kept_traces, refused_quota_spans, refused_quota_batches, refused_size_batches, refused_invalid_batches, redacted_fields, bodies_not_captured and body_bytes. The explorer sums them for the window shown (Explorer) and the ingestion page charts them by hour. Receipts and counters older than the project's traces retention are purged with the traces.

What is never sampled

EvalBook's own spans are the platform's evidence and are kept whole. Spans with purpose: evaluation or investigation are also never picked up by online scoring, so a judge is never judged recursively; see Online scoring.

For the producer

Treat 429 as "send this batch again after Retry-After", unchanged: the batch was not stored, and the same batch_id resent is idempotent. Treat 413 as "split the batch". A 202 with dropped_by_sampling > 0 is not an error; it is the policy working.