Redaction and capture
The project's capture policy — full, redacted or metadata-only — applied to every body before it is stored, the path and pattern rules, where redaction is recorded, and what an external judge may see.
Every project has a capture policy that decides what EvalBook stores of the bodies it receives: span inputs and outputs, tool arguments and results, retrieval queries. The policy is applied at ingestion, before anything is written, by the same code that validates the rules a person saves. A project with no policy cannot ingest at all (CAPTURE_POLICY_MISSING); nothing is stored by default.
The mode and rules belong to the project (Settings → Privacy, capability settings.privacy). Traces → Settings shows them read-only and holds the traces-specific choice described at the end: what an external judge may see.
Status (25 October 2026). The traces component is built in the application (ingestion, explorer, settings, exports and deletions pages are live in the shell); its requirements EV-002 to EV-009 are not yet marked implemented in the requirements ledger and are being verified now. Nothing on this page is marked verified.
Capture modes
| Mode | What is stored |
|---|---|
full |
bodies as sent |
redacted |
bodies after the project's rules; needs at least one rule; the paths changed are recorded |
metadata_only |
no body at all — the span records not_captured for each; attributes, timings, usage and errors are kept |
The receipt reports redacted_fields; the span drawer shows which paths were redacted. The original value of a redacted path is not kept anywhere.
What counts as a body
input, output, tool arguments and result, and the retrieval query are bodies. Attributes, events, links, model, usage and errors are metadata. OpenTelemetry content attributes (gen_ai.input.messages, gen_ai.prompt, output.value and the rest) are moved into bodies before the policy runs, so a GenAI exporter cannot carry a prompt past it; see OpenTelemetry.
Metadata is not a place to hide a body either: in redacted and metadata_only modes the pattern rules also run over every string attribute value (span, resource, event and link attributes) and over the error message.
Rules
Rules apply in position order; a project holds at most 100.
- Path rules name a dot path:
input.user.email,tool_arguments.token,messages.*.content.*matches any key or index at that level. A path whose first segment names a body is applied from that body; any other path is applied inside every body. The value found is replaced by the rule's replacement. - Pattern rules are regular expressions (at most 500 characters, written bare or as
/…/i) applied to every string leaf; every match is replaced. A pattern that nests one repetition inside another —(a+)+,(\w*)*— is refused when saved, because one crafted string could pin a CPU during ingestion.
A new project in redacted mode starts with three visible, editable pattern rules: Email addresses → [email], API keys and tokens (prefixes such as sk, pk, evb, evr, ghp, xox…) → [secret], and Bearer credentials → Bearer [secret].
Where redaction is recorded
For each payload EvalBook stores the mode it was captured under, the number of replacements and the changed paths (leaves only; a replaced subtree is one path). Up to 1,000 paths are kept per payload; beyond that the count is kept and the paths are not. A candidate curated from a span carries the source's capture mode and redacted paths into the dataset as metadata; see Curation and synthetic data.
Large bodies
A body is stored inline up to the project's max_body_bytes, never above 256 KB. A larger kept body goes to Bookbag SSO's file store, uploaded outside the ingestion transaction and removed again if the transaction fails. Its digest and size stay on the row.
What external judges may see
An LLM judge sends bodies to a model provider, outside EvalBook. The traces setting judge_body_access decides what a judge may be given, whatever the reader's own permission:
| Setting | In the app | What a judge gets |
|---|---|---|
forbidden (default) |
Never | nothing — the span is unavailable to judges |
redacted |
Redacted again | the stored body with the project's current rules applied once more |
stored |
As stored | the body as the capture policy stored it |
A span whose producer set the attribute evalbook.judge = forbidden is refused to judges under every setting. Only a project admin changes this; every change is audited and carries expected_revision. The same gate is what LLM judges go through when they ask for a span's content, and the span drawer's External judges line says what the answer would be.
What the policy does not do
The policy acts on what arrives. Producers are expected to redact before upload as well (the contract names client-side hooks); what never arrives never has to be redacted. A change of mode or rules applies to spans received afterwards; stored bodies are not rewritten. To remove what is already stored, use a deletion or retention: Retention, export and deletion.