# EvalBook

> EvalBook — Evaluations, traces and measured impact. CodeBook executes. EvalBook measures. MadeBook authorizes.

EvalBook is Bookbag’s evaluation product: traces from any application or CodeBook factory, datasets with immutable versions, scorers that are deterministic, executable or model judges, evaluation suites that issue digested reports and release gates, human review, improvement proposals, and impact measured from source events with its uncertainty shown. Identity, organizations, billing, files and AI come from Bookbag SSO; CodeBook and MadeBook are peers over a signed protocol.

Traces in, scorers over them, reports with a digest, and impact with its denominators. Nothing it reports is authority: it measures, and the people and products around it decide.

## Position in the Bookbag suite

- CodeBook (https://codebook.work) executes: factories run the work; a station can require an EvalBook suite and waits for the report; runs export as traces; outcomes are told to EvalBook; improvement proposals arrive as cards.
- EvalBook measures: traces, datasets, scorers, suites, immutable digested reports, review, improvement, impact.
- MadeBook (https://madebook.ai) authorizes: a report becomes evidence on a pull request only after MadeBook re-fetches and re-hashes it; protected EvalBook configuration changes are decided by MadeBook's policy and people.
- Bookbag SSO authenticates, grants model access and meters usage; it also holds files, billing, the peer directory and the peer secret.

## What it never does

- It never authorizes: a report is a measurement; consumers re-fetch it and decide. Protected configuration changes ask MadeBook or apply separation of duties.
- It never runs customer code in the API or worker: executable scorers run in a separately installed, outbound-only runner, in a container with no network and a read-only harness.
- It never calls a model provider directly: every model call goes through Bookbag SSO with a reserved budget and an idempotency key.
- It never shows an absent number as zero: every metric carries its source and denominators.
- It never takes the tenant from a request body, and never edits a version, a report or an audit row.

## Pricing

Free at launch within a Bookbag organization. The free plan holds three active projects per organization. Paid tiers are not decided; a plan change will never delete evidence.

## Vocabulary

- Project: The work container. One Bookbag organization owns many; a project links to a CodeBook workspace and a MadeBook workspace.
- Trace: A tree of spans sharing a trace id: runs, steps, model calls, tool calls, retrievals, evaluator verdicts.
- Dataset: Examples with a schema, frozen into immutable, digested versions with splits and slices.
- Scorer: Deterministic, executable in a runner, or a model judge. Versioned; a judge is calibrated against human labels.
- Suite: The scorers, dataset versions, thresholds and decision rule an evaluation runs under, frozen as a version.
- Report: The immutable, digested result of one evaluation of one exact subject. A consumer re-fetches and re-hashes it.
- Review queue: Where people annotate, score and adjudicate, and where judge calibration gets its labels.
- Proposal: A problem, a hypothesis and a bound, sent to CodeBook as one card with protected paths.
- Impact contract: A metric from the dictionary, a baseline, a cohort and a method that is named, never called causal.

## In the application today (built; none verified yet)

Each area is built and reachable in the application. The requirements ledger carries it as implemented but unverified, or as being verified now. Do not describe any of it as verified.

- Traces: Spans from the SDKs, the HTTP API, OTLP/HTTP (JSON and protobuf) and CodeBook runs, with durable receipts, batch idempotency, a 1,000-span / 5 MB cap per batch, per-trace sampling and quota backpressure that refuses a batch whole rather than half-storing it. (ledger: EV-002–009 not yet marked implemented; being verified)
- Capture policy and redaction: Per project: full, redacted or metadata-only capture, applied before anything is stored; path and pattern rules with the changed paths recorded; bodies over 256 KB in the organization’s SSO file store; an external-judge body policy that defaults to forbidden. (ledger: EV-007 not yet marked implemented)
- Retention, export and deletion: Retention per content class (raw bodies, traces, results, cost records, audit metadata) enforced by daily purgers with receipts; JSONL exports; deletions that need the project’s exact name, remove derived rows and files, leave a tombstone and withdraw earlier exports. (ledger: EV-009 not yet marked implemented)
- Datasets with immutable versions: Examples with a schema and correction history, frozen into versions whose digest is computed over the examples; splits with families kept together; slices; exact-digest and near-duplicate leakage scans on every freeze; imports that report every rejected row. (ledger: EV-010–018 implemented, unverified)
- Curation and synthetic data: Candidates from traces, reviews, proposals and incidents wait as pending and are never auto-labelled ground truth; generated examples and suggested expected outputs are labelled unvalidated until a reviewer approves them; holdout examples are never shown to the generator. (ledger: EV-015/016 implemented, unverified)
- Scorer registry: Versioned scorers of three kinds — deterministic, executable in a runner, model judge — each with a digest, a score type, a direction, a timeout and what it requires; test cases and test runs per scorer; pairwise judges; activation of a version is a governed action. (ledger: EV-019–028 not yet marked implemented; being verified)
- Model judges with calibration: Judges call models only through Bookbag SSO, with a budget reserved before the call and the charge deduplicated by idempotency key; rubrics are versioned; calibration against human labels reports Cohen’s kappa or Krippendorff’s alpha with its sample count. (ledger: EV-026/027 not yet marked implemented; being verified)
- Executable scorers in an outbound-only runner: Tests, builds, lint and scanners run on your machine in a container with no network, a read-only root and a read-only harness, claimed by a separately installed runner that dials out and never listens. The API and worker never execute customer code. No release gate relies on a runner result until the register records a live run. (ledger: EV-021 not yet marked implemented; runner drivers not yet verified live)
- Impact contracts and source events: A versioned metric dictionary; source events from signed webhooks and from CodeBook’s outcome.observed; baselines captured and approved; cohorts; a contract names its method — descriptive before/after, matched observational or randomized — and an association is never called causal. (ledger: IMP-001–022 not yet marked implemented; being verified)
- Usage and cost, allocated from SSO: Every model call EvalBook makes is a ledger row at Bookbag SSO; EvalBook’s own figures are labelled “allocated from SSO usage” with the ledger reference, missing cost is counted as missing, and spans quoting one ledger reference are counted once. (ledger: IMP-012/013 not yet marked implemented)
- Peer links and governed configuration: Links to CodeBook and MadeBook workspaces from Bookbag SSO’s peer directory; every call signed over its raw body and keyed for idempotency; protected configuration changes ask MadeBook for a decision bound to the change’s digest and fail closed; a standalone project applies separation of duties. (ledger: verified against stand-ins; never crossed to a real CodeBook or MadeBook)
- The capability register: Every adapter operation with its status (verified live, verified against a stand-in, implemented unverified, in progress, not built), the documentation it was built against and its evidence. No operation is verified live, and no gate relies on any of them yet. (ledger: register version 2026-10-15.1)

## Being finished (NOT available yet)

These areas have a contract and in most cases an API, but their pages are not in the application and the requirements ledger has not moved. Preserve the limits in each entry; do not describe them as available.

- Suites, experiments and reports (not available): A suite version freezes its scorers, dataset versions, thresholds and decision rule; an evaluation pins every version and the subject digest; the report is immutable, digested, with explicit check states (pass, fail, error, timeout, skipped, unsupported, abstained, inconclusive, missing, stale) and coverage. (ledger: API routed; pages not in the shell; EV-031–037 not started in the ledger)
- Release gates for CodeBook and MadeBook (not available): CodeBook asks for an evaluation of an exact candidate and waits; MadeBook asks for reports on a head commit. Both re-fetch the report and recompute its digest; pass only when every required check passed with complete evidence. The callback never carries the verdict they act on. (ledger: EV-038 CodeBook side verified; EvalBook side being finished)
- Statistical decision rules (not available): Wilson intervals, paired bootstrap with a recorded seed, exact McNemar, Holm–Bonferroni across slices, non-inferiority with a declared margin, group-sequential monitoring — frozen in the suite version before an experiment starts. (ledger: EV-033–035 not started in the ledger)
- Human review, online scoring and alerts (not available): Queues with assignment, annotation, correction and adjudication; inter-rater agreement; online scoring policies that sample completed application traces and never evaluation spans; alert rules over scores and drift. (ledger: API routed; pages not in the shell; EV-029/030/039/040 not started in the ledger)
- Failure clusters, investigations and proposals (not available): Failure signals deduplicated into clusters by subject and signature; an investigation assistant that calls models through SSO; proposals with a problem, a hypothesis, a reproduction, a bound and protected paths, dispatched to CodeBook as one card per proposal. (ledger: component is an empty shell; EV-041–047 not started)

## Pages

- [Home](https://nurobook.com/): the argument, the loop, what it measures, what it never does, what is in the product, pricing, what is being finished
- [How it works](https://nurobook.com/product): nine stages, the report and its ten check states, the peers
- [About](https://nurobook.com/about)
- [Contact](https://nurobook.com/contact)

## Documentation

A page marked (status note) carries a Status note: its area is being finished or is not yet verified.

### Getting started

- [What is EvalBook?](https://nurobook.com/docs/getting-started/what-is-evalbook): Bookbag's evaluation product in one page: what it measures, what it issues, how it fits beside CodeBook and MadeBook, and what it never does. (status note)
- [Sign in and organizations](https://nurobook.com/docs/getting-started/sign-in-and-organizations): EvalBook has no sign-in of its own. You sign in with your Bookbag account, the organization is the tenant, and people and roles are managed at Bookbag SSO.
- [Projects](https://nurobook.com/docs/getting-started/projects): A project is EvalBook's work container. Creating one, its settings, archiving it with its exact name, the free-plan limit, and how it links to CodeBook and MadeBook workspaces. (status note)
- [Environments and targets](https://nurobook.com/docs/getting-started/environments-and-targets): Environments say where a span came from or where an evaluation ran; targets say what is being evaluated. Both are named by the project and bound to spans, API keys and subjects. (status note)
- [API keys](https://nurobook.com/docs/getting-started/api-keys): Project API keys (evb_) for SDKs, collectors and CI, shown once, with scopes that are a ceiling under their creator's roles; and runner tokens (evr_) for executable scorers.
- [Roles and access](https://nurobook.com/docs/getting-started/roles-and-access): Organization roles come from Bookbag SSO; project roles are EvalBook's own and are separated duties, not a ladder. The capabilities each role holds, and how they combine. (status note)

### Traces

- [Ingestion](https://nurobook.com/docs/traces/ingestion): How a batch of spans reaches a project over the HTTP API, what is checked before anything is stored, and what the receipt says. (status note)
- [OpenTelemetry](https://nurobook.com/docs/traces/opentelemetry): Sending OTLP/HTTP traces to a project, how GenAI semantic-convention attributes are mapped onto EvalBook spans, and what the adapter does with content, environments and schema versions. (status note)
- [Explorer](https://nurobook.com/docs/traces/explorer): Searching traces, reading a trace and its spans, error groups, the live tail and saved views, and which capability each of them needs. (status note)
- [Redaction and capture](https://nurobook.com/docs/traces/redaction-and-capture): The project's capture policy — full, redacted or metadata-only — applied to every body before it is stored, the path and pattern rules, where redaction is recorded, and what an external judge may see. (status note)
- [Sampling and backpressure](https://nurobook.com/docs/traces/sampling-and-backpressure): Per-trace sampling policies by environment, the plan's daily span allowance, per-minute backpressure, and the counters that keep every drop and refusal visible. (status note)
- [Retention, export and deletion](https://nurobook.com/docs/traces/retention-export-and-deletion): Retention per content class with purge receipts and tombstones, JSONL trace exports with a manifest and digest, and deletion requests that remove every derived row and withdraw exports. (status note)

### Datasets

- [Datasets and examples](https://nurobook.com/docs/datasets/datasets-and-examples): A dataset's schema, its working copy of examples with stable keys, splits, tags, metadata and review status, the rules every write obeys, and the SDK endpoints. (status note)
- [Versions and splits](https://nurobook.com/docs/datasets/versions-and-splits): Freezing an immutable, digested version of the approved working copy, the three splits and the two split policies, comparing versions, and checking what a consumer can use. (status note)
- [Slices and leakage](https://nurobook.com/docs/datasets/slices-and-leakage): Named subsets of a dataset with versioned definitions and deterministic samples, required slices that suites must report on their own, and the exact and near-duplicate scans that find leakage across splits and versions. (status note)
- [Import and export](https://nurobook.com/docs/datasets/import-and-export): Importing JSON, JSONL and CSV with a dry run, diagnostics for every row and a resumable apply, and exporting a frozen version as reproducible JSONL or CSV with a manifest. (status note)
- [Curation and synthetic data](https://nurobook.com/docs/datasets/curation-and-synthetic-data): Candidates curated from trace spans, reviews, proposals and incidents, synthetic generation under a token budget, AI-suggested expected outputs, and why none of it is ground truth until a person approves it. (status note)
- [Protected data and governance](https://nurobook.com/docs/datasets/protected-data-and-governance): Protected datasets and holdout membership change only through governed change sets with a digest, applied when an approval for that exact digest arrives, and never authorized by EvalBook itself. (status note)

### Scorers

- [Registry and versions](https://nurobook.com/docs/scorers/registry-and-versions): Scorers are named checks whose behaviour lives in immutable, digested versions that move from draft to review, shadow and, through a governed activation, active. (status note)
- [Deterministic scorers](https://nurobook.com/docs/scorers/deterministic-scorers): Fourteen in-process checks over the recorded output or trace, with no type coercion, bounded pattern matching and explanations that never quote the case. (status note)
- [Executable verification](https://nurobook.com/docs/scorers/executable-verification): Executable scorers run a pinned command in a sandbox on a separately installed, outbound-only runner against an exact commit; EvalBook draws the verdict from what the runner observed. (status note)
- [LLM judges](https://nurobook.com/docs/scorers/llm-judges): A judge version pins a model from the SSO catalog, a prompt template, an optional rubric version and a strict verdict schema; every call goes through Bookbag SSO and a malformed answer is never a pass. (status note)
- [Calibration](https://nurobook.com/docs/scorers/calibration): A calibration measures how well one judge configuration agrees with approved human labels, with sample counts, confidence intervals whose method is recorded, and a stale mark when the judge changes. (status note)
- [Pairwise](https://nurobook.com/docs/scorers/pairwise): A pairwise judge compares a candidate with a baseline in a seeded random order, records every trial, and claims no preference when the order changed the answer. (status note)

### Evaluations

- [Suites and thresholds](https://nurobook.com/docs/evaluations/suites-and-thresholds): A suite version freezes which scorer and dataset versions are run, each check's threshold and minimum coverage, and the decision rule; only a governed activation makes it the suite's gate. (status note)
- [Experiments](https://nurobook.com/docs/evaluations/experiments): An experiment runs a baseline and up to five candidate variants against the same frozen suite version as separate evaluations and compares them example by example under the decision rule frozen in that version. (status note)
- [Reports](https://nurobook.com/docs/evaluations/reports): A report is the immutable, digested verdict of one evaluation; its envelope says whether it is current or superseded, and a reproducibility bundle exports everything it rests on with an explicit list of what is missing. (status note)
- [Release gates](https://nurobook.com/docs/evaluations/release-gates): An evaluation pins a suite version, a subject digest, a deadline and a budget at admission, runs to a report, and answers CI and linked products with a gate verdict and an exit code. (status note)
- [Playground](https://nurobook.com/docs/evaluations/playground): A playground run is an isolated experiment of prompt and model variants, run through SSO models under a token budget, that writes nothing outside itself and announces nothing. (status note)
- [Modes](https://nurobook.com/docs/evaluations/modes): An evaluation records two things — the purpose it serves (offline, online, simulation, release_gate) and how the output under test is produced (existing outputs, a managed prompt, an HTTP target, a repository change, a trace replay); there is no separate "rerun". (status note)

### Review

- [Queues and annotation](https://nurobook.com/docs/review/queues-and-annotation): How review queues take items, assign them to reviewers and record each answer, and how a correction is kept without rewriting ground truth. (status note)
- [Adjudication and agreement](https://nurobook.com/docs/review/adjudication-and-agreement): How reviewers' answers become a decision, how a disagreement is adjudicated, how agreement is measured, and how resolved labels reach calibration and evaluations. (status note)
- [Online scoring](https://nurobook.com/docs/review/online-scoring): Policies that score a sample of completed production traces with pinned scorer versions, within a daily limit and token budget, with every denominator kept. (status note)
- [Alerts](https://nurobook.com/docs/review/alerts): Rules that measure a quality metric over a window against a baseline, fire only with enough samples, name an owner, and record a failure signal for improvement. (status note)

### Improvement

- [Failure clusters](https://nurobook.com/docs/improvement/failure-clusters): The failure-signal interface that groups alerts, failed case results, online scores, review findings and incidents into clusters without flooding anyone with tickets. (status note)
- [Investigations](https://nurobook.com/docs/improvement/investigations): The investigation assistant's contract: model calls through Bookbag SSO under a budget, read-only access to evidence under the person's own permissions, and separation from production scoring. (status note)
- [Proposals](https://nurobook.com/docs/improvement/proposals): What an improvement proposal contains, who may write and dispatch one, the events it emits to peers, and how CodeBook turns it into a card with a bound and protected paths. (status note)
- [Remediation](https://nurobook.com/docs/improvement/remediation): How a proposal is dispatched to CodeBook as a card, how progress and outcomes come back, and what EvalBook does and does not do while a fix is attempted. (status note)
- [Release channels](https://nurobook.com/docs/improvement/release-channels): The channels a suite or scorer version moves through, from draft to shadow to active, and the governed activation that is the only way to make one the gate. (status note)

### Impact and usage

- [Metrics and sources](https://nurobook.com/docs/impact/metrics-and-sources): Where impact events come from, the event vocabulary, signed webhooks and CSV imports, the versioned metric dictionary, and how every figure keeps its source and denominators. (status note)
- [Impact contracts](https://nurobook.com/docs/impact/impact-contracts): The agreement that makes an impact claim checkable, with its baseline, windows, metrics, guardrails, comparison method, stated causal limits and benefit classification, and the reports and alerts that follow it. (status note)
- [Usage and cost](https://nurobook.com/docs/impact/usage-and-cost): What Bookbag SSO reports about EvalBook's model usage, how EvalBook's own calls are recorded, and how charges are allocated to work as figures labelled "allocated from SSO usage", never as an invoice. (status note)

### Integrations and peers

- [The capability register](https://nurobook.com/docs/integrations/capability-register): Every adapter operation EvalBook has or will have, with a status, the documentation it was built against, the version tested and its evidence. A named integration is a requirement, not a claim of support.
- [Peer links](https://nurobook.com/docs/integrations/peer-links): How an EvalBook project is linked to a CodeBook or MadeBook workspace, how every call between the products is authenticated, signed and made idempotent, and what the outbox and inbox guarantee.
- [CodeBook as a peer](https://nurobook.com/docs/integrations/codebook): What crosses between a CodeBook factory and a linked EvalBook project — a station gated on a suite, runs exported as traces, outcomes told to EvalBook, and proposals that arrive as cards — and the messages behind each. (status note)
- [MadeBook as a peer](https://nurobook.com/docs/integrations/madebook): How an EvalBook report becomes evidence on a MadeBook pull request — re-fetched and re-hashed, trusted only as far as its assurance — and how EvalBook asks MadeBook before it changes protected configuration. (status note)

### Administration

- [Platform admin](https://nurobook.com/docs/administration/platform-admin): The /admin area for the people who run EvalBook itself, the four platform capabilities, and the organization views that owners and admins get for their own tenant.
- [Readiness and operations](https://nurobook.com/docs/administration/readiness-and-operations): The liveness and readiness endpoints with their five checks, the job queue, the outbox and inbox, the daily retention purge, and what an operator does when a check fails.
- [Security model](https://nurobook.com/docs/administration/security-model): How EvalBook authenticates people, API keys, runners and peers, how it authorizes every action on the server, what it encrypts, where tenant isolation is enforced, and what it never does.
- [Self-observability](https://nurobook.com/docs/administration/self-observability): What EvalBook can tell you about its own health and work today (readiness, jobs, deliveries, audit, AI call records, retention receipts) and what the self-observability requirement will add. (status note)
- [Implementation status](https://nurobook.com/docs/administration/implementation-status): What exists in the application, what the requirements ledger says about it and which pages are in the shell, area by area, as of 25 October 2026. Nothing is verified yet.

### Local development

- [Local development](https://nurobook.com/docs/development/local-development): Ports, every environment variable, a local SSO, the database, peers, and the checks to run before handing work over.
- [The local e2e SSO](https://nurobook.com/docs/development/local-e2e-sso): scripts/e2e/sso.sh runs a separate, local Bookbag SSO on ports 8290, 3290 and 8299 for testing EvalBook with CodeBook and MadeBook; what it starts, what it writes, and the cookie caveat.
- [Deployment](https://nurobook.com/docs/development/deployment): Coolify builds the repository's Dockerfile from main and runs one container with the API, the worker and the web app; the environment it needs, what must be registered at SSO first, and the production addresses.