What is EvalBook?

EvalBook is Bookbag's evaluation product. It takes traces from any application or from a CodeBook factory, holds datasets with immutable versions and scorers of three kinds, runs evaluation suites that issue immutable, digested reports, puts people in the loop through review queues, turns failures into improvement proposals, and measures impact from source events with the uncertainty shown. It is useful on its own and strongest next to its siblings.

Identity, organizations, billing, files and models come from Bookbag SSO. CodeBook and MadeBook are peers over a signed protocol. The one sentence the whole suite is built on: CodeBook executes. EvalBook measures. MadeBook authorizes. SSO authenticates, grants model access and meters usage.

Status (25 October 2026). EvalBook is being finished. Traces, datasets, scorers, impact, usage, integrations and project settings are in the application; suites, experiments, reports, review, online scoring, alerts and the improvement pages are not yet in the shell. Nothing is marked verified in the requirements ledger. Every page of this manual says at the top what that means for its area, and Implementation status lists it all in one place.

What it measures

Reading	Where it comes from
Traces	Spans from the SDKs, the HTTP API, OTLP/HTTP exporters and linked CodeBook factories, under the project's capture policy. See Ingestion.
Scores	A deterministic scorer's computation, an executable scorer's observation inside a sandbox, or a model judge's answer, each with a status, evidence and an assurance level. See Registry and versions.
Reports	One evaluation of one exact subject under one frozen suite version: every check's state, the coverage, whether the evidence is complete, and a digest over the body. See Reports.
Agreement	A judge's agreement with people (Cohen's kappa or Krippendorff's alpha with the sample count), and reviewers' agreement with each other. See Calibration.
Impact	Outcomes and source events against a captured baseline, in cohorts, under a method that is named and never called causal. See Impact contracts.
Cost	Tokens and cost allocated from Bookbag SSO's ledger, labelled as allocated, never an invoice. See Usage and cost.

Every metric carries a source (measured, manual, estimated or unavailable) and its denominators. An absent number is shown as "—", never as 0.

The words

Word	Meaning
Project	The work container. A Bookbag organization owns many. A project links to one CodeBook workspace and one MadeBook workspace. See Projects.
Environment, target	Where a subject ran, and what was evaluated. See Environments and targets.
Trace, span	A tree of spans sharing a trace_id: runs, steps, model calls, tool calls, retrievals, evaluator verdicts, human steps.
Dataset version	An immutable snapshot of a dataset's approved examples with a digest, splits and slices.
Scorer version	An immutable definition of a deterministic, executable or judge scorer with a digest.
Suite version	The scorers, dataset versions, thresholds and decision rule an evaluation runs under, frozen together.
Report	The immutable, digested result of one evaluation. Supersession is on the envelope, never in the body.
Assurance	How far a result can be relied on: reported, independently_observed, platform_verified, human_attested.
Governed action	A change to protected configuration that MadeBook decides, or that separation of duties decides in a standalone project.

How it fits beside CodeBook and MadeBook

• A CodeBook station can require an EvalBook suite. CodeBook asks for an evaluation of the exact candidate, waits, fetches the report by id, recomputes its digest and decides. A fail, an error, an inconclusive result, a missing or stale check, or an unreachable EvalBook never passes. CodeBook also exports its runs as traces, tells EvalBook what became of finished work, and turns EvalBook's proposals into cards. See CodeBook as a peer.
• A MadeBook workspace can require "EvalBook suite N must pass" on a pull request. MadeBook re-fetches and re-hashes every report, counts only platform_verified passes on the exact head commit, and decides whether EvalBook may activate a protected suite, scorer, judge, dataset or threshold change. See MadeBook as a peer.
• Without either link, a project works alone. Protected changes then follow EvalBook's own separation-of-duties rule: the author of a change cannot activate it.

What it never does

• It never authorizes. A report is a measurement. Consumers re-fetch it and decide under their own rules.
• It never runs customer code in the API or the worker. Executable scorers run in a separately installed, outbound-only runner, inside a container with no network and a read-only harness. See Executable verification.
• It never calls a model provider directly. Every judge, pairwise, generation and investigation call goes through Bookbag SSO with a reserved budget and an idempotency key.
• It never takes the tenant from a request body. The organization and project come from the API key, the session or the peer link.
• It never edits a version, a report or an audit row. A change is a new version with a new digest; the audit log is hash-chained per organization.

Where to go next

1. Sign in and organizations
2. Projects, then API keys
3. Send a first trace
4. Roles and access before adding your team