Nine stages, one record.
Everything EvalBook does is one of nine stages, and every stage leaves rows with a tenant, a digest and an audit entry. This page walks them in order; the manual has a page for each.
- Stages
- 9
- Report states
- 10
- Customer code
- Runner only
- Authority
- None
01The loop
The loop
Traces in · impact out · people and peers between
01
Traces
Spans from your application, the SDKs, OTLP or a CodeBook run.
02
Datasets
Examples curated from traces and reviews, frozen into digested versions.
03
Scorers
Deterministic, executable in a runner, or a calibrated judge.
04
Suites
Scorers, dataset versions, thresholds and a decision rule, frozen together.
05
Reports
Immutable, digested, with every check’s state and the coverage.
06
Review
People annotate, adjudicate and feed calibration.
07
Improvement
Failure clusters, investigations, proposals sent to CodeBook as cards.
08
Impact
Outcomes and source events against a baseline, method named.
08 → 01: what impact observes becomes the next trace, the next candidate example, the next failure cluster.
02Stage by stage
- 01
A project, a capture policy, a key
A project is the work container, owned by your Bookbag organization. It has environments (development, staging, production, your own), targets, members with separated project roles, a capture policy (full, redacted or metadata only), retention per content class, and API keys whose scopes are a ceiling under their creator’s roles.
- 02
Traces arrive, policy applied first
Spans come from the SDKs, the HTTP API, an OTLP/HTTP exporter or a linked CodeBook factory, at most 1,000 spans and 5 MB per batch, with a batch id that makes a resend idempotent. The tenant is the credential’s, never the body’s. Redaction and capture policy run before anything is stored; a receipt is written durably before the answer.
- 03
Datasets freeze into versions
Examples with a schema are curated from traces, reviews, proposals and incidents, or imported. Candidates wait as pending and are never ground truth by themselves. A version copies every approved example and takes a digest over them; splits keep families together; every freeze queues a leakage scan against every other frozen version in the project.
- 04
Scorers of three kinds
Deterministic scorers compute. Executable scorers run tests, builds and scanners on your own machine in an outbound-only runner, in a container with no network and a read-only harness. Judges ask a model through Bookbag SSO under a reserved budget, and are calibrated against human labels. Each scorer is versioned with a digest and declares what it requires.
- 05
A suite version is frozen, then run
A suite names its scorers, dataset versions, thresholds and decision rule, and freezes them as a version. Activating a version is a governed action. An evaluation pins every version and the digest of its exact subject, runs as leased, fenced jobs, and issues an immutable report with every check’s state, the coverage and whether the evidence is complete.
- 06
Gates re-fetch the report
CodeBook asks for an evaluation of an exact candidate and waits; MadeBook asks which reports exist for a head commit. Each fetches the report by id, recomputes the digest and decides from the body: pass only when every required check passed with complete evidence. The callback that says “done” carries no verdict they act on.
- 07
People review, judges get calibrated
Review queues hold spans, case results, examples and pairwise comparisons. Reviewers claim, annotate, correct and adjudicate; agreement between them is measured; their labels are what a judge is calibrated against. Online scoring samples completed application traces and never evaluation spans, so a judge never grades its own grading.
- 08
Failures cluster, proposals go out as cards
Failure signals from case results, online scores, reviews, incidents and alerts are deduplicated into clusters by subject and signature. An investigation gathers the evidence. A proposal states the problem, a hypothesis, how to reproduce it, a bound and the paths that must not be touched, and CodeBook makes exactly one card of it.
- 09
Impact, with its denominators
Source events arrive from signed webhooks and from CodeBook’s own record of what became of finished work. A contract names a metric from the versioned dictionary, a captured baseline, a cohort and a method, and reports the reading with its uncertainty and coverage. Cost is allocated from SSO’s ledger and labelled as allocated.
03The report
What a report says, and what it refuses to say.
A report is immutable and digested. Supersession lives on the envelope, not in the body, so a digest never changes after it is issued. Every check has one of ten explicit states; nothing is folded into a single score.
- suite
- release-readiness · v7
- mode
- release_gate
- subject
- repository_change · acme/api
- commit
- <40 hex> · base <40 hex>
- subject_digest
- sha256:…
- dataset_refs
- 2 versions pinned
- issued_at
- —
- valid_until
- —
- CHK-01unit-tests · executablerequiredplatform_verifiedpass
- CHK-02json-schema · deterministicrequiredplatform_verifiedpass
- CHK-03answer-grounded · judgerequiredreportedfail
- CHK-04dependency-scan · executableoptional—unsupported
- CHK-05latency-p95 · deterministicoptionalplatform_verifiedpass
coverage
expected 5 · executed 4 · missing 0
evidence_complete
false · CHK-03 is reported, not verified
result
fail
digest sha256:… · the consumer recomputes it over the canonical body and refuses a mismatch · pass only when every required check passed with complete evidence
pass
The check ran and met its threshold with the evidence it needs.
fail
The check ran and did not meet it.
error · timeout
The check could not finish. Never counted as a pass.
inconclusive · abstained
A judge declined or the statistics did not reach a decision.
missing · stale
Expected but not executed, or superseded by a newer report.
skipped · unsupported
Not applicable to this subject, or the operation is not built — said so, not hidden.
04In the suite
CodeBook executes. EvalBook measures. MadeBook authorizes.
What crosses each boundary, and the rules every message obeys.
authenticates
Bookbag SSO
Accounts, organizations, roles, model access, the token ledger, files, billing, the peer directory and the peer secret.
executes
CodeBook
Factories run the work. A station can require an EvalBook suite and waits for the report; runs export as traces; outcomes are told to EvalBook; proposals arrive as cards.
As a peer →measures
EvalBook
Traces, datasets, scorers, suites, reports, review, improvement, impact. It issues reports and asks before it changes protected configuration.
authorizes
MadeBook
Reports become evidence on a pull request only after MadeBook re-fetches and re-hashes them; protected EvalBook changes are decided by its policy and people, never by a model.
As a peer →
- CodeBook → EvalBook
- evaluate this exact candidate · run and step spans · outcome.observed
- EvalBook → CodeBook
- evaluation.completed (re-fetch the report) · improvement proposals as cards
- MadeBook → EvalBook
- reports for this head commit? · evalbook_decision.resolved
- EvalBook → MadeBook
- evaluation.completed · may I activate this suite version? (decision bound to its digest)
Every call: bearer peer secret · X-Peer-App · X-Peer-Signature = HMAC-SHA256 over the raw body · Idempotency-Key on writes · every event carries the link it travels on · a callback never carries the verdict a consumer acts on.
Free at launch
CodeBook executes. EvalBook measures. MadeBook authorizes.
Traces in, scorers over them, reports with a digest, and impact with its denominators. Nothing it reports is authority: it measures, and the people and products around it decide.
Sign in with your Bookbag account · the organization is the tenant