Self-observability
What EvalBook can tell you about its own health and work today (readiness, jobs, deliveries, audit, AI call records, retention receipts) and what the self-observability requirement will add.
An evaluation system that cannot account for its own work is not evidence of anything. Today EvalBook accounts for itself through the readiness endpoint, the job, outbox and inbox views, the audit log, the per-attempt record of every model call it makes, and the receipts its retention purge writes. There is no metrics endpoint and no dashboard of EvalBook's own latency or error rates.
Status (25 October 2026). Self-observability (EV-050) is not started in the requirements ledger; what exists today is the readiness endpoint, the platform admin views and the job, outbox and inbox pages described here.
What exists today
| Signal | Where | Who |
|---|---|---|
| Liveness and readiness | /health, /ready; detail at Admin → Readiness or with EVALBOOK_DIAGNOSTICS_TOKEN |
anyone for the yes/no; platform admins and a monitor for the detail |
| Jobs, with attempt history and counts by status | Organization → Deliveries & jobs; Admin → Jobs & deliveries | organization owners and admins; platform admins |
| Outbox (sent) and inbox (received), with conflicts | the same pages | the same |
| Audit log with chain verification | Organization → Audit log; Admin → Audit log | the same |
Model calls: one AiCall row per attempt, with purpose, model asked and answered, usage figures SSO returned, and the outcome including outcome_unknown |
the project's Usage & cost page | usage.read |
| Retention receipts | the project's Settings → Privacy & retention | settings.privacy to change, project.read to see |
| Logs | the process's standard output, level set by LOG_LEVEL |
the operator |
The pages are described on Platform admin and the endpoints and queues on Readiness and operations. Usage figures are allocated from SSO's ledger, never an invoice; see Usage and cost.
Reading the job views
The counts tell you where work is. A rising queued count with a queue warning on the readiness page means no worker is claiming. awaiting_reconcile means an attempt ended with outcome_unknown: the side effect may or may not have happened, and the job waits for a reconcile step or a person rather than being retried blind. A failed delivery has exhausted its 17 attempts and will not move until someone sends it again. A conflict in the inbox means a peer sent the same idempotency key with different bytes; the first delivery stands and the second was recorded, not applied.
Every job carries the organization and project it belongs to, so an organization's view shows only its own work while the platform view crosses organizations.
What the logs say and do not say
Logs carry a correlation id per request and the names of the environment overrides in effect at boot (PEER_SECRET, PEER_CODEBOOK_URL and the rest), never their values. No credential, token, key or peer secret is written to a log by the application.
What is not there yet
The ledger entry EV-050, eval system self-observability, is not_started. The related operations requirements for operational SLOs (OPS-009) and incident response and alerts (OPS-021) are also not_started. Concretely, today there is:
- no metrics endpoint and no export of EvalBook's own request, queue or delivery timings;
- no dashboard of evaluation throughput, judge latency or runner health across projects;
- no alerting on EvalBook's own state; the alert rules in the review component are about the quality of what EvalBook measures, not about EvalBook;
- no advertised service-level targets. The targets in the specification are proposals until load tests run on the deployed topology, and they are not stated on this site.
Until EV-050 is built, an operator watches /health and /ready from outside, reads the job views, and relies on the container exiting when any of its three processes stops. The whole position of every requirement is on Implementation status.
Monitoring from outside
The production container probes its own web port every 30 seconds with a 5-second timeout, after a 90-second start period, and is marked unhealthy after 3 failures. A restart policy should watch /health; a load balancer or uptime monitor should watch /ready, and may read the detail with the diagnostics token. The readiness result is shared for 5 seconds, so frequent polling does not add load. See Deployment.