Executable verification

An executable scorer runs a command against an exact commit. EvalBook never runs customer code outside the separately installed, outbound-only runner; the API and worker execute no customer code. The runner dials out to EvalBook, claims a job for its project, runs it in a sandbox and reports what it observed. EvalBook draws the verdict.

Status (25 October 2026). The scorers component is built in the application (registry, versions, rubrics, calibration and runners pages are live in the shell); its requirements EV-019 to EV-028 are not yet marked implemented in the requirements ledger and are being verified now. The capability register lists the runner's container and process drivers as not built and no executable scorer has run against a live runner; until that evidence exists no release gate relies on one.

What an executable version pins

• Image: image_ref plus image_digest (sha256:…), run exactly as pinned.
• Command: up to 64 arguments, with ${CANDIDATE}, ${HARNESS}, ${CASE}, ${REPORT} and ${SCRATCH} placeholders.
• Environment: up to 64 variables, never secrets; a credential-like name or value is refused when the version is created.
• Network: deny (default) or an allowlist of up to 32 host:port names, never addresses.
• Protected harness: up to 200 files (256 KB each, 1 MB together), each digested, mounted read-only.
• Artifacts: up to 32 declared paths; one may be the report; content is kept inline up to 64 KB.
• Limits: by default 1,024 MB, 1 CPU, 256 processes, 262,144 bytes of output, 256 MB scratch, 600,000 ms; at most one hour.
• Driver: container (default), process or any, and labels a runner must carry.

A binary version passes on its exit code (pass_exit_codes, 0 by default). A continuous version reads its score at report_score_path in the JSON report. The case must name subject.repository.full_name (owner/name on GitHub) and a 40-character subject.commit_sha; it is written to /case/case.json (at most 256 KB).

The runner

The runner is one dependency-free file, evalbook-runner.mjs (Node 20+), plus git and Podman or Docker. A project admin (runner.manage) creates its token (evr_…) on Scorers → Runners; it is shown once and may expire after a chosen number of days. The runner registers its drivers, labels and concurrency, and acts within the authority of the person who created the token: if they lose runner.manage or leave the organization, the token authorizes nothing.

One job

1. Claim: a 60 s lease and a fencing token; the runner heartbeats every 15 s. A late result from a runner that lost its lease is 409 STALE_FENCING_TOKEN and never recorded. A job has 3 attempts and a deadline.
2. Check out the exact commit. For GitHub the runner asks EvalBook for a read-only installation token for that one repository, minted by SSO for at most an hour and never stored; git receives it through its environment, never inside the sandbox.
3. Write the protected harness, verify its digests, make it read-only.
4. Run the command in the sandbox below.
5. Report what it observed: driver, image digest, exit code, duration, output digests with the last 8 KB of each stream, the commit, the harness digests before and after, the artifacts, and whether the container was removed.

The sandbox

Control	Container driver
network	--network none; with an allowlist, a proxy admitting exactly the listed pairs (private, loopback, link-local and metadata addresses refused)
filesystem	read-only root; size-limited tmpfs scratch; /candidate writable, /harness and /case read-only
identity and resources	--user 65534:65534, --cap-drop ALL, no-new-privileges; --pids-limit, --memory, --cpus, the version's timeout, an output cap
environment	the version's variables and EVALBOOK_* paths; nothing of the machine, never the token
cleanup	the container's removal is confirmed; otherwise the runner takes no more work

The candidate cannot change the harness, but it can read it and the case, including an expected answer: keep ground truth a candidate must not see out of executable cases.

EvalBook's verdict

The verdict is computed from the report, in order: the commit checked out is the requested one; the harness digest before and after the run equals the version's; the image digest equals the pinned one; a timeout is timeout; every required artifact is present with a digest; then the exit code or the report score. A failed check is an error, never a pass.

Assurance

Driver	Assurance
container	platform_verified
process	development_unisolated: a plain child process with no isolation, refused unless EVALBOOK_RUNNER_ALLOW_PROCESS_DRIVER=1

A release_gate evaluation turns a development_unisolated pass or fail into unsupported (release gates). A runner that could not confirm a container's removal is blocked until a person restarts it.

The Runners page

Scorers → Runners lists each runner (online when heard from within 90 s) with its drivers, labels and jobs, and every runner job with its verdict, assurance and delivery. A result that found no evaluation listening is undelivered and offers Deliver again.