Skip to content
EvalBook

The local e2e SSO

scripts/e2e/sso.sh runs a separate, local Bookbag SSO on ports 8290, 3290 and 8299 for testing EvalBook with CodeBook and MadeBook; what it starts, what it writes, and the cookie caveat.

scripts/e2e/sso.sh runs a separate, local Bookbag SSO for testing EvalBook against CodeBook and MadeBook. It never touches the owner's development SSO and never production. It lives in its own checkout, the Bookbag/SSO-evalbook worktree on branch feat/evalbook, with its own databases, its own encryption key and a model provider that is a test double.

e2e SSO the owner's SSO (never touched)
checkout Bookbag/SSO-evalbook (feat/evalbook) Bookbag/SSO
API and issuer http://localhost:8290 :8090
web (sign-in, admin) http://localhost:3290 :3200
databases fresh components/*/db/e2e.sqlite3 in the worktree development.sqlite3
encryption key generated once into SSO-evalbook/.env.e2e (gitignored, chmod 600) its own
AI provider scripts/e2e/ai-stub.mjs on :8299, a test double whatever the owner set

Use

source ~/.nvm/nvm.sh && nvm use 24.19.0
export E2E_SECRETS_DIR=/somewhere/private        # optional; default log/e2e/secrets (gitignored)

scripts/e2e/sso.sh start     # first start bootstraps everything; later starts reuse it
scripts/e2e/sso.sh verify    # live checks, PASS/FAIL per line, exit code = failures
scripts/e2e/sso.sh status
scripts/e2e/sso.sh stop      # stops only the processes it started, by PID

Logs and PIDs are in log/e2e/.

What the first start does

  1. Writes SSO-evalbook/.env.e2e with a random 32-byte SSO_ENCRYPTION_KEY, after checking that git ignores it, and derives an e2e environment file (no secrets) from the development one: port 8290, issuer http://localhost:8290, UI http://localhost:3290, every context on its own e2e.sqlite3.

  2. Starts the AI stub, then creates the schema and applies the migrations.

  3. Registers three apps with SSO's own registration script:

    app origin peer path
    evalbook http://localhost:3006 /project/api/peer
    codebook (its e2e instance) http://localhost:3283 /workspace/api/peer
    madebook (its e2e instance) http://localhost:3281 /workspace/api/peer

    Each app token is written to $E2E_SECRETS_DIR/tokens.env as EVALBOOK_SSO_APP_TOKEN, CODEBOOK_SSO_APP_TOKEN and MADEBOOK_SSO_APP_TOKEN, with SSO_ISSUER. SSO shows a token once; if one is missing from the file the app is re-registered with --rotate, which retires the old token.

  4. Creates a local-only platform admin, e2e-admin@bookbag.test, with a generated password written to $E2E_SECRETS_DIR/admin.env (E2E_ADMIN_EMAIL, E2E_ADMIN_PASSWORD).

  5. Registers the stub as the platform's provider (custom, base URL http://localhost:8299/v1, chat model evalbook-stub-chat as the default, embedding model evalbook-stub-embed).

  6. Starts the SSO API (master=e2e) and the SSO web app on 3290 with its own dist directory and its API addresses pointing at :8290, so it never proxies to :8090.

Both .env files are chmod 600, and no token, password, key or peer secret is ever printed. The SSO API runs with SSO_ALLOWED_ORIGINS set to its web origin and the three product origins, SSO_CHAT_ORIGINS set to the three product origins, and SSO_PRIVATE_HOSTS covering localhost and 127.0.0.1 on 3006, 8086, 3283, 8283, 3281 and 8281, so events can reach the products' local receivers once their events URLs are set in the e2e SSO's app registry at http://localhost:3290/admin/apps.

Pointing EvalBook at it

In .env.local: SSO_ORIGIN=http://localhost:8290, SSO_UI_ORIGIN=http://localhost:3290, and SSO_APP_TOKEN copied from tokens.env with an editor (do not echo it). In nextjs-app/.env.local: NEXT_PUBLIC_SSO_ORIGIN=http://localhost:3290. EvalBook asks SSO for its client id and redirect address at /apps/api/bootstrap, and the products learn each other and the peer secret from /apps/api/peers. The rest of the local setup is on Local development.

The AI stub is a test double

ai-stub.mjs answers the chat-completions wire (blocking and streamed) and the embeddings wire deterministically, with usage counted from the text, so SSO meters and ledgers every call exactly as it would a real provider's. The reply is [ai-stub] <hash>, or whatever follows STUB_REPLY: in the last user message; STUB_TOOL: <name> <json> returns a tool call and STUB_FAIL: <status> an error. Every response is marked X-Test-Double: evalbook-ai-stub.

Nothing it says is a measurement. A judge run on it proves the plumbing (budget reservation, idempotent charge, the AiCall record), not the output, and nothing scored by it counts as evidence. See LLM judges.

Caveats

  • Cookies are not port-scoped. The e2e SSO's session cookie and the owner's SSO on :8090 share the name sso on localhost; signing in to one in a browser replaces the other's session. Use a separate browser profile for the e2e SSO.
  • Do not run a production build in SSO-evalbook/nextjs-app while its dev server is up; stop it first with sso.sh stop.
  • Starting fresh: sso.sh stop, then delete SSO-evalbook/components/*/db/e2e.sqlite3 and the secrets files; the next start bootstraps again. Keep .env.e2e unless you also delete the databases, because the signing key and the platform AI key are sealed with it.
← PreviousLocal developmentNext →Deployment