Impact contracts
The agreement that makes an impact claim checkable, with its baseline, windows, metrics, guardrails, comparison method, stated causal limits and benefit classification, and the reports and alerts that follow it.
An impact contract fixes, before anyone claims an improvement, what is claimed, where, how it is measured, how it is compared and what may be concluded. Every impact claim links to a contract and to the version in force. Missing data makes a result incomplete, never zero, and no comparison method except randomized assignment lets EvalBook call a difference the intervention's doing. In the app, contracts live under Impact → Impact contracts.
Status (25 October 2026). The impact component is built in the application (Impact and Usage & cost pages are live in the shell); its requirements IMP-001 to IMP-022 and EV-049 are not yet marked implemented in the requirements ledger and are being verified now.
What a version pins
A contract has immutable versions, each digested over its content:
- Purpose, intervention and
causal_limits, in words. - Metrics from the dictionary, each pinned to a metric version: exactly one
primary, anysecondary, at least oneguardrailwith a limit (max,min,max_increase,max_decrease). - Sources (at least one), scope and rules (
include,exclude, andtreatmentfor matched designs). - Windows: a baseline and an observation window, each with a cohort definition in words.
- Comparison:
comparison_method,minimum_samples,maturity_days(0–365),alpha(default 0.05). - Cost methodology and the cost categories it requires; a missing category makes cost incomplete.
- Human effort:
observed,estimated,self_reportedornot_measured; measuring people's effort needs the consent statement they agreed to. - Benefit classification:
observed_operational_change,estimated_capacity_value(needs an approved loaded hourly rate) orfinance_verified_cash_saving(needs a finance-verified source).
A revision is a new version with a change_note; earlier versions and what was measured under them stay.
Lifecycle
A contract moves draft → active, active ⇄ paused, and from any of those to closed. Activation captures the baselines, enrolls units into cohorts and computes the first observations.
A baseline is captured per metric over the baseline window as of activation; a window with no data is not_measured, a stated limitation, never an invented historical value. It is proposed until a person approves it. An observation exists per metric and arm: pending → provisional → mature, or insufficient_data / not_measured. Every recomputation appends a revision; a late event that changes a mature value moves it to corrected and keeps the earlier value in the history.
Comparison methods and what they allow
comparison_method |
Arms | What EvalBook says |
|---|---|---|
descriptive_before_after |
baseline window vs observation window | anything else that changed between the windows can explain a difference; an association, not causation |
matched_observational |
treatment vs control, matched exactly on the declared variables; unmatched units are reported, never paired loosely | unmeasured differences can explain the result; an association, not causation |
randomized |
treatment vs control by a recorded seed over the unit key at the declared treatment_share; an arm is assigned once |
a supported difference can be attributed to the intervention, within the stated uncertainty and for the units in scope |
phased |
groups of one dimension switched on at recorded times | changes over time and between groups can still explain a difference; an association, not causation |
This is the architecture's rule: attribution states its method and never calls an association causal. The comparison applies Holm's correction across metrics at the contract's alpha, evaluates each guardrail as ok, breached or unknown, and flags a task-mix shift of more than 10 % between the windows. The result is named Association, not causation unless the design is randomized.
Costs and benefits
Benefit terms are computed from the contract's own figures (eligible tasks, minutes per task, the approved hourly rate, finance-verified cash events, and the costs allocated to the contract; see usage and cost), with unknown cost items counted rather than priced. The separation is printed with the result: estimated capacity is not cash, cash savings are not revenue, and only finance-verified events count as realized cash benefit.
Reports, alerts and surveys
Under Impact → Reports & alerts, a schedule (weekly or monthly, over the last 1–366 days) builds an immutable run: the metric versions and snapshots it showed, its completeness and its caveats. An incomplete run says so in its first line; nothing incomplete is presented as an improvement. Recipients are organization members resolved from Bookbag SSO, and every delivery attempt through SSO mail is recorded with SSO's answer; the template evalbook_impact_report does not exist at SSO yet, so deliveries record its refusal until it does. Impact alerts watch a metric above or below a threshold, a guardrail_breach, a source_stale or a metric that is data_incomplete; unknown is reported as such.
Under Impact → Surveys, developer-experience surveys are voluntary and anonymous: answers are totals per question, shown only once at least min_group_size people (never fewer than 3) have answered, with no individual scores.
Reading needs impact.read; writing needs impact.write. See roles and access.