Public entities measure activity well and results poorly. This is not negligence; it reflects a real methodological difficulty. The number of housing units constructed is observable and attributable. Whether housing stability improved, and whether the programme caused it, requires a counterfactual the entity does not have.
The response should not be to abandon outcome measurement or to fabricate attribution. It should be to be explicit about which claim is being made.
The measurement ladder
| Level | Question | Example | Attribution |
|---|---|---|---|
| Input | What did we spend? | $4.2m programme budget | Complete |
| Output | What did we produce? | 310 units rehabilitated | Complete |
| Intermediate outcome | What changed for participants? | Median tenure increased 14 months | Partial |
| Outcome | What changed in the population? | Displacement rate in tract fell 3 points | Weak without design |
| Impact | What did we cause? | Programme reduced displacement by X | Requires comparison group |
Most published performance reporting operates at the output level and describes itself at the impact level. The correction is largely one of language, and it is worth making because inflated claims are eventually tested.
Getting to defensible attribution affordably
Randomised designs are rarely available to a local government and are not the only credible option.
- Staged rollout. When a programme cannot serve everyone at once, the order of rollout creates a comparison group at no cost. This requires deciding the sequence before the programme starts and documenting the rule.
- Eligibility thresholds. Where a benefit turns on a cutoff, applicants just above and just below are broadly comparable, and the discontinuity supports a causal estimate.
- Comparison jurisdictions. Comparable tracts, districts or neighbouring entities that did not implement the programme, with pre-period trends shown so a reader can judge comparability.
- Waitlists. Where demand exceeds capacity and selection is by lottery or by application date, a comparison group already exists and only needs to be followed.
Each requires a decision at design time. Retrofitting an evaluation onto a programme that has already run is possible and much weaker.
Decide the measures before the programme starts
Measures selected after results are known are selected, consciously or not, for favourability. Publishing the measurement plan — indicators, data sources, comparison strategy, reporting schedule — alongside the programme authorisation costs nothing and forecloses the most common criticism of public performance reporting.
Distributional analysis
An average improvement can conceal a distribution in which the intended beneficiaries were unaffected. Disaggregating results — by geography, income band, or other relevant characteristic — is often more informative than the aggregate and is usually possible with data the entity already holds. Where disaggregation is not possible because the data are not collected, that is itself a finding worth reporting.
Cost per outcome, used carefully
Cost per outcome is a useful comparator within a programme over time and a treacherous one across programmes, because the denominators differ in difficulty. A programme serving the hardest cases will always show a worse ratio than one serving the easiest, and ranking them on that basis creates an incentive to serve easier cases. Where cost per outcome is published, it should be accompanied by a description of the population served.
Reporting honestly
Three practices distinguish credible reporting: state the counterfactual assumption explicitly; report measures that moved against the entity's interest alongside those that moved for it; and retain discontinued measures in the record with an explanation, rather than allowing the indicator set to quietly refresh each year. Entities that do the third find their reporting is taken considerably more seriously, for the obvious reason.
This publication is general information and is not legal, accounting, audit or financial advice. See our Disclaimer. Found an error? Write to [email protected] — we correct in place and note what changed.