Explainability debates in the private sector usually resolve into a question about model architecture: is this a decision tree or a gradient-boosted ensemble, and can we attribute the output to inputs? For a public entity, that framing misses the obligation. The obligation is procedural. A resident, vendor, or grantee affected by a decision is entitled to know the basis for it in terms they can contest. Whether the underlying computation is linear is largely beside the point.
Two different audiences, two different artefacts
Conflating these is the source of most confusion.
| Technical explanation | Decision rationale | |
|---|---|---|
| Audience | Validators, auditors, model owners | Affected party, elected officials, press |
| Content | Features, weights, attributions, evaluation results | The facts relied upon and the rule applied to them |
| Test | Can a reviewer reproduce the output? | Can the affected party identify what to dispute? |
| Produced | At validation | At the moment of the decision |
A feature-attribution chart handed to a vendor whose invoice was held is not a rationale. "Your invoice was held because the unit price exceeded the contract schedule by more than fifteen per cent on lines 4 and 7" is a rationale. Note that the second statement is generated from the case, not from the model, and remains true regardless of what technique flagged it.
The design consequence
If the rationale must be expressible in the entity's own policy language, then the system should be built so that a flag is always accompanied by a policy citation. In practice this means pairing statistical detection with a rules layer:
- The statistical component ranks — it decides what gets looked at.
- The rules component articulates — it identifies which policy provision the item potentially violates.
- The human decides — and the recorded decision cites the provision, not the score.
Where no policy provision is implicated but the item is still statistically unusual, that is a legitimate reason to look and an illegitimate reason to act. Systems that permit action on unexplained anomaly alone eventually produce a decision nobody can defend.
The question to ask in procurement
"When your system flags an item, what text does it produce that we could put in a letter to the affected party?" Vendors who answer with a confidence score have not built for this environment. Vendors who answer with a policy reference and the specific values that triggered it have.
Contestability
An explanation that cannot be acted upon is decoration. A workable contestability process has four elements: a stated route to challenge, a decision-maker who is not the original reviewer, a defined response time, and a record of the outcome that feeds back into evaluation. That last loop is what turns disputes into a data source. Entities that log dispute outcomes discover systematic errors far faster than entities that rely on periodic validation alone.
What to publish
A short public description of each moderate- or high-tier system: what it does, what data it uses, what decisions it influences, who is accountable, and how to challenge an outcome. Not the model. Not the thresholds — publishing thresholds for fraud detection creates an obvious gaming problem, and it is reasonable to withhold them, provided the entity says clearly that it is doing so and why.
The balance is not novel. Public finance has managed the same tension for decades with audit sampling methodologies: the existence and general approach are public, the specific selection criteria are not. Applying the established convention is more defensible than inventing a new one.
A note on procurement scoring
Automated scoring of bids and proposals deserves particular caution. Procurement decisions are among the most litigated actions a public entity takes, and the record has to withstand a protest. A scoring system whose output cannot be traced to the evaluation criteria published in the solicitation is a liability regardless of how well it performs. Where automation is used, restrict it to mechanical checks — responsiveness, completeness, arithmetic verification — and leave evaluative judgement with the panel.
This publication is general information and is not legal, accounting, audit or financial advice. See our Disclaimer. Found an error? Write to [email protected] — we correct in place and note what changed.