A test that cannot fail proves nothing

This week we released the starter kit for our governance engine. It lets a second product team build a governed, permission-aware index of their customers' data, prove it correct, and ship it without us sitting beside them. The kit hands that team thirteen conformance checks they have to pass. Which means the checks themselves are now the thing that has to be right, and we built all of them around one rule: every check ships with a deliberately careless build that has to fail it.

The counter that never counted

Here is the rule earning its keep.

One of the thirteen checks says that a governed read must bypass the product's own cache, so an answer never comes from a stale copy of a record. The engine spec writes the test plainly: perform the read, then confirm the cache hit counter is still zero.

That test passes when the cache is correctly bypassed. It also passes when the counter is broken and never counts anything at all. A number that never moves satisfies "still zero" perfectly, and reports success while measuring nothing.

So the harness now carries a positive control. After the governed read, it performs a second read through the ordinary cached path, and that read has to move the counter. If the counter does not move, the harness fails and says why. The check can now tell a cache that was bypassed from a meter that was never wired up.

The same shape, four more times

Once you look for it, that pattern is everywhere. A check written to confirm the presence of something is passed by anything present, including something inert. Four of the thirteen needed the same treatment.

• Revision on reads. Every governed read has to carry the revision of the record it read. A check that looks for the field is satisfied by a connector that stamps a constant, or by a number read once at start-up and never again. The harness now requires the value to move with the record.
• Canonical queries. A figure the engine answers has to agree with the product's own number for it. Seed one value and a constant agrees with it. So each figure now has to take at least two different values across the test cases before its query counts as proven.
• A check that throws. Before a product runs a command a rule covers, it asks the engine. If that call raises an exception, the product has not been told yes. It has been told nothing. The harness requires an actual verdict, not merely the absence of a refusal.
• The wrong record type. A command path that names the wrong entity type quietly skips every rule covering the right one, and then every check passes without looking at anything. In 0.3.1 the engine refuses that call outright: it refuses a command asked about a record no rule of its package covers, one the engine cannot read, and one of a type the package does not define.

The guide cannot rot unseen

The kit includes a step-by-step guide in the engine repository. Documentation that drifts away from the code is worse than no documentation, because people follow it.

So the guide is wired into the build. Every link, every engine type and member name, every capability and every repository path quoted in those pages is asserted against the built assemblies. Rename a method and the guide's own test fails, in the same run, before the release is packed. The wiring the guide shows a reader is itself a test that runs.

A host wired wrong does not start

A package that declares action types needs the host product to supply its own approval seams. Without them, approvals would be the one part of the chain nobody checked.

So the engine refuses to start, and the start-up check names each missing seam. A package with no action types starts cleanly without them, on purpose. The failure arrives in a terminal, by name, at start-up, instead of at the moment somebody approves a payout.

Several of these came out of a review of the kit right after the release, and they ship as 0.3.1: the host check tightened as described above, the project template's change feed now handles a change that commits late, gov-lint prints the content hash the host stamps, and a package file spelled .YAML or .yml is refused rather than silently embedded by one build and ignored by another.

We score the tests, not just the code

Writing a careless build by hand covers the mistakes you thought of. For the rest, we change the engine's source on purpose, one small break at a time, run the whole suite, and count how many of those breaks the suite notices.

When a change does survive, we say so and say why, and the answer has to be that the change altered nothing a person could observe. Anything else is a gap in the suite, and the suite is what gets fixed.

What shipped

Version 0.3.0 went out on 25 October 2026: seventeen packages on our private feed, a project template that creates a package and a test project with one test per conformance item already in place, and three command-line tools for linting a package, verifying an auditor export and evaluating answers.

The release follows its own rule. The whole suite runs, the seventeen packages are packed, a product is then built from the packed files alone, and that product is tested and linted against nothing but those packed files before any of it reaches the feed. A package proven only inside the source tree it was built in has not been proven to ship.

This is the same standard we hold when we put a product into somebody else's environment, next to the roles, the approval doors and the audit trail. A check that tells you that you are safe is worth having only when it could have told you that you are not. That work is described on our implementation page.