The gate caught me four times

Every figure on this page names what produces it. That started as a stated provenance — a string next to a number saying where it came from — and stated provenance is worth very little, because nothing checks it. So it became a build gate: a recipe derives each value from the source, the build compares, and a mismatch fails before Next.js starts.

This essay is the list of things it caught in me, while I was writing the page it protects. Unlike the other two here, every source it cites is public: the gate, the recipes and the manifest are all in this repository.

One · the number was right and the citation was wrong

A figure read 1,078 tests, sourced to a file called AGENTS.md. The number is correct. That file contains no test counts at all — the figures live in a different document, along with the phrase about zero skips that the label also borrowed. I had cited the wrong file and would never have noticed, because the number was true.

Two · counting files and calling them something else

A figure read 10 collectors, sourced to a directory. The directory holds ten Python files, so counting them gives ten. Two of them are not collectors — one is shared HTTP behaviour, one adapts third-party output — and the project's own handover says, in as many words, nine sources.

This is the same defect class as a snippet of SQL an earlier revision of this page displayed, which declared a policy on a table that exists in no migration. Both looked sourced because a path was printed next to them.

Three · the check was checking itself

The worst one, because it made the gate look like it was working. A claim's expected fragments were written out by hand in the registry, right next to the check. So the check compared the registry's copy of the claim against the source file — and the page was not involved at all.

I found it by mutation testing: I edited a published count from 73 to 74 and the gate stayed green. The page said one thing, the source said another, and the mechanism designed to catch exactly that reported success.

Fragments can now be built from the page's own content, so editing the page changes what gets checked. A check with its own copy of the answer is not a check.

Four · two ways to count a line

I published a document as 742 lines. I had measured it by splitting on newlines and taking the length, which counts one more than there are newlines. The recipes count newline characters, matching wc -l. The file is 741, and the recipe said so the first time it ran.

And a fifth, in the gate itself

A check that no absolute filesystem path reaches this public repository ran for four sprints as a shell one-liner, and was broken the entire time: written as a double-quoted grep pattern, the backslash in a Windows path was consumed before grep ever saw it. The search ran without it and matched nothing, so it reported clean while a committed document carried the paths to every private repository on my machine.

It is now a structural assertion inside the verifier, it runs in every mode including on the deploy host, and it was mutation-tested by injecting a path into a tracked file. It immediately flagged its own explanatory comment, which had spelled the broken pattern out literally.

What this is actually evidence of

Not that I am careless. Four of these five are the ordinary drift of writing about work while doing it: a citation goes stale, a label outruns what was counted, a convention differs by one. The interesting fact is that none of them was found by reading. I read every one of those lines several times.

They were found because something mechanical disagreed with them, and the disagreement stopped the build.