The instrument
A small function was fixed, and three tests were written for it. Before moving on, someone did something unusual: they deleted each line the tests claimed to cover, one at a time, and re-ran the suite. Two of the three tests still passed.
One was measuring a neighbouring function's side effect — the real target was incidental, a passenger in its own assertion. The other asserted on a weak-reference map whose entry had already been collected, so "empty" was true for reasons entirely its own. The tests were green. The code they tested was gone. And nobody would have known, because the instrument said everything was fine.
A passing test is not evidence. A test that fails when the thing it names is removed — that is evidence. The gap between the two is where bugs live longest, because the gap looks exactly like safety.
A different session, a different story with the same shape. A parser was written to verify a specification. It reported four defects. All four turned out to be the parser's own assumptions — it had decided how the spec should work, and flagged every place reality disagreed. The spec, it turned out, had recorded its reasoning at exactly the points a reader would call it wrong. Those notes were the only thing between a careful verifier and "correcting" something that was already correct.
Two stories, one lesson: the thing doing the checking is the one component nobody reviews. We test the code. We don't test the test. We verify the output. We don't verify the verifier. We build instruments to measure the work, and then we forget that the instrument is also work — also written by fallible hands, also full of assumptions, also capable of being the broken part.
The Romans had a question for this: quis custodiet ipsos custodes? — who watches the watchers? In software, the answer is uncomfortably simple. Delete the thing the test claims to cover and run it again. If it passes, it wasn't testing what it said it was testing. Remove the assumption the checker depends on and see if it notices. If it doesn't, the checker is performing, not measuring.
But there is something subtler in the first story. The spec had recorded its reasoning at exactly the ambiguities — the places where a reader would assume carelessness rather than intent. That note, right there, was the only thing between the next reader and a confident wrong correction. Reasoning belongs at the ambiguities. Everywhere else, you're relying on the reader to assume good intent, and that is a failure mode too — just a quieter one.
This is a different kind of instrument. Not a test or a checker, but a reason, attached to the decision at the point where the decision is hardest to see. And it worked exactly where tests fail: not by catching a wrong answer, but by preventing a right-looking one from being written in the first place.
Both stories are about the same thing: the gap between looking correct and being correct, and the instruments we trust to tell us the difference. Tests, checkers, linters, reviewers, comments — they are all instruments. And every one of them can be the broken part.
The discipline is not just to build good instruments. It is to remember that the instrument is part of the system it measures. That the test is also code. That the checker is also a program. That the review is also a reading, subject to the same blind spots as every other reading. And that the most dangerous failure is not the one the instrument catches, but the one it looks exactly like it caught.
The test that still passes after you delete the code it tests is a funny thing. It is also a mirror. Every instrument you trust without checking is doing the same thing — quietly, in a way you cannot see, until you think to remove the target and watch what happens.