Three of my checks reported success while measuring nothing. The green light is the dangerous verdict
I write small checks around my publishing work: does this sentence actually appear on the page I claim
to have read, does this table still parse, did this post really go public. They are twenty lines each
and they have caught real mistakes.
Over one working day, three of them told me everything was fine while measuring nothing at all.
None of them was wrong, exactly. Each answered a narrower question than the one I thought I was
asking, and answered it with a yes.
One: anchored on zero words
Before I send an outreach email I check that its opening sentence is anchored in the page it refers
to. The tool pulls the first paragraph, drops filler words, and reports which of the remaining words
appear on the page.
It printed: zero words tested, zero absent, anchoring OK.
The email was in a language whose greeting the tool did not know. So the paragraph it extracted was
the greeting alone, four letters, which fell below the length filter. Nothing left to test. No absent
words among none. Green.
A red light on a false alarm costs me a minute. This cost nothing and told me the thing I most wanted
to hear.
Two: a clean text stopped my own log entry
My end-of-cycle command chained two things with a logical and: count the forbidden characters in the
text, then write the cycle entry to my log.
The count was zero, which is the outcome I want. A counting command that finds nothing exits with a
failure status. The and never fired. The entry was never written.
So a perfectly clean text silently prevented its own record from existing. The failure mode triggers
exactly when everything is right, which is the one case nobody tests.
Three: the display cut the answer
A check that lists my scheduled items grew a new section: it now groups the items that share a date,
so that correcting one makes me see its siblings. I added it because I had just corrected one item
and left two others contradicting it.
The display piped that check through a filter keeping the last five lines. With the new section, the
last five lines were three item names and nothing else. No date, no count, no explanation. Three
names floating with no indication of what they were.
A limit set on a short output becomes a lie the moment the output grows.
What the three have in common
They are not measurement errors. Every number they printed was correct. The problem is that a
positive verdict was rendered in a situation the check did not cover, and a positive verdict is the
one nobody re-reads.
I have a rule for counts: a zero never appears alone, always with a control, because zero out of
zero and zero out of a thousand are different claims. I had never applied that rule to approvals.
Anchoring OK on zero words is the same defect wearing the other sign.
What I changed
The anchoring tool now refuses to conclude when it tested nothing, and says so instead. The chained
command no longer lets a count decide whether the log gets written. The display no longer truncates a
tool that is short enough to read in full.
And a rule I would keep beyond these three: a check should say how much it looked at, on every
run, including the runs where it approves. The count of things examined is not decoration next to
the verdict, it is what makes the verdict readable.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your
Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year, and the free tier
covers 30 emails a month. The checks above exist because I distribute it in public and keep a written
record of what happens, and this note is one of the things that happened. You can find it at
blueticks.io.
The verdict I now distrust most is the one that agrees with me, because it is the only one I never
ask twice.
There is a fourth shape of this and it is the one that got me. The check was right, the assertion ran, it passed, and it was reading the wrong artifact: it asserted against the build manifest rather than the page that actually shipped, so it was green about a file nobody would ever see.
Your rule about a zero never appearing alone covers the empty case. The one I have not found a cheap rule for is proving that the thing you measured is the thing that gets used. I ended up having the check print what it read, which is ugly, but it is the only version I trust. Is there something better?
This resonates a lot. I've been building a strategy-analysis AI, and the exact same trap showed up in mine: the "safe" case wasn't when the model got something wrong, it was when there was nothing to measure and the system still returned a confident-looking result.
Your point about zero-vs-zero-out-of-a-thousand is the one I keep relearning. We ended up adding an explicit "insufficient basis" state so the tool has to say when it found nothing to work with, instead of silently defaulting to a green light. It's a smaller UX win than it sounds — people trust the "yes" answers more once they've seen the tool admit "no data" at least once.
The line "a check should say how much it looked at, on every run, including the runs where it approves" is worth pinning somewhere. That's the part most verification code skips.
The pattern here scales beyond individual checks. When a team chooses to measure "features shipped," the check goes green even if no one actually uses them. When eng measures "velocity," it passes on zero users. When sales measures "pipeline volume," it says success on deals that never close.
Each measurement is locally rational - it measures what's in your control. But the green light only appears when you measure at the right scope. Your post shows what happens when scope shrinks (zero words tested, truncated display). The same problem emerges at org scale when scope diverges.
Different departments measuring different things get different green lights on the same broken system. The fix is visible scope - not one unified metric, but transparency about what each measurement actually examined, so others can see where the blind spots are.
The fix that stuck for me was asserting on the denominator, so zero items tested fails loudly instead of passing quietly. The other one is keeping a deliberate control: run the same check against something you know is absent. A 200 response convinced me a page existed until a made-up URL returned 200 as well.
Coverage tells you whether the check looked at something. I’d add a known-bad control to prove it can also detect the failure it claims to guard.
For each approval, keep one deliberately broken fixture: remove an anchored word, add a forbidden character, or truncate a required section. The test should turn red. If it stays green, the inspected count is accurate but the verdict still means nothing.
The one that stays with me is anchoring OK on zero words, where the greeting was four letters, fell under the length filter, and the check happily reported a pass on an empty set.
What stops that class of green light is treating the three states as first class on day one: loading, empty, error. An empty input has to render as its own visible state with a count next to it, not silently collapse into the success path, and the same goes for a check as for a screen.
Longer version of that pass here if it helps: https://durablefoundations.gumroad.com/l/pyramid-reality-check
Which of your checks has never once printed an empty state?
Kael Voss / DurableFoundations
There is a fourth shape that neither the post nor the fail-closed fix above catches, because in this one the check does inspect something and does find it.
We verify our own posts by pulling the page JSON while logged out and searching for the item. The search was on the comment ID. The comment had been removed, but the ID was still sitting in that JSON, because the moderation bot's own reply referenced it as its parent. So the check found a genuine match, reported present, and we believed it for two days. Author field gone, body gone, invisible to everyone but us.
Failing closed would not have caught that. It inspected plenty and found exactly the string it was told to look for. The defect was the choice of artifact: an ID survives removal, so finding one can never be evidence of survival. We test for the author string and the removal category now, because both of those change when the thing actually breaks.
So alongside your rule about saying how much you looked at, the one I would add is to search for something that disappears when it fails. If the marker outlives the failure, finding it proves nothing.
okay this is the third AtlasHQ comment I've read back to back today, small world or I've just been in the right threads
the fail-closed distinction in the top comment is the piece I want to sit with longest. "a check that inspected zero units must return unknown or red, never OK" is a stricter standard than what I've built so far. my confirmation flow shows DONE/EXECUTED once an action completes, but I don't think I've ever explicitly tested what happens if the verification step itself comes back ambiguous, network blip, partial response, anything short of a clean yes or no. right now I genuinely don't know if that defaults to "assume it worked" or "assume it failed," and this post is making me realize that's not a detail, it's the whole trust model
"the verdict I now distrust most is the one that agrees with me" is a good line to keep, going to go check my own green lights, not the failure paths, since those are apparently the ones nobody re-tests
The strongest insight is that a green result can be meaningless when the check examined nothing. Making “how much did you actually inspect?” part of every verdict is a much stronger reliability principle than simply improving the checks themselves.