Five rounds of adversarial review on a payment path, and every bug lived in the previous fix
I don't hand-write the code. I direct an AI to build it, and lately I've been learning what that does and doesn't protect you from on the part of a product that moves money.
We had a refund path with a green test suite and a working demo. Then we ran it through repeated adversarial review: each round a fresh reviewer whose only job was to break what the last round fixed. Five rounds. Every round found a real defect, and every defect was hiding inside the previous round's fix.
The pattern never changed. The fix was correct for the exact case we tested and wrong for the case one step to the side. A refund that fails at the bank. The same webhook delivered twice. A retry that succeeds after a failure. None of them showed up in a unit test, because a unit test asserts the case you already thought of.
What actually caught them, in order: a second reviewer who assumed the fix was wrong and went looking; reading the database state back under concurrency instead of trusting what the function returned; testing the money path with two connections racing, not one call at a time.
What made it converge wasn't smarter code. It was refusing to treat "the tests pass" as evidence that the money is right. On a payment path, correct-in-the-common-case is the failure mode, because the common case is the one everyone tests and the one that never breaks.
If you're shipping anything that touches money and you didn't write the code by hand, one review pass is not enough. Curious what others here have used to verify a money path from outside the code itself.
I have not shipped a payment path, so discount this accordingly - my version of the same problem moves files between two machines, not money.
The line I would push on is "every defect was hiding inside the previous round's fix." I hit that shape last week. My worker's auto-commit used a pathspec, which bypasses the index, so it quietly swept unstaged and unrelated edits into job commits. Every test I had stayed green the whole time, because they were all asserting the case I already thought of.
What changed afterwards was not the fix, it was a rule about what a fix is allowed to ship with: the test has to FAIL against the old code. I wrote four checks and ran them against the pre-fix version. Three failed, and those three are the only ones I trust. The fourth passed against the broken code, so whatever it is measuring, it is not that bug.
If the fixes from those rounds went in as separate commits, this costs you almost nothing, because the expensive part is already paid for: run each round's test against the commit immediately before its own fix. Anything that passes there is not the thing standing between you and that defect, and you can decide today whether it gets rewritten or dropped. One caveat from your own list - the racing connections and the twice-delivered webhook are timing-dependent, so a single green run against the old commit settles nothing there; those have to be repeated before either result means anything.
It also gives aryan_sinh's question an answer from the repo rather than from memory: revert one fix at a time and see which of the other tests light up. If dropping round two's fix breaks round four's test, those rounds were poking the same workflow.
The repeated-fix pattern is the interesting part.
Did each fresh review catch a different class of failure, or mostly edge cases around the same workflow?