Built a claim-level fact-checker into our AI rewrite tools — here's how it actually works
Shipping something this week I think this group will find interesting: a fact-check + auto-fix step across 19 of our AI writing tools (paraphrase, humanize, grammar fix, tone, translate, etc).
The problem: any AI rewrite can quietly change what the text actually claims, not just how it sounds. Ran a real test fed in a paragraph with a specific stat, a hedge ("preliminary result, more time needed"), and a specific team size. Asked for a cleaner rewrite. The stat survived. The hedge became a flat claim of proof. The team size changed and picked up a timeline that was never mentioned. Nothing about the output looked wrong it read like clean, confident writing.
How the check works:
User provides (or the tool's own input doubles as) the source document
Each claim in the rewritten output gets checked against it and labeled: Supported / Contradicted / Not found in source / Unclear
"Unclear" is a real, allowed output false accusation of fabrication is worse than a miss, so ambiguous claims don't get forced into a verdict
Auto-fix then corrects flagged claims against the source and re-verifies its own correction
Deliberately scoped narrow: it only checks against the source you give it, doesn't search the web or rule on general truth. That's a reliability tradeoff I'd make again an AI checking a document you control beats an AI guessing at open-ended "truth."
Full writeup with the real before/after example: https://letsflw.com/blog/ai-hallucinations-catch-before-you-publish?utm_source=indiehackers&utm_medium=social&utm_content=ih_post
Curious if anyone else building AI writing/content tools has run into this specific failure mode rewrite reads perfectly, quietly changes a claim. How are you handling it, if at all?
This is a really useful distinction: a rewrite can preserve the general topic while quietly changing the strength or meaning of a claim.
I especially like that “Unclear” is a valid result. Forcing the checker to make a binary decision could make it look more confident while actually reducing reliability.
One test case I’d be interested in is numerical consistency: percentages, dates, sample sizes and words such as “approximately” or “at least.” Those small modifiers are easy to lose, but they can completely change the claim.
Do you compare each corrected version with the original source again, or only re-check the claims that were previously flagged?
The claim-level approach is interesting because rewrites can preserve the wording while quietly changing the underlying meaning. Treating “unclear” as a valid result also seems important—forcing every claim into supported or contradicted would create a different kind of reliability problem.
Exactly, and the "unclear" case ended up mattering more in practice than I expected going in. Early on I assumed most claims would cleanly resolve to supported or contradicted, and unclear would be a rare edge case. It's actually a meaningful chunk of real output, mostly because source documents are often genuinely ambiguous or just don't address a specific detail one way or the other the model didn't necessarily get it wrong, the source just doesn't settle it.
Forcing a binary verdict there would've pushed those into "contradicted" by default (safer-sounding, but wrong) or "supported" by default (which defeats the whole point). Either one trains the user to stop trusting the flags after a few false alarms. Letting it say "I don't have enough here to call this" turned out to be the thing that keeps the supported/contradicted labels actually meaningful when they do fire.
The harder version of this problem, still not fully solved on my end: confidence calibration on the model's own "unclear" calls. Right now it's binary flagged unclear or not but there's probably a real difference between "genuinely 50/50 ambiguous" and "I can tell this is probably fine but I'm not fully certain," and collapsing those into one label loses information. Haven't found a clean way to surface that distinction without making the UI more confusing than it's worth yet.