I built a public dashboard that shows almost nothing, on purpose
I shipped a two-agent sales kit a couple weeks ago. Two agents, one shared attribution layer, sold as a self-deploy kit for $349. And the dashboard that's supposed to prove the thing works currently shows the honest empty state, because no deals have closed yet.
That's not a launch problem I'm hiding. It's the design.
Here's the thing about selling anything that claims to measure AI's contribution to revenue: everyone claims it, almost nobody can show it, and a buyer who's paid attention knows the difference between a vendor number and a real one. So before I worried about what the dashboard would eventually show, I spent real time on whether the thing generating those numbers could be trusted not to lie to itself.
The two agents, InboxCopilot for inbound email and EnrichmentMessenger for CSV enrichment, both write to the same database as capped, conservative attribution credit. Before shipping either one, I ran a cold verification pass: the skill file exactly as written, fed to the model as its only instructions, output checked against a strict schema, no retries, no hand-holding. Six test cases total. Five passed clean.
The sixth one was the interesting failure. A CSV row with basically nothing usable in it. Correct behavior: skip it, log nothing. What actually happened: the agent invented a plausible-sounding "yes, this became a real lead" result out of a row that had no basis for one. If I hadn't caught that before launch, the sourced pipeline number, the single cleanest metric this whole product is built around, would have been quietly wrong from day one. Not dramatically wrong. Just wrong in a way nobody would have noticed until they went looking, which is worse.
The root cause wasn't the model being dumb. It was my own instructions letting ambiguity resolve toward "sure, I'll call it something" instead of toward "I don't have enough to call it anything." I rewrote the decision logic as a single ordered check that rules out "not enough information" first, before anything else gets evaluated. Re-ran the same six cases. Six for six.
I bring this up because it's the actual answer to the question every attribution product should get asked and mostly doesn't: how do you know your own tool isn't lying to you? Mine failed a test once, in a way that would have mattered, and I have the commit where I caught it and the commit where I fixed it. That's a better answer than a clean track record I couldn't prove.
The dashboard being empty right now is just the same discipline pointed at the next question. I'm not going to publish a number I can't defend, and I'm not going to borrow someone else's case study to fill the gap while I wait for my own data. It fills as the real deals close. If you'd rather watch a real number arrive slow than trust one that showed up too fast, the empty dashboard is at stack.chadtdyar.com/proof and the kit itself is at stack.chadtdyar.com.
The sixth case is the whole post. Ambiguity resolving toward "sure, I'll call it something" is the default failure of anything that reports on itself, and reordering the checks does fix it.
What made it stick for me was moving it into the output instead. Our format-currency report can only return current, behind or unknown, with one written rule next to it: never round unknown up to current. An ordering can be quietly re-broken by the next edit, whereas a third value that something downstream has to handle cannot be, and it gives a reviewer a word to point at.
Did you leave the honest empty state visible to buyers, or is it behind the login?
This is the measurement insight everyone understands intellectually but almost nobody actually ships: integrity of the metric matters more than the presence of the metric. You chose an empty state over a plausible-but-wrong number, and that decision ripples through the whole product.
Most founders reverse this. They put together a dashboard that "shows progress" even when there's no actual progress, because the empty state feels like failure. So they decorate it. They add activity metrics, success rates, estimated impact - anything that looks like the thing is working. The dashboard becomes theater, and because it's theater, your measurement system starts lying to you.
What happens next is predictable: you optimize based on what the dashboard says is working, but the dashboard is measuring noise, not signal. You end up building more of what the metric says is good while missing that the underlying thing isn't actually working. The sixth test case that invented a lead is exactly how this breaks in production for everyone else - they never catch it because the dashboard looked healthy enough.
You built the dashboard backwards. Instead of starting with "what should be here," you started with "what should NOT be here" - requiring high standards before you emit any signal at all. That's measurement discipline. How much of your product decision-making gets cleaner when your measurement system can't lie to you?"
Your ordered “insufficient evidence” check does more than reduce hallucination: it gives the system a valid null state. I would make that null state first-class in the dashboard too.
For every attribution claim, keep an immutable event record with the source, decision-rule version, evidence pointer, and status: pending, verified, rejected, or reversed. Then publish aggregate counts for rejected and unresolvable events alongside closed deals. A buyer can distinguish “no outcomes yet” from “the system discarded weak evidence” without seeing customer data.
I would also freeze each claim under the rule version that made it. Otherwise a classifier update can silently rewrite historical attribution. An empty dashboard plus a visible rejection trail is stronger proof than a full dashboard with no audit path.
An honest empty state is a real test of nerve, and I think you’ve got the right mechanism: a conservative number someone can verify is more valuable than an impressive number they just have to trust.
I build measurement tools and publish the error bands, including cases where the method simply can’t produce a number. Counterintuitively, that refusal is often the thing users mention when they recommend the tools.
One suggestion for the dashboard: put the definition right next to the empty state (what actually counts as a closed deal, and what doesn’t). Making the criteria visible turns “nothing yet” from awkward into credible.
Has any prospect actually pushed back on the empty dashboard or has it mostly worked as a positive filter so far?
The confident-hallucination catch on row 6 is the real story here most people ship the clean 5/6 and never think to ask what the model does with nothing to work with. Ruling out "not enough info" first is such a simple fix but it's the kind of thing you only find if you're actually trying to break your own tool before a customer does.
Your sixth test failure is the hardest metric to detect and the most important one to catch: the confident false positive. It reads as signal, feels like data, but it's pure noise that happens to be organized.
The pattern you caught is what separates leading indicators from false indicators. A real leading indicator correlates with outcomes. A false indicator is just correlated with your output instructions. When your agent hallucinates a lead from nothing, it's not measuring anything - it's detecting your own confidence threshold, not buyer behavior.
"Confidently wrong beats obviously wrong" because people act on confident signals. The empty dashboard is actually the most honest lagging indicator: real deals that closed. No correlation ambiguity, no instruction bleed, no room for the system to lie to itself by following your rules too well.
This is why measurement systems designed to detect their own failure (cold tests, commits with failure modes visible) are worth more than ones that just never failed in your test set. You shipped the hard part - not the dashboard, but the discipline to catch when your system mistakes confidence for accuracy.
Caught the same class of bug dogfooding our analytics agent (amami.dev): it didn't fail loudly, it produced a confident, plausible-sounding number from a row with nothing usable. Ordering "not enough information" first is the right call - we shipped the same rule, and now every answer has to show the pages/dates behind it or it gets flagged as a guess. The empty dashboard is the honest version of that discipline.
The attribution hallucination failure you caught is exactly the right failure to worry about. "Confidently wrong" is worse than "obviously wrong" because it does not trigger alarm bells until someone goes looking - and in sales attribution, by the time someone goes looking, you have already made decisions off it.
The fix (ruling out "not enough information" before anything else gets evaluated) is a pattern worth noting anywhere you are asking a model to make a credibility claim. Models are trained to complete tasks, and that pressure beats "admit I cannot" in ambiguous cases unless you explicitly structure around it first.
The empty dashboard as design choice rather than gap is the thing I would actually pitch to buyers. Any vendor can show you a number. Almost none of them can show you the commit where they caught their own tool lying and the commit where they fixed it. That is a more compelling proof-of-methodology story than a full pipeline chart.
The failed sixth test is more compelling than a clean 6/6 would have been. You found a failure mode that could have produced believable but indefensible attribution, which is exactly the kind of failure worth exposing before customers depend on the number.