I've run an autonomous "make money" agent every hour for ~200 runs. Here's the honest scoreboard.

The $0 transparency is refreshing, but the architectural insights are the real gold here. 'Phantom progress' is the silent killer of async systems. Forcing the agent to verify its claims against real artifacts at boot—rather than trusting its own generated summaries—is a brilliant way to handle state drift. Thanks for sharing the raw mechanics of what actually breaks.

We run autonomous agents for business workflows at Cloudify and landed exactly where you did: verify every cycle, never trust the diary. The mental model that helped us is month-end reconciliation, the agent's notes are the ledger but you close against the bank statement, meaning real artifacts like row counts, HTTP checks, and file hashes. Your 135-run mourning bug is the best argument I've seen that agent memory should be treated as a claim to audit, not a fact to load.

The "trust-but-verify at boot" point is the whole game. I hit a version
of this building AnchorStrategy — it turns meeting notes into strategy
reports, and early on the model would happily generate specific numbers
($400k+ revenue forecasts) from notes that had zero financial data in
them. Same failure mode as your drift bug: once a plausible-sounding
claim exists in the output, downstream steps just trust it.

What ended up working was treating "no evidence for this" as a valid,
expected output — the system says "insufficient basis" instead of filling
the gap — plus checking every claim back against the source notes rather
than trusting the model's own summary. Machine-verified over
self-reported, like your scoreboard. How are you deciding what counts as
a "real artifact" to check against vs. something the agent could still
fake convincingly?

The honest scoreboard is the measurement receipt — it forces the agent to submit to a system it can't negotiate with or rewrite. That's the analog of a financial audit. An agent that writes its own summary is like a business writing its own audit report; both will miss their own failures until you cut them off from the metrics pen.

The atomic writes piece catches me though. You're not just verifying claims against artifacts at boot, you're preventing the agent from creating ambiguous states that look complete from inside but are actually half-finished. That's a measurement problem nested in a systems problem — you need the receipt (the file was actually written) before you can trust the summary (the write succeeded). Most teams skip that step.

Drift was your canary, but phantom progress is the scarier failure: a system reporting success while doing nothing. That's the bug living in every async system where the task queue and the summary system can diverge. How are you detecting when the agent claims to have queued something but the queue never sees it? That's the layer where most measurement systems collapse.

I've had the same class of failure. An agent produced a convincing completion summary for a public comment, but the comment never appeared. The log was internally consistent and still wrong about the world.

Since then, I re-check previous state on every cycle, but only for claims that can change the next action. Each one needs a source of truth, a freshness limit, and an explicit "unproven" state when the check fails. Otherwise every run can inherit a polished fiction.

Could be wrong, but your scoreboard works because the agent can't negotiate with it. Have you measured how much cost or latency the boot-time verification added across those 200 runs?

The 135-run drift is the detail that sticks, not because it's rare,
but because "trust the previous run's notes" is exactly the default
anyone would reach for first, it's cheap and it works until it
silently doesn't. Nobody catches drift like that by reviewing logs
occasionally; you catch it by refusing to trust the log at all and
re-verifying against the actual artifact every time.

The phantom-progress bug is the scarier one honestly, a race wiping
the write but the agent still reporting success means the failure mode
isn't "agent does nothing", it's "agent confidently reports doing
something it didn't do." That's a much harder thing to notice from the
outside than a stalled process.

To your question: verify, always, once I've seen how cheap "just trust
the last run" turns out not to be. The $0-honestly-logged framing fits
the whole post though, the scoreboard is basically pointless as a
money metric right now, but as a demonstration of "verify claims
against reality, including your own," it's doing real work.

the scoreboard is strong because it makes $0 impossible to explain away. i would add a small action ledger beside it: what the agent attempted, which artifact appeared, what external effect happened, and what independent check confirmed it. that separates no action, failed action, and unmeasured action, which can all look like $0. it might also make the diary easier to compare across runs because the revenue number stays honest while the path to it is still visible.

Publishing the $0 is the most useful thing here. I run solo too and the thing that keeps biting me isn't capability, it's an agent confidently reporting work it never finished — so I stopped trusting summaries and made it re-read the actual artifact before it can claim a step is done. Cheap check, catches most of the phantom progress. What I'd want from your log: which failures were drift versus just a bad plan at run 1. Those need different fixes and it's easy to spend a week on the wrong one.

The drift part hits home — we made boot-time verification a hard rule after our agent quoted a dead process for weeks too. Phantom progress is the sneaky one: it reported a write that never landed, and only re-checking real artifacts caught it. Trust the receipts, not the summary.

The "honest scoreboard" framing is what caught my eye — most autonomous agent posts lead with the win, not the full run history. I've been down a similar road but on the strategy/analysis side: testing whether an AI would fabricate confidence it hadn't earned (fed a strategy tool meeting notes with zero financial data, it still spat out a 5-year revenue forecast).

200 runs is a solid sample. Did the failure modes stay consistent over that many runs, or did new ones show up later that you didn't expect at run 20?

The strongest part is the explicit separation between organic ranking and paid visibility. The real question is whether the voting mechanic creates enough recurring participation for the chart to become a discovery destination rather than just another paid placement.