Ran the numbers before deciding how much to automate. 38% of doc work turned out to be low hanging fruit
Im working on DocsKoala. it generates docs from merged github prs. before building out more automation i wanted to actually know how much of the problem was automatable versus how much genuinely needs a human, so i ran a real audit instead of guessing.
pulled 14,891 merged prs across 220 public repos. classified each as customer facing feature, internal, bug fix, or unclear. for the 3,623 customer facing ones, checked whether the pr diff and title alone gave enough to draft a doc, and separately whether it ever got documented at all.
1,386 features, 38.3%, were automatable based on the diff alone, no extra context needed from the dev. 541 features, 14.9%, never got documented period, automatable or not.
cost estimate if a human wrote docs for just the automatable chunk manually, 20-30 min each, comes to 462-693 hours and 34.6k-52k dollars across these 220 repos.
this number is basically why DocsKoala focuses where it does. it drafts automatically for the 38% that's clear cut, and flags the rest for a human instead of pretending it can guess context that isn't in the diff. figured people building similar tools or just curious about the "how automatable is documentation actually" question would find the raw split useful.
This is nice. It shows that more attention is needed for accuracy
Interesting that you quantified it before automating — most people (myself included) jump straight to "automate everything" without measuring where the return is highest. How did you measure that 38%? Was it time spent per task or another metric?
Flagging the rest for a human instead of pretending it can guess is the right instinct, and it is rarer than it should be. I built the same principle into FounderFlow, it grades its own confidence instead of presenting a guess as a fact. Founders trust these tools a lot more once they can see where the tool itself is unsure.
The finding that 38% is automatable from the diff alone is the more useful number here than the aggregate automation rate, because it tells you where the ceiling actually is. I've hit the same wall building on top of PR diffs, a diff shows you what changed but not always why, and the why is usually what a real doc or a real review comment needs. I would chase the 14.9% that's never documented at all before chasing a higher percentage on the automatable slice, since that gap is pure signal that nobody is even trying to answer, not a hard technical ceiling.
I like the name DocsKoala, and I think all companies should run a test similar to yours. Do you have any other projects planned? If so, which ones?
This is a great approach. Most people rush to automate everything without looking at the data first. 38% is a huge chunk of low-hanging fruit to optimize, and it gives you a clear ROI right away.
Did you find that automating this 38% freed up enough time to handle the more complex edge cases manually, or are you planning to tackle the remaining 62% with a more advanced AI setup later?
The 38% figure feels believable, and calling out the rest instead of making up context is where trust comes from. That missing 14.9% is probably more than a documentation problem. The reason for a change often lives in Slack, on a call, or just in one person's head. I'd try a one-sentence intent prompt at merge time, then let the draft use that with the diff. You can automate the first draft without pretending the diff knows why the change happened.
agreed, and the one sentence intent prompt at merge time is basically what "/koala" tagged commits are for right now, but its opt-in and i dont think most devs would bother with it
Running the audit first before deciding what to automate is exactly the right call, and the 14.9% that never got documented at all is the more interesting number to me. That's not an automation problem, that's a workflow problem. Those features are invisible to users no matter how good the tooling is. The 38% automatable slice is a clean wedge though. Scoping to what's clear cut and flagging the rest is more honest than most doc tools which pretend the ambiguous 62% doesn't exist. How are you positioning this to dev teams who've already got a process (however broken) versus teams who have nothing at all?
that would require two different pitches, but, its probably easy to sell to teams who have nothing because any docs would obviously beat no docs.
however, teams with (broken) tools could be tricker. the pitch isnt to "replace your process", its acutally, DocsKoala could automate that process while still having a human on loop for checking the docs before approving so the human attention could be directed towards something which actually needs human attention.
havent tested that though, feel free to give it a go yourself, id love your recommendations.
The number I'd chase isn't the 38% automatable, it's the 541 that never got documented at all. Docs you'd have written anyway are a time save; docs that would never have existed are net-new coverage, and that's a far easier thing to sell. How much of that 541 overlaps with the automatable 38%? Also worth a caveat: this is public repos, where PR titles tend to be tidier than internal ones, so the automatable share on private repos with 'fix stuff' messages might look different.
havent oulled the overlap yet. i need to check how much of the 541 undocumented also falls inside the automatable 38%, If it happens to be high, thats a much better pitch than the time save framing.
That overlap is the whole ballgame, so split the 541 into two buckets before you run it. The ones inside the automatable 38% are your headline: docs that would never have existed, draftable with zero extra dev input. Basically free coverage. The ones outside it (undocumented, and not automatable from the diff alone) are the harder story — those are exactly what a merge-time intent prompt has to catch, and they're the changes nobody bothered to write up the first time round. If that outside slice is big, the intent prompt stops being a nice-to-have and turns load-bearing. Curious which way it splits once you pull it.
Thats nice!!! A 38% is absolutely wild
Yeah, it is.
I think the most valuable part of this isn't the 38%.
It's that you identified the boundary where automation stops being trustworthy. A lot of AI products try to maximize automation. Defining where a human should take over often creates more confidence than squeezing out a few extra percentage points.
i appreciate that. i went in wanting to push the 38% higher, but the audit convinced me the boundary matters more. and every draft still goes through human review before it publishes, so it's not only "AI ships stuff," but "AI drafts what it's confident about, a human checks it, and the hard 62% gets flagged instead of being guessed
I like where your thinking landed.
Your reply raised a question for me about how you'll decide where that boundary should move over time. I'd rather understand that in the context of your product before offering a conclusion.
What's the best email to reach you on?