I could not tell if Google's AI Overview was citing my sites, so I built a way to check

I run a couple of small content sites. One of them answers questions about PFAS in consumer products, which is exactly the kind of informational query Google now answers with an AI Overview before anyone gets near a blue link.

Last spring my analytics started doing the thing everyone warns you about. Impressions held up. Clicks didn't. The obvious suspect was the AI Overview sitting above my results, summarizing an answer I'd written and citing somebody else. Obvious, but I couldn't prove it, and I couldn't have told you which queries it was happening on.

So I did the thing everyone does first. Incognito window, type the query, squint.

The manual version, and why it didn't last a week

Checking one query by hand takes about forty seconds. Search, wait for the overview to render, read the citation chips, write it down.

Forty seconds is nothing for one query. I had roughly two hundred I cared about across two sites, and the answer changes over time, so it's a chore that comes back every week. Two hundred queries is a bit over two hours of clicking, and what you get for it is a spreadsheet that starts going stale the moment you close it.

There's a worse problem underneath that one. Eyeballing doesn't turn into anything you can analyze later. I wanted to ask things like "across these forty queries, which domains does Google's AI cite most, and am I anywhere on that list." A folder of screenshots can't answer that.

What I actually needed was datasets for analysis

What I kept reaching for was structured output. One row per query, the answer text, and every cited source broken out with its title, link, and domain.

Once each check is a row, the questions I cared about get boring to answer, which is the point. Which of my pages get cited. Which competitor keeps turning up instead of me. Whether an overview shows at all, because Google skips plenty of queries entirely.

Building it was less about parsing than I expected and more about one quirk I didn't see coming. Google doesn't always hand over the overview with the search result. Sometimes it defers and gives you a short-lived token instead. That token dies in about a minute, so you can't hand it to a user and let them deal with it later; it has to get resolved inside the same run or the data is just missing. A naive scraper logs those queries as "no overview" when there was one sitting right there.

I'd guess that quirk accounts for most of the gap between a check you can trust and one that quietly under-reports.

Wiring it into AI agent workflows

The part I didn't plan was how much I'd end up using it through Claude instead of through anything I built.

I made it callable as a tool over MCP mostly to see if it'd work. Now it's how I actually use the thing. I ask in plain language, something like "is my domain cited in the overview for these ten queries," and the answer comes back in the conversation. No dashboard, no spreadsheet. For the recurring version I run it on a schedule and let the history pile up, which is the part that matters, because a single snapshot tells you nothing about whether you're gaining or losing.

Where it stands a month in

It's public on Apify now as the Google AI Overview API. Today it's at 162 total users, 80 of them active in the last month, and a little over 8,000 runs. Four reviews, all five stars, which I'll caveat myself: four is a small number and I'd rather say so than round it into something it isn't.

It costs a cent to start a run and a cent and a half per retrieval, so a normal query runs about two and a half cents and a deferred one about four. My original two hundred queries now cost a few dollars a week instead of two hours.

The two use cases I built first are still the two I run: checking whether a brand appears in Google AI Overviews, and seeing which sites an overview cites for a keyword. If you want the technical write-up rather than the story, that's over on Dev.to, and there's a shorter walkthrough on Peerlist.

The limitations are worth more to you than the pitch, so: Google mostly shows overviews for English, question-style queries, in a limited set of countries. A real chunk of queries come back with nothing. That's a legitimate answer rather than a failure, but coverage is patchier than you'd assume going in.

What I'd tell another founder

I spent about two months worried about a trend I couldn't measure and about a week building the measurement. That ratio still bugs me.

If something is quietly eating your traffic and you're guessing, turning the guess into a row in a table is cheap. Cheaper than the guessing was, anyway. Mine told me I was cited less than I thought and that one competitor kept showing up instead, which was annoying but at least it was actionable.

Two things people keep asking me

Someone asked me what AI agent workflow automation actually means here, because it gets used for everything. In this case it's narrow: you make a tool an agent can call, and the agent handles the fetching and the deciding instead of you clicking a UI. Mine calls the overview check, reads the citations, and tells me what moved since last week.

The other one is which platform lets you chain multiple AI agents in workflows. Anything that speaks MCP will let an agent call a tool like this directly, and the usual automation platforms cover the scheduled multi-step version if you'd rather keep a person in the loop.