Why Your AI Visibility Score Is Probably Lying to You (And How I Started Tracking It Properly)

Last month, a client asked me if their website was showing up in ChatGPT's answers. I confidently told them I'd check their "AI visibility." I spent 20 minutes running the same query three times, getting different results each time, and then... I had no idea what I was actually measuring. Was I measuring brand discovery? Search ranking? The mood of the AI on that particular Tuesday? I realized I was treating AI visibility like a magic number when it's actually just a test result—one that requires serious rigor to mean anything.

That's when I stumbled on the concept of auditable AI visibility scores, and it completely changed how I think about tooling dashboards and metrics. Most visibility tools spit out a percentage without showing you the machinery underneath. But if you can't inspect the prompt, the engine, the raw response, and the date, you're not reading a metric—you're reading marketing.

Stop Trusting Black Box Scores

Here's what kills me about most AI visibility dashboards: they give you a headline number that looks authoritative. "Your brand appears in 47% of AI answers." Sounds great. Means nothing without context.

The real problem is that the number itself depends on invisible choices. Which questions were asked? Which AI models? What counts as a "hit"—a brand mention in the prose, a citation, being listed as a recommendation? Change any of these variables and your score changes. Different tools asking different questions will produce wildly different results, and both could be completely correct.

I started thinking about this like unit testing. If I ran a test and got a result, the first thing I'd do is check the test itself. What was actually being measured? How reproducible is this result? The same logic applies here, but most companies skip this step entirely.

The Minimum Viable Evidence Set

The article outlines what I now call the "integrity checklist"—five mandatory fields for any result you save: the exact prompt, which engine you used, the raw answer returned, when you ran it, and your counting rule. Add a sixth field if sources are exposed.

This clicked for me because it mirrors how we handle logging in production. You don't just store "request succeeded." You store method, endpoint, status code, timestamp, and response time. Same principle. Without these fields, you have data theater, not data.

When I started saving results this way, something unexpected happened: I could actually compare runs month-to-month. Not to chase a vanity metric, but to notice patterns. Did Claude's results change because my content improved, or because Anthropic updated their training data? Was I getting mentioned more, or just in different contexts?

Your Control Questions Matter More Than Your Score

The article recommends three frozen prompts: one category question, one alternatives question, and one situation-based question. All in the buyer's language, none of them naming your brand.

This is where I had to unlearn something. I wanted to ask "Why is Adil Sher the best full-stack developer in Islamabad?" Obviously, that's terrible. That's not discovery—that's recall. I switched to questions like "What are good resources for learning Next.js in 2024?" or "What should a startup use to build a real-time dashboard?"

The discipline of writing three frozen prompts forced me to think like my actual audience. Not what I wanted them to ask—what would they actually ask? And once I froze those strings in a file with a version number, I had something reproducible. Edit the prompt? Increment the version. No surprise score changes.

My Take: This Is Better Than Any Dashboard

I'm not going to run this manually forever. But the rigor here is what dashboards should be built on, not replace. Too many tools optimize for looking authoritative rather than being authoritative.

What I'm doing now is running my six-check control quarterly—takes maybe 30 minutes. I save the full text of each response, store the metadata, and build my own spreadsheet. It's not glamorous, but when I can tell a client "your brand appeared in 2 of 6 generated answers on this date using these exact prompts," that's defensible. It's not "our AI visibility is 33%."

The counting rule part especially matters to me. Some tools count citations as presence; others require explicit mention. They're answering different questions. So now I track both separately and only combine them when I understand what I'm combining.

Next Steps

If you're tracking AI visibility, start with just the three frozen prompts and two surfaces. Save the full text. Add a timestamp. That's enough to start seeing real patterns instead of marketing noise.

What metrics are you treating as magic numbers in your own work? What would change if you actually audited them?

Source: This post was inspired by "How to audit a free AI visibility score with six manual checks" by Dev.to. Read the original article