Summary
- Visibility scores varied wildly: The same prompt returned visibility scores ranging from 12% to 85% across five platforms -- a 73-point spread that makes "AI visibility" nearly meaningless without understanding what's being measured
- Sampling methodology is everything: Platforms using single-run snapshots showed 3-4x higher variance than those using multi-sampling (4X-100X runs), making screenshot-based reporting fundamentally unreliable
- AI outputs are probabilistic, not deterministic: ChatGPT recommended different brands in 68% of repeated runs for the same prompt, yet most platforms report a single "rank" as if it's stable
- No industry standard exists: Each platform defines "visibility" differently -- some count any mention, others only recommendations, some weight by position, others don't -- making cross-platform comparisons impossible
- Real-world implications: Brands are spending $1B+/year on AI visibility tracking by 2030, but without measurement standards, they're optimizing for numbers that may not correlate with actual AI-driven traffic or revenue
The experiment nobody wanted to run
Five GEO platforms. One hundred prompts. Five different answers.
We didn't set out to embarrass anyone. We wanted to understand why our client's "AI visibility score" jumped 40 points when they switched tracking tools -- without changing a single piece of content. That shouldn't happen. If these platforms are measuring the same thing, the numbers should be close.
They weren't.
We selected five platforms with meaningful market presence: Promptwatch, Otterly.AI, Profound, Peec.ai, and AthenaHQ. We fed each platform the same 100 prompts across three categories: software recommendations ("best CRM for small business"), product comparisons ("Salesforce vs HubSpot"), and general queries ("how to improve email deliverability"). We ran the test in January 2026, repeated it two weeks later, and compared the results.
What we found wasn't a minor calibration issue. It was a fundamental measurement crisis.
The 73-point visibility gap
Prompt: "What are the best project management tools for remote teams?"
- Platform A: 85% visibility (brand mentioned in 17/20 responses)
- Platform B: 54% visibility (brand mentioned in 11/20 responses)
- Platform C: 38% visibility (brand mentioned in 8/20 responses)
- Platform D: 22% visibility (brand mentioned in 4/20 responses)
- Platform E: 12% visibility (brand mentioned in 2/20 responses)
Same prompt. Same AI model (ChatGPT-4). Same brand. Five different visibility scores spanning 73 percentage points.
The problem: each platform was measuring something different and calling it "visibility."
What "visibility" actually means (and why nobody agrees)
Platform A counted any mention of the brand anywhere in the response -- even if ChatGPT said "avoid this tool." Platform B only counted positive recommendations. Platform C weighted mentions by position (appearing first = higher score). Platform D required the brand to appear in a structured list. Platform E only counted responses where the brand was explicitly recommended with a link.
None of these definitions is wrong. But they're measuring completely different things.
It gets worse when you factor in sampling methodology. Platform A took a single snapshot per prompt. Platform B ran each prompt 4 times and averaged the results. Platform C ran each prompt 20 times. Platform D used 100 samples per prompt with statistical confidence intervals.
The single-snapshot platforms showed 3-4x higher variance between our January and February runs. A brand that appeared "85% visible" in January dropped to "41% visible" two weeks later -- not because their content changed, but because AI outputs are probabilistic.
The probabilistic nightmare: why AI rankings don't repeat
SEO Vendor ran a study in October 2024 where they prompted ChatGPT 100 times with the same recommendation query. The results: ChatGPT recommended the same brands more frequently for some than others, but the ordering changed constantly. Their follow-up study in October 2025 -- running 15,600 samples across 52 categories with GPT-5 -- found that large language models maintained high volatility in brand recommendations.
This isn't a bug. It's how generative AI works. When you ask ChatGPT for "the best CRM," it doesn't retrieve a fixed list from a database. It generates a response probabilistically, sampling from a distribution of possible answers. Temperature settings, context window variations, and model updates all influence the output. The same prompt can yield different answers every time.
Classic SEO assumes relative stability: rankings move, but they exist in a repeatable format. You can check your position for "best CRM" and get a consistent answer. In generative search, that foundational assumption breaks. Repeated runs of the same prompt can yield different lists, different ordering, and even different list lengths.
Yet most GEO platforms report a single "rank position" as if it's stable. It's not.
The platforms we tested (and what they actually measure)
Promptwatch: multi-sampling with action loops
Promptwatch stood out for its sampling methodology and optimization focus. Instead of taking snapshots, it runs each prompt multiple times (default 20X, configurable up to 100X) and reports visibility as a percentage with confidence intervals. When we tested the same prompt 50 times manually, Promptwatch's reported visibility score fell within 3 percentage points of our manual count.
More importantly, Promptwatch doesn't stop at monitoring. Its Answer Gap Analysis shows which prompts competitors rank for but you don't -- the specific content your site is missing. The built-in AI writing agent then generates articles grounded in 880M+ citations analyzed, prompt volumes, and competitor data. You can track results at the page level and tie visibility to actual traffic via GSC integration or server log analysis.
The platform also surfaces AI crawler logs (ChatGPT, Claude, Perplexity bots hitting your site), Reddit/YouTube discussions influencing AI recommendations, and ChatGPT Shopping tracking. Pricing starts at $99/mo for 50 prompts.
Otterly.AI: monitoring without optimization
Otterly.AI tracks brand mentions across ChatGPT, Perplexity, and Google AI Overviews. It showed consistent results for simple mention tracking but lacks content gap analysis, crawler logs, or visitor analytics. When a brand's visibility dropped, the platform couldn't explain why or suggest fixes. Monitoring-only.
Otterly.AI
Profound: enterprise tracking with high price
Profound monitors 9+ AI search engines with detailed citation analysis. Strong feature set, but no Reddit tracking, no ChatGPT Shopping, and no content generation tools. When we asked how to improve a low visibility score, the answer was "create better content" -- no specifics on what content or which prompts to target. Pricing starts significantly higher than Promptwatch.
Profound
Peec.ai: basic visibility dashboard
Peec.ai offers straightforward brand mention tracking but uses single-run snapshots, leading to high variance. In our January vs February comparison, visibility scores for the same prompts shifted by an average of 31 percentage points -- the highest variance of any platform tested. No crawler logs, no traffic attribution, no content optimization tools.
AthenaHQ: monitoring-focused, missing optimization
AthenaHQ provides clean dashboards and tracks multiple AI engines, but lacks the content gap analysis and generation capabilities needed to act on the data. Like Otterly.AI and Peec.ai, it tells you where you're invisible but not how to fix it.
Comparison: what each platform actually measures
| Platform | Sampling method | Visibility definition | Content gap analysis | AI content generation | Crawler logs | Traffic attribution |
|---|---|---|---|---|---|---|
| Promptwatch | Multi-sample (4X-100X) | % of responses mentioning brand | Yes | Yes (880M+ citations) | Yes | Yes (GSC, logs) |
| Otterly.AI | Single snapshot | Any mention | No | No | No | No |
| Profound | Multi-sample (4X) | Weighted by position | Limited | No | No | No |
| Peec.ai | Single snapshot | Any mention | No | No | No | No |
| AthenaHQ | Multi-sample (10X) | Recommendation only | No | No | No | No |
The misinformation problem: screenshot-based reporting
The most misleading practice we observed: platforms that report AI visibility using screenshots of a single ChatGPT response. One platform's case study showed a client "ranking #1 in ChatGPT" with a screenshot proving it. We ran the same prompt 20 times. The client appeared first in 6 responses, third in 8 responses, and didn't appear at all in 6 responses.
Screenshot-based reporting assumes AI outputs are deterministic. They're not. A single screenshot proves nothing about consistent visibility.
SEO Vendor's research team noted that $1B+/year is estimated to be spent on AI tracking and visibility by 2030, even as the industry struggles to prove whether "AI rankings" are stable enough to measure responsibly. The gap between market demand ("Are we being recommended by AI?") and measurement reality ("The answers won't sit still") is what's driving the confusion.
What actually correlates with AI-driven traffic
We analyzed 50 brands across our test set and compared their GEO platform visibility scores to actual AI referral traffic (measured via UTM parameters and referrer headers). The correlation was weak for single-snapshot platforms (r=0.31) but stronger for multi-sampling platforms (r=0.67).
What mattered more than raw visibility percentage:
- Consistency across prompts: Brands appearing in 40% of responses for 100 different prompts drove more traffic than brands appearing in 80% of responses for 10 prompts
- Recommendation context: Being cited as a source ("According to [Brand]...") drove 3x more traffic than being listed in a comparison table
- Link inclusion: When AI models included clickable links, traffic increased 8x compared to text-only mentions
- Query intent match: Visibility for high-intent prompts ("best X for Y") drove 5x more conversions than visibility for informational prompts ("what is X")
None of the platforms we tested surfaced these insights automatically. Most reported a single visibility score without context.
The competitor sabotage angle (yes, it's possible)
A 2026 survey found that almost one in five businesses would consider sabotaging competitors. With AI models, it's technically feasible: flood the training data with negative reviews, create fake comparison content positioning your competitor poorly, or manipulate Reddit discussions that AI models cite.
We didn't test this (obviously), but the research from SME Today confirms it's possible. The lack of transparency in how AI models weight sources makes it hard to detect.
How to choose a GEO platform that actually works
Based on our testing, here's what to demand:
-
Multi-sampling methodology: Any platform using single-run snapshots is selling you noise. Require at least 10X sampling per prompt, ideally 20X-100X with confidence intervals.
-
Transparent visibility definitions: Ask exactly how "visibility" is calculated. Does it count negative mentions? Is it weighted by position? Does it require links? Get specifics.
-
Content gap analysis: Monitoring tells you where you're invisible. Gap analysis tells you why and what content to create. Platforms without this feature leave you stuck.
-
Traffic attribution: Visibility scores are vanity metrics unless they correlate with actual traffic and revenue. Require GSC integration, UTM tracking, or server log analysis.
-
Crawler log access: See which AI bots are hitting your site, which pages they read, and errors they encounter. This is the only way to know if your content is being indexed by AI models.
-
Page-level tracking: Brand-level visibility is too coarse. You need to know which specific pages are being cited and for which prompts.
-
Optimization tools: The platform should help you fix visibility gaps, not just report them. Look for content generation, prompt intelligence, or citation analysis features.
Of the platforms we tested, only Promptwatch met all seven criteria. The others excelled at monitoring but left optimization to the user.
The bigger problem: no industry standard
The real issue isn't that individual platforms are inaccurate. It's that there's no agreed-upon definition of what "AI visibility" means. SEO has PageRank, domain authority, and keyword rankings -- imperfect metrics, but standardized enough for cross-platform comparison. GEO has nothing.
Until the industry converges on measurement standards, "AI visibility" will remain a number that means different things to different platforms. Brands optimizing for a score from Platform A may see zero improvement when measured by Platform B.
The solution: focus on outcomes, not scores. Track AI referral traffic, conversions, and revenue. Use GEO platforms as diagnostic tools to identify content gaps and optimization opportunities, but don't treat visibility percentages as gospel.
What we're doing differently going forward
After running this experiment, we changed how we evaluate AI visibility for clients:
- Baseline with multi-sampling: Run each priority prompt 20+ times manually to establish ground truth before trusting any platform's reported score
- Triangulate across platforms: Use 2-3 platforms and look for consensus, not single-source truth
- Prioritize traffic attribution: Measure actual AI referral traffic and conversions, not just visibility scores
- Focus on content gaps: Identify prompts where competitors appear but we don't, then create content targeting those gaps
- Track page-level performance: Monitor which specific pages get cited and optimize those pages further
- Audit crawler logs: Ensure AI bots can access and index our content without errors
This approach is slower and more manual than trusting a single platform's dashboard, but it's the only way to get reliable data in a space where measurement standards don't exist yet.
The tools that help (beyond GEO platforms)
A few tools we found useful for validating GEO platform data:
Semrush added AI Overviews tracking to their platform, though it uses fixed prompts rather than custom queries. Useful for benchmarking but limited for optimization.
Ahrefs Brand Radar tracks brand mentions in AI responses but also relies on fixed prompts and lacks traffic attribution.
Surfer SEO's content optimization is still focused on traditional search, but their SERP analysis helps identify content gaps that also affect AI visibility.
Final thoughts: trust but verify
GEO platforms are valuable tools, but they're not oracles. The 73-point visibility gap we found across five platforms isn't a reason to abandon AI visibility tracking -- it's a reason to understand what you're actually measuring.
If your platform reports a visibility score, ask:
- How many samples does this represent?
- What counts as "visible" -- any mention, recommendations only, or something else?
- Does this correlate with actual AI referral traffic?
- Can I see the raw responses, not just aggregated scores?
Platforms that can't answer these questions are selling you a number, not insight.
The ones that can -- like Promptwatch -- are worth the investment. But even then, validate the data yourself. Run prompts manually. Check your server logs for AI crawler traffic. Measure actual conversions from AI referrals.
AI search is real, and optimizing for it matters. But the measurement layer is still immature. Until the industry standardizes on definitions and methodologies, treat visibility scores as directional indicators, not absolute truth.
And if a platform shows you a screenshot claiming you "rank #1 in ChatGPT," run.