Key takeaways
- Score every vendor on the same 25 criteria across six groups: data collection, metrics, tracing and diagnosis, action, measurement, and enterprise controls plus commercials.
- Weight the groups before you see a demo. A dashboard-only tool can score well on monitoring and still leave your team with all the work.
- Engines behave differently and change often, so multi-engine coverage, prompt-type segmentation and change annotations are scorecard items, not nice-to-haves.
- Run a 30-day pilot on your own prompts before signing. A demo with the vendor's sample brand tells you almost nothing.
- Treat SSO, audit logs, API limits and data handling as pass/fail gates, not points.
Why a scorecard at all
Most AI visibility demos look the same. A score goes up, a line chart appears, a competitor bar chart makes you feel behind. Then you sign, and three months later nobody on your team can say what to do with the number.
A scorecard fixes that by forcing the comparison onto criteria you picked before the sales call. Several published buyer guides take a similar approach. get3rd's checklist uses 14 criteria plus demo questions and red flags, and seoClarity's enterprise comparison applies eleven criteria from engine coverage to enterprise controls. I wanted something more granular for enterprise buyers, so this one has 25.
One caveat. Several of those guides are published by vendors who also appear on their own shortlists. Read them for the criteria, not the rankings.
How to use the scorecard
Score each criterion 0 to 3:
- 0: missing or unclear
- 1: exists but shallow or manual
- 2: solid and documented
- 3: strong, verified in your own pilot
Then apply weights. A reasonable enterprise default is 20% data collection, 15% metrics, 15% tracing, 20% action, 10% measurement, 20% enterprise controls and commercials. Move the weights if your situation differs. A regulated bank should push controls to 30%. A content-heavy publisher should push action up.
Mark criteria 21 to 24 as gates. A vendor that fails them is out regardless of total score.
Group 1: data collection (criteria 1-5)
Everything downstream depends on whether the underlying responses are trustworthy.
1. Engine coverage
Which engines are tracked, and which are real interfaces versus API calls? At minimum you want ChatGPT, Google AI Overviews, Google AI Mode, Gemini, Perplexity and Copilot. Add Claude, Meta AI and others if your audience uses them.
Citation behavior differs a lot by engine. In Promptwatch Data's average sources per response report, ChatGPT cited about 5 sources per web-search answer while Google AI Overviews and Perplexity cited about 10. Copilot swung from under 2 to nearly 17 sources within weeks. A tool that averages these engines into one number hides what is happening.
2. Real UI versus API collection
Ask directly: do you collect from the consumer interface or from the model API? User-facing answers, citations and shopping results can differ from API outputs. API data is still useful for seeing what a model believes from training, but it should not be the only source for engines that browse the web.
3. Prompt-set design and prompt types
Can you segment prompts into organic (no brand named), brand-specific and competitor comparison? It matters. In the ChatGPT unique domains report (Sep 28 to Oct 4, 2026), 52.5% of organic prompts cited 10 or more domains, while only 21.3% of brand-specific prompts did. Blend them and you learn nothing.
4. Query fan-out capture
AI engines turn one prompt into sub-searches, and your content has to match those, not the user's original wording. Ask whether the platform records the fan-out queries per response. The measured averages move over time. Promptwatch's fan-out data shows searches per response falling from 2.15 in early December to about 1.84 by early March, and ChatGPT began using the site: operator at scale on Aug 8, 2026 (report).
5. Geography, language and persona control
Can you fix a monitor to a country, language, state or city, and prompt as different buyer personas? Check that markets don't mix inside one monitor. For multi-location brands, ask about coordinate-level local targeting.
Group 2: metrics (criteria 6-10)
6. Transparent score definition
An AI visibility score is typically a 0 to 100 number reflecting how often and how prominently a brand appears. Campaign Creators' explainer describes it as combining frequency, platform coverage and citations. The problem is that every vendor builds it differently. Ask for the formula in writing. If they call it proprietary and won't say what goes in, mark it 0.
7. Share of voice and position
Mentions are not enough. You need share of voice against named competitors and your position inside the answer. Being listed fifth of seven is a different outcome from being the first recommendation.
8. Citations versus mentions
Can the tool separate being named from being cited as a source? A brand mentioned without a link to its own domain usually points to a content or crawlability problem. See Promptwatch's academy page on citations vs mentions for the distinction.
9. Sentiment and claim-level narrative
A single sentiment number is weak. The better version extracts the specific claims AI makes about you ("expensive for small teams", "weak mobile app") and groups them by theme. Bonus points if you can mark a claim as inaccurate and find the page it came from.
10. Volatility handling and annotations
Models change. Citations per ChatGPT response dropped around the GPT-5.3 rollout on March 4, 2026 (report), and reddit.com's share of ChatGPT citations fell from roughly 4% to 0.5% on August 14, 2026 (report). If a vendor can't tell you whether your drop was your fault or the engine's, your board report will be fiction. Ask how often prompts run (daily, hourly) and whether model changes are annotated.
Group 3: tracing and diagnosis (criteria 11-15)
11. Citation source analysis
Which of your pages are cited, where they land in the source list, and which third-party domains win instead. Look for classification by content type and source type.
12. AI crawler logs
This is the criterion that separates platforms fast. Most tools measure outputs. Few show whether AI bots can actually read your site. Ask whether crawler logs are IP-verified (spoofed user agents are common), which bots are covered, and whether the tool links crawl activity to citations per page. Promptwatch's crawler data shows how quickly bot mix shifts. Meta-WebIndexer, for example, went from about 2% to nearly 38% of tracked requests between mid-July and August 9, 2026.
13. Social and community citations
Reddit, YouTube, LinkedIn and Instagram matter differently by engine. Per Promptwatch Data (Sep 28 to Oct 4, 2026), ChatGPT drew 0.46% of citations from social platforms versus 12.2% for Google AI Overviews. A vendor with one blended social number is guessing.
14. Offsite mentions
Can it find third-party pages that name your brand without linking, especially pages AI already cites? Those are your PR and partnership targets.
15. Shopping and ads coverage
If you sell products, check shopping carousel tracking. If competitors bid on your prompts, check for ad visibility. See ChatGPT ads over time and shopping usage for how fast these surfaces are changing.
Group 4: action (criteria 16-20)
This is where enterprise budgets get wasted. A tool that diagnoses and stops leaves your team to turn insight into pages.
16. Content gap analysis
Does it map your existing pages against the prompts AI answers and tell you where coverage is missing? Ask to see it on your own site.
17. Content generation grounded in data
Generators without citation data write generic copy. Look for briefs that use the fan-outs and competitor pages AI already cites, plus a brand book or knowledge base for tone and facts.
18. CMS publishing
Can it publish to WordPress, Webflow or Framer as draft or live, with approval workflows? Ask whether it tracks the live URL afterwards.
19. Prioritized to-do list
A board of ranked, evidence-backed actions beats a wall of charts. Check whether tasks can be assigned and commented on.
20. Refresh of existing content
Half-life matters. In Promptwatch's content half-life data, landing pages and docs stayed cited longest while news lost half its cited pages within 36 days. Your platform should flag pages that decay.
Group 5: measurement (criteria 21-22 are gates in group 6; 21 here is not a gate)
To keep numbering clean, measurement is criterion 21 and 22, and gates follow in 23 to 25.
21. Visitor analytics and conversion attribution
Can the platform show traffic and conversions from AI engines? Attribution is getting harder. ChatGPT stopped sending a referrer on October 8, 2026, so ask how the tool handles utm-tagged visits. Promptwatch's academy covers attributing AI traffic.
22. Page-level feedback loop
The test: publish a page, then see whether it was crawled, cited and clicked, all in one place. Without that loop you can't prove ROI.
Group 6: enterprise controls and commercials (criteria 23-25)
These are your gates. Fail one and move on.
23. Security and access controls
SSO, role-based access, project-level permissions, an audit log of every change (including API and agent-made changes), scoped API keys and a security certification such as ISO/IEC 27001. A tool tracking ten engines with no SSO is not enterprise-ready, as GrowByData's enterprise guide puts it.
24. API, MCP and data export
Check rate limits per hour, date range limits per request, and whether you can pipe data into your warehouse or BI tool. Looker Studio, CSV and MCP access save weeks of analyst time.
25. Pricing model and contract terms
Compare cost per response, not headline price. Ask what happens when you exceed prompt or response limits, whether seats are unlimited, and whether tracking continues during a billing dispute. Beware per-seat or per-engine pricing that scales badly.
Putting vendors through it
A rough comparison of how different categories of tools tend to fare:
| Vendor type | Typical strengths | Typical gaps | Examples |
|---|---|---|---|
| End-to-end GEO platform | Tracing, action, crawler logs, attribution | Higher entry price | Promptwatch |
| Enterprise monitoring platform | Broad engine coverage, governance | Light on content execution | Profound, Scrunch |
| Lightweight trackers | Low price, fast setup | No crawler logs, no content tools | Otterly.AI, Peec AI |
| SEO suites with AI add-ons | Familiar UI, existing contract | Fixed prompt sets, shallow AI data | Semrush, Ahrefs Brand Radar, seoClarity |
| Content-only tools | Writing speed | No visibility data to ground it | AirOps, Jasper |
These are generalizations. Score each one yourself against the 25 criteria.
Promptwatch is the one I would put first on a shortlist where criteria 12, 16 to 18 and 21 to 22 carry weight, because it combines IP-verified crawler logs, the Content Agent, WordPress, Webflow and Framer publishing, and visitor analytics in one loop. It is ISO/IEC 27001 certified, tracks the real interfaces of the major engines, and has a 7-day free trial on monthly plans. Business starts at $579/mo and adds the Content Agent, hourly tracking and Fact Check, so confirm current terms at promptwatch.com/pricing. There is also a head-to-head page if Profound is on your list: Promptwatch vs Profound.
Profound
Otterly.AI
For a wider view, the GEO software directory at bestgeosoftware.com lists many more options you can score.
Demo questions that expose weak spots
Ask these in every call and write down the answers verbatim:
- Show me this on my brand and three competitors, not your sample account.
- Are responses collected from the consumer interface or the API?
- Can I see fan-out queries per response?
- How is your visibility score calculated, in writing?
- Do you segment by organic, brand and comparison prompts?
- Are crawler logs IP-verified? Which bots?
- Show me one page from publish to crawl to citation to click.
- What happens to my history if a model is retired?
- What is your audit log coverage for API and agent actions?
- What does the overage policy look like?
Red flags
- A single blended score with no engine breakdown.
- No way to view the raw response behind a data point.
- "Proprietary" metric definitions.
- Dashboards only, with no path from insight to fix.
- Pricing that hides prompt or response caps.
- Claims of 100% accurate tracking. Responses vary run to run, so honest vendors talk about sampling and confidence.
A 30-day pilot that settles it
Week 1. Load the same 50 to 100 prompts into each finalist: a mix of organic, brand and comparison prompts, in your real markets. Add the same competitors.
Week 2. Compare raw responses across vendors for ten prompts by hand. Differences in what each tool "saw" will tell you who is collecting properly.
Week 3. Connect crawler logs and analytics if available. Pick three content gaps and have each tool help you act on them.
Week 4. Score all 25 criteria with your team. Pull the numbers into one sheet, apply your weights, and check the gates.
If you want outside help running the pilot or building the roadmap afterwards, an agency like 1001 SEO Media handles GEO and AI search work for teams without in-house capacity.
The scorecard template
| # | Criterion | Weight | Vendor A | Vendor B |
|---|---|---|---|---|
| 1-5 | Data collection | 20% | ||
| 6-10 | Metrics | 15% | ||
| 11-15 | Tracing and diagnosis | 15% | ||
| 16-20 | Action | 20% | ||
| 21-22 | Measurement | 10% | ||
| 23-25 | Controls and commercials (gates) | 20% |
Copy this into a spreadsheet, add one row per criterion, and score 0 to 3. Whoever wins on your data, in your pilot, with every gate passed is the one to sign.
Final word
The number a vendor puts on a dashboard matters less than what you can do with it on Monday morning. Weight action and measurement as heavily as monitoring, run the pilot on your own prompts, and don't let a polished demo skip the gates.