Key takeaways
- DarkVisitors rebranded to Known Agents in 2026 and now covers crawler analytics, AI chat referral tracking, and an agent identification API, but it requires installing a script or plugin on your own site.
- Cloudflare's AI Crawl Control (formerly AI Audit) is free on every plan, including the free tier, and shows crawler requests, robots.txt compliance, and content-format mismatches with zero setup if you're already on Cloudflare's network.
- Promptwatch's Agent Analytics is the only one of the three that connects crawl data to citation data, telling you not just that GPTBot visited a page but whether that page actually got cited in an AI answer.
- None of these tools solve the biggest blind spot in crawler logging: a block in your robots.txt or firewall produces the exact same "zero visits" signal as a bot that's never heard of you. You still have to check your own configuration directly.
- If you only have budget or technical capacity for one source, start with whichever CDN or platform you're already running (Cloudflare, Vercel, Fastly), then layer a citation-aware tool on top once you want to know if the crawling is turning into anything.
Why this question keeps coming up
Every few weeks someone asks a version of the same question in a Slack channel or a GEO Twitter thread: "what's the best way to see which AI bots are hitting my site?" The honest answer is that there isn't a single best way, because the three obvious options measure different things and were built for different jobs. DarkVisitors (now Known Agents) started as a crawler directory and blocking tool. Cloudflare's AI Crawl Control is a byproduct of Cloudflare already sitting in front of a huge share of the web. Promptwatch's crawler logs, branded Agent Analytics, were built specifically to answer a GEO question: is my content actually making it into AI answers, and if not, where in the pipeline did it fall apart.
I've used variations of all three setups on different sites, and the gap between "I can see bots hit my server" and "I understand why my content isn't showing up in ChatGPT" is bigger than most people expect going in.
DarkVisitors is now Known Agents, and that matters
If you bookmarked DarkVisitors a year ago, go check that link again. The product rebranded to Known Agents in 2026, and the scope expanded well past the original crawler-blocking tool. Known Agents now offers four things: real-time agent analytics, AI chat referral tracking (measuring human click-through from ChatGPT, Claude, Gemini, Perplexity and others), an auto-updating robots.txt, and an agent identification API.
Pricing is straightforward: a free tier with one project and 100,000 events a month, a $29/month Startup plan with three projects and 1 million events, a $299/month Growth plan with ten projects and 10 million events, and custom enterprise pricing. The company claims 5,000+ websites use the service and publishes an aggregate "Agentic Web Index" built from that install base.
The catch, and it's a meaningful one: Known Agents requires you to install a script, plugin, or API integration on the site you want to measure. That means you can't point it at a competitor's domain or a prospect's site to see how exposed they are to AI crawlers. It's a monitoring tool for sites you control, full stop.
Cloudflare's AI Crawl Control is free and already running for a lot of people
Cloudflare's version of this, now called AI Crawl Control (it used to be called AI Audit), ships on every Cloudflare plan including the free one, and it needs zero configuration if your DNS already runs through Cloudflare. That's a real advantage over anything that requires a separate script tag.
The dashboard has three tabs worth knowing about. Overview gives you total requests, which crawlers are hitting you hardest, and your robots.txt compliance status. Crawlers breaks activity down per bot operator, OpenAI, Anthropic, Google, ByteDance, Meta, with per-crawler drilldowns added in late 2025. Metrics is the deepest tab: requests over time, status code distribution (so you can catch 403s and 5xx errors AI bots are hitting), a content-format comparison showing what AI systems request via their Accept header versus what your server actually serves, and a list of your most-crawled paths.
One nuance that trips people up: referral analytics, meaning which AI platforms are sending actual human click-through traffic to specific paths, are gated to paid Cloudflare plans. The free tier shows you crawler requests, not the referral side of the picture. And if you want Cloudflare's full enterprise-grade Bot Management with dedicated Bot Analytics inside Security Analytics, that's a different product tier entirely, and enterprise Cloudflare contracts commonly start somewhere north of $5,000 a month. Don't confuse the free AI Crawl Control dashboard with that.
What Promptwatch's crawler logs add that the other two don't
Promptwatch calls its version Agent Analytics, and the framing is blunt: if your pages aren't being crawled, they don't exist in AI search. That's the same starting point as DarkVisitors/Known Agents and Cloudflare. What's different is what happens after the crawl is logged.
Agent Analytics ties crawl events to citation outcomes. A customer-facing example on Promptwatch's own feature page shows 1,248 pages crawled, 312 of them actually cited in an AI response, for a 25% citation rate. Cloudflare and Known Agents will both tell you a bot showed up. Neither tells you whether the crawl turned into anything. That citation-rate number is the whole point of running a GEO program: crawling without citation means your content is technically reachable but not winning, which is a content or structure problem, not a crawl-access problem.
Promptwatch also connects to Cloudflare, Vercel, Fastly, AWS CloudFront, Netlify, Akamai, and Google Cloud CDN, plus raw server logs, so it's designed to sit on top of whatever you're already running rather than replace it. And because it ties crawl gaps to a Content Agent that can generate and publish pages against those gaps, the tool closes a loop the other two leave open: see the problem, then actually do something about it from inside the same product.
According to Promptwatch's own three-month production data covering June through September 2026, OpenAI's crawlers accounted for 79.8% of verified AI crawler requests in the first week of September, down from 94.8% in the week of June 8, as other providers grow their share. That's the kind of industry benchmark worth comparing against your own logs: if a provider makes up a large share of that aggregate and shows up near zero on your own site, you're probably blocked or unreachable for that specific crawler, not just unlucky. You can check the full breakdown in Promptwatch's AI crawler traffic data.
Anthropic's crawler is also worth watching specifically. Promptwatch's tracking of Claude's citation crawler visits shows Claude-User and ClaudeBot traffic growing more than 100x between mid-December 2025 and mid-April 2026, with an inflection point in late March where daily visits jumped from the hundreds into the thousands within two weeks. A robots.txt rule that blocked Claude harmlessly in December is excluding a fast-growing channel now.
Side by side
| Tool | Requires install on target site | Shows crawl requests | Shows human referral clicks | Ties crawling to citations | Can generate content to close gaps | Starting price |
|---|---|---|---|---|---|---|
| Known Agents (formerly DarkVisitors) | Yes | Yes | Yes | No | No | Free (100K events/mo), $29/mo for more |
| Cloudflare AI Crawl Control | No, if already on Cloudflare | Yes | Paid plans only | No | No | Free |
| Promptwatch Agent Analytics | Yes (site connection) | Yes | Yes | Yes | Yes | $95/mo (Essential) |
The blind spot all three share
Here's the thing nobody's marketing page says out loud. A third-party comparison of AI Crawler Check versus Known Agents put it well: if your robots.txt blocks ClaudeBot, your traffic log shows zero ClaudeBot visits. That zero looks identical whether Claude has never heard of your site or Claude would happily crawl you every day if you let it. Absence of traffic in any of these three tools doesn't tell you the cause of the absence.
That means crawler-log tools, Known Agents, Cloudflare, Promptwatch, all included, are monitors, not diagnostics. They tell you what already happened. If a crawler is conspicuously missing from your logs, the next step isn't to stare at the dashboard longer, it's to go read your own robots.txt and firewall rules directly. A free tool like aicrawlercheck.com's checker or Promptwatch's own free AI crawler URL inspector can confirm whether a specific bot is technically allowed in, which is a different question from whether it's showing up.
Which one should you actually use
If you're already on Cloudflare and just want a sanity check on which AI bots are hitting your site without installing anything new, start there. It's free, it's already live, and the content-format comparison chart alone (what AI systems request vs. what you serve) catches real bugs.
If you want referral click-through tracking and don't run Cloudflare, Known Agents is a reasonable standalone option, particularly at the free and $29 tiers, though remember it only works on sites you've installed it on.
If your actual goal is GEO, meaning you care less about "did a bot visit" and more about "is my content winning in AI answers, and what do I do about pages that get crawled but never cited," that's the specific gap Promptwatch's Agent Analytics was built to close, and it's the only one of the three that turns the answer into a prioritized action list rather than a number on a dashboard.
For a broader look at how crawler logging fits into a full GEO stack, the directory of agentic SEO tools at agenticseotools.com and the GEO software comparisons at bestgeosoftware.com are worth a browse before you commit to one platform.