Key takeaways
- A missing citation has three possible causes: AI bots can't reach the page (access), the page is reachable but not worth citing (content), or the platform simply prefers other domains (source bias).
- Check access first. If a crawler never hits the page, no amount of rewriting will help.
- If bots fetch the page but it never gets cited, you have a content problem. If it was cited and then dropped, suspect ordinary content decay before you suspect blocking.
- Compare your own crawler logs against the market mix. A provider that is a big share of the market but near zero in your logs usually means you are blocked.
- Citation errors are sometimes the platform's fault. The Tow Center found AI search tools often failed to link back to the original source even when they used it.
Why this diagnosis matters
Most teams react to "we're not showing up in ChatGPT" by writing more content. Sometimes that is exactly right. Other times the robots.txt file, a CDN rule, or a WAF has been quietly turning away GPTBot for six months, and the new articles are going into a void.
The two problems look identical from the outside. Your brand is absent from the answer. But the fixes are completely different, one is an engineer's afternoon and the other is a content sprint. Guess wrong and you burn weeks.
So think of it as a path with stages. A page has to be discovered, fetched, understood, selected, and then cited. A failure at each stage has its own evidence, and you can walk the path from left to right until you find where it breaks.
The crawl-to-citation path in five stages
| Stage | Question | Evidence you need | Typical fix |
|---|---|---|---|
| 1. Permission | Are bots allowed in? | robots.txt, CDN and WAF rules | Update rules, allow verified bots |
| 2. Fetch | Do bots actually request the page? | Crawler logs by bot and URL | Fix internal links, sitemap, server errors |
| 3. Render | Can they read what they fetched? | Status codes, rendered HTML, JS dependence | Server-side render or prerender |
| 4. Selection | Is the page chosen for answers? | Citation data per prompt | Improve content, fill gaps |
| 5. Attribution | Is it cited correctly and kept? | Citation position, freshness over time | Refresh content, fix source claims |
Stages 1 to 3 are access. Stages 4 and 5 are content (with a side of platform behavior). The boundary between them is the moment a bot successfully reads your page.
Stage 1: permission, the boring place problems hide
Access failures are usually self-inflicted and invisible. A few classics:
- A blanket
Disallowin robots.txt copied from a staging site. - A bot-protection rule that challenges anything without a browser fingerprint, which includes most AI fetchers.
- A CDN setting that rate-limits unfamiliar user agents.
- Different treatment for training crawlers and answer-time fetchers, where you blocked one family and accidentally the other.
That last one matters. OpenAI, Anthropic, Perplexity, and others each run more than one bot, and the ones that fetch pages live to answer a question are not the same as the ones that collect training data. Blocking GPTBot is a defensible choice. Blocking the search or user-triggered fetchers means you've opted out of being cited.
A quick sanity check: pull up your robots.txt and look for every AI user agent you care about. The AI bots and crawlers directory lists 400+ of them with robots.txt guidance, and the robots.txt generator for AI can draft a cleaner file. Then run the AI crawler URL inspector against a page you expect to be cited.
If you'd rather see how each crawler is treated across your whole stack, the Agentic Readiness Check shows which AI crawlers can read your site and which agent standards you support.
Stage 2: fetch, what your logs say
Permission on paper is not the same as access in practice. The real evidence is in server or CDN logs: did GPTBot, OAI-SearchBot, ClaudeBot, or PerplexityBot actually request this URL, and what did your server answer?
Two cautions before you trust a log line:
- User agents can be faked. Anyone can send a request that claims to be GPTBot. Count only requests whose source IP matches the operator's published ranges.
- Provider totals hide the interesting split. Look per crawler, because the bot doing the work at answer time is often not the bot with the biggest volume.
Compare your logs against the market mix
Here's a trick I like because it needs no special tooling. Promptwatch Data tracked IP-verified AI crawler requests weekly from June 8 to September 6, 2026 in its AI crawler traffic report. OpenAI made up 79.8% of verified requests in the week of Aug 31 to Sep 6, down from 94.8% in the week of Jun 8 to 14. Anthropic, Google, Perplexity, and Mistral split the rest.
Now open your own logs. If a provider is a large share of the market but close to zero in your data, you are probably blocked or unreachable for that provider. If your mix roughly resembles the market, access is likely fine and you should look at the next stage.
Another data point worth knowing: Promptwatch's Meta-WebIndexer report shows Meta's crawler going from about 2% to nearly 38% of tracked AI crawler requests between mid-July and August 9, 2026. The mix moves fast. A rule written in 2025 may be handling bots that barely existed then.
Promptwatch has AI crawler logs built for exactly this check, with IP-verified requests per bot, per page, and plain-language explanations of crawl errors. Setup runs through Cloudflare, Vercel, Fastly, other CDNs, or the WordPress plugin.
Other tools that can show bot traffic in some form include Siteline AI, Scriptbee, and xSeek, and classic log analyzers like JetOctopus and OnCrawl work if you filter for AI user agents yourself.
Stage 3: render, fetched but unreadable
A 200 status code doesn't mean the bot saw your content. Many AI fetchers don't execute JavaScript the way a browser does. If your product details, pricing, or article body only appear after client-side rendering, the bot may have fetched an empty shell.
Signs this is your problem:
- Logs show successful fetches, but the page never appears in any citation.
- View-source shows almost no body text.
- Pages that are server-rendered get cited while the JS-heavy ones don't.
Fixes include server-side rendering or prerendering. Prerender.io and DataJelly exist for this, and LovableHTML targets sites built with AI site builders.
Crawl errors count here too. Timeouts, 403s on certain paths, redirect chains, and orphan pages the bot never discovers all belong to the access side. Weekly technical audits and crawlers like Screaming Frog or Sitebulb catch many of these. Ahrefs has also been adding AI-crawl checks to Site Audit, covering whether AI bots can crawl the site and a list of AI-specific error issues.
Stage 4: selection, bots read it and still don't cite it
Once the logs show verified bots fetching and reading the page, you've crossed the line. From here, the problem is content (or platform preference), and the evidence changes. You need citation data per prompt: which prompts should this page answer, and which sources does the AI cite instead?
Questions to ask:
- Does the page actually answer the prompt, or just mention the topic?
- Which content type wins for this kind of prompt? Promptwatch Data's content type report found product pages were the most cited type in ChatGPT at 22.4% of citations (Sep 28 to Oct 4, 2026), followed by landing pages and listicles. A blog post chasing a prompt that ChatGPT answers with product pages is fighting the format.
- What do the cited competitors have that you lack? Specific data, clearer structure, a direct answer in the first paragraph?
- How wide is the net? Almost half of ChatGPT responses cite 10 or more unique domains, so there is room, but you need to be one of the sources worth including.
Query fan-outs help here. ChatGPT turns one prompt into several web searches, and your page may rank for none of them. The free ChatGPT query fan-out generator gives an idea of what to cover, and the fan-out academy guide explains the mechanics.
Content gap analysis closes the loop: map your pages against the prompts AI answers and see where coverage is thin. Promptwatch does this with coverage scores, and its Content Agents can draft and publish pages against those gaps. Alternatives that lean toward content work include AirOps, Frase, and Clearscope, though without citation or crawler data they can't tell you which of the two problems you have.
The third factor: platform source preference
Sometimes both your access and your content are fine, and the platform just likes other domains. The NDSS 2027 paper from Fudan University (arXiv 2610.11932) analyzed 17,211 citation instances across 10 AI-search platforms and 6,356 unique source domains. The top 20 domains captured between 20.5% and 70.8% of citations on each platform, depending on the platform.
Concentration like that means some prompts are effectively closed to unfamiliar domains on a given platform. The fix is not rewriting your page. It is earning mentions on the sources the platform already trusts, or targeting prompts where the cited set is wider. Promptwatch Data's industry reports show how different those sets are: for software prompts, GitHub, Microsoft's docs, and G2 lead, while for travel, Skyscanner, Going, and Tripadvisor do.
The same data shows source weight shifting quickly. reddit.com's share of ChatGPT citations fell from roughly 4% to 0.5% on August 14, 2026. If your strategy depended on Reddit threads, that wasn't an access problem or a content problem. It was the platform changing its mind.
Stage 5: attribution and decay, cited once, then gone
This is the stage people misread most. A page that was cited and then stopped being cited feels like a punishment, and teams jump to blaming access. Often it's just aging.
Promptwatch Data's content half-life study tracked every page ChatGPT cited on July 6, 2026 for 90 days, to October 4. Highlights:
- After 30 days, ChatGPT still cited 78% of landing pages and 76% of docs. After 60 days: 69% and 67%.
- Product pages, reviews, and how-tos sat around 55% still cited at 60 days, with a half-life of 74 to 77 days.
- News decays fastest. Almost a quarter of news pages were never cited again after day 0, and half were gone after 36 days.
My reading, and it is an inference rather than a claim in the report: if a news-style or how-to page drops out after a month or two, ordinary decay is the more likely explanation than a blocked bot. Check the logs first (cheap, definitive), then check freshness.
For evergreen types like docs and landing pages, a drop earlier than the typical curve deserves more suspicion. That's when to go back to stages 1 to 3.
Mis-citation is a platform issue too
There is one more wrinkle. The Columbia Journalism Review's Tow Center tested eight generative search tools with live search and found platforms often failed to link back to the original source. Chatbots were also bad at declining questions they couldn't answer accurately, and premium tools gave more confidently wrong answers than free ones.
Translation for diagnosis: if a bot fetched your page and the answer clearly used your information but cited someone else (or nobody), that isn't necessarily your content failing. It's attribution noise. Track it over many responses rather than reacting to one.
A decision flow you can run in an hour
- Pick one page you expect to be cited and three prompts it should win.
- Check robots.txt and WAF/CDN rules for every AI user agent, separating training bots from answer-time fetchers.
- Look for that URL in verified crawler logs over the past 30 days. No hits from any AI bot means an access or discovery problem. Fix internal links, sitemap inclusion, and rules.
- Hits from some bots but not others means a provider-specific block. Compare against the market mix.
- Hits with 4xx, 5xx, or empty rendered HTML means a render or server problem.
- Clean hits, no citations means a content problem. Read the pages that are cited instead and note format, depth, and directness.
- Previously cited and now dropped? Check the page's age against the half-life curve for its content type before touching anything else.
- Cited by competitors only on domains you have no presence on? Treat it as a source-preference problem and work on third-party mentions.
Which tool covers which stage
| Stage | What you need | Options |
|---|---|---|
| Permission | Rule checks | Free robots.txt and crawler inspectors, Agentic Readiness Check |
| Fetch | Verified bot logs | Promptwatch, Siteline AI, Scriptbee, CDN logs plus a log analyzer |
| Render | Rendered HTML audits | Screaming Frog, Sitebulb, Prerender.io |
| Selection | Citation and gap data | Promptwatch, Profound, Peec AI, Otterly.AI |
| Attribution | Per-page tracking over time | Promptwatch Page Tracker, manual spot checks |
Most prompt trackers answer "was I mentioned?" and stop. The reason to use something that joins logs and citations is that the diagnosis lives in the join: this page was fetched by these bots, and it was cited (or not) for these prompts. Promptwatch shows that crawl-to-citation path per page, so you can tell an access problem from a content problem without stitching two tools together. For a wider view of the market, the GEO software directory lists alternatives.
Profound
Common mistakes
Rewriting before checking logs. This is the most expensive one. Ten articles for a site that blocks the answer-time fetcher is ten articles nobody will cite.
Trusting a user-agent string. Spoofed bots inflate your numbers and hide real blocks.
Treating one missing citation as a trend. Responses vary. Look at patterns across prompts and days.
Confusing training access with citation access. You can block training crawlers and still be cited if the retrieval bots are allowed.
Forgetting decay. A page cited in July and absent in October may simply have aged out, especially if it is news-style.
Fix the right thing
The point of walking the path is to stop guessing. Access problems are binary and cheap to confirm in logs. Content problems need prompt-level citation evidence. Platform preference needs a different plan entirely. If your logs are clean and your content is solid, the next lever is where you are mentioned, not what you publish.
To get a first read without any setup, run the free AI visibility report, which includes a crawler access check, then use the 7-day free trial if you want per-page logs and citations in one place. Pricing is at promptwatch.com/pricing.
If you'd rather have specialists run the technical and content audit for you, 1001 SEO Media does SEO and GEO work for brands that want to be cited as well as ranked.