Key takeaways
- Promptwatch Data shows Meta-WebIndexer rising from roughly 2.2% of tracked AI crawler requests in mid-July 2026 to a peak of 37.8% on August 9, a roughly 17x jump from one crawler.
- The figure is a single-day peak, based on user-agent counts. It is a share of the AI crawler requests Promptwatch tracks, not of the web and not of bandwidth.
- Meta says allowing Meta-WebIndexer helps it cite and link to your content in Meta AI responses. It does not guarantee visibility.
- The reason for the build-out is unconfirmed. A secondhand report says Meta wants its own web index so Meta AI searches stop going through Google.
- Check robots.txt, your logs, and your hosting bill. Some site owners reported very heavy Meta crawl loads.
What actually happened
In mid-July 2026, Meta-WebIndexer was a background player. Promptwatch measured it at about 2.2% of all AI crawler requests in its logs, roughly 1 in 50. By August 9 it had reached 37.8%, more than 1 in 3, which made it the heaviest AI crawler in that dataset.
The timeline from Promptwatch's Meta-WebIndexer report, published August 10, 2026:
- The share hovered around 2% until July 17.
- It spiked to about 23% between July 20 and 22.
- It climbed past 26% around August 6.
- It rose from August 5 to a peak of 37.8% on August 9. August 10 was a partial day.
The report also says the surge comes from the indexing crawler specifically, not from Meta's AI-training or link-fetching bots. That distinction matters. Training crawlers collect data once to build a model. An indexing crawler is what you build when you want a searchable, refreshable copy of the web.
What Meta-WebIndexer is
PPC Land describes Meta-WebIndexer as the crawler Meta uses to build and refresh the search index behind Meta AI, the assistant embedded in Facebook, Instagram, WhatsApp and Messenger. That assistant needs live content to answer questions about news, prices or sports.
What Meta itself documents is fairly short:
- The crawler identifies as
meta-webindexer/1.1. - It answers to the robots.txt token
meta-webindexer. - Meta says it navigates the web to improve Meta AI search result quality.
- Meta tells publishers that allowing it makes their content eligible for citation and linking in Meta AI responses.
The NeuralAdX guide to Meta AI optimisation makes the same point and adds a useful caution: allowing the crawler addresses access, not ranking. It does not guarantee your pages get cited.
Why the surge, and what we do not know
Here is the honest version. Crawling at this volume and pace is what building a web index from scratch looks like. On August 6, the developer @levelsio posted that Meta staff had told him by DM that Meta is "allegedly" building its own web index, so Meta AI web searches do not go through Google, which could use them for its own training. He also said the scraping triggered load alerts on one of his servers.
That is unconfirmed and secondhand. I would treat it as a plausible explanation, not a fact. Meta has not announced a search engine. What we can say is that the crawler behavior is consistent with index building, and that Meta's own documentation ties the crawler to Meta AI search and citations.
How to read the 37.8% number correctly
A headline like "38% of AI crawler traffic" invites misreading. A few limits worth keeping in mind:
- It is a one-day peak on August 9, not a current share. I found no later data confirming it held.
- It is a share of requests, not bytes. A crawler making many small requests can look bigger than one pulling fewer large files.
- It is the
meta-webindexeruser agent divided by requests from all AI crawlers Promptwatch tracks. Meta's other crawlers (meta-externalagent,meta-externalfetcher,FacebookBot) count only in the denominator. Googlebot, Bingbot, FacebookExternalHit and human traffic are excluded. - It is user-agent based. Promptwatch's separate AI crawler traffic report covers IP-verified requests only, from OpenAI, Anthropic, Google, Perplexity and Mistral, and it does not show Meta's share. In that dataset OpenAI was 79.8% of verified requests in the week of August 31 to September 6, down from 94.8% in June 8 to 14. Do not mix the two numbers.
Other vendors see the same direction. Fastly's threat research, covering mid-April to mid-July, found AI crawlers made up almost 80% of AI bot traffic, with Meta generating the largest share of crawler requests.
The infrastructure side
This is not only a visibility story. Several site owners on Reddit's r/webdev reported heavy Meta crawling: one said Meta's AI crawler hit their site 7.9 million times in 30 days, with 900+ GB of bandwidth, and others posted numbers of 741,900 and 11 million requests in a month. These are anecdotes from individual sites, and I have not verified them. But they match Promptwatch's advice to budget for sustained crawl load if your infrastructure bills by request.
If you pay per request or per GB egress, check this before you decide anything about robots.txt.
The decision: allow, limit, or block
There is a real tradeoff here. Allowing the crawler makes your content eligible for Meta AI citations. Blocking it protects bandwidth and keeps content out of Meta's index, but if Meta ships a search product built on this crawl, you will not be in it.
| Situation | Likely approach |
|---|---|
| Publisher or brand that wants citations in AI answers | Allow Meta-WebIndexer on public pages |
| Site with expensive per-request hosting | Allow, but rate limit at the CDN or WAF and monitor |
| Paywalled or licensed content | Block or restrict to the pages you want indexed |
| Site with a data processing agreement limiting reuse | Review the terms with legal before allowing |
| Audience has no overlap with Meta AI users | Block is defensible, but revisit later |
There is no universally right answer. For most businesses that want to be recommended by AI assistants, I lean toward allowing it and managing load, because the cost of being absent from a new index is hard to reverse.
How to check your own exposure
Step 1: look at robots.txt
Find rules that touch the four Meta user agents. Promptwatch recommends checking all of them: Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and FacebookBot. A blanket Disallow or a copied AI-bot blocklist may already be shutting Meta out.
A minimal example that allows the indexing crawler on everything except a private path:
User-agent: meta-webindexer
Allow: /
Disallow: /account/
To block it entirely:
User-agent: meta-webindexer
Disallow: /
Step 2: check CDN and WAF rules
robots.txt is a request, not a wall. A bot protection rule can block a crawler you explicitly allowed. The AI crawler traffic report gives this advice: compare your own logs against the overall mix. A provider that is large in the mix but near zero in your logs suggests you are blocked or unreachable.
Step 3: read your server logs
Filter for the meta-webindexer user agent and look at request counts per day, status codes, and which paths get hit most. Watch for 429s and 5xx errors, which tell you the crawl is straining your origin. Because user agents can be spoofed, verify the source IPs if you plan to make firewall decisions from them.
Step 4: measure it continuously
Log review by hand does not scale. Several tools cover AI crawler analytics, with different depth:
| Tool | Crawler log analysis | Notes |
|---|---|---|
| Promptwatch | IP-verified logs of many AI crawlers, with errors and crawl-to-citation paths | Also tracks Meta AI responses and ties crawls to citations |
| Siteline AI | AI agent traffic analytics | Focused on agent traffic |
| Scriptbee | AI crawler monitoring | Unlimited domains |
| xSeek | GPTBot crawler monitoring | ChatGPT-focused |
| Screaming Frog | Log file analysis alongside crawling | General SEO crawler, not AI-specific |
If you want to see how crawls translate into actual mentions, tools like Promptwatch can track Meta AI responses next to ChatGPT, Perplexity and Google AI Overviews, so you can see whether being crawled turns into being cited. For a wider scan of options, the GEO software directory at bestgeosoftware.com lists more.
Making content worth citing in Meta AI
Crawl access is step one. Citation is a separate question, and Meta has not published how it ranks sources. Without invented rules, what we can reasonably lean on is what Promptwatch Data shows about AI citation behavior generally:
- Product pages are the most cited content type in ChatGPT, at 22.4% of citations from Sep 28 to Oct 4, 2026, followed by landing pages and listicles, per Promptwatch's content type report.
- Landing pages and docs stay cited longest, while news loses half its pages within 36 days, according to the content half-life report.
- Almost half of ChatGPT responses cite 10 or more unique domains, so there is room for more than the biggest sites, per the unique domains report.
These are ChatGPT numbers. Meta AI may behave differently, and I would not assume it does not. Still, clear, specific, factual pages that answer a question directly tend to be what retrieval systems pull. Fresh facts, prices, specs and original data are good candidates for a live-search assistant.
A short working checklist:
- Keep important pages server-rendered so a crawler sees real content without running scripts.
- Make sure pages return 200 quickly and are in your sitemap.
- Put proprietary data and direct answers high on the page.
- Update time-sensitive pages. Meta AI is described as answering questions about news, prices and sports.
- Check sentiment and accuracy of what Meta AI says about you, not just whether it mentions you.
What about the meta description myth?
One tangent, since it comes up. A six-week test by Seer Interactive found that altering meta descriptions had no measurable effect on how often AI crawled or cited their pages. Do not spend this quarter rewriting descriptions for Meta AI. Spend it on access and content quality.
What to do this week
- Open robots.txt and search for
meta-webindexer,meta-externalagent,meta-externalfetcherandfacebookbot. - Pull 30 days of logs and count Meta requests, errors and bandwidth.
- Decide on allow, rate limit, or block, and write the reason down so the next person knows why.
- Ask your host or CDN for request-based billing details if you do not have them.
- Set up recurring monitoring so a future crawler surge does not surprise you.
If you want help auditing crawler access or building a plan for AI search visibility, the team at 1001 SEO Media, which publishes this site, does technical SEO and GEO work.
More data on how AI systems crawl and cite is collected at Promptwatch Data, including the Meta-WebIndexer report that started this piece. Re-check the numbers there, because crawler shares move week to week.
Bottom line
Meta-WebIndexer peaked at 37.8% of the AI crawler requests Promptwatch tracks on August 9, 2026, up from about 2% a month earlier. Meta documents that allowing it makes your content eligible for Meta AI citations. Whether Meta is building a full search index is unconfirmed. Either way, the practical move is the same: know whether you are allowing it, know what it costs you, and decide on purpose.