Key takeaways
- Promptwatch Data shows Meta-WebIndexer rose from about 2.2% of tracked AI crawler requests in mid-July 2026 to 37.8% on August 9, 2026. That is roughly 17x in under a month, from one crawler.
- Meta-WebIndexer is the search and citation crawler. Meta says allowing it "helps us cite and link to your content in Meta AI's responses." It is a separate robots.txt token from the training crawler, Meta-ExternalAgent.
- Allow-or-block is a real tradeoff: blocking likely means no Meta AI citations, allowing means sustained crawl load.
- Robots.txt is a request, not enforcement. Meta says Meta-ExternalFetcher may bypass it, and FacebookExternalHit can skip it for security checks. Hard limits need CDN or WAF rules.
- Check your logs for all four Meta crawlers before you change anything, and verify by IP range, because user-agent strings can be faked.
What happened: a single crawler became the heaviest in the logs
Promptwatch published a Meta-WebIndexer crawl report in August 2026 with an uncomfortable chart. In mid-July, meta-webindexer accounted for about 2.2% of all AI crawler requests in its logs, roughly 1 in 50. By August 9 it was 37.8%, more than 1 in 3. It is now the heaviest AI crawler in that dataset.
The timeline was not smooth:
- The share hovered around 2% until July 17.
- It spiked to about 23% between July 20 and 22.
- It passed 26% around August 6.
- It climbed from August 5 to a peak of 37.8% on August 9. August 10 was a partial day.
The surge comes from the indexing crawler specifically, not Meta's AI-training or link-fetching bots. PPC Land's explainer on Meta-WebIndexer says it overtook Meta's training crawler in June 2026.
How to read the 38% number
The method matters, because 38% sounds like "Meta is 38% of your traffic." It is not. The figure is meta-webindexer requests divided by requests from all tracked AI crawlers, identified by user agent. It is not total site traffic. Googlebot, Bingbot, FacebookExternalHit and human visitors are excluded from the calculation, and Meta's other crawlers count only in the denominator.
The tracked set covers OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot, claude-web), Google, Perplexity, xAI, Mistral, Cohere, DeepSeek and Meta. So the claim is narrower and still striking: among AI bots, Meta's indexer went from a footnote to the biggest line item in about a month.
Why is Meta doing this?
On August 6, 2026, @levelsio posted that Meta staff had allegedly told him Meta is building its own web index, so Meta AI searches don't have to go through Google. He also said Meta was scraping his sites hard enough to trigger load-average alerts. That claim is unconfirmed by Meta, and I would treat it as a plausible explanation, not a fact. The crawl pattern is at least consistent with someone building a search index.
Meta's four crawlers, and why they need separate decisions
The most common mistake I see in these discussions is treating "Meta's AI crawler" as one thing. Meta's web crawler documentation (updated May 21, 2026) lists several, each with its own purpose and robots.txt behavior.
| Crawler | Purpose per Meta | User agent | Robots.txt behavior |
|---|---|---|---|
| Meta-WebIndexer | Improves Meta AI search quality; helps Meta cite and link to your content | meta-webindexer/1.1 | Honors the token, per Meta's guidance |
| Meta-ExternalAgent | Training foundation AI models, or improving products by indexing content directly | meta-externalagent/1.1 | Block with a Disallow line |
| Meta-ExternalAds | Improving advertising and business products | meta-externalads/1.1 | Block with a Disallow line |
| Meta-ExternalFetcher | Fetches individual links at a user's request, supports agentic AI functions | meta-externalfetcher/1.1 | Meta says it may bypass robots.txt |
| FacebookExternalHit | Link previews on Facebook, Instagram and Messenger | facebookexternalhit/1.1 | Might bypass robots.txt for security or integrity checks |
Promptwatch also tracks FacebookBot, which the scraped Meta page doesn't describe. I would log it and decide later.
Two things follow from this table. First, blocking training (Meta-ExternalAgent) and allowing search (Meta-WebIndexer) are independent choices because they are different tokens. Second, the two crawlers that can ignore robots.txt are exactly the ones you cannot control with robots.txt.
The tradeoff: citations versus crawl load
Meta states the deal plainly. Allowing Meta-WebIndexer helps Meta cite and link to your content in Meta AI responses. The implication is that blocking it likely means you don't get cited there. Meta hasn't published numbers on how much traffic Meta AI sends, and I didn't find any, so you are weighing a visibility channel of unknown size against a crawl load you can measure.
Here is how I would frame it by site type:
- Publishers and brands that want to be recommended: allow Meta-WebIndexer. If people ask Meta AI about your category, you want to be in the source list.
- Small community or hobby sites on metered hosting: the cost side is real. One Towards AI writer reported paying an extra £25 a month because of Meta's crawler load. Rate limiting is likely a better answer than a full block.
- Sites with content licensing concerns: block Meta-ExternalAgent (training), and decide on the indexer separately.
- Sites that bill per request, such as serverless or edge setups: budget for sustained crawl load, which is also Promptwatch's own advice.
Complaints you will find, and how much weight to give them
The loudest evidence is anecdotal. A Reddit r/webdev thread reports 7.9 million requests and 900+ GB in 30 days, with other commenters citing 11 million and 741,900 requests. A Towards AI post claims Meta's crawler sent 226,391 requests to disallowed addresses, 98% of everything it asked for.
A Cloudflare community thread describes excessive repeated requests from meta-externalagent and meta-externalfetcher.
I would not read these as proof that Meta-WebIndexer ignores robots.txt. They are unverified, and several name other Meta crawlers, including ExternalAgent and ExternalFetcher. Two of those are the ones Meta says can bypass the file. The honest summary is that the compliance question is open, and the only way to answer it for your own site is to check your own logs.
Step 1: audit your logs before touching robots.txt
Pull 30 days of access logs and filter by these user-agent substrings:
meta-webindexer
meta-externalagent
meta-externalfetcher
meta-externalads
facebookexternalhit
FacebookBot
For each one, count requests, bytes served, top paths and status codes. You are looking for three things:
- Volume per crawler, so you know which one is actually hurting.
- Hits to paths you already disallow, which tells you whether the token is respected.
- Repeated requests to the same URL, which points to a caching or rate-limiting problem on your side as much as theirs.
User-agent strings can be faked. Meta lists IP addresses for allowlisting and verification, which it calls more secure than UA strings. I couldn't see the ranges on the scraped page, so pull them from Meta's documentation directly. Anything claiming to be Meta but arriving from an unlisted IP is not Meta.
If you would rather not grep logs by hand, tools built for this exist. Promptwatch has IP-verified AI crawler logs that show which pages each AI bot reads, how often, and where requests fail, with spoofed bots filtered out against operators' published IP ranges.
Other options for crawl analysis include log-focused crawlers like [tool:jetoctopus] and AI-agent traffic analytics like [tool:siteline-ai]. If you already run a technical crawler, [tool:screaming-frog] helps you see what a bot would actually find on your site.
Step 2: set your robots.txt
Meta's own example syntax is a user-agent line followed by a Disallow path. Robots.txt changes can take up to 24 hours to take effect, because crawlers may cache the file. Here are three common configurations.
Option A: allow search, block training
This is where I land for most publishers and brands. You stay eligible for Meta AI citations and opt out of model training.
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalagent
Disallow: /
Option B: allow search, but keep private areas out
User-agent: meta-webindexer
Disallow: /private/
Disallow: /account/
Disallow: /search
User-agent: meta-externalagent
Disallow: /
Blocking internal search result pages is worth doing for any crawler. Infinite URL spaces are where crawl budgets go to die.
Option C: block everything from Meta
User-agent: meta-webindexer
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalads
Disallow: /
User-agent: meta-externalfetcher
Disallow: /
Expect no Meta AI citations. Also expect that ExternalFetcher may ignore the last rule, and note that you should not block FacebookExternalHit unless you are fine with broken link previews on Facebook, Instagram and Messenger.
Crawl-delay is not the answer
I didn't find a Meta crawl-delay policy, and crawl-delay is not part of the robots.txt standard that Meta points to. Don't count on it. Rate limit at your edge instead.
Step 3: enforce what robots.txt can't
Google's own documentation says robots.txt instructions "cannot enforce crawler behavior to your site; it's up to the crawler to obey them." Meta's page tells you to add a Disallow line to block its crawlers, but it also says ExternalFetcher may bypass the file. For anything robots.txt can't stop, you need a layer that actually refuses requests.
- Match on user agent plus Meta's published IP ranges in your CDN or WAF. UA alone is spoofable, and IP alone can catch legitimate shared ranges.
- Rate limit by user agent rather than blocking, if you want to stay citable but cap the load. A few requests per second per crawler is a sensible starting point for a small site.
- Return 429 or 503 with Retry-After for overload, not 200 with an error page. Bots that see 200s keep coming.
- Cache aggressively. Many of the "900 GB" stories are really about uncached dynamic pages being fetched repeatedly.
- Keep FacebookExternalHit working. Meta requires gzip and deflate support and Open Graph properties within the first 1 MB. If your previews break, the Sharing Debugger helps troubleshoot.
If Meta's behavior looks like a genuine violation, Meta lists [email protected] as the contact.
Step 4: treat Meta as an emerging AI search surface
The interesting part of this story isn't the load. It is that a major assistant may be building an independent index. For years, AI answers have leaned on Google and Bing results. Meta crawling at this scale suggests Meta AI may eventually cite sources the way ChatGPT does.
Source behavior differs a lot by engine. Promptwatch Data shows ChatGPT got only 0.46% of its citations from social platforms from Sep 28 to Oct 4, 2026, while Google AI Overviews got 12.2% (social media citations by AI model). Whatever Meta ends up citing will probably have its own quirks, so don't assume it behaves like another engine.
Practical steps:
- Keep the pages you want cited fast, crawlable and server-rendered. Crawlers like this rarely run your JavaScript.
- Check what the indexer actually fetches. If it spends most of its requests on tag archives and filter URLs, your important pages are being starved.
- Track whether Meta AI mentions or cites you at all. Promptwatch monitors Meta AI's real interface alongside ChatGPT, Gemini, Perplexity and Google AI Overviews, so you can see whether more crawling turns into citations.
- Compare crawl to citation. A bot reading a page doesn't mean the page is cited. Per-page crawl-to-citation rates separate an access problem from a content problem.
For broader background on how crawler access feeds citations, the AI crawler traffic report breaks down weekly verified requests from OpenAI, Anthropic, Google, Perplexity and Mistral. To check whether your site is readable at all, the free Agentic Readiness Check and the Robots.txt Generator for AI are useful starting points, and the AI bots and crawlers directory lists 400+ user agents with robots.txt guidance.
Comparison: ways to handle Meta's crawlers
| Approach | Stops Meta-WebIndexer | Stops ExternalFetcher | Keeps Meta AI citations | Effort |
|---|---|---|---|---|
| Robots.txt allow indexer, block ExternalAgent | No (by design) | No | Yes | Low |
| Robots.txt block all Meta tokens | Yes, if honored | Not reliably | No | Low |
| WAF rule by UA and IP | Yes | Yes | No | Medium |
| Rate limiting by UA | Caps load | Caps load | Yes | Medium |
| Better caching | Reduces cost, not requests | Reduces cost | Yes | Medium |
Most sites should combine rows one, four and five. Block hard only if the cost is genuinely unmanageable or the content is licensed.
Tools that help with crawler monitoring
Log analysis and crawler tooling can all cover parts of this job. The directory at bestgeosoftware.com lists GEO platforms if you want to compare options for tracking AI citations, and several tools in the catalog watch AI crawlers specifically, such as [tool:scriptbee] and [tool:xseek]. For a site audit alongside crawler review, [tool:sitebulb] and [tool:oncrawl] both handle large sites. The point is to see your own numbers before you decide, not to trust anyone's headline percentage, including this one.
A short decision checklist
- Pull 30 days of logs for all Meta user agents.
- Verify those requests against Meta's published IP ranges.
- Allow meta-webindexer if you want Meta AI citations. Block meta-externalagent if you don't want training use.
- Add rate limits and caching before you reach for a full block.
- Use WAF rules for ExternalFetcher, since robots.txt isn't reliable there.
- Recheck in a month. This share moved 17x in under four weeks, so it can move again.
The 38% figure is a snapshot of one dataset on one day, and the numbers will have changed since August. What won't change is the shape of the decision: Meta ties citations to crawl access, and you now have separate switches for each purpose. Use them deliberately.