AI crawlers are automated programs that AI companies use to collect web content, either to train language models or to find and fetch pages for AI search answers. Most companies publish separate user agents for these purposes so that site owners can allow or block each one in robots.txt.[1][2]
Major AI crawlers
| Company | User agent | Purpose | Follows robots.txt |
|---|---|---|---|
| OpenAI | GPTBot | Content that may be used to train models | Yes[1] |
| OpenAI | OAI-SearchBot | Surfacing websites in ChatGPT search | Yes[1] |
| OpenAI | ChatGPT-User | Fetches pages for user requests | Rules "may not apply"[1] |
| Anthropic | ClaudeBot | Content that may be used for training | Yes[2] |
| Anthropic | Claude-SearchBot / Claude-User | Search indexing / user requests | Yes[2] |
| Perplexity | PerplexityBot | Surfacing websites in Perplexity search | Yes[3] |
| Perplexity | Perplexity-User | Fetches pages for user questions | "Generally ignores" robots.txt[3] |
| Apple | Applebot / Applebot-Extended | Search features / training control | Yes[4] |
| Common Crawl | CCBot | Open web archive | Yes[5] |
Training and search agents
OpenAI says each of its robots.txt settings is independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to opt out of training; sites that block OAI-SearchBot do not appear in ChatGPT search answers.[1] Anthropic says its three agents all follow robots.txt and that blocking its IP addresses is not a reliable way to opt out.[2] Perplexity says PerplexityBot is not used to collect content for training AI models.[3]
Google-Extended and Applebot-Extended
Google and Apple use robots.txt tokens that control how content may be used rather than separate crawlers. Google-Extended governs use of content Google crawls for training Gemini models and grounding in Gemini apps, without affecting Google Search;[6] Applebot-Extended governs whether content Applebot has crawled may train Apple's foundation models, and pages that disallow it can still appear in Apple's search features.[4]
Common Crawl
Common Crawl, a nonprofit founded in 2007, maintains a free archive of more than 300 billion web pages collected over 15 years.[7] Its crawler, CCBot, checks robots.txt before fetching pages.[5] Filtered Common Crawl data made up 60% of the training mix for OpenAI's GPT-3 model, according to the 2020 paper describing it.[8]
llms.txt
In September 2024 Jeremy Howard proposed llms.txt, a Markdown file placed at /llms.txt that gives language models a short summary of a site and links to key content.[9] Google states that Google Search does not use such files, so having one neither helps nor hurts visibility in its search results.[10] An IETF working group, AIPREF, is developing a standard vocabulary for expressing AI-usage preferences through robots.txt and HTTP headers.[11]
Blocking and payment
On July 1, 2025, Cloudflare began blocking AI crawlers by default for new domains unless site owners give permission, and asked AI companies to declare whether each crawler is for training, inference or search.[12] It also introduced "pay per crawl," in private beta, letting publishers charge crawlers a flat price per request using the HTTP 402 Payment Required status.[13]
In August 2025 Cloudflare accused Perplexity of using undeclared "stealth" crawlers to get around blocks;[14] Perplexity responded that its agents fetch pages on users' requests rather than crawling, and that Cloudflare had misattributed traffic from a third-party service.[15]
References
- ^ ^ ^ ^ ^ "Overview of OpenAI Crawlers". OpenAI. developers.openai.com
- ^ ^ ^ ^ "Does Anthropic crawl data from the web, and how can site owners block the crawler?". Anthropic, April 7, 2026. support.claude.com
- ^ ^ ^ "Perplexity Crawlers". Perplexity. docs.perplexity.ai
- ^ ^ "About Applebot". Apple, September 4, 2026. support.apple.com
- ^ ^ "Frequently asked questions". Common Crawl Foundation. commoncrawl.org
- ^ "Google's common crawlers". Google Crawling Infrastructure, updated October 15, 2026. developers.google.com
- ^ "Common Crawl". Common Crawl Foundation. commoncrawl.org
- ^ Brown, Tom B.; et al. (2020). "Language Models are Few-Shot Learners". arXiv. arxiv.org
- ^ Howard, Jeremy (September 3, 2024). "The /llms.txt file". llmstxt.org. llmstxt.org
- ^ "Google's guide to optimizing for generative AI features on Google Search". Google Search Central, updated July 10, 2026. developers.google.com
- ^ "AI Preferences (aipref) working group". IETF Datatracker. datatracker.ietf.org
- ^ "Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large". Cloudflare, July 1, 2025. cloudflare.com
- ^ "Introducing pay per crawl: Enabling content owners to charge AI crawlers for access". Cloudflare Blog, July 1, 2025. blog.cloudflare.com
- ^ "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives". Cloudflare Blog, August 4, 2025. blog.cloudflare.com
- ^ "Agents or bots? Making sense of AI on the open web". Perplexity, August 4, 2025. perplexity.ai