AI crawlers
AI crawlers are the automated programs AI companies use to read the web, just as search engines use Googlebot. Each AI assistant relies on one or more of them:
| Crawler | Used by |
|---|---|
| GPTBot, OAI-SearchBot | ChatGPT (OpenAI) |
| ClaudeBot | Claude (Anthropic) |
| Google-Extended | Gemini (Google) |
| PerplexityBot | Perplexity |
| Bingbot | Copilot (Microsoft) |
If your blocks these crawlers, the corresponding assistants may be unable to read your pages - which usually means they cannot cite or recommend your content in their answers. Many AI crawlers also do not run , so content that only appears after scripts execute may be invisible to them even when they are allowed in. For which assistants people actually use, see AI assistant market share.
Learn more
- Overview of OpenAI crawlers - GPTBot and OAI-SearchBot
- Google's common crawlers - Google-Extended
- Does Anthropic crawl data from the web? - ClaudeBot
- Perplexity crawlers - PerplexityBot
- Which crawlers does Bing use? - Bingbot