AI crawler directory
AI crawlers are bots that AI companies use to read websites. They do three different jobs: reading a page for a person right now, indexing pages for AI search, and collecting training data. The job decides what a visit means for you and whether you want to allow it.
This directory lists the crawlers Sitenumbers recognizes, with each one's owner, robots.txt token and how to check that a request is genuine. Every fact comes from the company's own documentation, linked on each bot's page. Where a company publishes nothing, we say so.
Check your site's AI crawler access
Reading for a person, right now
These agents fetch a page because someone asked an AI assistant a question. They are the closest thing to a direct measure of how often assistants use your content. OpenAI and Perplexity say robots.txt may not apply to these fetches, because a user started the request. Anthropic says its bots, Claude-User included, honor robots.txt.
| Bot | Owner | What it does | robots.txt token | Verification |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | Visits pages for user actions in ChatGPT and custom GPTs | ChatGPT-User (may not apply) |
IP list |
| Claude-User | Anthropic | Fetches pages when a Claude user asks a question | Claude-User |
IP list (shared by Anthropic's bots) |
| Perplexity-User | Perplexity | Fetches pages for user actions in Perplexity | Perplexity-User (generally ignored) |
IP list |
Indexing for AI search
These crawlers add pages to the index that AI search answers draw from. Blocking them can keep your site out of those answers.
| Bot | Owner | What it does | robots.txt token | Verification |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search | OAI-SearchBot |
IP list |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search | PerplexityBot |
IP list |
| Googlebot | Crawls for Google Search and its features | Googlebot |
Reverse DNS or IP list | |
| Bingbot | Microsoft | Crawls for Bing search | bingbot |
Reverse DNS (search.msn.com) |
| Applebot | Apple | Crawls for Spotlight, Siri and Safari search features | Applebot |
Reverse DNS (applebot.apple.com) or IP list |
Apple says Applebot data may also be used to train Apple's foundation models. That use is controlled with the separate Applebot-Extended token, which does not crawl on its own.
Collecting training data
These crawlers gather material for training AI models. A visit does not mean your page will appear in AI answers.
| Bot | Owner | What it does | robots.txt token | Verification |
|---|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's models | GPTBot |
IP list |
| ClaudeBot | Anthropic | Collects content that could contribute to training Anthropic's models | ClaudeBot |
IP list (shared by Anthropic's bots) |
| Meta-ExternalAgent | Meta | Crawls for uses such as training AI models or indexing content | meta-externalagent |
Not published |
| Bytespider | Attributed to ByteDance | Not published by ByteDance | Bytespider |
Not published |
| Google-Extended | robots.txt token only. Controls whether Google may use your content for Gemini training and grounding. No separate user agent. | Google-Extended |
Not applicable |
Google says Google-Extended does not affect inclusion or ranking in Google Search.
What the columns mean
- robots.txt token: the name you use in a
User-agent:line to allow or block the bot. - IP list: the company publishes the IP ranges its bot uses. A request is genuine only if its IP is on the list.
- Reverse DNS: look up the hostname of the IP, check the domain, then look the hostname up again to confirm it points back to the same IP.
- Not published: the company gives no way to verify the bot. Anyone can send that user agent.
Do AI crawlers run JavaScript?
Of the crawlers above, only Google (Googlebot) and Apple (Applebot) document rendering JavaScript. OpenAI, Anthropic, Perplexity, Meta and ByteDance do not document it. That is why script-based analytics is the wrong place to count AI crawlers. Read why analytics misses AI crawlers and how to count them.
How Sitenumbers uses this list
Sitenumbers counts the bots in this directory on your server or CDN, checks each hit against the company's IP list or reverse DNS where one is published, and shows the result in an AI crawler table: verified hits, pages read, last seen and last response status. Bots without a published verification method are always shown as unverified.
| Bot | Verified hits | Pages | Last seen | Status |
|---|---|---|---|---|
| GPTBot | 1,284 | 312 | 2 h ago | 200 OK |
| ClaudeBot | 846 | 207 | 5 h ago | 200 OK |
| OAI-SearchBot | 391 | 118 | 1 day ago | 403 blocked |
| ChatGPT-User | 120 | 34 | 20 min ago | 200 OK |
FAQ
What is an AI crawler?
A bot run by an AI company that fetches web pages. Depending on the bot, it reads a page for a user's question, indexes it for AI search, or collects it as training data.
How do I block AI crawlers?
Add a User-agent: rule with the bot's robots.txt token and Disallow: / to your robots.txt. Block training crawlers and search crawlers separately, so you can opt out of training without leaving AI search. Each bot page has a ready-to-copy example.
How do I know if a crawler is really from OpenAI or Anthropic?
Check the request's IP against the company's published list. OpenAI, Anthropic and Perplexity publish IP lists. A user agent alone can be faked.
Which AI crawlers ignore robots.txt?
OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a user started the request. ByteDance has not said whether Bytespider follows robots.txt.