AI crawlers: see which AI bots read your site
AI crawlers rarely show up in web analytics, because analytics counts visits by running a script in the browser and filters out bot user agents. Sitenumbers counts AI crawlers on your server or CDN instead, checks that each one is genuine, and shows which pages they read.
Why your analytics does not show AI crawlers
Script-based analytics, including Google Analytics and the Sitenumbers script, records a visit when the page runs a small piece of JavaScript. Two things keep AI crawlers out of those numbers:
- Most AI crawlers do not document running JavaScript. Google and Apple say their crawlers render pages. OpenAI, Anthropic and Perplexity do not document JavaScript rendering for their bots. A bot that fetches only the HTML never runs the analytics script.
- Analytics scripts drop known bots on purpose. Hits from bot user agents are filtered out so they do not inflate visitor numbers. Sitenumbers does the same for its visitor stats.
The result: an AI assistant can read a page many times a day and your analytics shows nothing. The only complete record is on the server or CDN that answered the request.
Three kinds of AI crawlers
Not every AI bot visit means the same thing. Sitenumbers groups them by the job the company says they do.
| Type | What it means for you | Examples |
|---|---|---|
| Reading for a person, right now | Someone asked an AI assistant a question and it fetched your page to answer | ChatGPT-User, Claude-User, Perplexity-User |
| Indexing for AI search | Your page is added to the index that AI search answers draw from | OAI-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot |
| Collecting training data | Your page may be used to train a model. It does not mean you will appear in answers. | GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider |
The full list, with each bot's owner, robots.txt token and verification method, is in the AI crawler directory.
The groups matter when you decide what to block. OpenAI, for example, documents separate tokens for training (GPTBot) and search (OAI-SearchBot), so you can opt out of training and stay in ChatGPT search.
How Sitenumbers counts AI crawlers
Sitenumbers reads bot hits where they happen: on your CDN or web server. You pick one connection:
| Connection | Status | How it works |
|---|---|---|
| Cloudflare Worker | Ready | Passes every request through untouched. When the user agent matches a known bot, it sends the hit to Sitenumbers in the background, so it never delays the response. |
| nginx log shipper | Ready | A small script run by cron every minute. It reads new lines from your access log, keeps only bot hits and sends them in batches. |
| AI assistant setup prompt | Coming soon | |
| Cloudflare one-click setup | Coming soon | |
| WordPress plugin | Coming soon | |
| Next.js and Node package | Coming soon |
For each hit Sitenumbers keeps only the bot name, the page, the response status and the time. The IP address is used once to verify the bot and then discarded.
How Sitenumbers verifies AI crawlers
Anyone can send a user agent that says GPTBot or ClaudeBot. A hit is marked verified only when it passes the check the company publishes:
IP list: OpenAI, Anthropic and Perplexity publish the IP ranges their bots use. Sitenumbers fetches these lists daily and checks the hit's IP against them.
Reverse DNS: Google, Apple and Microsoft let you confirm a crawler by looking up the hostname of its IP and checking that it resolves back to the same IP.
Nothing published: some bots, such as Bytespider, have no official verification method. Their hits are always shown as unverified.
Spoofed bots are never counted as verified.
What you see in the dashboard
The AI crawler table shows, for each bot: verified hits, the pages it read, when it was last seen and the last response status. That answers practical questions:
- Which pages do AI assistants read for their users, and how often?
- Can OAI-SearchBot and PerplexityBot reach your pages, or do they get 403 or 5xx errors?
- Did your robots.txt or firewall change actually stop GPTBot or ClaudeBot?
- Are bots claiming to be AI crawlers really from OpenAI, Anthropic or Perplexity?
| Bot | Verified hits | Pages | Last seen | Status |
|---|---|---|---|---|
| GPTBot | 1,284 | 312 | 2 h ago | 200 OK |
| ClaudeBot | 846 | 207 | 5 h ago | 200 OK |
| OAI-SearchBot | 391 | 118 | 1 day ago | 403 blocked |
| ChatGPT-User | 120 | 34 | 20 min ago | 200 OK |
Next to the crawler table, Sitenumbers shows the people who click through from ChatGPT, Perplexity, Claude, Gemini and Copilot. Comparing the two gives you read vs clicked: how often an assistant read a page for someone, and how often that led to a visit.
Honest limits
Sitenumbers shows what reaches your server. It cannot know what a user asked an AI assistant, what the assistant answered, whether it cited your page, where you ranked in the answer, or whether your content ended up in a trained model. No analytics tool can.
Check your site's AI crawler access
FAQ
Why does Google Analytics not show AI crawlers?
Google Analytics counts visits by running JavaScript in the browser. Most AI crawlers do not document running JavaScript, and bot traffic is filtered out of analytics anyway. AI crawler hits are visible in server or CDN logs.
Do AI crawlers run JavaScript?
Google and Apple document that their crawlers render pages. OpenAI, Anthropic and Perplexity do not document JavaScript rendering for their bots. Make sure your important content is in the HTML the server sends. The free AI crawler checker shows what a page looks like without JavaScript.
Do I need Cloudflare to use Sitenumbers for AI crawlers?
No. If your site runs on nginx, the log shipper sends bot hits from your access log. Other connections are coming soon.
Does Sitenumbers store bot IP addresses?
No. The IP is used once to check the bot against the company's published list or reverse DNS, and then discarded.
Which AI crawlers should I block?
That depends on what you want. Blocking training crawlers such as GPTBot or ClaudeBot opts you out of model training. Blocking search crawlers such as OAI-SearchBot or PerplexityBot can keep you out of AI search answers. Each bot page shows the robots.txt rules.
See pricing for plans.