AI crawler checker
Enter a URL to see your page the way most AI crawlers see it: fetched without JavaScript. The checker shows how much text is there, the title and headings, and what your robots.txt allows each AI bot to do.
The checker is coming soon.
What the checker shows
| Result | Why it matters |
|---|---|
| Text length without JavaScript | Major AI crawlers don't render JavaScript. If your content is loaded by scripts, they may get an almost empty page. |
| Title | The first thing an AI system reads about the page. |
| Headings | Show whether the page structure survives without JavaScript. |
| robots.txt rules for AI bots | Whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Meta-ExternalAgent, Bytespider, Google-Extended, Applebot and others are allowed or disallowed for this URL. |
How to read the results
Little or no text
If the text length is near zero but the page looks full in a browser, the content is rendered with JavaScript. AI crawlers that don't run JavaScript can't read it. Server-side rendering or static generation fixes this.
Missing title or headings
A page without a title or H1 in the raw HTML gives AI systems little to go on. Check that your framework outputs them on the server.
A bot is disallowed
robots.txt is a request, not a lock. Training and search crawlers such as GPTBot, ClaudeBot and PerplexityBot say they follow it. Fetchers that act for a user, such as ChatGPT-User and Perplexity-User, may not apply robots.txt rules, according to their owners' documentation. Blocking a search crawler such as OAI-SearchBot can keep your pages out of that assistant's search answers.
Each bot's purpose and robots.txt token: AI bot directory.
What the checker can't tell you
- Whether bots actually visit. The checker reads your rules. It doesn't see your traffic.
- Firewall and CDN rules. Your server may answer AI bots differently from the checker, for example with a 403 from a bot-blocking rule.
- Whether you appear in AI answers. No tool can see what an assistant answered.
To see which AI bots really fetch your pages, how often, and whether they get errors, you need server-side counting. That is what Sitenumbers does: AI crawler hits per bot and page, verified against each owner's published IP ranges, next to the visits AI assistants send you. More: AI crawlers.
| Bot | Verified hits | Pages | Last seen | Status |
|---|---|---|---|---|
| GPTBot | 1,284 | 312 | 2 h ago | 200 OK |
| ClaudeBot | 846 | 207 | 5 h ago | 200 OK |
| OAI-SearchBot | 391 | 118 | 1 day ago | 403 blocked |
| ChatGPT-User | 120 | 34 | 20 min ago | 200 OK |
How the checker works
The checker fetches the URL once from our server, without running JavaScript, and reads /robots.txt from the same host. It follows at most 2 redirects, stops after 10 seconds or 2 MB, and does not fetch private or internal network addresses. Checks are rate-limited per IP address.
FAQ
Can AI crawlers read JavaScript-rendered pages?
Major AI crawlers, including GPTBot and ClaudeBot, don't render JavaScript according to Vercel's 2024 analysis. Content that only appears after scripts run is likely invisible to them.
How do I block AI crawlers?
Add a Disallow rule for the bot's user agent token in robots.txt. Each bot page in the AI bot directory has copy-paste examples. Remember that user-initiated fetchers may not follow robots.txt.
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot crawls for model training. ChatGPT search uses OAI-SearchBot, which has its own robots.txt token.
Is the checker free?
Yes. It is free and needs no account.
Does the checker store the URLs I check?
Plans for server-side AI crawler tracking: pricing.