ClaudeBot

ClaudeBot is Anthropic's web crawler that collects public web content that could be used to train Anthropic's AI models. Disallowing it in robots.txt tells Anthropic to exclude your site's future content from its training datasets.

ClaudeBot at a glance

Anthropic runs three bots, each with its own robots.txt token: ClaudeBot (training data), Claude-User (fetches pages when a Claude user asks a question) and Claude-SearchBot (indexes content to improve Claude's search results). This page covers ClaudeBot. See the AI crawler directory for the others.

What a ClaudeBot visit means for your site

A ClaudeBot hit means Anthropic fetched the page as possible training material. It does not mean:

• a person asked Claude about your page right now (that is Claude-User),
• your page will appear in Claude's answers or search results (that is Claude-SearchBot),
• your content ended up in a trained model. No analytics tool can know that.

So ClaudeBot traffic is a signal about training collection, not about AI search visibility. Blocking ClaudeBot does not block Claude-User or Claude-SearchBot, because Anthropic lists them as separate bots with separate tokens.

How to allow or block ClaudeBot

Anthropic says that opting out requires robots.txt. Add the rule to the robots.txt file in your top-level directory, and repeat it on every subdomain you want to opt out.

Block ClaudeBot from the whole site:

User-agent: ClaudeBot
Disallow: /

Slow it down instead of blocking it. Anthropic supports the non-standard Crawl-delay extension:

User-agent: ClaudeBot
Crawl-delay: 1

Block training but keep Claude's search and user fetches:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Avoid blocking Anthropic by IP address only. Anthropic states that IP blocking may not work correctly or persistently as an opt-out, because it stops ClaudeBot from reading your robots.txt.

Check your site's AI crawler access

How to find and verify ClaudeBot in your logs

Any client can send a user agent that says ClaudeBot. To tell the real crawler from an impostor, check both the user agent and the source IP.

1. Find requests whose user agent contains ClaudeBot:

grep -i "claudebot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn

2. Check each IP against Anthropic's published list. Anthropic says that a crawler whose source IP is on this list is coming from Anthropic. The list covers Anthropic's bots as a group, not ClaudeBot alone.

import ipaddress, json, sys, urllib.request

url = "https://claude.com/crawling/bots.json"
prefixes = json.load(urllib.request.urlopen(url))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
ip = ipaddress.ip_address(sys.argv[1])
print("on Anthropic's list" if any(ip in n for n in nets) else "not on the list")

Run it as python3 verify.py 203.0.113.10. A ClaudeBot user agent from an IP that is not on the list is not Anthropic.

How Sitenumbers shows ClaudeBot

ClaudeBot does not show up in JavaScript analytics, so Sitenumbers counts it on your server or CDN with a Cloudflare Worker or an nginx log shipper. The AI crawler table shows ClaudeBot's hits, the pages it read, when it was last seen and the last response status, so you can spot pages that return errors or are blocked. Hits from an IP on Anthropic's list are marked verified. The IP is used once for that check and then discarded.

Read more about how Sitenumbers counts AI crawlers.