GPTBot

GPTBot is OpenAI's web crawler for content that may be used to train OpenAI's generative AI foundation models. Disallowing GPTBot in robots.txt tells OpenAI that your content should not be used for that training.

GPTBot at a glance

When GPTBot fetches robots.txt itself, OpenAI may add a robots.txt marker to the user agent, so you can tell those requests apart in logs without paths.

What a GPTBot visit means for your site

A GPTBot hit means OpenAI fetched the page as possible training material. It does not mean your page appears in ChatGPT search or answers. ChatGPT search uses OAI-SearchBot, and pages read for a user's question use ChatGPT-User.

OpenAI says each setting is independent. You can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. If you allow both, OpenAI may use one crawl for both purposes to avoid crawling twice, so you may see fewer GPTBot hits than you expect.

No analytics tool can tell you whether your content ended up in a trained model. GPTBot hits only tell you the page was fetched.

How to allow or block GPTBot

Block GPTBot from the whole site:

User-agent: GPTBot
Disallow: /

Block training but stay in ChatGPT search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Block only part of the site:

User-agent: GPTBot
Disallow: /members/

Check your site's AI crawler access

How to find and verify GPTBot in your logs

1. Find requests whose user agent contains GPTBot:

grep -i "gptbot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn

2. Check each IP against OpenAI's published list. A GPTBot user agent from an IP outside the list is not OpenAI.

import ipaddress, json, sys, urllib.request

url = "https://openai.com/gptbot.json"
prefixes = json.load(urllib.request.urlopen(url))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
ip = ipaddress.ip_address(sys.argv[1])
print("on OpenAI's list" if any(ip in n for n in nets) else "not on the list")

How Sitenumbers shows GPTBot

GPTBot does not appear in JavaScript analytics, so Sitenumbers counts it on your server or CDN with a Cloudflare Worker or an nginx log shipper. The AI crawler table shows GPTBot's verified hits, the pages it read, when it was last seen and the last response status. If you block GPTBot, you can see that it now gets 403s or stops coming. Hits are marked verified when the IP is in OpenAI's published ranges. The IP is used once for that check and then discarded.

Read more about how Sitenumbers counts AI crawlers.