OAI-SearchBot

OAI-SearchBot is OpenAI's crawler for search: it is used to surface websites in ChatGPT's search features. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.

OAI-SearchBot at a glance

When it fetches robots.txt, OAI-SearchBot may add a robots.txt marker to the user agent: ...; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot.

Note that the user agent starts like a regular Chrome on macOS. Match on the OAI-SearchBot token, not the start of the string.

What an OAI-SearchBot visit means for your site

OAI-SearchBot hits mean your pages are being crawled for ChatGPT search. Of OpenAI's three agents, this is the one that matters if you want to be found in ChatGPT search. OpenAI recommends allowing it in robots.txt and allowing requests from its published IP ranges.

What OAI-SearchBot hits do not tell you: whether ChatGPT cited your page in a specific answer, or where your page ranked in it. No analytics tool can see that. What you can see is whether the bot reaches your pages, which pages it reads, and whether it gets errors.

OAI-SearchBot and GPTBot are independent. You can allow search and opt out of training. If both are allowed, OpenAI may use one crawl for both purposes.

How to allow or block OAI-SearchBot

Allow ChatGPT search and opt out of training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Block OAI-SearchBot (your site will not be shown in ChatGPT search answers):

User-agent: OAI-SearchBot
Disallow: /

If you use a WAF or bot protection, check that it does not block OpenAI's published IP ranges. A robots.txt that allows OAI-SearchBot does not help if the firewall returns 403.

Check your site's AI crawler access

How to find and verify OAI-SearchBot in your logs

1. Find requests whose user agent contains OAI-SearchBot:

grep -i "oai-searchbot" /var/log/nginx/access.log | awk '{print $1, $9}' | sort | uniq -c | sort -rn

The second column is the response status, so you can spot 403 and 5xx responses.

2. Check each IP against OpenAI's published list:

import ipaddress, json, sys, urllib.request

url = "https://openai.com/searchbot.json"
prefixes = json.load(urllib.request.urlopen(url))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
ip = ipaddress.ip_address(sys.argv[1])
print("on OpenAI's list" if any(ip in n for n in nets) else "not on the list")

How Sitenumbers shows OAI-SearchBot

OAI-SearchBot does not appear in JavaScript analytics, so Sitenumbers counts it on your server or CDN with a Cloudflare Worker or an nginx log shipper. The AI crawler table shows OAI-SearchBot's verified hits, the pages it read, when it was last seen and the last response status. That tells you which pages ChatGPT search can index and which ones it hits errors on. Hits are marked verified when the IP is in OpenAI's published ranges. The IP is used once for that check and then discarded.

Read more about how Sitenumbers counts AI crawlers.