Bytespider

Bytespider is a web crawler that identifies itself with the Bytespider token and is attributed to ByteDance, the company behind TikTok. ByteDance does not publish documentation for it, so its purpose, IP addresses and robots.txt behavior are not officially confirmed.

Bytespider at a glance

What a Bytespider visit means for your site

Because ByteDance has not said what Bytespider is for, a Bytespider hit tells you only that a client using that name fetched the page. It could be the real crawler or something else using its name: without a published IP list there is no reliable way to tell them apart.

In the AI crawler directory, Bytespider is grouped with bots that collect training data, because that is how it is commonly described. ByteDance has not confirmed it.

How to block Bytespider

You can add a robots.txt rule, but ByteDance has not said whether Bytespider follows it:

User-agent: Bytespider
Disallow: /

If you want a hard block, refuse the requests on your server or CDN by user agent. In nginx:

if ($http_user_agent ~* "bytespider") {
    return 403;
}

This blocks anything that uses the name. It cannot stop a crawler that uses a different user agent, and with no published IP list there is no way to allowlist the genuine bot.

Check your site's AI crawler access

How to find Bytespider in your logs

Find requests whose user agent contains Bytespider, with the response status:

grep -i "bytespider" /var/log/nginx/access.log | awk '{print $1, $9}' | sort | uniq -c | sort -rn

If you added a block, the status column should show 403. If you only added robots.txt, a steady number of 200 responses after the change shows the rule is not being followed by whatever is using the name.

How Sitenumbers shows Bytespider

Bytespider does not appear in JavaScript analytics, so Sitenumbers counts it on your server or CDN with a Cloudflare Worker or an nginx log shipper. The AI crawler table shows Bytespider's hits, the pages it read, when it was last seen and the last response status, so you can check whether a block works. Because ByteDance publishes no IP list or reverse DNS hostname, Sitenumbers cannot verify Bytespider: its hits are always shown as unverified.

Read more about how Sitenumbers counts AI crawlers.