AI Crawlers Are Reading Your Website: A Guide to robots.txt and AI Bots

A client's server logs showed a bot nobody had heard of reading 400 pages in ninety seconds. GPTBot, ClaudeBot, Google-Extended: how to see who's visiting your site, and how to control it.

One night, a client's server logs showed something unusual: a bot they'd never heard of reading four hundred pages in ninety seconds. It wasn't Googlebot. It was an AI crawler, and it was quietly deciding whether the client's website would be quoted in someone's AI answer the next morning.

Somewhere on your server logs tonight, there's probably a bot doing the same thing. Depending on your goals, that's either an opportunity or something you'd rather limit. Either way, it's worth understanding what these bots are and what you can control.

Why this suddenly feels like a decision you have to make

Ten years ago, a website owner never thought about crawlers at all. Googlebot came, Googlebot read, and everyone agreed that was good. AI crawlers broke that silent agreement, because they are not all doing the same thing: some read your site to quote it in live answers, others read it to train models that might replace the need to visit you at all.

The discomfort most owners feel is not technical, it is psychological: a loss of control. Something you never invited is consuming the thing you paid to build, and the rules of the old web (everyone crawls, everyone benefits) suddenly feel one-sided. That discomfort is legitimate, and the good news is that you actually do have choices. Most people just don't know they have them.

The bots on your server logs

Most AI companies operate crawlers with distinctive names. You'll see them in your access logs or analytics, usually as user agents that read many pages in quick succession:

• GPTBot. OpenAI's crawler, used for training and for answering questions about your site.
• OAI-SearchBot. Another OpenAI bot, used specifically to power ChatGPT's search and citations.
• ClaudeBot. Anthropic's crawler for Claude.
• Google-Extended. Google's way of letting sites opt out of being used for Gemini and Vertex AI training, separate from regular Google search.
• PerplexityBot. The crawler behind Perplexity's answers.
• CCBot. Common Crawl's bot, which feeds many AI training datasets.

If you use Cloudflare or most analytics tools, you can filter your logs by user agent and see which of these visit you and how often.

What robots.txt can and can't do

robots.txt is a small text file at the root of your site that tells crawlers which paths they may or may not visit. It works on the honour system: well-behaved crawlers read it and comply. It's not a security measure, since a determined crawler can ignore it, but in practice the major AI companies do respect it.

It can also be specific: you can block a particular bot while leaving Googlebot untouched, or allow AI crawlers but block them from specific paths like your admin area or private files.

Block or allow? The tradeoff

This is the decision every site owner is now facing, and there's no universal answer:

• Allow AI crawlers. Your content can be cited in AI answers and included in training data. For a public business site that wants to be found, this is usually the right default: an AI that recommends you is free marketing, and it is the modern version of being quoted in a trade publication.
• Block AI crawlers. You keep your content out of AI products. That might matter if your content is the core product you sell, if you're concerned about copyrighted or proprietary material, or if you're paying for traffic you'd rather not give away.

Many businesses take a middle path: block training crawlers (like GPTBot's training use) but allow search bots (like OAI-SearchBot and Google-Extended), because being cited in live answers drives real visits. Think of it as choosing who gets to quote you: you can be happy to be cited in a live answer that sends someone your way, while declining to be absorbed into a training set that never points back to you.

A sensible default robots.txt

If you want to allow search citation but block training, a file like this is a reasonable starting point:

User-agent: *\nAllow: /\nUser-agent: GPTBot\nDisallow: /\nUser-agent: CCBot\nDisallow: /

If you'd rather welcome everyone, the default User-agent: * / Allow: / is fine. Just remember that robots.txt applies to your whole site unless you're more specific, and that some AI products read your public pages regardless of the file, so keep anything sensitive out of the public site entirely.

How to see who's crawling you

1. Check your hosting or Cloudflare logs and filter by these user-agent names.
2. Look at your analytics for sessions that read many pages in seconds with no clicks.
3. Set up a simple check: if you update a page, see whether a known AI crawler requests it within a day or two.

Seeing the actual traffic is worth more than reading articles about it. The pattern on your site is what matters, and the numbers usually resolve the anxiety: either the bots are barely there, or they are clearly interested in you, and each answer points to a different sensible policy.

Your decision in three questions

1. Is being recommended by AI worth being read by AI? For most businesses, yes.
2. Is any of your public content something you'd rather not be absorbed? For most businesses, no; the sensitive stuff should not be public anyway.
3. Are the bots actually visiting you? Check the logs before deciding; a policy for a crawler that never comes is a policy without stakes.

The bottom line

AI crawlers are part of how the web works now, and you get to decide your stance: opt in to be quotable in AI answers, opt out to keep your content private, or pick a middle path. Whatever you choose, make the choice deliberately. The default of not knowing these bots exist is the only wrong answer.

If you'd like us to look at your logs, tell you which bots are visiting, and set up a robots.txt that matches your actual goals, send us a message. It takes us an afternoon, and you'll never wonder again.