Blog · Website fixes

Should you block AI crawlers? Search bots vs training bots

OAI-SearchBot, PerplexityBot and Claude-SearchBot decide whether you can appear in AI answers. GPTBot, ClaudeBot, CCBot and Google-Extended are about training. Here is what each one does, from the vendors' own documentation.

· 6 min read

Over the last two years a lot of websites started blocking "AI bots" in their robots.txt file, often because a hosting company, a security plugin or a well-meaning web designer switched it on. For a publisher worried about its articles being used to train models, that can make sense. For a local business that wants to be recommended by AI assistants, a blanket block can quietly remove you from the answers you want to be in.

The key is that AI companies run different bots for different jobs. Some fetch pages so the assistant can show them in search answers. Others collect pages to train future models. You can treat them separately, and most local businesses should.

A 30-second primer on robots.txt

robots.txt is a plain text file at the root of your website (for example yourbusiness.com/robots.txt). It lists which crawlers may visit which parts of your site. A rule looks like this:

User-agent: GPTBot
Disallow: /

That says: the crawler that identifies itself as GPTBot should not fetch any page. Well-behaved crawlers, including all of the ones below that say they honour robots.txt, follow these rules. You can open your own file in a browser right now to see what it says.

The search bots: block these and you can't be cited

These crawlers fetch pages so that an AI assistant can find them and link to them when it answers questions.

  • OAI-SearchBot (OpenAI). OpenAI's crawler documentation (external site) says it is used to surface websites in ChatGPT's search features, and that sites which opt out will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says changes to robots.txt take about 24 hours to take effect.
  • PerplexityBot (Perplexity). Perplexity's bot documentation (external site) says it is designed to surface and link websites in Perplexity's search results, and that it is not used to crawl content for AI foundation models.
  • Claude-SearchBot (Anthropic). Anthropic's crawler page (external site) says it crawls the web to improve the relevance and accuracy of search results shown to Claude's users, and that blocking it prevents your content from being indexed for search.
  • Googlebot (Google). Google's AI Overviews and AI Mode are part of Google Search, so they rely on Google's normal search crawler. Google's AI features guide (external site) says a page has to be indexed and eligible to show with a snippet to appear as a supporting link. Blocking Googlebot removes you from Google Search altogether, which no local business wants.

For a local business, our recommendation is to allow all four. They are how you get into the answers.

The user-triggered fetchers

Some assistants also visit a page at the moment a user asks about it, for example when someone pastes your link into a chat. OpenAI calls this ChatGPT-User, Perplexity calls it Perplexity-User, and Anthropic calls it Claude-User. OpenAI and Perplexity both say that because these visits are started by a person, robots.txt rules may not apply to them. OpenAI also says ChatGPT-User is not used to decide whether content appears in search. There is little reason for a local business to block them.

The training bots: a separate decision

These crawlers collect pages that may be used to train future AI models. Blocking them does not, according to the vendors, affect whether you appear in search answers.

  • GPTBot (OpenAI). OpenAI says disallowing GPTBot indicates your content should not be used to train its foundation models, and that each of its bot settings is independent of the others. Blocking GPTBot while allowing OAI-SearchBot is an explicitly supported combination.
  • ClaudeBot (Anthropic). Anthropic says blocking ClaudeBot signals that your site's future content should be excluded from its training data.
  • CCBot (Common Crawl). Common Crawl (external site) is a non-profit that publishes an open archive of the web, which many AI developers have used as training data. Its page explains how to block CCBot in robots.txt.
  • Google-Extended (Google). This one is unusual. Google's crawler documentation (external site) describes it as a product token, not a separate crawler: Google keeps crawling with its normal user agents, and the Google-Extended rule controls whether that content may be used to train future Gemini models and for grounding in some Gemini products. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal there.

Whether to block training is a values decision, not a visibility one. A local business's homepage is mostly facts it wants repeated (name, services, area, hours), so many owners won't mind. Others prefer not to contribute to model training on principle. Either choice is reasonable, as long as it's a choice and not an accident.

One caution on Google-Extended: Google's wording mentions grounding in "certain Gemini and Vertex AI products" as well as training. It's not fully clear which consumer features that covers, so if appearing in Gemini matters to you, it is the one training-side rule we'd be slowest to add.

A sensible robots.txt for a local business

If you want to be found by AI search but keep your site out of model training, this is a reasonable starting point. Keep any existing rules your site already needs (for example, blocking an admin area) and add these:

# AI search: allow, so assistants can find and cite us
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# AI training: opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

If you'd rather allow everything, the simplest file is no AI-specific rules at all. Don't add a Google-Extended rule unless you've decided you want it.

How to check what your site is doing now

  1. Open yourdomain/robots.txt in a browser. Look for any of the names above, and for a catch-all block such as User-agent: * followed by Disallow: /, which blocks every crawler that obeys the file.
  2. Check your security or CDN settings too. Some hosting and firewall services offer a one-click "block AI bots" setting that works at the server, outside robots.txt. If your file looks fine but you never appear as a cited source, this is worth checking.
  3. If you're on a website builder, its help pages will explain where robots.txt is managed; some builders let you edit it and some generate it for you.

A note on llms.txt: you may see advice to add an llms.txt file, a community proposal (external site) for a plain-text summary of a site aimed at language models. It does no harm and takes ten minutes, but none of the vendor documents linked in this article say their search products read it, so treat it as optional.

How Saymeter checks this

Saymeter reads your robots.txt when it reviews your website. If it finds a rule that blocks an AI search crawler, it raises a fix with the corrected lines to paste, then re-reads the file to confirm the change is live. Training-bot rules are left as your decision. The full list of checks is on our methodology page, and a free scan shows how often AI answers name you today.