Skip to content
Vekord
EN
Check your site

Blog ·

Is robots.txt hiding your client from AI search?

Before you rewrite content for ChatGPT or Perplexity, check whether their bots may read the site at all. It takes two minutes.

A client asks why ChatGPT recommends their competitor and never them. The usual answer is content: more pages, clearer answers, better sources. There is a cheaper question to ask first — can the assistant's search bot read the site at all?

Often it can't, and nobody decided that on purpose. A line copied from a template years ago, a security plugin or a CDN setting can lock out exactly the bots that AI search depends on.

Two kinds of AI bots

AI companies run separate crawlers for separate jobs. One kind collects text to train models. The other fetches pages so the assistant can find and cite them when someone asks a question. Blocking the first keeps the site out of training data. Blocking the second keeps it out of the answers.

Company Search and answers Model training
OpenAI OAI-SearchBot GPTBot
Anthropic Claude-SearchBot ClaudeBot
Perplexity PerplexityBot —
Google Googlebot Google-Extended

Google-Extended is not a separate crawler. It is a name in robots.txt that tells Google whether pages fetched by Googlebot may be used for Gemini; blocking it does not remove the site from Google Search. Each company also has an agent for pages a user asks about directly, for example by pasting a link (ChatGPT-User, Claude-User, Perplexity-User), and those follow their own rules.

Reading robots.txt in two minutes

Open /robots.txt on the client's domain and look for three things:

  1. A blanket block. User-agent: * followed by Disallow: / closes the site to every bot that has no group of its own, AI search bots included. It is common on sites that started life as a staging copy.
  2. Named search bots. A group for OAI-SearchBot, Claude-SearchBot or PerplexityBot with Disallow: / is a decision to stay out of AI answers. If the client didn't make it, find out who did.
  3. Lists copied from "block AI" guides. Many of them put search bots and training bots in one group. The intent was to protect content; the effect is invisibility.

A bot follows the group that names it and ignores User-agent: *, so a block in a named group wins even when the general group allows everything.

If the client wants to stay out of training but appear in answers, write it down explicitly:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

robots.txt is not the only gate

A permissive robots.txt says nothing about the firewall. CDNs and security plugins can block AI crawlers on their own — Cloudflare, for instance, has a setting for exactly that — and the bot then gets an error page whatever robots.txt says. Check the CDN's bot settings and the security plugin's block list too.

A quick test from the terminal shows how the server answers a given bot name:

curl -sI -A "OAI-SearchBot" https://example.com/

A 200 is a good sign. A 403 or a challenge page means something blocks that name. Firewalls that verify bots by their IP address may treat this request differently from the real bot, so a clean result here is a hint, not proof.

Where Vekord fits in

The AI visibility module in Vekord puts a fixed set of questions about the client's industry to a language model and to Perplexity, shows the client's share against competitors and lists the AI bots blocked in robots.txt. A blocked OAI-SearchBot appears right next to the number it explains.

All posts