named
Get a free AI audit

Blog

Which AI crawlers should you allow to show up in AI answers?

Author: Saad Khan, founder of Named (named.khanex.ai)

Published:

Last updated:

Drafted with AI help from the sources below and Named's live site. Saad checks every claim before it goes live. Named sells AI visibility work to small businesses, so read this with that in mind.

If you want AI answers to cite your site, allow the search crawlers: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for AI Overviews and AI Mode, plus Bingbot and Applebot. Training is a separate choice with one new shared robots.txt line. Opting out keeps you in search answers, but on Google it can cost Gemini app citations.

Updated Nov 3, 2026: added the shared AI-training opt-out (__usage_ai-training), which Google, Microsoft, OpenAI and Anthropic each announced support for in late October 2026, and what it changes for AI visibility.

Which AI crawlers decide whether you show up in AI answers?

The search crawlers do. Each AI company documents one crawler that builds the index its answers cite from, and if that crawler can't read your page, the engine has nothing of yours to quote.

Engine Search crawler (allow it) Training crawler (your choice) Fetches when a user asks
ChatGPT OAI-SearchBot GPTBot ChatGPT-User
Claude Claude-SearchBot ClaudeBot Claude-User
Perplexity PerplexityBot none listed (PerplexityBot is not used for training) Perplexity-User
Google AI Overviews and AI Mode Googlebot Google-Extended (a control token, see below) n/a
Apple (Siri, Spotlight, Safari) Applebot Applebot-Extended (a control token) n/a

OpenAI is the clearest about the stakes. Its bot documentation says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." Anthropic says blocking Claude-SearchBot "may reduce your site's visibility for user-directed web search." Perplexity recommends allowing PerplexityBot "to ensure your site appears in search results."

For Bing and Microsoft's AI answers, Bingbot is the crawler to allow. If you want to see what those answers cite, Bing now reports it in Webmaster Tools; we covered that in what Bing's AI Performance report shows.

Can you block AI training and still get cited?

Yes, for the engines that split the two jobs. OpenAI's docs say each setting "is independent of the others," and give the exact example: allow OAI-SearchBot to appear in search results while disallowing GPTBot. Anthropic splits ClaudeBot (training) from Claude-SearchBot (search) the same way.

Apple works differently. Applebot does both jobs, and Applebot-Extended doesn't crawl anything. It only tells Apple whether it may train on what Applebot already collected. Apple says pages that disallow Applebot-Extended "can still be included in search results."

So "should we block AI?" is really two questions. Do you want your content in training data? That's a rights and business call. Do you want to be named when a buyer asks an AI engine for a recommendation? Then the search crawlers have to get in.

Until October 30, 2026, saying no to training meant one rule per company. The next section covers the shared line that replaces that, and the catches that come with it.

What is the new shared AI-training opt-out?

It's a robots.txt group named __usage_ai-training that tells every participating AI company how you want your content used for training, in one place. Google, Microsoft, OpenAI and Anthropic each announced support for it, with Google and Microsoft posting on October 30, 2026.

To opt your whole site out of training, add this:

User-Agent: __usage_ai-training
Disallow: /

The token comes from an IETF Internet-Draft that Kevin Kelley of Anthropic submitted the same day. Three details in the draft matter for anyone who cares about AI visibility:

  • It's a preference about use. The draft says the token "is not a directive for crawling behavior." Your rules for GPTBot, ClaudeBot or any other named crawler keep their current meaning.
  • User-agent: * doesn't count. The draft says wildcard groups "MUST NOT be interpreted as matching the policy token." If your file has no __usage_ai-training group, you've expressed no preference at all, even if you block everything under *.
  • The spelling has to match. Consumers match the token "exactly, apart from case." It starts with two underscores. Some copy-paste versions online drop them, and a group named usage_ai-training is a different token.

Nobody honors it everywhere yet. Here's where each company stood when we checked on Nov 3, 2026:

Company When it starts What keeps working alongside it
Anthropic Already started implementing; Anthropic says adding support across its services will take time ClaudeBot rules
OpenAI "In the coming months" GPTBot and its other crawler rules
Microsoft February 1, 2027 Page-level training and grounding settings; NoArchive and NoCache stop counting after March 1, 2027
Google Early 2027 Google-Extended

There are two more caveats. The draft is an individual submission, and the IETF's own page says it is "not endorsed by the IETF and has no formal standing." And on Nov 3, 2026, neither OpenAI's crawler page nor Anthropic's crawler help article mentioned the token yet. Keep your per-company rules in place.

Will the shared opt-out hurt your AI visibility?

Not in ChatGPT, Claude or Perplexity search answers, because those run on crawlers the token doesn't touch. The costs show up on Google and Microsoft, where training and grounding sit closer together.

Google ties it to Gemini grounding. Google says that when a site opts out with the token and doesn't name Google-Extended, it will treat that "as if the publisher had disallowed Google-Extended." Google-Extended controls Gemini training and grounding in the Gemini app, where Gemini pulls in live search results to answer. Google's post spells out the trade: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." So a site-wide training opt-out on Google also opts you out of being a grounded source in the Gemini app. AI Overviews and AI Mode aren't affected. They run on Googlebot.

Microsoft splits training from grounding. Starting February 1, 2027, Bing reads the shared token for training. Grounding gets its own page-level setting, set with a meta tag or header, not robots.txt:

<meta name="Bingbot" content="ai-grounding:disallow">

The options are Allow (the default), DisallowExtended, which keeps grounding inside Microsoft's own products such as Copilot, and Disallow. Microsoft says choosing Disallow or DisallowExtended "will significantly reduce your content's visibility in AI-generated experiences," though it doesn't affect whether you can appear in Bing search results. So on Bing you can say no to training and still be a source for Copilot answers.

My take for a small business that wants to get recommended: leave training allowed. Our own robots.txt does. I don't know of any published study showing that being in training data gets you named more often, so this is a judgment call. What's documented is the downside. On Google, the shared opt-out also costs you Gemini app grounding.

What does Google-Extended actually control?

Less than the name suggests for Google Search, and more than most crawler lists say for Gemini. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Blocking it won't remove you from AI Overviews or AI Mode. Those run on Googlebot, and Google says there are "no additional technical requirements" beyond being indexed and eligible for a snippet.

Plenty of crawler lists stop there and call Google-Extended a training opt-out. Google's own page says more: the token also controls whether your content is used "for grounding" in Gemini Apps and in Grounding with Google Search on Vertex AI. Grounding is the step where the model pulls in fresh search results while it writes an answer. So if you care about being cited inside the Gemini app, read that line before you block Google-Extended.

From early 2027 there's a second way to switch it off. Google says it will read a __usage_ai-training disallow as a Google-Extended disallow unless your file names Google-Extended on its own.

Why can a crawler still be blocked when robots.txt allows it?

Because robots.txt is only a request, and your CDN or firewall can turn the bot away before it ever reads the file. Google lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" as a basic for AI features.

The most common case right now is Cloudflare. In July 2025 it started asking new domains whether to allow AI crawlers, with blocking as the default. On September 15, 2026 it changed its controls again: "Block" and "Block on pages with ads" now also apply to mixed-use crawlers, including Applebot, Bingbot and Googlebot, so those settings can affect regular search too. Cloudflare's new "Disallow AI Training" setting is the one that refuses training while keeping you in search. If your site sits behind Cloudflare and someone picked "Block" months ago, check it again.

Perplexity adds one more trap in its own docs: Cloudflare's Bot Fight Mode "can't be skipped with WAF custom rules," so it can still challenge PerplexityBot even when you wrote a rule to let it through.

What about Grok?

There's no robots.txt line that targets Grok. xAI doesn't publish a crawler name, user agent or IP list for Grok. We couldn't find one on Oct 29, 2026, and DeployRadar's crawler directory, last verified Sept 2, 2026, says the same and reports that Grok fetches pages with ordinary browser user agents. Tokens like "GrokBot" that show up in copy-paste lists aren't documented by xAI. xAI also wasn't one of the four companies that backed the shared training opt-out on Oct 30. The practical move is to keep your site open to normal visitors and not block datacenter traffic you can't identify.

What should a robots.txt for AI visibility look like?

Allow the search crawlers by name, then make a deliberate choice about training. Here's a starting point for a site that wants to be cited and is fine with training. If you're not, swap Allow for Disallow on the training lines and in the shared group at the bottom.

# Search crawlers: these decide whether AI answers can cite you
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Applebot
Allow: /

# Fetches a person triggers inside an assistant
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

# Training and grounding controls: your call
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

# Shared training preference (honored as each company rolls it out)
User-agent: __usage_ai-training
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

If you opt out of training but still want Gemini app citations, keep an explicit Google-Extended group set to Allow. Google says it reads the shared token as a Google-Extended block only when Google-Extended isn't named, and that sites which specifically allow Google-Extended stay eligible for Gemini app grounding. That also allows Gemini training, because Google-Extended covers both.

One rule to remember: a crawler follows the most specific group that names it and ignores User-agent: *. If you give OAI-SearchBot its own group, your general disallow lines won't apply to it, so repeat any paths you still want hidden.

For what it's worth, our own robots.txt allows every agent and names nine of these bots explicitly, including the training ones. It doesn't have a __usage_ai-training group yet. Under the draft, no group means no preference, and Microsoft treats no preference as Allow.

How do you check that it worked?

Give it a day, then look at the evidence instead of the file. OpenAI says robots.txt changes take about 24 hours to reach its search systems, and Perplexity says up to 24 hours.

  1. Open yoursite.com/robots.txt and confirm no group disallows the search crawlers above. Our free site check tests whether robots.txt blocks GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot.
  2. If you added __usage_ai-training, check that it starts with two underscores and that your Google-Extended group says what you mean.
  3. If you use a CDN or firewall, check its bot settings as well as robots.txt.
  4. Check your server logs for OAI-SearchBot, Claude-SearchBot and PerplexityBot requests. OpenAI and Perplexity publish IP lists so you can tell real visits from fakes.
  5. Ask the engines the questions your buyers ask and see whether your pages get cited. Answers vary from run to run, so ask more than once.

Crawler access only makes you eligible. Whether an engine names you still depends on whether your pages answer the buyer's question and whether other sites mention you. Our guide to getting recommended by AI covers the next steps.

FAQ

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, and OpenAI treats the two settings separately.

Does the __usage_ai-training line block crawlers?

No. The draft calls it a preference about how content is used, not a crawling rule. Crawlers still follow the groups that name them. The draft does let a company skip fetching pages it can't use.

If I add the shared opt-out, can I delete my GPTBot and ClaudeBot rules?

Not yet. OpenAI says the shared signal "will not replace" its existing robots.txt controls, Anthropic says it complements them, and Google and Microsoft don't start until 2027.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended doesn't affect inclusion in Google Search, and AI Overviews and AI Mode use Googlebot. It does affect grounding in the Gemini app, and from early 2027 a __usage_ai-training disallow will count as a Google-Extended block unless you name Google-Extended yourself.

Do I need llms.txt or special schema to get into Google's AI features?

Google says no: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."

Can I block the bots that fetch pages when a user asks?

You can try, but OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." Anthropic says blocking Claude-User may reduce your visibility in user-directed search.

How long until a robots.txt change shows up?

OpenAI says about 24 hours for ChatGPT search, and Perplexity says up to 24 hours. Google recrawls on its own schedule, and Google says that can take several days to several months.

Sources

Named works to get small businesses recommended by every AI engine you care about, starting with ChatGPT, Claude, Gemini, Perplexity, Grok and Muse. We keep one fixed prompt set and repeat it to check changes. The Namies draft fixes, and a person checks every change. See how Named works, or if you want to see where you stand, start with a free AI audit.