Robots.txt for AI Crawlers: What to Allow, What to Block

Q: Do I need to list every AI bot in robots.txt to allow it?

No. Any bot without its own group follows your User-agent: rules. Only name a bot when you want to treat it differently, and then repeat every rule it should follow in its group.

A good robots.txt for AI crawlers treats two kinds of bots differently. Retrieval bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch your pages so AI engines can cite and link you. Unless you have a very unusual reason, always allow them. Training bots like GPTBot, ClaudeBot and Google-Extended collect content for future models. Whether to allow those is a business decision, and you can make it separately.

Most D2C brands get this wrong in one of two ways. Some copied a "block all AI" snippet in 2023 and are now invisible in ChatGPT search. Others think they allow everything, but one small grouping mistake quietly cancels their own rules. This post covers both, with three robots.txt files you can copy.

What's the difference between retrieval bots and training bots?

A retrieval bot fetches your page so an AI product can answer a question right now. If it's blocked, you can't be cited or linked in that engine's answers. That's lost visibility today.

A training bot collects pages that may be used to train future models. Blocking it keeps your new content out of future training. It doesn't remove you from AI search results, because search is handled by a different bot.

There's a third kind worth knowing: user-triggered fetchers. These visit a page when a person asks the AI to open it, or when the AI needs it mid-answer. They behave more like a browser than a crawler, and some of them don't follow robots.txt at all.

The main AI companies now publish separate user agents for each job:

Company	Retrieval / search	Training	User-triggered	Follows robots.txt?
OpenAI	`OAI-SearchBot`	`GPTBot`	`ChatGPT-User`	Yes for search and training. "Robots.txt rules may not apply" to ChatGPT-User.
Anthropic	`Claude-SearchBot`	`ClaudeBot`	`Claude-User`	Yes, all three
Perplexity	`PerplexityBot`	None (says PerplexityBot isn't used for training)	`Perplexity-User`	PerplexityBot yes. Perplexity-User "generally ignores robots.txt rules".
Google	`Googlebot` (Search, including AI features in Search)	`Google-Extended` (a control token, not a separate crawler)	—	Yes
Apple	`Applebot` (Siri, Spotlight, Safari)	`Applebot-Extended` (a control token, not a separate crawler)	—	Yes

Sources: OpenAI's crawler documentation, Anthropic's crawler help article, Perplexity's crawler documentation, Google's crawler list and Apple's Applebot support page. All checked on 2 October 2026.

What do the official docs actually say?

The details matter, so here's each company in its own words.

OpenAI. OAI-SearchBot is for ChatGPT search, and OpenAI is direct about the cost of blocking it: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." GPTBot is the training crawler: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." OpenAI also says it "can take ~24 hours" after you update robots.txt for its systems to adjust.

Anthropic. ClaudeBot collects content that could contribute to training. Claude-SearchBot "navigates the web to improve search result quality for users." Claude-User fetches pages when people ask Claude questions. Anthropic says blocking Claude-SearchBot or Claude-User reduces your visibility in Claude's search and user-directed answers. It also supports the non-standard Crawl-delay rule. And it warns that blocking its IP addresses instead of using robots.txt may not give you a reliable opt-out, because it stops Anthropic reading your robots.txt.

Perplexity. PerplexityBot is "designed to surface and link websites in search results on Perplexity" and is "not used to crawl content for AI foundation models." Perplexity-User handles user actions and generally ignores robots.txt. Perplexity says changes can take up to 24 hours to show.

Google. Google-Extended is "a standalone product token" that controls whether Google may use your content for training future Gemini models and for grounding in Gemini Apps and Vertex AI. Google is clear that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode are part of Google Search, which is crawled by Googlebot. So Google-Extended isn't the switch for those.

One nuance people miss: because Google-Extended also covers grounding in Gemini Apps, blocking it is not purely a training decision. It can affect how your content is used when someone asks the Gemini app a question.

Apple. Applebot powers search features in Siri, Spotlight and Safari. Applebot-Extended "does not crawl webpages". It only decides whether content Applebot already crawled can be used to train Apple's models. Disallowing it doesn't remove you from Apple's search results.

What should a D2C brand allow?

Here's our honest view for a typical D2C brand selling skincare, snacks, supplements or clothes.

Always allow: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Applebot. These decide whether you show up when a buyer asks "best sunscreen for oily skin under ₹500". Blocking them costs you visibility and gets you nothing in return.

Probably allow: GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. If you want AI models to know your brand, your ingredients and your price range, being in the training data can only help. Your product pages aren't secret. They're marketing.

Consider blocking training bots if: your content is the product. A recipe brand with a paid meal-plan library, a fitness brand selling course content, or a brand that publishes original research might reasonably say no. Even then, keep the retrieval bots allowed so you can still be cited.

Don't bother blocking: ChatGPT-User and Perplexity-User. Their own docs say robots.txt may not apply to them, and they only visit when a real person asks about you. That's usually a buyer.

Robots.txt for AI crawlers: three copy-paste examples

Replace yourbrand.com and the private paths with your own. Cart, checkout and account paths differ by platform, so check yours.

Example 1: Allow everything (the right default for most brands)

# Every crawler, including AI search and training bots
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://www.yourbrand.com/sitemap.xml

That's it. You don't need to list each AI bot by name to allow it. Any bot without its own group falls back to User-agent: *. The simplest file is often the safest.

Example 2: Allow AI search, block AI training

# Training bots: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including OAI-SearchBot, Claude-SearchBot,
# Claude-User, PerplexityBot, Googlebot, Bingbot and Applebot
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://www.yourbrand.com/sitemap.xml

Stacking several User-agent lines above one set of rules is valid and keeps the file short. The search bots aren't named, so they follow the * group and stay allowed.

Example 3: Block training on one section only

Say a protein-bar brand publishes a free recipe hub at /recipes/ and is happy to be cited for it, but doesn't want it used as training data. Product pages stay fully open.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /recipes/
Disallow: /cart
Disallow: /checkout
Disallow: /account

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://www.yourbrand.com/sitemap.xml

Notice the cart and checkout rules appear twice. That's not a typo. It's the mistake covered next.

Why do some robots.txt files block the wrong bots?

This is the most common robots.txt error we see on D2C sites, and it's subtle. A crawler obeys only one group: the most specific one that names it. Google's robots.txt documentation puts it plainly: "Only one group is valid for a particular crawler." The rules in User-agent: * don't get added on top.

So this file looks harmless but isn't:

User-agent: *
Disallow: /cart
Disallow: /checkout

User-agent: GPTBot
Disallow: /blog/

GPTBot reads its own group, sees only Disallow: /blog/, and is now free to crawl /cart and /checkout. The reverse is worse. If you add a group like User-agent: OAI-SearchBot with Allow: / "to be safe", that bot ignores every disallow in your * group.

The fix: every named group must repeat all the rules that bot should follow. Or don't name a bot unless you need to treat it differently, as in Example 1.

A few other traps:

Paths are case-sensitive. Disallow: /Checkout doesn't block /checkout.
One file per host. yourbrand.com/robots.txt doesn't cover shop.yourbrand.com. Each subdomain needs its own.
robots.txt isn't a security tool. It's a polite request. Well-behaved bots follow it. Scrapers that ignore it won't be stopped by it.
Blocking crawling isn't the same as removing a page from search. Google says a blocked URL can still appear in results without a snippet. Use noindex if you want a page gone.

What else can block AI crawlers besides robots.txt?

A perfect robots.txt can still be overruled one layer up. Check these too:

Cloudflare and other CDNs. Bot-protection and "block AI bots" settings can return errors to OAI-SearchBot or PerplexityBot even when robots.txt says allow. Look at your bot or firewall settings and decide what each one does.
Shopify apps and WordPress security plugins. Some anti-scraping apps block unfamiliar user agents by default.
Rate limits. An aggressive limit can block crawlers that request many pages quickly.
Your platform's default file. On Shopify, robots.txt is edited through the robots.txt.liquid theme template. Changes there sit on top of Shopify's defaults, so read the live file at /robots.txt after editing, not just the template.

If you need to verify that a request really came from OpenAI or Perplexity, both publish IP range lists linked from their crawler docs. Anyone can fake a user agent string. Faking an IP from those ranges is much harder.

How to check yours

You can do this in about ten minutes.

Open https://yourbrand.com/robots.txt in a browser. If it's missing or empty, every bot is allowed by default.
Search the file for User-agent: * followed by Disallow: /. That one line blocks every crawler without its own group, including all the AI search bots.
Look for named AI bots. For each named group, check it contains every rule you meant it to follow.
Run your domain through our free AI crawler checker. It reads your live robots.txt and tells you which AI search and training bots can reach your pages. It saves you parsing groups by hand.
Check your CDN and security app settings for AI or bot blocking.
After any change, give it a day. OpenAI and Perplexity both say updates can take around 24 hours to take effect.

Crawler access is step one of AI search visibility. It fails silently, so it's worth checking before anything else. If you're past that and wondering why ChatGPT still doesn't mention you, read why your brand doesn't show up on ChatGPT.

FAQ

Should I block GPTBot?

Only if keeping your content out of future model training matters more to you than having AI models know your brand. Blocking GPTBot doesn't affect ChatGPT search, which uses OAI-SearchBot. Most D2C brands are better off allowing both.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended doesn't affect inclusion in Google Search, and AI Overviews are part of Search, crawled by Googlebot. Google-Extended controls Gemini model training and grounding in Gemini Apps and Vertex AI.

Do I need to list every AI bot in robots.txt to allow it?

No. Any bot without its own group follows your User-agent: * rules. Only name a bot when you want to treat it differently, and then repeat every rule it should follow in its group.

Do all AI crawlers respect robots.txt?

The named search and training bots from OpenAI, Anthropic, Perplexity, Google and Apple say they do. OpenAI's ChatGPT-User and Perplexity's Perplexity-User say robots.txt may not apply to user-triggered visits. Anthropic says Claude-User does follow it.

How long do robots.txt changes take to work?

OpenAI says about 24 hours, and Perplexity says up to 24 hours. Google says it generally caches robots.txt for up to 24 hours. Plan for a day before you test again.

If you want to see what happens after the bots get in, which buyer questions mention your brand and which mention competitors, get the free AI visibility report. Or talk to us if you'd like a second pair of eyes on your setup.