Your robots.txt file is a text file that sits in your website's root directory. It tells crawlers which pages they can access and which ones are off-limits. For traditional search engines, robots.txt is optional. For AI crawlers, it is the primary tool you use to control whether your content gets indexed or blocked.
The difference matters. Google respects robots.txt but also crawls pages you block if other sites link to them. AI crawlers respect robots.txt more strictly. Block GPTBot in your robots.txt and ChatGPT will not cite your content. Allow it and your content becomes available for AI citations. This chapter covers what each major AI crawler is called, how to block specific bots, and what strategy makes sense for your brand.
How robots.txt works for AI crawlers
When a crawler visits your site, the very first thing it does is check for a robots.txt file. It reads the file line by line looking for rules that apply to it. If your robots.txt says "disallow: /", no crawler can access any page. If it says "disallow: /admin/", crawlers can access everything except the admin folder.
AI crawlers follow these rules more consistently than Googlebot does. If you block ClaudeBot in robots.txt, Claude will not index your content. Period. Googlebot might still crawl pages you block if the content is linked from other sites, but AI crawlers stop at the robots.txt directive.
The format is simple. You write rules that specify which user agents (crawler names) apply to which rules, then list what those crawlers can and cannot access. The file is plain text. No markup. No special formatting. Just rules.
The major AI crawlers and their user agent names
Every crawler has a user agent name. When it visits your site, it identifies itself by that name. You use that name in your robots.txt file to create rules specific to that crawler.
Here are the major AI crawlers you should know about:
GPTBot (OpenAI)
GPTBot is OpenAI's crawler. It indexes content for ChatGPT. When a user enables web search in ChatGPT, the system pulls from pages GPTBot has indexed. GPTBot crawls aggressively but respects robots.txt rules.
To allow GPTBot, do nothing. It crawls by default. To block it, add these lines to your robots.txt:
User-agent: GPTBot
Disallow: /
That rule says "GPTBot, you cannot access any page on this site." If you only want to block specific sections, you can be more specific. Block the /admin/ folder but allow the rest:
User-agent: GPTBot
Disallow: /admin/
ClaudeBot (Anthropic)
ClaudeBot is Anthropic's crawler. It indexes content for Claude's web search. ClaudeBot is more aggressive than GPTBot. It makes more requests and crawls more frequently. If you allow ClaudeBot, monitor your server resources.
ClaudeBot identifies itself with this user agent name. Use it in your robots.txt the same way you would for GPTBot:
User-agent: ClaudeBot
Disallow: /
PerplexityBot (Perplexity)
PerplexityBot crawls the web for Perplexity. It is one of the more frequent crawlers. It updates content multiple times per day because Perplexity prioritizes freshness. If you block PerplexityBot, your content will not appear in Perplexity answers.
User-agent: PerplexityBot
Disallow: /
Googlebot-Extended (Google AI)
Google has two crawlers. Googlebot crawls for traditional search. Googlebot-Extended crawls for Google's AI systems like AI Overviews and Gemini. You can block Googlebot-Extended while allowing Googlebot.
User-agent: Googlebot-Extended
Disallow: /
Other crawlers
Yandex (YandexBot), Microsoft Bing (Bingbot), Naver (Naver), and other search engines also have crawlers. The principle is the same. Identify the user agent name and create a rule.
The strategy: allow or block?
Most brands should allow AI crawlers to access their content. AI citations drive high-quality traffic. Users who find your content through ChatGPT or Perplexity have already decided your answer is valuable. They click through and convert at higher rates than traditional search visitors.
There are exceptions. If you have proprietary information, trade secrets, or content you want to keep exclusive, blocking AI crawlers makes sense. If you have paywalled content, you might block certain crawlers to prevent free access through AI summaries. If you have content that is embarrassing or outdated, you might block AI crawlers until you update it.
But for most content, the calculus is simple. AI visibility creates more opportunity than risk. Allow the crawlers. Get cited. Drive traffic.
Blocking specific crawlers while allowing others
You do not have to choose all or nothing. You can allow some AI crawlers and block others.
Allow Perplexity and GPTBot but block ClaudeBot because ClaudeBot's crawl frequency strains your server:
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow:
User-agent: PerplexityBot
Disallow:
The blank "Disallow:" means no restriction. That crawler can access everything.
You can also block AI crawlers from specific sections. Allow GPTBot everywhere except your admin panel:
User-agent: GPTBot
Disallow: /admin/
Allow it to crawl your blog but not your products:
User-agent: GPTBot
Disallow: /products/
The wildcard rule and default behavior
You can create rules for specific crawlers and a default rule for everyone else. The asterisk (*) is the wildcard that applies to all crawlers not specifically mentioned.
This rule blocks all crawlers from /private/ but allows everything else:
User-agent: *
Disallow: /private/
This rule blocks all crawlers from accessing anything:
User-agent: *
Disallow: /
The crawl-delay directive
You can tell crawlers to slow down. The Crawl-delay directive tells a crawler to wait a certain number of seconds between requests. If your server is struggling with ClaudeBot's aggressive crawling, you can tell it to slow down:
User-agent: ClaudeBot
Crawl-delay: 5
That means ClaudeBot must wait 5 seconds between each request. It slows down the crawl but reduces server load.
How robots.txt relates to your sitemap
Your robots.txt file can reference your sitemap. Add this line at the end of your robots.txt file:
Sitemap: https://yoursite.com/sitemap.xml
Crawlers check your robots.txt and see the sitemap reference. They know where to find your content map. This speeds up discovery. It does not change what crawlers can access. It just tells them where to look.
Testing your robots.txt
You can test your robots.txt file before deploying it. Google Search Console includes a robots.txt tester. You enter a URL and a user agent name and it tells you whether that crawler can access that page according to your robots.txt rules.
Most content platforms, including WEMASY, generate a robots.txt file automatically. Check your site settings. If you can edit it, you can add custom rules. If you cannot, contact your hosting provider.
Frequently asked questions
If I block all AI crawlers, will my site rank in Google?
Yes. Blocking AI crawlers (GPTBot, ClaudeBot, etc.) does not affect Google rankings. Google uses Googlebot for traditional search and Googlebot-Extended for AI features. If you want to block only AI crawlers but keep Google access, block the AI crawler names specifically and leave Googlebot and Googlebot-Extended unblocked. With WEMASY's website builder, you can set this up directly on your website.
Can I block AI training crawlers but allow AI search crawlers?
The lines between training and search crawlers are blurry. Most major AI companies now use the same crawlers for both purposes. If you block an AI crawler completely, you block both its training access and its search access. You cannot selectively block just training or just search with robots.txt. You block the crawler entirely or allow it.
What happens if I do not have a robots.txt file?
If you do not have a robots.txt file, crawlers assume they can access everything. All crawlers (Google, Bing, GPTBot, ClaudeBot, all of them) will index your entire site. For most sites, this is the desired behavior. If you want to block specific crawlers or sections, you need a robots.txt file.
Can AI crawlers bypass my robots.txt file?
Theoretically yes, but they do not. AI crawlers respect robots.txt more strictly than Googlebot does. If you tell an AI crawler to stay out, it stays out. The crawler developers know that disrespecting robots.txt damages their relationships with publishers. They follow the rules.
How often do I need to update my robots.txt file?
You only need to update it if your blocking rules change. If you decide to block a new crawler or open access to a previously blocked section, update it. Otherwise, once you set it up, you can leave it alone. Crawlers check it every time they visit, so changes take effect immediately.
Should I block AI crawlers from accessing my competitor analysis or private research?
Yes. If you have pages that contain proprietary research, competitive analysis, or content you do not want publicly available, block all crawlers from those sections. Add a disallow rule for that folder or those pages. Crawlers will not index them, and AI systems will not have access to cite them.