Search engines crawl your site constantly, following links and indexing pages. But not every page deserves to be indexed. You have private pages, admin areas, test pages, and duplicate pages that waste crawl budget. Robots.txt is a simple text file that tells search engines which pages they can and cannot crawl. It is the gatekeeper between your site and search engine bots.
Robots.txt is a plain text file that sits in your website's root directory (yoursite.com/robots.txt). It contains instructions for search engine crawlers on which pages and sections of your site they are allowed to crawl. You do not need a login to access robots.txt. It is public. Anyone can visit yoursite.com/robots.txt and see your instructions.
Search engines are supposed to follow robots.txt rules. Most do. Some do not. Well-behaved crawlers (Google, Bing, reputable search engines) respect robots.txt. Bad actors and malicious bots often ignore it.
What does robots.txt look like?
A simple robots.txt file looks like this:
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /blog/
Sitemap: https://www.example.com/sitemap.xml
This tells all bots (User-agent: *) that they cannot crawl /admin/ or /private/ sections. It explicitly allows /blog/ (which is useful if a parent folder is blocked). It also points bots to your sitemap.
Here is another example with more specific rules:
User-agent: Googlebot
Disallow: /temp/
User-agent: *
Disallow: /test/
Disallow: /*.pdf
Crawl-delay: 5
This blocks only Google's crawler from /temp/. All other bots are blocked from /test/ and all PDF files. The crawl-delay tells bots to wait 5 seconds between requests.
How robots.txt helps SEO
Robots.txt helps SEO in one main way: it preserves crawl budget. Search engines allocate a limited amount of crawl budget to each site. For small sites, crawl budget is rarely a problem. For large sites with thousands of pages, crawl budget matters.
If your site has 10,000 pages but only 500 are important, you want bots crawling the 500 important pages multiple times rather than spreading their crawl budget across all 10,000 pages. Robots.txt blocks bots from wasting time on unimportant pages.
Common pages to block: admin areas, login pages, thank you pages, test pages, duplicate content, session pages, and temporary content. Blocking these pages tells bots to focus on content that matters.
What not to block in robots.txt
Do not block pages you want to rank. Do not block your blog section or main content areas. Do not block CSS, JavaScript, or image files needed to _blocked_ pages properly (unless you specifically want to prevent indexing).
Google recommends not blocking CSS, JavaScript, and image files even if you do not want them indexed. Blocking these resources can prevent Google from properly rendering and understanding your pages.
Do not use robots.txt to hide sensitive information. Robots.txt is public. Anyone can read yoursite.com/robots.txt. If you want to truly hide something, use password protection or server authentication, not robots.txt.
How to create a robots.txt file
Creating robots.txt is simple. It is a plain text file. Most website builders and CMS platforms generate one automatically.
If your CMS did not create one, create it manually. Open a text editor. Write your rules. Save it as "robots.txt" (not "robots.txt.txt"). Upload it to your site root directory (yoursite.com/robots.txt).
Test your robots.txt file. Go to Google Search Console, select your property, go to the Robots.txt Tester tool, and paste in your rules. Google will show you what the file blocks and allows. This confirms your rules work as intended.
Common robots.txt rules and examples
Here is the most common rule: block all bots from a specific directory.
User-agent: *
Disallow: /admin/
Here is how to block a specific file type (like PDFs):
User-agent: *
Disallow: /*.pdf
Here is how to block all bots except Google:
User-agent: *
Disallow: /
User-agent: Googlebot
Disallow:
Here is how to add a crawl delay (wait X seconds between requests):
User-agent: *
Crawl-delay: 10
Here is how to point bots to your sitemap:
Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap2.xml
Important robots.txt limitations
Robots.txt is not a security tool. It is a courtesy system. Bad bots, hackers, and malicious crawlers ignore robots.txt completely. Do not use it to hide sensitive data or protect important pages. Use password protection for those.
Robots.txt does not guarantee exclusion from search results. Google sometimes indexes pages listed in robots.txt if they are linked from other indexed pages. The block prevents crawling, but not indexing if the page is linked elsewhere.
Robots.txt is not a noindex. If you want to prevent indexing completely, use a robots meta tag or X-Robots-Tag header, not robots.txt. Robots.txt only controls crawling, not indexing.
Frequently asked questions
Does robots.txt affect SEO rankings?
Indirectly. Robots.txt does not directly impact rankings. However, by directing crawl budget to important pages, you help search engines crawl and index your best content more frequently. Better crawling leads to better indexing leads to better ranking potential. So robots.txt helps rankings indirectly through crawl budget management. With WEMASY's website builder, you can set this up directly on your website.
Can I block Google from crawling my site?
Yes, but you should not. You can disallow Googlebot in robots.txt, but that prevents Google from indexing your site. Your site will disappear from Google search. The only reason to do this is if you are taking your site offline or moving it and do not want the old version indexed. In almost all cases, you should allow Google to crawl.
Is robots.txt case sensitive?
Partially. The User-agent and Disallow commands are not case sensitive. The paths you disallow are case sensitive on most servers. /Admin/ and /admin/ are different paths. Test your rules in Google Search Console to be sure they work as intended.
Should I block duplicate pages with robots.txt?
It depends. Robots.txt blocks crawling but not indexing. If you want to prevent duplicate pages from being indexed, use the canonical tag or robots meta noindex instead. Use robots.txt to block crawling of unimportant duplicates you do not care if Google ever finds.
Do I need to notify Google when I change robots.txt?
No. Google crawls robots.txt frequently. Changes take effect usually within hours. You do not need to resubmit your site or notify anyone. Just update robots.txt and wait for the next crawl.
What is the difference between robots.txt and robots meta tags?
Robots.txt blocks crawling at the file level. Robots meta tags control indexing at the page level. Robots.txt says a page cannot be crawled. Robots meta tags (like noindex) say a page can be crawled but should not be indexed. Use robots.txt for large sections you want blocked. Use robots meta tags for individual pages you want crawled but not indexed.