What Is a Robots.txt File?
A robots.txt file is a plain text file at the root of your site that tells crawlers which pages they may visit. Search engines read it before they crawl, so a good file keeps bots away from carts, admin screens and endless search result pages, and points them at your sitemap instead.
It is also the place where you decide which AI crawlers can use your content. You can block bots that collect text for AI training while still letting AI search tools show and cite your pages.
How to Use the Robots.txt Generator
- Choose the default. Live sites should allow crawling. Block the whole site only on staging copies.
- List paths to block, one per line. Use
*to match any characters and$to match the end of a URL. - Add exceptions for files inside a blocked folder that crawlers still need.
- Paste your sitemap URL, or several if you have more than one.
- Tick the AI crawlers you want to keep out, then copy or download the file and upload it to your site root.
Robots.txt Rules Google Follows
| Rule | What it does |
|---|---|
User-agent | Starts a group of rules for one crawler, or * for all of them |
Disallow | Blocks paths that start with the value. Paths are case sensitive |
Allow | Opens a path inside a blocked folder. The longest matching rule wins |
Sitemap | Full URL of a sitemap. You can list as many as you need |
Crawl-delay | Not supported by Google, so this tool leaves it out |
A crawler only follows the most specific group that names it. If you add a group for Googlebot, it ignores the * group completely, so repeat any rules it should still obey. Google explains this in how Google interprets robots.txt.
Tips for a Safe Robots.txt
- Do not use it to hide pages from Google. A blocked URL can still be indexed from links. Use a
noindextag and leave the page crawlable. - Never block CSS or JavaScript your pages need, or Google cannot render them properly.
- Block crawl traps like internal search, filters and add to cart links that create endless URLs.
- AI choices are separate. Blocking
Google-ExtendedorGPTBotdoes not change your Google rankings.
Learn More About Crawling
Our complete robots.txt guide covers every rule in depth, crawl budget explained shows why blocking waste matters on big sites, and opting out of Google AI search walks through your AI choices. Already have a file? Check it with the robots.txt tester.
Frequently Asked Questions
1Where do I put my robots.txt file?
Upload it to the root folder of your site so it opens at yourdomain.com/robots.txt. Crawlers never look for it in subfolders, and every subdomain needs its own file.
2Does blocking AI crawlers hurt my Google rankings?
No. Google Extended, GPTBot, ClaudeBot and CCBot are separate from Googlebot, so blocking them does not change how Google Search crawls or ranks your pages.
3Will robots.txt remove a page from Google?
No. It only stops crawling. A blocked page can still appear in results if other sites link to it. To remove a page, allow crawling and add a noindex meta tag.
4Should I block wp admin in WordPress?
Blocking the wp admin folder is common and safe, as long as you allow the admin ajax file inside it, which some themes and plugins load on public pages.