Robots.txt Explained: Complete Guide with Examples

A single misplaced line in robots.txt can hide your entire site from Google. It is one of the smallest, oldest files on the web, plain text, four main directives, and it still controls billions of dollars of traffic every single day. This guide covers the full syntax, real AI crawler rules for 2026, and the one mistake that quietly de-indexes sites.

Robots.txt is a plain-text file placed at the root of a website that tells search engine crawlers and AI bots which URLs they are allowed or not allowed to access.

It follows the Robots Exclusion Protocol, formalized as RFC 9309, a standard respected by Google, Bing, and most major AI crawlers.

We covered the page-level counterpart to this file in our meta robots tags guide. Robots.txt blocks crawling before a page is ever fetched; meta robots tags control indexing after it already has been.

Basic Robots.txt Syntax

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/public/

Sitemap: https://www.example.com/sitemap.xml

Must live at exactly yoursite.com/robots.txt; a file at /blog/robots.txt is simply ignored

The Four Core Directives

Wildcard Patterns for Precise Matching

According to Ixieverse's complete robots.txt setup guide, combining the asterisk and dollar sign builds precise patterns matching specific URL structures rather than entire directories.

# Block every PDF file across the entire site
Disallow: /*.pdf$

# Block any URL containing a query string starting with ?sort=
Disallow: /*?sort=

# Block an entire path and everything beneath it
Disallow: /search

The dollar sign anchors the match to the end of the URL; without it, the pattern matches anywhere

Blocking or Allowing AI Crawlers in 2026

According to ToolsDock's complete 2026 robots.txt guide, every major AI company now runs its own crawler, some for training data, some for real-time browsing, and some for both.

# Allow Google and Bing as normal
User-agent: Googlebot
Allow: /

# Block OpenAI's training crawler specifically
User-agent: GPTBot
Disallow: /

# Allow Anthropic's on-demand fetch bot, block its training crawler
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /

Different bots from the same company often serve different purposes; block selectively, not all at once

Should You Block AI Crawlers Entirely?

This depends entirely on your goals. Tap through both perspectives.

Robots.txt Rule Tester

Enter a Disallow pattern and a URL path to see if the rule would block it.

Robots.txt and Crawl Budget

According to Conductor's guide to robots.txt essential rules, one incorrect rule can block major sections of a site from search engines and AI bots simultaneously, making careful review essential before any change goes live.

Directives only apply to the exact domain and protocol where the file is hosted, meaning http and https versions of the same site need matching files if both are technically reachable.

How Robots.txt Groups and Multiple User-Agents Work

According to InstarankSEO's complete 2026 robots.txt guide, a crawler looks for its specific User-agent block first, and falls back to the wildcard block only if no matching specific entry exists.

Multiple user-agents can also share the exact same set of rules, listed as consecutive User-agent lines followed by one shared block of directives, avoiding the need to repeat identical rules for each bot separately.

Common Mistakes That Break Robots.txt

Testing Your Robots.txt Before Deploying

According to SE Ranking's robots.txt syntax and setup guide, a bot can technically choose not to obey the file at all, though major search engines and reputable AI crawlers do respect it in practice.

According to DominateTools' complete 2026 reference guide, testing changes in Google Search Console's report before deploying catches accidental blocks before they affect live crawling.

According to Fieldari's guide to examples, rules, and mistakes, the gap between what this file does and what it doesn't do is exactly where even experienced developers trip up, making a test pass before publishing a genuinely worthwhile habit.

According to ValidateHTML's complete syntax reference, running the final file through a dedicated syntax checker catches typos and malformed directives a quick manual read-through can easily miss.

FAQs on Robots.txt