Meta Robots Tags Explained: Index, Noindex, Follow, Nofollow, and More

Meta robots tags control something more specific than most site owners realize: not whether Google can visit a page, but whether it can include that page in search results and whether it should follow the links sitting on it. Get the directive wrong and you can accidentally deindex a page that took months to rank.

The confusion usually starts with terminology. Robots.txt, meta robots tags, and X-Robots-Tag all sound like the same thing, but they control different layers of how search engines interact with a page, and mixing them up is one of the most common technical SEO mistakes.

These directives operate at the individual URL level, sitting inside a page's head section rather than applying sitewide. That precision is exactly what makes them useful, and exactly what makes a single misplaced tag dangerous.

This guide breaks down every meta robots directive worth knowing, how index, noindex, follow, and nofollow actually behave, and the mistakes that quietly cost sites their rankings.

01What Meta Robots Tags Actually Do

Devender Gupta's technical breakdown defines it precisely: a meta robots tag is an HTML snippet placed in a document's head that tells search engines whether they can include the page in their index and whether they should follow its links, operating at the individual URL level rather than sitewide.

MDN's official reference notes this isn't part of any formal web specification, it's a de-facto standard for communicating with search bots, crawlers, and similar user agents, cooperatively honored by Google, Bing, and other major engines.

02Index vs Noindex: What Each Directive Controls

Olibro's guide to meta robots directives puts it simply: index tells search engines to include the page in results, while noindex hides it from view entirely. Without any such tag present, Digital Applied's complete SEO reference confirms Google defaults to index, follow automatically.

Noindex doesn't stop crawling on its own. Devender Gupta's breakdown, referenced above, explains Google continues visiting a noindexed page periodically to check whether the directive has changed, which is why removing thousands of pages via noindex takes real time to fully process.

03Follow vs Nofollow: What Changed in 2019

Follow tells search engines to crawl and potentially index the pages a link points to. Nofollow tells them to skip that, historically preventing any link authority from passing through.

Devender Gupta's breakdown, referenced above, notes a shift worth knowing: since 2019, Google treats nofollow as a hint for crawling and indexing purposes rather than a strict directive. It usually still blocks PageRank transfer, but Google may still discover and follow the link if it finds the same URL referenced elsewhere.

04Common Meta Robots Tag Combinations

Index and follow behave independently, which means all four combinations are valid and each serves a distinct purpose. Victorious' guide to noindex tags offers a useful mental model: think of your site as connected rooms, where index or noindex controls whether a search engine can enter a room, and follow or nofollow controls whether it explores the doorways inside.

Olibro's guide, referenced above, ties each combination to real use cases: noindex, follow suits "thank you" pages you still want to pass discovery signal through, while index, nofollow fits pages you want visible in search but don't want passing authority onward, like user-generated content pages.

05Additional Meta Robots Directives Worth Knowing

Index, noindex, follow, and nofollow aren't the whole list. MDN's reference documents several more: noarchive prevents a cached version from appearing in results, nosnippet blocks any text snippet or video preview, and max-image-preview:large is essential for Google Discover eligibility specifically.

Search Roost's directive guide adds unavailable_after, which schedules a page for automatic removal from search results at a specific date and time, useful for time-limited content like event pages or expiring promotions without needing a manual cleanup later.

06Meta Robots vs Robots.txt vs X-Robots-Tag

This distinction causes more damage than any single directive mistake. iRankly's guide states it plainly: robots.txt controls crawling, meta robots controls indexing, and confusing the two is one of the most common points of failure in technical SEO setups.

Digital Applied's complete reference issues the critical warning directly: never combine disallow with noindex. If robots.txt blocks a URL, Googlebot cannot fetch the page and will never see the meta noindex tag inside it, so the URL can remain indexable indefinitely despite your intent to remove it.

X-Robots-Tag solves a different problem entirely. Google's own Search Central documentation on robots meta directives confirms it carries the same values as meta robots but works as an HTTP header, making it the right choice for non-HTML files like PDFs and images where editing a head section isn't possible.

07Watch: Noindex vs Disallow Explained by Google

For a walkthrough of the exact mistake covered above, mixing a robots.txt disallow with a page-level noindex tag, this video breaks down troubleshooting the "Excluded by noindex tag" status directly inside Search Console.

08Meta Robots Directive Combinations at a Glance

User Growth's comprehensive guide to robots meta tags and Victorious' room analogy both map onto the same practical set of use cases.

09Meta Robots and X-Robots-Tag Code Examples

Here are the exact tag formats for the most common scenarios, plus the server-side equivalents for non-HTML files.

<!-- Standard noindex, follow -->
<meta name="robots" content="noindex, follow">

<!-- Fully hidden, no link equity passed -->
<meta name="robots" content="noindex, nofollow">

<!-- Target Googlebot specifically -->
<meta name="googlebot" content="noindex">

<!-- Schedule automatic removal -->
<meta name="robots" content="unavailable_after: 2026-12-31T23:59:59+00:00">

# Apply noindex to all PDF files
<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

# Apply noindex to image file types sitewide
<Files ~ "\.(png|jpe?g|gif)$">
  Header set X-Robots-Tag "noindex"
</Files>

10Meta Robots Implementation Checklist

A disciplined process prevents the accidental deindexing that meta robots mistakes are most known for.

11Priority Decision Workflow

Five questions cover most meta robots decisions. Tap each one for guidance.

12Which Directive Should You Use Score

Answer a few questions about the page you're configuring to see the recommended meta robots directive.

13FAQs on Meta Robots Tags