How Google Crawls Websites: From Discovery to Indexing

Google indexes only 30 to 60% of a typical website's pages, according to Google's own John Mueller. The rest sit in some in-between state, crawled but excluded, discovered but never fetched, or fetched but never rendered properly. Understanding the actual pipeline between discovery and indexing explains exactly where most of that other 40 to 70% quietly gets lost.

Google crawls websites through a multi-stage pipeline: discovering URLs through links, sitemaps, and prior crawl history, fetching their content, rendering JavaScript-heavy pages in a headless browser, and finally evaluating that content for inclusion in the index. Crawling and indexing are related but genuinely separate processes.

We covered why Google sometimes skips pages entirely in our crawl budget guide. This article covers the full pipeline that budget gets spent moving through.

The Full Pipeline, Start to Finish

Stage 1: How Google Discovers a URL

According to Fudugo's technical deep dive into how Google does indexing, discovery starts from seed URLs: previously crawled pages, known backlinks, submitted XML sitemaps, and pages manually submitted through Search Console.

According to Incremys' guide to understanding Google's site exploration, a sitemap is a discovery and prioritization signal, not an "index now" button. Submitting one never forces crawling or indexing directly.

Stage 2 and 3: Fetching, Then Rendering

Modern Googlebot processes pages in two distinct waves.

According to SEO Hacker's step-by-step guide to Google crawling, rendering requires significant computational resources, which is why Google prioritizes which pages get rendered based on page importance and freshness requirements.

Stage 4: What Happens During Indexing

According to Incremys' 2026 guide to managing Googlebot, indexing analyzes the fully rendered content to determine its topic and decide whether it earns a place in the index at all.

Why Crawling Doesn't Guarantee Indexing

Crawl-to-Index Diagnostic

Select the Search Console status you're seeing to identify which pipeline stage is the actual problem.

Why Understanding Crawling and Indexing Together Matters

According to SEO Kreativ's guide to crawling and indexing explained simply, treating these as one combined process is a common mistake. Every page you want visible must be crawlable, renderable, and indexable, three distinct hurdles rather than one.

According to a detailed explainer on how Google Search works, crawling and indexing together form the foundation the entire ranking system depends on, powering everything from traditional results to AI Overviews and AI Mode.

Internal Linking's Role in the Discovery Stage

According to Incremys' guide to SEO crawling referenced above, internal linking plays a dual role in both crawling and indexing outcomes: it aids discovery of new pages and clarifies the site's overall hierarchy for Google to interpret.

The deeper a page sits, meaning the more clicks required from an entry point to reach it, the harder it becomes to discover and the less frequently it gets revisited once it is crawled and indexed for the first time.

What Slows This Pipeline Down

According to Viacon's beginner's guide to Google crawl versus index, excessive JavaScript reliance without server-side rendering delays the second wave significantly, since the rendering queue is inherently slower than the initial HTML pass.

According to Digital Strategy Force's guide to how Google crawls and indexes, architectural clarity, every URL reachable, renderable, and unambiguous in its canonical identity, matters more to full indexing than content quality alone.

FAQs on How Google Crawls Websites