Web Scraper for Competitor Content Audits: Build a Complete Python Tool in 9 Steps

Checking twenty competitor pages by hand means twenty open tabs, twenty word counts, and twenty chances to lose track of what you already reviewed. A web scraper for competitor content audits does the same job in the time it takes to make a coffee.

A client once asked us to review how forty competitor blog posts compared to their own content on word count, heading structure, and publish dates. Doing that by hand would have meant a full day of copying numbers into a spreadsheet.

The script we actually used finished the same forty pages in about ninety seconds, and it did not miss a single page or mistype a word count along the way.

That is the real value of a web scraper for competitor content audits. It is not about replacing your judgment on what makes content good.

It is about removing the copy and paste work so you can spend your time on the analysis instead of the data collection.

This guide builds a complete, respectful scraper in Python. You will fetch pages politely, respect robots.txt, pull the metrics that actually matter for a content audit, crawl a whole competitor sitemap automatically, and turn the results into a report you can act on.

A proper content audit looks at dozens or hundreds of pages across word count, heading structure, freshness, and media use. Reviewing that many pages by hand is slow, and slow work tends to get inconsistent.

One page gets a careful word count. The next gets an estimate because you are tired of counting. That inconsistency quietly wrecks the comparison you are trying to make.

A script treats every page the same way, every time. It counts words the same way on page one and page one hundred, and it never skips a field because it got bored.

Search Engine Journal and Search Engine Land both publish regular guidance on running content audits at scale, and consistency is the theme that comes up again and again.

There is also a coverage argument. A person can realistically review twenty or thirty competitor pages in a sitting before quality drops.

A script can work through an entire competitor sitemap, which might be five hundred pages, without losing accuracy on page four hundred and ninety nine.

Keep the toolkit small. You need a way to fetch pages, a way to parse the HTML, a place to hold results, and a retry layer for the occasional slow or failed request.

# create an isolated environment so package versions don't clash
python -m venv venv
source venv/bin/activate      # Windows: venv\Scripts\activate

pip install requests beautifulsoup4 lxml pandas tenacity

The requests library handles the HTTP calls, Beautiful Soup parses the returned HTML, lxml makes that parsing fast, pandas holds and organizes the results, and Tenacity retries a request that fails without crashing the whole run.

Set a clear User-Agent header before you send a single request. Many sites use it to identify who is visiting, and an honest, identifiable value is part of scraping politely rather than pretending to be a regular browser.

HEADERS = {
    "User-Agent": "BrandellaContentAuditBot/1.0 (contact: audits@example.com)"
}
REQUEST_DELAY_SECONDS = 1.5
REQUEST_TIMEOUT = 15

Every respectful web scraper for competitor content audits starts by checking what the site actually allows. The Robots Exclusion Protocol is a published standard, and robots.txt is how a site tells automated tools which paths are off limits.

Python's standard library already includes a parser for this, so there is no reason to skip the check.

from urllib.robotparser import RobotFileParser
from urllib.parse import urljoin

def can_fetch(base_url: str, path: str, user_agent: str = "*") -> bool:
    parser = RobotFileParser()
    parser.set_url(urljoin(base_url, "/robots.txt"))
    parser.read()
    return parser.can_fetch(user_agent, path)

if __name__ == "__main__":
    site = "https://example.com"
    target = "https://example.com/blog/sample-post"
    if can_fetch(site, target):
        print("Allowed, proceed with the audit")
    else:
        print("Blocked by robots.txt, skip this path")

Build this check into every audit script from the start rather than adding it later. A site that disallows crawling on certain paths, like login areas or internal search results, is telling you plainly where the boundary sits.

• Check robots.txt first: run can_fetch before requesting any URL, not after you notice a problem.
• Identify yourself honestly: use a real User-Agent string with a contact method, not a spoofed browser signature.
• Add real delays: a second or two between requests protects the target site and your own IP address from getting flagged.
• Stay on public pages: never attempt to scrape content behind a login wall or paywall for a competitor audit.
• Cache what you already fetched: save responses locally so a rerun does not hit the same server twice for no reason.

With permission checked and headers set, the core function is short. It fetches a page, retries on a transient failure, and hands the HTML to BeautifulSoup for parsing.

import time
import requests
from bs4 import BeautifulSoup
from tenacity import retry, wait_exponential, stop_after_attempt
from config import HEADERS, REQUEST_DELAY_SECONDS, REQUEST_TIMEOUT

@retry(wait=wait_exponential(multiplier=1, min=2, max=20), stop=stop_after_attempt(4))
def fetch_page(url: str) -> BeautifulSoup:
    response = requests.get(url, headers=HEADERS, timeout=REQUEST_TIMEOUT)
    response.raise_for_status()
    time.sleep(REQUEST_DELAY_SECONDS)   # be polite between requests
    return BeautifulSoup(response.text, "lxml")

def extract_basics(soup: BeautifulSoup, url: str) -> dict:
    title = soup.title.get_text(strip=True) if soup.title else ""
    meta_desc_tag = soup.find("meta", attrs={"name": "description"})
    meta_description = meta_desc_tag["content"] if meta_desc_tag else ""
    h1 = soup.find("h1")
    body_text = soup.get_text(separator=" ", strip=True)

    return {
        "url": url,
        "title": title,
        "meta_description": meta_description,
        "h1": h1.get_text(strip=True) if h1 else "",
        "word_count": len(body_text.split()),
    }

The retry decorator matters more than it looks. A single competitor site will occasionally hand back a slow response or a brief server error, and without a retry layer that one failure can quietly drop a page from your whole audit.

A web scraper for competitor content audits earns its real value once you look past word count alone. A content audit gets useful when you add heading structure, link counts, media presence, and whether the page carries schema markup.

def extract_content_metrics(soup, url: str, domain: str) -> dict:
    headings = {
        "h1_count": len(soup.find_all("h1")),
        "h2_count": len(soup.find_all("h2")),
        "h3_count": len(soup.find_all("h3")),
    }

    links = soup.find_all("a", href=True)
    internal_links = [a for a in links if domain in a["href"]]
    external_links = [a for a in links if domain not in a["href"] and a["href"].startswith("http")]

    schema_tags = soup.find_all("script", type="application/ld+json")

    return {
        "url": url,
        **headings,
        "internal_link_count": len(internal_links),
        "external_link_count": len(external_links),
        "image_count": len(soup.find_all("img")),
        "has_schema_markup": len(schema_tags) > 0,
    }

A single page tells you very little about a competitor's overall content strategy. The real value of a web scraper for competitor content audits shows up once you point it at their whole sitemap.

Most sites publish one at a predictable path, so pulling every blog URL is usually a short step.

import requests
import pandas as pd
from bs4 import BeautifulSoup
from scraper import fetch_page, extract_basics
from metrics import extract_content_metrics
from config import HEADERS

def get_sitemap_urls(sitemap_url: str) -> list:
    response = requests.get(sitemap_url, headers=HEADERS, timeout=15)
    soup = BeautifulSoup(response.text, "xml")
    return [loc.get_text() for loc in soup.find_all("loc")]

def audit_competitor(sitemap_url: str, domain: str, limit: int = 50) -> pd.DataFrame:
    urls = get_sitemap_urls(sitemap_url)[:limit]
    rows = []

    for url in urls:
        try:
            soup = fetch_page(url)
            basics = extract_basics(soup, url)
            metrics = extract_content_metrics(soup, url, domain)
            rows.append({**basics, **metrics})
        except Exception as error:
            print(f"Skipped {url}: {error}")

    return pd.DataFrame(rows)

if __name__ == "__main__":
    df = audit_competitor("https://competitor.com/sitemap.xml", "competitor.com")
    df.to_csv("competitor_audit.csv", index=False)
    print(f"Audited {len(df)} pages")

Notice the try and except block wrapped around every page. A single broken page should never take down a fifty page audit, so one failure gets logged and the crawl moves on.

A CSV full of numbers is not a report yet. The last step is comparing what you found against your own content, or against the other competitors you audited.

import pandas as pd

def summarize_audit(df: pd.DataFrame) -> pd.DataFrame:
    summary = pd.DataFrame({
        "avg_word_count": [df["word_count"].mean()],
        "median_word_count": [df["word_count"].median()],
        "avg_h2_count": [df["h2_count"].mean()],
        "pct_with_schema": [df["has_schema_markup"].mean() * 100],
        "pages_below_800_words": [(df["word_count"] < 800).sum()],
    })
    return summary.round(1)

if __name__ == "__main__":
    df = pd.read_csv("competitor_audit.csv")
    print(summarize_audit(df))

Numbers like these turn into decisions fast. If a competitor averages 1,800 words on their top ranking posts and your own average sits at 900, that is a specific, evidence backed gap to close rather than a vague feeling that your content should be longer.

A web scraper for competitor content audits will eventually run into a page that fights back. Some sites block automated requests outright, and others render their content with JavaScript, so the HTML that requests receives is nearly empty.

• 403 or 429 responses: slow down further, respect a Retry-After header if one is present, and never retry aggressively against a site that is actively refusing you.
• Empty or near empty HTML: if word count comes back near zero on a page you can see content on in a browser, the site likely renders with JavaScript and needs a headless browser instead.
• Encoding issues: set response.encoding explicitly when special characters show up as garbled text.
• Timeouts: a generous but finite timeout, paired with the retry decorator, keeps one slow server from stalling the whole audit.
• Sites that require JavaScript: reach for Playwright instead of requests. It runs an actual browser engine and waits for the page to render before you read the HTML.

from playwright.sync_api import sync_playwright

def fetch_rendered_html(url: str) -> str:
    with sync_playwright() as pw:
        browser = pw.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        html = page.content()
        browser.close()
        return html

Reach for this only when you actually need it. A headless browser is slower and heavier than a plain request, so keep it as a fallback for JavaScript rendered pages rather than your default approach.

Real Python's guide to the requests library covers most of the timeout and error handling patterns worth knowing before you run a large crawl.

Scraping publicly visible pages for a competitor analysis sits in genuinely nuanced territory, and this guide is not legal advice. The details depend on your jurisdiction, the site's terms of service, and exactly what you collect.

What is worth knowing is that courts have addressed this question directly. In hiQ Labs v. LinkedIn, a United States appeals court found that scraping data a site has made publicly accessible, without logging in, did not violate federal computer fraud law on its own.

That ruling did not settle every question. A site's terms of service can still create separate contract obligations, and scraping personal data brings privacy rules like GDPR into play regardless of what a court decided about one specific statute.

For a content audit built for internal strategy work, the practical guidance is simple. Stay on public pages, respect robots.txt, identify your bot honestly, keep request rates low, and never collect personal information you do not need.

If you want data with a clearer license and none of this ambiguity, an official API is always the safer starting point, the same way it is for keyword research tools that offer one.

Watching the request and parse cycle run in real time makes it click faster than reading code alone. freeCodeCamp.org, a well known nonprofit coding education channel, published a full crash course covering exactly the requests and BeautifulSoup pattern used throughout this guide.

Video credit: freeCodeCamp.org.

The fundamentals in that video, fetching a page and walking its parsed tags, are the same fundamentals this entire pipeline builds on. Everything past that point is really just organizing what BeautifulSoup already gives you.