Python Keyword Research Automation: 7 Proven Ways to Query SEO APIs at Scale

Most SEO teams still build keyword lists by hand, one browser tab and one export at a time. Python keyword research automation replaces that slow loop with a script that runs in seconds and scales to thousands of terms.

The first time I automated a client's keyword list with a script instead of a spreadsheet, it pulled 4,800 keyword variants with search volume, CPC, and competition data in under two minutes.

The equivalent manual export from Google Keyword Planner, cleaned and deduplicated by hand, had taken our team most of a working day the week before.

That gap is the whole case for python keyword research automation. It is not about replacing strategic judgment, since a script cannot decide which keywords fit your content plan or your business.

What it does remove is the part that is repetitive and prone to errors: logging into an API, paging through results, merging data from more than one source, and running the same query again next month without starting from zero.

This guide is a working build, not a theory piece. You will set up a Python environment and authenticate against two different keyword APIs.

From there you will pull and merge live search volume data, cluster the results into topics, and schedule the whole pipeline to run on its own. Along the way we will cover the failure modes that actually show up in production, including rate limits, quota errors, and mismatched locales.

Manual keyword research has a ceiling. A person can comfortably review a few hundred terms before quality starts to drop.

Fatigue sets in, formatting gets inconsistent, and duplicate or near duplicate keywords slip through. Manual work also does not refresh itself, so last quarter's volume numbers sit untouched until someone remembers to export them again.

A script has none of those limits. Once the authentication and request logic are written, running it again next month costs almost nothing.

This matters even more once you are trying to align keyword targeting with search intent across a large site. That kind of classification is far easier to apply consistently when you are working with one structured dataframe instead of a scattered set of CSV exports from different tools.

There is also a data quality argument. Ahrefs, Semrush, DataForSEO, and Google's own Keyword Planner rarely agree exactly on volume for the same term, because each pulls from different sample sizes and methods.

Ahrefs' own study comparing Search Console and Keyword Planner volumes puts that gap in concrete numbers. Blending two or three sources programmatically, then averaging or flagging outliers, is realistic in Python. Doing the same thing by hand across multiple browser tabs is not, as Ahrefs' comparison of Search Console and Keyword Planner data makes clear.

Every python keyword research automation project starts with the same small, reproducible environment. You need an HTTP client, a place to hold results, a way to keep API keys out of your codebase, and a retry layer for the moments when an endpoint throttles you partway through a run.

# create an isolated environment so package versions don't clash
python -m venv venv
source venv/bin/activate      # Windows: venv\Scripts\activate

pip install requests pandas python-dotenv tenacity google-ads gspread

Store credentials in a .env file and load them with python-dotenv. Never hardcode API keys into a script you might commit to a public repository.

DATAFORSEO_LOGIN=your_login
DATAFORSEO_PASSWORD=your_password
GOOGLE_ADS_DEVELOPER_TOKEN=your_dev_token
GOOGLE_ADS_CLIENT_ID=your_client_id
GOOGLE_ADS_CLIENT_SECRET=your_client_secret
GOOGLE_ADS_REFRESH_TOKEN=your_refresh_token
GOOGLE_ADS_CUSTOMER_ID=1234567890

import os
from dotenv import load_dotenv

load_dotenv()

DATAFORSEO_LOGIN = os.getenv("DATAFORSEO_LOGIN")
DATAFORSEO_PASSWORD = os.getenv("DATAFORSEO_PASSWORD")

if not DATAFORSEO_LOGIN:
    raise EnvironmentError("Missing DATAFORSEO_LOGIN in .env")

This structure matters. A dedicated config.py that fails loudly when a credential is missing saves you from a much worse failure mode: a script that silently sends empty auth headers and returns a generic 401 three modules deep into a pipeline.

Every keyword API trades off differently on cost, setup complexity, and freshness. Before writing a single request for python keyword research automation, decide which one actually fits the job.

Pulling seed keyword volume for a content calendar has very different requirements than continuously monitoring competitor rankings. For a broader look at the tools themselves, read our comparison of the best keyword research tools. Here is how the four most commonly scripted APIs stack up for Python automation specifically.

For most teams, the practical answer is to combine two APIs. Use the Google Ads API for authoritative search volume and keyword ideas sourced directly from Google, and use DataForSEO for the bulk keyword expansion and clustering data that Google's own API does not expose.

That pairing is what the rest of this walkthrough uses.

The first practical step in python keyword research automation is pulling real search volume, not theoretical seed terms. DataForSEO's Keywords Data endpoints accept a list of seed terms and return monthly search volume, CPC, and competition in a single request.

For lists under 1,000 keywords per call, no pagination is required.

import base64
import requests
import pandas as pd
from tenacity import retry, wait_exponential, stop_after_attempt
from config import DATAFORSEO_LOGIN, DATAFORSEO_PASSWORD

API_URL = "https://api.dataforseo.com/v3/keywords_data/google_ads/search_volume/live"

def _auth_header():
    token = base64.b64encode(
        f"{DATAFORSEO_LOGIN}:{DATAFORSEO_PASSWORD}".encode()
    ).decode()
    return {"Authorization": f"Basic {token}", "Content-Type": "application/json"}

@retry(wait=wait_exponential(multiplier=1, min=2, max=30), stop=stop_after_attempt(5))
def get_search_volume(keywords: list, location_code: int = 2840, language_code: str = "en"):
    """Fetch search volume for up to 1000 keywords in one call."""
    payload = [{
        "keywords": keywords,
        "location_code": location_code,
        "language_code": language_code,
    }]
    response = requests.post(API_URL, json=payload, headers=_auth_header(), timeout=30)
    response.raise_for_status()
    data = response.json()

    rows = []
    for task in data.get("tasks", []):
        for item in (task.get("result") or []):
            rows.append({
                "keyword": item.get("keyword"),
                "search_volume": item.get("search_volume"),
                "cpc": item.get("cpc"),
                "competition": item.get("competition"),
            })
    return pd.DataFrame(rows)

if __name__ == "__main__":
    seeds = ["python for seo", "keyword research api", "seo automation tools"]
    df = get_search_volume(seeds)
    df.to_csv("volume_report.csv", index=False)
    print(df.sort_values("search_volume", ascending=False))

The @retry decorator from Tenacity is doing real work here, not just decoration. DataForSEO, like most SEO APIs, will occasionally return a 429 or a brief 500 under load.

Exponential backoff is the difference between a script that fails cleanly on a genuine outage and one that fails on a blip that would have resolved itself two seconds later.

The Google Ads API's KeywordPlanIdeaService is the closest thing to raw Keyword Planner data you can script against. Setup is heavier than DataForSEO.

You need a developer token approved on a Google Ads manager account, plus an OAuth2 refresh token. In exchange, the data comes straight from Google.

developer_token: "YOUR_DEV_TOKEN"
client_id: "YOUR_CLIENT_ID"
client_secret: "YOUR_CLIENT_SECRET"
refresh_token: "YOUR_REFRESH_TOKEN"
login_customer_id: "1234567890"

from google.ads.googleads.client import GoogleAdsClient

def generate_keyword_ideas(customer_id: str, seed_keywords: list, page_url: str = None):
    client = GoogleAdsClient.load_from_storage("google_ads.yaml")
    keyword_plan_idea_service = client.get_service("KeywordPlanIdeaService")

    request = client.get_type("GenerateKeywordIdeasRequest")
    request.customer_id = customer_id
    request.language = "languageConstants/1000"   # English
    request.geo_target_constants.append("geoTargetConstants/2840")  # US
    request.keyword_plan_network = (
        client.enums.KeywordPlanNetworkEnum.GOOGLE_SEARCH
    )
    request.keyword_seed.keywords.extend(seed_keywords)

    results = keyword_plan_idea_service.generate_keyword_ideas(request=request)

    ideas = []
    for result in results:
        metrics = result.keyword_idea_metrics
        ideas.append({
            "keyword": result.text,
            "avg_monthly_searches": metrics.avg_monthly_searches,
            "competition": metrics.competition.name,
        })
    return ideas

Two setup details trip people up every time. First, login_customer_id must be the manager account, not the client account, if you are accessing data through an agency's MCC structure.

Second, the refresh token has to be generated with the exact OAuth scope that Google's OAuth2 documentation specifies for Ads API access. A token generated for a different Google API scope will authenticate, but it will fail silently on every call to the Ads API.

Raw volume data is not a content plan. It is a spreadsheet with thousands of rows until it gets grouped into topics someone can actually write against.

This is where python keyword research automation earns its keep a second time. Clustering by hand past a few hundred rows is genuinely impractical, but a script can group thousands of keywords by similarity in seconds.

A lightweight approach uses term frequency vectors and cosine similarity. It needs no external embedding API, which keeps the pipeline free to run:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
import pandas as pd

def cluster_keywords(df: pd.DataFrame, n_clusters: int = 12):
    vectorizer = TfidfVectorizer(stop_words="english", ngram_range=(1, 2))
    vectors = vectorizer.fit_transform(df["keyword"])

    model = KMeans(n_clusters=n_clusters, random_state=42, n_init="auto")
    df["cluster"] = model.fit_predict(vectors)

    # label each cluster with its highest volume keyword
    labels = (
        df.sort_values("search_volume", ascending=False)
          .groupby("cluster")["keyword"]
          .first()
          .to_dict()
    )
    df["cluster_label"] = df["cluster"].map(labels)
    return df

Set n_clusters conservatively at first. Clustering too aggressively fragments closely related terms across separate topics, which defeats the point.

A good starting rule is one cluster for every 15 to 25 keywords in your input set, then adjust by hand once you see the output. Once clusters are labeled, feeding them into a proper topic cluster structure is a much shorter step than starting from an unsorted keyword list.

A script you run manually once is only a proof of concept, not python keyword research automation. A script that runs itself on a schedule and writes results somewhere your team actually looks, that is the real deliverable.

Two practical options exist here. Use a cron job on a server you control, or use a GitHub Actions workflow if your code already lives in a repository.

name: Weekly Keyword Report
on:
  schedule:
    - cron: "0 6 * * 1"   # every Monday at 06:00 UTC
  workflow_dispatch: {}

jobs:
  run_report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - run: python run_pipeline.py
        env:
          DATAFORSEO_LOGIN: ${{ secrets.DATAFORSEO_LOGIN }}
          DATAFORSEO_PASSWORD: ${{ secrets.DATAFORSEO_PASSWORD }}

For storage, writing straight to Google Sheets with gspread keeps results visible to teammates who are not developers, without building a dashboard:

import gspread
import pandas as pd

def push_to_sheet(df: pd.DataFrame, sheet_id: str, worksheet_name: str = "Keywords"):
    gc = gspread.service_account(filename="service_account.json")
    sheet = gc.open_by_key(sheet_id)
    try:
        ws = sheet.worksheet(worksheet_name)
    except gspread.WorksheetNotFound:
        ws = sheet.add_worksheet(title=worksheet_name, rows="2000", cols="10")

    ws.clear()
    ws.update([df.columns.values.tolist()] + df.values.tolist())

Every python keyword research automation pipeline that runs unattended will eventually meet an edge case a manual workflow would have caught by eye. Build for these before they show up in an early morning scheduled run with nobody watching.

• Rate limit responses (429): back off exponentially and cap total retries. Do not loop indefinitely against an API that is actively throttling you.
• Keywords with zero volume: DataForSEO and Google Ads both return None or 0 for terms that get very little search traffic. Filter or flag these rather than treating them as errors.
• Locale and currency mismatches: a location_code or language_code set incorrectly returns data that looks valid but is wrong. It will not throw an error, so validate outputs against a known baseline keyword.
• Duplicate keywords across sources: normalize casing and whitespace before merging DataForSEO and Google Ads results, or you will count the same term twice under two different labels.
• API quota running out partway through: log progress to disk as you go, so a failed run at keyword 8,000 of 10,000 does not force a full restart from zero.

None of these are exotic. They are the same categories of failure covered in Real Python's guide to the requests library, and building error handling for them up front is far cheaper than debugging a silent data quality issue three weeks later.

Two things get overlooked once a pipeline works: where the credentials live, and whether the method of collection is actually allowed. Neither is optional to think through.

Keep every credential in environment variables or a secrets manager, never in a script committed to version control. Add .env to .gitignore on day one, not after a key leaks.

If the pipeline runs in CI, store credentials as encrypted repository secrets, as shown in the GitHub Actions example above, rather than pasting them into workflow files directly.

On compliance, everything in this guide queries official, documented APIs: Google Ads, DataForSEO, Semrush, and Keywords Everywhere. Each is governed by its own terms of service and rate limits that are meant to be respected, not routed around.

That is a meaningfully different activity from scraping Google's live search results pages directly, which sits outside Google's terms and carries its own risk. Search Engine Land and Search Engine Journal have both covered enforcement actions against tools that scrape rather than query approved endpoints, a good reminder to build python keyword research automation on official APIs, not around them.

Seeing python keyword research automation run outside a code editor helps it click. For a walkthrough of querying Google's keyword data directly from a script, JC Chouinard covers the request and response structure end to end in this video.

He is a Python and SEO practitioner whose scripts and tutorials at jcchouinard.com are widely referenced in the technical SEO community.

Video credit: JC Chouinard, jcchouinard.com.

The core logic in the video maps closely to the DataForSEO and Google Ads examples above. Authenticate, send seed keywords, then parse the JSON response into rows you can actually work with in pandas.