Keyword Research with AI Agents and Google Search Console

In the world of digital marketing, manual keyword research has long been a tedious and disconnected process. This article explores a modern approach that leverages AI agents and Google Search Console data to automate keyword clustering, intent analysis, and content gap detection for websites.

Manual keyword research is plagued by guesswork. Researchers rely on competitor sites, keyword tools with volume estimates, and predictions about what users might search for. However, Google Search Console, a powerful tool, already provides actual search query data grouped by page, complete with impressions, clicks, and average position. Unfortunately, extracting actionable insights from thousands of rows of query data is a daunting task.

This is where AI agents come into play. The system consists of three main components: a data source (Google Search Console Search Analytics API), an AI agent (a coding agent with access to GSC data and the ability to execute scripts), and the cluster output (a JSON mapping of queries to pages, intents, and optimization opportunities).

To begin, you need API access to Search Console data. This requires setting up a Google Cloud project with the Search Console API enabled, configuring Application Default Credentials, and adding the Search Console property to your account. The necessary ADC credentials are obtained using the command `gcloud auth application-default login`. Additionally, the quota project must be set (required since 2026) using the command `export GOOGLE_CLOUD_QUOTA_PROJECT = your-project-id`.

The core of the workflow is a Node.js script that fetches GSC data and processes it. The script authenticates with the GSC API, fetches query data for a specific period, and handles pagination for larger datasets. It then maps routes to pages, aggregates queries by page, identifies intent clusters based on query phrasing and page relevance, and detects potential cannibalization when multiple pages rank for similar queries.

The clustering logic involves several steps: mapping queries to canonical routes, aggregating queries by page, identifying primary, secondary, and long-tail clusters, and detecting cannibalization through a detectCannibalization function. The final output is a JSON file that maps each page to its keyword clusters, serving as the foundation for all content and SEO decisions.

By automating keyword clustering, intent analysis, and content gap detection, this AI-driven approach streamlines the keyword research process, making it more efficient, accurate, and aligned with real user behavior.