I Analyzed 1,000 Keywords with Python to See How AI Overviews Relate to Google Rankings

In this research project, I examined the relationship between Google AI Overviews and organic search rankings by analyzing approximately 1,000 keywords. Initially, I gathered a large set of product-category keywords from Semrush, which initially totaled nearly 100,000 keywords. After removing irrelevant terms, I narrowed the dataset to around 50,000 keywords.

To streamline the analysis, I focused on queries in the format “product category + recommended,” such as “T-shirt recommended” or “hair dryer recommended.” This allowed me to assess the likelihood of specific brands appearing in AI Overview responses.

The primary goal of the research was to determine the extent to which traditional Google rankings affect visibility in AI-generated answers. While some brands seem relatively SEO-unfocused yet frequently recommended by AI platforms, others perform well in Google Search but are less often mentioned in AI-generated responses. This led to questions about the impact of ranking positions on visibility in AI search.

Should brands aim for the Top 3, Top 5, or even lower positions like 15th? These insights could inform SEO strategies in the era of AI Overviews.

I used Python, LangChain, and LLM APIs to automate the analysis of the collected data, which included Google organic rankings, AI Overview responses, citations, ads, and shopping results. By extracting and linking these datasets, I could compare the two sources to identify patterns and draw conclusions about the relationship between traditional search rankings and AI Overview visibility.