I ran an SEO experiment on my side project and broke my own control group

In September, a side project featuring a tarot card reference in four languages encountered an unexpected issue. The Ukrainian card pages were positioned 6-9 in Google search results, but received minimal clicks. Over three months, these pages garnered 2,614 impressions and only 2 clicks. On September 21, all 78 Ukrainian meta descriptions were rewritten to improve clarity, while the other three languages remained unchanged as a control group. The experiment was to be monitored for two weeks before any conclusions were drawn.

Two weeks later, the results revealed a discrepancy. Impressions had doubled to 463, but the average position worsened to 21. At first glance, this appeared contradictory, as higher positions typically correlate with more clicks. However, further investigation unveiled that Google had started showing previously unimpressed pages, pushing them into positions 20-35.

This phenomenon dragged down the average position, even though the existing high-ranking pages remained relatively stable. When only considering pages with average positions 10 or better, the impression count stood at 157 and the click count remained at 2, resulting in a click-through rate (CTR) of 1.3%. This CTR is comparable to what one would expect for pages ranked 6-8 on mobile devices.

The experiment was compromised due to a rule violation. The original plan was to compare CTR only if page positions remained stable throughout the testing period. However, the single most important page, "Six of Wands," experienced a drastic shift in rankings—from 7.4 to 37.5—during the experiment's timeframe. Additionally, an FAQ block was added to all 312 card pages across all four languages on the same day as the meta description changes.

Consequently, the control group became invalidated, as the Russian card pages saw a decline in impressions from 379 to 133, and the cause of this decline was unclear.

Moving forward, it is essential to maintain the integrity of a control group. A control group must not experience any alterations, even those that seem innocuous. The Ukrainian meta description experiment serves as a cautionary tale of how seemingly minor changes can disrupt a carefully designed study.

A surprising outcome emerged from the experiment: the French "Card of the Day" page achieved remarkable results. On September 20, when the main local query "Tarot du jour" was added to its title, the page garnered 18 impressions and 5 clicks over the subsequent two weeks. This performance accounted for half of all clicks on the site during the same period.

It is hypothesized that the page title plays a crucial role in attracting clicks, while the meta description only becomes relevant once the page is clicked upon. The next step in the experiment will involve testing this theory with a control group that remains untouched.

An interesting observation concerning Yandex search engine occurred. Between September 21 and October 5, Yandex excluded 76 out of the 88 Russian pages marked as "low value." On October 1, all 76 affected Russian pages were resubmitted for recrawling through Yandex Webmaster tools. This action resulted in a gradual increase in search engine rankings for these pages: 12 on October 1, 21 on October 2, and 30 on October 5.

Despite no new links or changes in analytics, the recrawl request seemed to significantly impact the pages' visibility. The same update also affected 20 English card pages with a similar "low value" status, which last appeared in Yandex's search results between September 21 and 23. This suggests that Yandex may pay closer attention to recrawl requests than previously assumed. The next step will involve monitoring the situation in mid-October.