Measuring whether AI search names your product: six ways the measurement lied to me
I build DOS AI (dosai.pro), an AI assistant that answers a small business's customers in WhatsApp, Instagram and Telegram and keeps the leads in a built-in CRM. Search consoles show where a page ranks. Nobody shows you what an AI assistant says when a buyer asks it for a recommendation, so I wrote a script for that. The script is under 300 lines. Getting it to tell the truth took longer than…
The author created an AI assistant called DOS AI and developed a script to measure how many times the AI mentions their product based on user queries. Initially, the script counted an answer as a mention if the text contained the company name. However, this led to inaccurate results as the AI would sometimes repeat the question instead of mentioning the domain.
The author then adjusted the script to only count a mention if the answer cited the company domain. They also encountered issues with the script's behavior when no arguments were provided, leading to incorrect results. To fix this, the script now exits with an error code when no arguments are given. Additionally, the author discovered that running the script multiple times at short intervals could produce inconsistent results due to the AI's decision-making process.
To address this, the author requires a change in both the unbranded count and the number of runs for a result to be considered real. The author found that their AI assistant performs well by domain but struggles by name alone when it comes to identifying the product. They also noted that their published articles and sitemap had not significantly impacted the AI assistant's results.
The next step for the author is to create pages on other domains that mention the product to improve the AI assistant's performance.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
