My AI visibility score was 29.2. Here is what the one number was hiding
The daily run of my AI citation tracker finished on 21 October with a blended score of 29.2. On its own that reads like a mediocre week. Split by question type it is something else entirely, and the split is the only part of it I would trust. How the number is built Every prompt goes to two web-grounded models, Gemini and OpenAI's search model. Each answer is scored on a 0 to 3 scale for how well…
My AI citation tracker posted a blended score of 29.2 on October 21st, but this single number can be misleading. By examining the breakdown by question type, a different picture emerges. Each prompt is sent to two web-grounded models, Gemini and OpenAI's search model, which score the answers on a 0-3 scale for how well they identify and describe the entity being tracked. On that day, the blended score was 28 out of 96 points, calculated from 16 prompts, with two models answering 32 questions, each worth three points.
When analyzed by question bucket, the story changes. The "Direct" bucket, where the entity is asked about by name, accounted for 88.9% of the points and included 6 out of 32 answers. This bucket alone made up more than a third of the total. The "Genre recommendation" bucket contributed 50.0%, equaling 12 of the 32 answers. The other buckets, including "Genre," "Comparison cluster," and "Research," scored 0%, leaving a total of 29.2.
This blended score is largely a reflection of the mix of question types rather than the models' ability to handle the entity consistently. Adding more direct questions that scored similarly to today's performance would increase the blended score to 38.6 without changing the models' performance. Smaller buckets are less reliable, as one answer can change the percentage by up to 25 points.
The model gap between Gemini and OpenAI's search model is smaller than it appears at first glance - a 12.5-point difference on that day corresponds to just two fully correct answers.
Instead of focusing on the blended score, it's more insightful to look at the points, possible points, and the number of answers (n) for each bucket. Reporting these metrics without relying on percentages helps avoid misleading interpretations. For instance, the "Direct" bucket asks whether the model knows who the entity is when asked directly, while the "Genre" bucket assesses whether the model can understand the entity without being prompted by name.
These are distinct problems with different solutions, and averaging the two does not provide a complete picture.
On October 21st, the honest reading is that both models performed well when asked by name, but struggled with open genre questions. Tracking this second line - how the models handle the entity when not specifically named - could provide more meaningful insights when analyzed week to week with more prompts per bucket. It's essential to remember that one day of data may not represent the true performance of the models, and pooling data over a week or more can provide a clearer view before comparing different periods.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
