I measured my AI search. The numbers were humbling 📏
In part 1 I built Because I liked : you describe something you loved and why , and it finds films, shows, books and games that match the why , then explains each pick. Part 1 ended with "the misses are the most useful thing you can send me". People sent them, across several platforms. This post is about what I did with them: building a test set, a fix that worked, a fix that should have worked…
In this story, the author conducts experiments with an AI search algorithm to understand its limitations and improve its performance. The first lesson learned is to stop judging search results solely by looking at them. By writing a test set of searches and comparing the AI's results to expected titles, the author finds that the recall@12 metric, which measures how many expected titles appear in the top 12 results, yields a mean score of 28%.
The second lesson is that similarity alone cannot distinguish a classic from its clone. The author provides an example of a search query asking for a hard but fair game where dying teaches you something. The AI returns a list of games that fit the description, but only one of the expected titles (Celeste) appears in the top 12 results.
To address this issue, the author introduces a tiebreaker based on audience opinion, which improves the recall@12 score to 40%. However, the author also discovers that the AI struggles to differentiate between classics and their clones due to similarity in descriptions. The third lesson learned is that offline improvements may fail when tested on real user queries.
The author provides an example of a search query asking for an analytical drama with a sarcastic stoic lead character. The AI returns a list of titles that fit the description, but none of the expected titles appear in the top 12 results. The author then tries to improve the AI's performance by appending a second fingerprint vector to each title, containing information about the lead character.
While this approach works offline, it fails when tested on real user queries. Finally, the author concludes that users search in ways that are not easily predictable. In one example, a user simply lists their favorite shows without providing a clear why. The author discovers that the AI drops titles that have been named by the user, even if they fit the search criteria.
This highlights the need to adapt the search algorithm to accommodate a variety of search patterns and user behaviors.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
