I benchmarked 8 LLMs for a niche production app. The flagship cost 5.8x more - and lost.
My app generates personalized readings for BaZi — Chinese "Four Pillars" birth charts. Every reading is an LLM call, every call costs money, and the domain is full of trap terminology that models love to botch. So before launch I benchmarked every candidate model on my actual workload, and then built the routing layer around what the benchmark found. The results generalize to any "LLM in a niche…
The report detailed the results of benchmarking eight language models (LLMs) for a niche application, specifically an app that generates personalized Chinese Four Pillars birth charts. The app's success depended on accurate language model outputs, which were critical due to the domain's specialized terminology. The author discovered that while the flagship model was five and eight times more expensive than other options, it failed in domain accuracy and cost per reading.
Mid-tier models, although cheaper, had accuracy issues and used more reasoning tokens, making them unsuitable for the app. Ultimately, the article concluded that a cheap-and-accurate small model was the best choice for the free tier, and a mid-tier model was the most cost-effective option for paid users. The routing system used to select models was based on evaluation results, and the article emphasized the importance of understanding providers' error taxonomy to handle failures effectively.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
