The LLM in my app is not allowed to decide anything
I build software in the single worst domain for LLM truthfulness: fortune-telling. A BaZi (Chinese Four-Pillars astrology) reading app, where the model's job is to sound like a wise master — and where user reviews of competing AI products converge on one complaint: "pure nonsense." An LLM asked to "read a birth chart" will hallucinate chart elements that aren't there, invent rules that don't…
In the world of AI-powered fortune-telling, using Large Language Models (LLMs) to generate readings can lead to problematic hallucinations. A BaZi (Chinese Four-Pillars astrology) reading app is plagued by such issues, where the model generates nonsensical chart elements, invents nonexistent rules, and confidently delivers it all. To combat this, the engineering solution is clear: the deterministic engine computes the facts, while the LLM only decides how to present them.
The first challenge lies in handling uncertain input, specifically the birth hour. Since this pillar is essential for generating accurate readings, the solution is to compute all possible charts and take the intersection of facts that hold in every candidate chart. This ensures that only concrete, consistent facts are included in the output.
Another issue arises when the model generates output, which needs to be validated to ensure it adheres to the rules. The validator scans the output for invented pillars and checks for forbidden patterns, such as fatalistic or fear-mongering constructions. If any violations are detected, the model is flagged for review.
Additionally, the validator checks for the presence of star and deity terms not computed by the engine itself. If any such terms are found, the output is flagged for review. This ensures that the model only generates factual information based on the provided chart, without introducing fabricated elements.
Lastly, the model's prompt is written in Chinese, but it also serves English readings. To avoid the model generating hybrid sentences in production, an explicit paragraph must be included stating that quoted sentences are tone demonstrations only and must not be copied or translated. The output should contain no Chinese characters except for glossed pinyin.
Implementing these guardrails not only improves truthfulness but also reduces costs, as the model writes fewer words of stylized prose. Moreover, consistent readings are generated for users with the same chart, eliminating stylistic variations. By applying these measures, AI-powered fortune-telling apps can maintain accuracy and credibility, even in a domain with minimal external ground truth.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
