LLM Evaluation for Software Engineers Without an ML Background: the 90+ Checks We Actually Run
I run a content pipeline where AI writes every article — and where AI is, on principle, not trusted. Before any piece ships to our site, it survives more than ninety separate verifications: research checks, fact cross-referencing, a deterministic validator with dozens of rules, integration guards. We never sat down and said "let's build an LLM evaluation harness." We sat down and said "let's not…
The software engineering team behind a content pipeline that uses AI to write all articles runs over ninety separate verifications for each piece before it is published. These verifications include research checks, fact cross-referencing, a deterministic validator, and integration guards. The team didn't intentionally create an LLM evaluation harness; instead, they added checks to prevent embarrassing text from being published.
The stack progresses from code verification (which is free and consistent) to LLM judgment (which is expensive and variable). The deterministic layer performs structure, range, forbidden patterns, link integrity, and metadata checks. The LLM's role is limited to claim verification, determining if a claim is supported by the researched fact sheet.
The team emphasizes that a narrow focus is crucial; asking the LLM to rate the entire article is unreliable. They do not use a general quality judge and only use deterministic checks or closed factual lookups, as these are the most reliable for production.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
