The Audit Became a Build Step
I have audited my travel site's legal-status dataset twice. The first audit found the same fact stored in four places, drifting independently. The second found the drift had a fifth home — the prose — and ended with me writing that a half-derived page contradicting itself "is not a risk, it's a schedule." Six weeks later the schedule delivered again. A country's status was wrong in one file and…
The audit of the travel site's legal-status dataset was conducted twice, each time revealing inconsistencies in stored facts. The first audit found the same fact stored in four places, with a fifth occurrence in prose. The second audit discovered that the drift had a fifth home - the database itself. A country's status was incorrectly recorded in one file while being accurate in another. A different country was placed in the wrong continent in the database, causing dead links on live pages.
Both audits were essential and accurate when executed. However, the problem lies in the fact that an audit only certifies a moment. The dataset has multiple writers, including an admin panel, an agent editing files, the reporter, and occasional migrations. After the auditor's departure, no system holds agreement among these writers.
The check is not just a session; it is a script that runs before anything else in the build. Two design decisions are crucial: loading the app's real modules instead of a regex scrape and using the app's own slug transforms, name normalization, and merge logic. These ensure that the check accurately reflects the rendering code's behavior.
The check distinguishes between errors and warnings. A contradiction between two sources where a reader can reach - the database says one status, and the search dropdown says another - is an error and triggers a build failure. On the other hand, a country that exists in the static files but has no database row is a warning, indicating an unmanaged page that will be edited in the future.
The check also has two important rules: failing to run is a failure, and running silently without verifying is worse than not running at all. If the database is unreachable, the check exits with a nonzero status and prints "this is not a pass."
The check initially produced 28 known conflicts, which couldn't be resolved by blocking every deploy. Therefore, known differences live in a baseline file, but any baseline entry that stops matching reality is reported. The baseline starts at 28 and eventually shrinks to 0, forming a contract for any future disagreements.
The audit exposed the true sources of truth in the dataset, revealing different authorities for the same facts on various pages of the site. This realization led to the creation of a definition that resolves ambiguities and clarifies which copy should be trusted. This definition has proven beneficial over time, ensuring consistency in data interpretation and maintenance.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
