Three of the First Four Alerts Were the Question's Fault
Last week I turned my data audit into a build step : a check that runs before anything else and fails the build when the database and any static copy of my travel site's legal-status data disagree. It ended the era of the site contradicting itself. It did nothing about the site agreeing with itself on something false. That's not a hypothetical. The most expensive error the whole project found was…
Three of the initial four alerts raised by the new data audit system were due to a discrepancy between the model's response and the database. The alerts indicated that certain states graded as medical by the database were considered illegal by the AI model. However, upon investigation, it was discovered that the model was correct, and the database was using a different grading criteria.
This highlighted the importance of using a consistent rubric when comparing data sources. The issue was resolved by updating the prompt to include the dataset's written definition, ensuring that the model graded the data using the same rubric as the database. One alert, however, proved to be a real issue, involving a state that was incorrectly graded as medical in the database despite the model stating that it was decriminalized.
The dataset had a written definition that prioritized possession laws over medical programs for individuals, and this discrepancy was identified and corrected in the database and static files.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
