Filtered should never mean deleted
We shipped a filter that threw away bad GPS readings. Months later somebody asked whether it was working, and I could not answer. The evidence was gone. That question changed how I build anything that rejects data. The obvious version, and why it rots Mileage tracking depends on trustworthy distance, and GPS lies constantly. So the first version of our cleanup did what everyone's first version…
A filter was added to discard faulty GPS data, but the decision to delete it meant the data was gone forever. Past mistakes prompted a redesign where rejections became labeled instead of deleted. Only two cases are still removed: impossible coordinates and impractical accuracy. All other data is kept and categorized into different groups based on the reason for rejection.
This approach allows for tracking the reasons behind filtering, enabling easier dispute resolution and threshold adjustments. By retaining the rejected data, it's possible to verify the accuracy of the filtering process and make necessary changes. The key takeaway is that filtering should never result in permanent deletion of data, as it hinders the ability to audit the system's effectiveness.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
