The head of your CSV is lying: how 9,291 invoice numbers almost vanished
Real transaction data is never clean — and the worst part is that it looks clean. This is a short story from a real dataset (UCI Online Retail: 541,909 e-commerce transactions) about the quietest way to destroy data: silent type coercion. All numbers below come verbatim from an executed notebook. The head looks perfect Peek at the first rows of the file and InvoiceNo parses as clean integers —…
The head of a CSV file may appear clean, but hidden within its numbers lies a potential data loss issue. In a study using the UCI Online Retail dataset of 541,909 e-commerce transactions, 9,291 invoice numbers were silently lost due to type coercion. These invoice numbers starting with 'C' were transformed into NaN values when coerced to numeric, erasing an entire class of business events - cancelled orders worth 8.4% of everything sold.
The issue is particularly insidious as the pipeline continues to function without any exceptions or crashes. The lost invoice numbers include the dataset's top-selling product, 'PAPER CRAFT, LITTLE BIRDIE', which was actually a single order of 80,995 units that was fully cancelled within minutes of being placed. The fix involves treating InvoiceNo as an identifier rather than a quantity, a task that can be accomplished by revealing the library's assumptions and creating a replayable plan.
This approach ensures that cancellation invoices are preserved, and downstream revenue numbers remain accurate. The key lessons from this incident include never trusting the head of a file, measuring the parse rate of the entire file, and explicitly declaring identifiers. By following these guidelines, data issues like silent type coercion can be avoided, allowing for more reliable and accurate data analysis.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
