How we tested an AI classifier on marketplace data: two error rates, not one accuracy number
Disclosure: we build CompsAPI, the sold-price API whose filter is measured in this post. It was drafted with AI assistance, and every figure below comes from CompsAPI's published test notes. The labels in the test were made by us, which is a limit I come back to at the end. "The model is 92% accurate" is a sentence that sounds like a result and tells you almost nothing. This post takes one real…
The article examines the limitations of a single accuracy percentage when evaluating an AI classifier used on marketplace data. It uses CompsAPI, the sold-price API, as the main example, noting that its test numbers reveal more than just a simple accuracy figure.
Key points from the source:
1. CompsAPI's classifier labels marketplace sales as item, lot, parts, accessory, box_only, mystery, replica, digital or other. Anything not item is considered junk.
2. The classifier was tested on three key points: two independent labels per listing, listings not used in training, and testing on marketplaces never seen by the model during training (e.g., Reverb, Mercari, Poshmark).
3. On these unseen marketplace searches, the classifier achieved about 92% accuracy. However, this number hides two types of errors:
- Real items wrongly removed as junk (5.8%)
- Junk wrongly kept as item (10.8%)
4. When broken down further, on the main marketplaces:
- 1.5% of real items were wrongly removed
- 1% of junk items were wrongly kept as item
5. The accuracy number is misleading because it doesn't account for the base rate of junk in the data. On eBay, where junk is more common, the classifier performs better compared to a baseline that labels everything as item.
6. The choice of classifier depends on the intended use:
- For median price calculations, the 10.8% junk miss rate is more important
- For finding specific deals, the 5.8% real item removal rate is more critical
7. The article provides a simple Python script to test the classifier on any search, allowing users to manually evaluate the model's confidence and the possible errors.
8. Finally, it emphasizes that these numbers should be taken with caution and used as a starting point for further manual examination, as the classifier's performance can vary greatly depending on the specific listings it encounters.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.