Out-of-Order Isn't Late: What Building an IoT Ingestion Engine Taught Me About Event-Time Chaos
The scary part about a bug isn't when the numbers look wrong. It's when they look wrong in a way you can talk yourself into accepting. That's what nearly happened building ingest-sentinel , a telemetry ingestion service exploring how to preserve state correctness when sensor events arrive out of order, duplicated, or late. The Setup A FastAPI ingestion endpoint accepts timestamped events from…
Building an IoT ingestion engine exposed the fragility of event-time chaos. The core challenge was preserving state correctness when sensor events arrived out of order, duplicated, or late. Three key invariants had to hold: deduplication, out-of-order handling, and a late-arrival rejection policy. A small test case confirmed the logic with 3 accepted readings, 2 late rejections, and correct aggregates.
Scaling up to thousands of events, the dashboard showed inflated late rejection numbers, suggesting a bug. The initial explanation was that stacking and legend formatting caused the visualization issue. However, querying raw Prometheus metrics revealed the actual numbers: 2,980 accepted events and only 20 late rejections. The Grafana visualization was misleading due to series stacking and legend formatting misconfiguration.
Out-of-order handling worked correctly, with the service managing all shuffled packets. The lesson is to trust raw data over visualizations when discrepancies arise. Always verify metrics against domain invariants and confirm the root cause before accepting any explanation.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
