{
  "id": 5070524,
  "title": "Out-of-Order Isn't Late: What Building an IoT Ingestion Engine Taught Me About Event-Time Chaos",
  "url": "https://urgent.news/2026/09/02/out-of-order-isnt-late-what-building-an-iot-ingestion-engine-taught",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-02T11:05:38.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/0xkikiola/out-of-order-isnt-late-what-building-an-iot-ingestion-engine-taught-me-about-event-time-chaos-5ci2"
  },
  "original_language": "en",
  "account": "Building an IoT ingestion engine exposed the fragility of event-time chaos. The core challenge was preserving state correctness when sensor events arrived out of order, duplicated, or late. Three key invariants had to hold: deduplication, out-of-order handling, and a late-arrival rejection policy. A small test case confirmed the logic with 3 accepted readings, 2 late rejections, and correct aggregates. Scaling up to thousands of events, the dashboard showed inflated late rejection numbers, suggesting a bug. The initial explanation was that stacking and legend formatting caused the visualization issue. However, querying raw Prometheus metrics revealed the actual numbers: 2,980 accepted events and only 20 late rejections. The Grafana visualization was misleading due to series stacking and legend formatting misconfiguration. Out-of-order handling worked correctly, with the service managing all shuffled packets. The lesson is to trust raw data over visualizations when discrepancies arise. Always verify metrics against domain invariants and confirm the root cause before accepting any explanation.",
  "summary": "The scary part about a bug isn't when the numbers look wrong. It's when they look wrong in a way you can talk yourself into accepting. That's what nearly happened building ingest-sentinel , a telemetry ingestion service exploring how to preserve state correctness when sensor events arrive out of order, duplicated, or late. The Setup A FastAPI ingestion endpoint accepts timestamped events from…",
  "key_points": [
    "IoT ingestion engine revealed fragility of event-time chaos",
    "Three core invariants: deduplication, out-of-order handling, late rejection policy",
    "Grafana visualization misleading due to series stacking and legend formatting"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}