{
  "id": 10310555,
  "title": "You decide when done is done: Manual offset commits in Kafka.",
  "url": "https://urgent.news/2026/09/27/you-decide-when-done-is-done-manual-offset-commits-in-kafka",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-27T23:06:07.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/william_rodriguez_65a5898/you-decide-when-done-is-done-manual-offset-commits-in-kafka-ha8"
  },
  "original_language": "en",
  "account": "Manual offset commits in Kafka allow you to decide precisely when messages are considered processed and permanently committed to the system. This approach eliminates the risk of data loss that can occur when using auto-commit, where messages are marked as processed before database writes are completed.\n\nThe manual commit architecture, implemented in the wkafka library, ensures that data remains intact even if a consumer crashes during the processing of a message. By committing offsets only after successful database writes, the system maintains consistency and accuracy.\n\nIn the `on_financial()` function, after saving the message to the database, the `msg.commit()` method is called. This ensures that the offset is only advanced once the message has been fully processed and persisted to storage.\n\nThis approach provides several benefits over auto-commit:\n1. Zero data loss: Premature offset advancement is prevented, and messages are reprocessed upon recovery if they are uncommitted.\n2. Clean replay capability: Messages that were in-progress at the time of a consumer crash can be reprocessed immediately.\n3. Compatibility with various Python versions (3.9 to 3.14) and strict typing support.\n4. Proven reliability, as it has been tested and verified against real Kafka broker clusters.\n\nTo implement manual offset commits using wkafka, you can disable auto-commit by setting `auto_commit=False` when creating a Kafka consumer instance. Then, within your consumer callback function, you can move the offset commit to a separate step after the message has been successfully processed and written to the database. This architecture gives you full control over when offsets are committed, ensuring data integrity and eliminating the risk of silent data loss.",
  "summary": "You decide when done is done: Manual offset commits in Kafka. Day 06 of the WKafka Open-Source Engineering Series. If your consumer crashes right after reading a message with auto_commit=True, that message is gone forever. Manual commit puts you in control. The Pain Points We Faced Auto-commit marking messages as 'processed' before database writes finish Silent data loss when consumer crashes…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 2,
    "also_reported_by": [
      {
        "outlet": "Dev.to",
        "title": "You decide when the deed is done: Manual commit of offsets in Kafka.",
        "url": "https://urgent.news/2026/09/27/tu-decides-cuando-hecho-es-hecho-commit-manual-de-offsets-en-kafka",
        "published": "2026-09-27T23:06:27.000Z"
      }
    ]
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}