You decide when done is done: Manual offset commits in Kafka.

Manual offset commits in Kafka allow you to decide precisely when messages are considered processed and permanently committed to the system. This approach eliminates the risk of data loss that can occur when using auto-commit, where messages are marked as processed before database writes are completed.

The manual commit architecture, implemented in the wkafka library, ensures that data remains intact even if a consumer crashes during the processing of a message. By committing offsets only after successful database writes, the system maintains consistency and accuracy.

In the `on_financial()` function, after saving the message to the database, the `msg.commit()` method is called. This ensures that the offset is only advanced once the message has been fully processed and persisted to storage.

This approach provides several benefits over auto-commit:

1. Zero data loss: Premature offset advancement is prevented, and messages are reprocessed upon recovery if they are uncommitted.

2. Clean replay capability: Messages that were in-progress at the time of a consumer crash can be reprocessed immediately.

3. Compatibility with various Python versions (3.9 to 3.14) and strict typing support.

4. Proven reliability, as it has been tested and verified against real Kafka broker clusters.

To implement manual offset commits using wkafka, you can disable auto-commit by setting `auto_commit=False` when creating a Kafka consumer instance. Then, within your consumer callback function, you can move the offset commit to a separate step after the message has been successfully processed and written to the database. This architecture gives you full control over when offsets are committed, ensuring data integrity and eliminating the risk of silent data loss.