Zenoh's put is fire-and-forget, get isn't — a read-after-write race in Elixir

This article dives into an issue observed while using Zenoh's Elixir bindings for its put/get storage feature. While the functionality mostly worked, there were moments when the data retrieved was one step behind the expected state. To investigate, a simple loop was created that performed a put followed by a get on the same key.

Out of 2000 iterations, around 78 (3.9%) of the runs produced a "stale!" message indicating that the retrieved payload was not the one that was put. However, querying again almost always returned the correct value, suggesting a brief lag before the write became visible.

The Zenohex.Session.put/4 function is a thin Rustler wrapper around zenoh-rust's put. The NIF implementation queues the message for local processing but does not wait for the remote side (the zenohd router storing the data) to receive and apply it. In contrast, the Zenohex.Session.get/4 function is registered as a DirtyIo NIF and genuinely waits for a response from the remote side within a timeout, making it a true request/response operation.

This discrepancy between put (cast-like) and get (call-like) behavior can lead to race conditions if one assumes the effect of put is immediately visible after the get is performed.

Zenoh itself has acknowledged this issue in their issue tracker (zenoh/zenoh#2511), where it is pointed out that Zenoh's pub/sub path is fire-and-forget, meaning the session.put() returns when the message is sent, not when it's stored. The recommended solution is to wrap put and get in a small module that performs the put, then immediately gets the same key back and only returns once the written payload can be read.

This approach introduces a retry mechanism at a short interval until a timeout is reached, ensuring that the put is indeed successful before relying on the subsequent get.