Two 32 MB tokenizers: hunting a memory floor in a Rust proxy

Two 32 MB tokenizers were found to be consuming excessive memory in a Rust proxy called cliproxy-rs. This proxy acts as a local server for various tools, including Codex CLI and Claude Code, forwarding requests to configured APIs. While testing on a small Linux box, the proxy's resting memory increased significantly from 26 MB to 89 MB over seven hours, with peaks reaching up to 400 MB. This type of memory leak typically starts with an upward trend that never decreases.

To investigate the issue, a soak test was conducted using a mix of Claude and Codex requests of 1 to 2 MB, with four sessions lasting an hour. The results showed that the 0.2.0 version of cliproxy-rs consumed 128 to 145 MB of memory at rest, which was deemed excessive for a proxy without caching functionality.

Heaptrack analysis revealed that 67.1 MB of memory was still allocated after two batches of 100 field-mix turns, with 63.8 MB of that attributed to two o200k_base tokenizer encoders - one built by Claude count_tokens and the other by Codex count_tokens. These encoders were lazily built and never freed, leading to up to seven copies existing and consuming a total of 164 MB of memory.

The fix implemented a shared encoder per encoding, built only when a count first needs it. After implementing the fix, the resting memory dropped to 80.6 to 97.6 MB, and the peak memory usage reduced from 169.0 MB to 169.0 MB. However, the issue of memory growth under certain conditions remains unsolved, with potential causes including the field server building encoders hours apart and the impact of request shaping on memory usage.