cached_tokens is 0 because your system prompt isn't stable

In the story "cached_tokens is 0 because your system prompt isn't stable" by Chad Priest, the author discovered a significant issue with their system prompt while using the Fireworks service. The problem stemmed from the system prompt not being stable, leading to the same prefix being used repeatedly, which resulted in full pricing for the same content. This issue cost the author more money than anticipated.

The author initially suspected session affinity, but it turned out not to be the main culprit. They then found that caching was automatic and total, requiring the prefix to be identical for the cache to work. They identified two main problems: the ~24KB workspace bootstrap rode turn 1 and vanished on turn 2, and the ~15K query-dependent memory recall was concatenated straight onto the system message.

To fix this, the author implemented a branch in the code, freezing the system prompt to make it cacheable. They then relocated the memory for system use to a late system turn, before the current user message. This allowed the cached prefix to be recognized and utilized effectively. The author also fixed the bootstrap issue by resending it every turn, resulting in a more efficient billing structure.

In conclusion, the key rule for a stable system prompt in a frozen-prefix design is to treat the system prompt as an invariant data structure, with any query-dependent elements placed within the message array. Understanding the discount provided by caching is crucial, as the effectiveness of a cache can vary significantly depending on the amount of input it can handle.