How do you debug something that is allowed to be wrong?
Yesterday afternoon a runtime I wrote created 70 paragraphs on the same draft article. It was supposed to create one. The runtime was an experiment in explicit state, the kind where the model gets the task spec, a small JSON, the last observation, and nothing else; it had a commit rule I was proud of: nothing counts as done until a read-back confirms it. After every paragraph it read the draft…
Yesterday afternoon, a runtime that I had created produced 70 paragraphs on the same draft article. The runtime was designed to generate one paragraph, but it failed to do so. It was an experiment in explicit state, where the model received the task specification, the last observation, and nothing else. After every paragraph, it re-read the draft, found no paragraph, and asked the model for another one.
This process repeated seventy times. My initial suspicion was that the model was the source of the problem, as it was the last component to produce output. After investigating, I discovered that the issue lay in the read-back process, which served the paragraph list from a cache with a 60-second lifetime. The model had behaved as expected, following the instructions given by the runtime, while the runtime had received incorrect information from its own read path.
The key to debugging this issue was to ask three crucial questions in a specific order: what was the runtime allowed to get wrong, which layer was responsible for the error, and how long did the error persist. By answering these questions, one can effectively debug a system that is allowed to be wrong.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
