I turned Jev into a (lousy) chatbot

I turned Jev into a chatbot, a lousy one at that. The project's name, Jevchat, is derived from the fact that it turns Jev into a chat model. The model functions by repeatedly asking Jev questions: given the user's question and the reply written so far, which symbol comes next? Jev must then provide a probability for each possible symbol, including an alphabet and an option to stop emitting.

The sampler then selects the next symbol from the normalised distribution based on these probabilities. The process continues until Jev draws the "STOP" symbol, at which point generation halts. The tool supports several alphabets and sampling strategies, but its primary purpose is for fun, and the cost is somewhat impractical. The experiment was conducted using Claude accelerated methods, and the code was implemented accordingly.

To run the chatbot, users need to input their Jev API key in a .env file next to the pyproject.toml. Alternative keys such as TYPESAFE_API_KEY are also accepted. The chatbot's output is displayed in a panel with a live readout of the generation rate, including symbols per second, characters per second, milliseconds per API call, and elapsed time.

The bot also provides the top few symbols that Jev scored at the last step, allowing users to observe the distribution the sampler is drawing from. The user can cancel the generation process by pressing Ctrl-C, with the first press stopping the generation once the in-flight request returns and keeping the partial reply. A second press aborts the generation immediately. In chat mode, the partial reply remains in the conversation history.

The chatbot offers various commands, such as /help, /alphabet [name], /temp v, /stop-bias v, /reset, /stats, and /exit. Additionally, two aspects are swappable: the distribution over the next symbol (-s/--strategy) and the alphabet (-a/--alphabet). The user can choose from different strategies, including choice, bisect, and even revert to the original symbol-based method.

However, when the width is greater than 1, temperature, top_p, and top_k stop applying, as beams are ranked by probability instead of being drawn randomly.

It is possible to ask Jev the same question using different methods. When using the --presentation symbol option, the user can choose from the symbols themselves or Jev has to append the option to the reply in its head before judging it. The latter method is considered a significant improvement in the project, as it roughly triples the top-1 score and doubles the probability mass landing on the right symbol when using character alphabets.

This improvement is achieved with fewer input tokens compared to the original symbol-based method. The project underwent 158 offline tests using a scripted fake client for the generation loop and an httpx.MockTransport for the HTTP layer, with no API key or network required. The benchmarking component, jevchat bench, performs the API calls.