Introducing Lev

Jev, introduced by TypeSafe a few weeks ago, sparked excitement due to its ability to play Doom in real-time. Jev functions as a hosted classifier model, which processes unstructured state input and generates typed probabilistic decisions in a single pass, completing the task in just a few hundred milliseconds. The model's structured responses eliminate the possibility of hallucination, although it might still provide incorrect classifications.

The concept behind Jev is not entirely novel, as numerous open-source projects have implemented similar ideas. Curiosity led the author to attempt implementing their own Jev-like model, known as Lev, using Jolt. This decision engine offers the same wire API as Jev, aiming to explore the performance of open weights models running locally and address potential shortcomings.

The core of Lev's classifier tier involves decoding input, marking options, and assigning probabilities to each option. Unlike generation, which is ideal for open-ended tasks, classification requires a known set of answers and is comparatively more efficient. The models behind Lev and Jev read text bidirectionally in one pass, significantly reducing computational costs compared to encoder models like ModernBERT.

However, these models do have limitations. They struggle with deductive reasoning, frequently misinterpret adversarial phrasings, and tend to overconfidence in their answers. Moreover, they often choose the "insufficient-evidence" option even when there's enough information to answer the query. These failures were intentionally included in Lev's adversarial set to test model robustness.

Another issue is overconfidence, where the model's high certainty doesn't always align with actual accuracy. Calibration errors, indicating the mismatch between predicted confidence and actual correctness, have been observed. For instance, Jev displayed a calibration error of 0.246 in an independent measurement. Despite these limitations, the approach proves effective for specific scenarios, such as when integrated with a different type of model to handle complex tasks that the classifier can't manage alone.