OpenJev benchmarks against a value it never explains
OpenJev runs a local language model in your browser and does one specific thing with it. Given a prompt and a fixed set of options, it reads the model's logits for those option tokens and applies a softmax across only them. There is no decoding and no generated text; a single forward pass yields a probability distribution over the answers you allowed. The site calls these "semantic ifs": a branch…
OpenJev measures the performance of a local language model by comparing two methods: a direct readout of logits for a set of options versus the model generating JSON with probabilities. The direct readout is a single forward pass that yields conditional probabilities over the displayed options, which is cheaper than generating prose.
The option labels range from A to T, with weights coming from quantized GGUF models served through wllama. The demo runs MiniCPM5 2B as the default model, with Qwen3 0.6B for phones and a 4B option requiring more memory. The quality table scores local models on balanced accuracy against two reference columns, TypeSafe and Jev. However, Jev's published value lacks a clear definition and source, making it harder to assess the accuracy of the comparison.
The direct readout method is transparent and can be tested on personal hardware, while the accuracy numbers are less reliable due to potential divergence between quantized browser weights and native checkpoints.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
