{
  "id": 16115313,
  "title": "OpenJev benchmarks against a value it never explains",
  "url": "https://urgent.news/2026/10/22/openjev-benchmarks-against-a-value-it-never-explains",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-22T22:25:07.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/randomchaos/openjev-benchmarks-against-a-value-it-never-explains-4ng"
  },
  "original_language": "en",
  "account": "OpenJev measures the performance of a local language model by comparing two methods: a direct readout of logits for a set of options versus the model generating JSON with probabilities. The direct readout is a single forward pass that yields conditional probabilities over the displayed options, which is cheaper than generating prose. The option labels range from A to T, with weights coming from quantized GGUF models served through wllama. The demo runs MiniCPM5 2B as the default model, with Qwen3 0.6B for phones and a 4B option requiring more memory. The quality table scores local models on balanced accuracy against two reference columns, TypeSafe and Jev. However, Jev's published value lacks a clear definition and source, making it harder to assess the accuracy of the comparison. The direct readout method is transparent and can be tested on personal hardware, while the accuracy numbers are less reliable due to potential divergence between quantized browser weights and native checkpoints.",
  "summary": "OpenJev runs a local language model in your browser and does one specific thing with it. Given a prompt and a fixed set of options, it reads the model's logits for those option tokens and applies a softmax across only them. There is no decoding and no generated text; a single forward pass yields a probability distribution over the answers you allowed. The site calls these \"semantic ifs\": a branch…",
  "key_points": [
    "OpenJev compares two methods for evaluating local language models.",
    "Direct readout is cheaper than generating JSON with probabilities.",
    "Jev's published value lacks a clear definition and source."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}