Laya: replace your LLM-as-a-judge with a 322M-parameter decision engine
Originally published at AI Frontier Post . You have a support inbox, a moderation queue, or an agent that needs to route its own work. The decision you need is embarrassingly simple — which department, how angry, should this escalate — but the standard implementation is absurd: serialize the text into a prompt, send it to a frontier model, wait for tokens to dribble out, then regex the answer out…
Laya is a new decision engine that aims to replace the traditional LLM-as-a-judge approach with a more efficient and cost-effective solution. The standard implementation of routing decisions using LLMs involves serializing text into a prompt, sending it to a powerful model, waiting for results, and then extracting the relevant information. This process is slow, expensive, and can produce inconsistent results due to variations in model outputs.
Laya offers a non-autoregressive decision engine that takes text along with a set of typed questions (choice, score, or yes/no) and provides answers in a single forward pass through an encoder. This means no tokens are generated, the response reports output_tokens: 0, and the process is both fast and deterministic, similar to System 1 thinking. The project gained significant attention within a short period, with over 3,000 stars in just nine days, indicating a strong interest in the solution.
The repository provides three checkpoints on Hugging Face, covering English, multilingual (covering 100+ languages), and a typed-decisions variant. It requires Python 3.10 or newer, pip, and PyTorch (CPU build is sufficient). The entire setup can be done locally without any API keys or costs, making it accessible to users without specialized hardware or accounts.
Using Laya is straightforward. The CLI version allows for quick testing, while the Python code enables integration into existing systems. The engine's performance is impressive, with a 421M-parameter English model achieving 33ms per question on a T4 GPU, dropping to 7.2ms per question when batched. In addition to the triage preset, Laya includes presets for email, guard, moderation, and router use cases, all of which generate typed decisions, confidence scores, and honesty in abstaining when necessary.
This approach reduces the reliance on expensive frontier-model tokens and ultimately accelerates decision-making processes.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
