Ask ten questions, read the text once: making a decision model 6.7x faster
A lot of "AI" in production isn't generation at all. It's a pile of small decisions about a piece of text: which team should handle this ticket? how urgent is it? is the customer threatening to leave? is this e-mail phishing? Sending each of those to an LLM and parsing the answer is slow, expensive and occasionally creative in ways you didn't ask for. Laya is an open model for exactly this job:…
AI implementation lags behind in many areas, often involving numerous small decisions about text, such as assigning tickets to teams, determining urgency, and detecting customer dissatisfaction. Laya, an open model, addresses these issues by taking a state (ticket, email, JSON document) and typed questions — choices, scores, or yes/no — and returning calibrated probabilities in a single forward pass.
The key to its speed is a single row layout that encodes the document and questions together, allowing Laya's checkpoint to work efficiently without relearning the task. This layout saves tokens and positions, enabling a small call to be dominated by overhead rather than math. The rest of the call is optimized with fast tokenization, CUDA graph replay, and cached operations.
Benchmarks show significant performance improvements with cbjev compared to Laya, particularly on single-question and short-ticket cases. However, cbjev still falls short of Laya on specific benchmarks, such as support triage, prompt injection, DAIR emotion, and AG News. The project is open-source and available for installation through pip or direct cloning from the GitHub repository, along with a server that communicates using TypeSafe JeV's wire format.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
