Zet: an open-source layer on Laya that knows when to ask a human
Confidence scores aren't error rates. Zet sits on top of Laya and marks every answer sure or unsure, using conformal prediction sets and a Learn-then-Test threshold calibrated per language, so the answers it automates stay within an error budget you choose (say 5%). On MASSIVE (human-labeled, 300 examples per language, 5% budget), it automated 84% of English and 80% of Swedish requests on a…
Zet is an open-source tool designed to work alongside Laya, a language model. Unlike traditional models, Zet does not provide confidence scores as error rates. Instead, it uses conformal prediction sets and a Learn-then-Test threshold that is calibrated per language. This allows Zet to ensure its automated answers remain within a user-defined error budget, such as 5%.
In extensive testing on a task with 300 examples per language and a 5% error budget, Zet automated 84% of English and 80% of Swedish requests on a 6-scenario task, with no errors in the 75 sure English answers. However, as the task complexity increased to 18 scenarios, Zet ceased automation, yet its prediction sets still contained the correct answer 97-99% of the time.
This capability to refuse when the task becomes too challenging is Zet's key feature. The tool runs locally on ONNX Runtime without requiring PyTorch, and offers a web interface for creating tasks, uploading labeled examples, calibrating, and reviewing answers that are uncertain. Zet is licensed under the Apache-2.0 license and is built on Laya, though it is not endorsed by the authors of Laya.
More information can be found on its website at https://yoosseph.github.io/Zet/ or its repository at https://github.com/Yoosseph/Zet.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
