A 421M encoder beat a 1.5B LLM at running my agent, inside a browser tab

A team of researchers has created an AI agent that outperformed a larger language model (LLM) in tasks performed entirely within a web browser tab. The agent, called Laya, utilizes a small typed-decision encoder model with only 421 million parameters, compared to a 1.5-billion-parameter LLM.

The design of Laya is based on the concept of split thinking, where System 1 (Laya) handles routine decisions using fast thinking, while System 2 (the small instruction-tuned LLM) is used for more complex reasoning. Laya operates entirely without any server or API key, and the agent's actions only proceed with the agent's approval.

When faced with a decision, Laya first attempts to answer a multiple-choice question using extracted spans from the given goal and last observation. If Laya is unsure, it passes the decision to System 2. Each tool in the agent is either read-only or requires user confirmation before execution.

The researchers found that the extracted arguments from the goal were accurate 91% of the time, and Laya's argument choices were correct 71% of the time overall. The gate that determines whether a step runs automatically is controlled by the probability of the chosen option and various other thresholds.

In conclusion, Laya, with its more modest parameter count, was able to successfully complete tasks that a larger LLM struggled with, demonstrating that not all agent work requires a generative model.