Urgent.News

What's breaking now, across thousands of outlets.

AI

Does model size matter for LLM agent reliability? 11 open models, 1.5B to 32B: host-held state exact on 10 of 11

When the host holds the state and checks every change the model proposes, the state the host holds was exactly right on every step on 10 of 11 open models, from 1.5B to 32B parameters, in 3,200-step runs with 0 corrupted steps. Model size mattered a lot in other places: when requests were written in ordinary words, a plain model got 2.31 percent of steps right at 1.5B and 97.91 percent at 32B.…

When the host holds the state and verifies every alteration the model suggests, the host's state aligned flawlessly with the true state on 10 out of 11 open models, ranging from 1.5B to 32B parameters. This experiment, involving 3,200 steps using vLLM 0.31.0 and an NVIDIA L4 GPU, revealed that model size significantly impacted performance when models processed natural language requests.

Among the tested models, Qwen2.5-32B (4-bit) achieved the highest exact share at 97.91%, while Qwen2.5-1.5B and Phi-3.5-mini attained 100% exactness. Conversely, DeepSeek-R1-Distill-Qwen-7B faltered, as its replies exceeded the 96-token limit, causing the host to drop them and fall behind the true state. In contrast, when the model had to comprehend ordinary words, model size became crucial, as larger models read more words and improved accuracy.

Despite the host reading the sentence, no weak model achieved full exactness in this scenario. The findings, documented in the AXI runtime repository, demonstrate that model size influences reliability but does not guarantee accuracy when dealing with natural language processing.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

This story

This is one outlet's version. Read the fullest account.

Every outlet that reported this, and when →

Read the original at dev.to →

More in AI

Adding AI Safety with Amazon Bedrock Guardrails: Protecting Inputs and Outputs in a RAG Application

Introduction Building an AI application that can retrieve company information and generate useful responses is exciting. But getting the model to produce a relevant answer is not the only engineering…

  • Amazon Bedrock Guardrails add AI safety to RAG applications
  • Input guardrail checks prompts before retrieval and generation
  • Output guardrail evaluates generated responses before delivery

LLM agent memory compared: whole chat, summary, sliding window, retrieval and host-held state on a 14B model

Six ways of remembering what an agent has done, measured on one 14B model over 600-step jobs: the whole chat, a generic summary, a summary that writes the state out, a sliding window, retrieval, and a…

  • Host-held state is cheapest and fastest method at 0.3034 seconds per step
  • Whole chat method limited to 16,000 tokens after 418 steps
  • Host-held state achieved 81.89% correct identification of model's reply

More from Wednesday 18 November →