{
  "id": 263657,
  "title": "I Kept Retrying a Local Model Into the Right Shape. Turns Out I Didn't Have To Retry At All.",
  "url": "https://urgent.news/2026/08/07/i-kept-retrying-a-local-model-into-the-right-shape-turns-out-i-didnt",
  "topic": "world",
  "section": "World",
  "published": "2026-08-07T12:28:55.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/naitik_kapatel_f96f1fb424/i-kept-retrying-a-local-model-into-the-right-shape-turns-out-i-didnt-have-to-retry-at-all-44d9"
  },
  "original_language": "en",
  "account": "The challenge wasn't with the AI technology, but rather the client requirements: data couldn't leave the device. The entire processing was confined to a quantized model, running through llama.cpp, with the sole responsibility of categorizing text into one of five labels. Initially, the model struggled with the task, but eventually achieved accuracy. However, the final solution came from using shapecraft, a tool that allowed for a more robust approach. By defining a grammar for the labels, the model was forced to only generate valid outputs, eliminating the need for retries. This not only improved the efficiency of the process but also ensured reliability. The key takeaway is that by integrating a grammar into the model's generation process, the need for error correction or retries was eliminated, leading to a more streamlined and efficient pipeline.",
  "summary": "The constraint on this one wasn't the AI part, it was the client: nothing leaves the device. No API calls, no cloud model, full stop, because the documents being classified weren't allowed off the machine they landed on. So the whole pipeline runs on a quantized model through llama.cpp, and its one job is boring on purpose - read a chunk of text, output exactly one of five category labels,…",
  "key_points": [
    "Client requirements confined data processing to a local model",
    "shapecraft tool forced model to generate valid outputs",
    "Grammar integration eliminated retries and improved efficiency"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}