{
  "id": 4603241,
  "title": "My LLM Critic Flip-Flops on Every Run. That's Fine — Because a Frozenset Decides What's Fatal.",
  "url": "https://urgent.news/2026/08/31/my-llm-critic-flip-flops-on-every-run-thats-fine-because-a-frozenset",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-31T08:09:45.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/debashish_ghosal/my-llm-critic-flip-flops-on-every-run-thats-fine-because-a-frozenset-decides-whats-fatal-4ep9"
  },
  "original_language": "en",
  "account": "This article explores a design principle extracted from fixing a flaw in an LLM critic. The critic returns different verdicts and explanations every time it is run on the same input. The article explains that this non-determinism is due to the architecture assigning different responsibilities to different components.\n\nThe \"dangerous\" direction is under-claim, where a defective plan gets zero blockers. This is the critical path where safety is paramount. The \"noisy\" direction is over-claim, where a sound plan gets an unnecessary blocker. These are handled by different authorities in the system - deterministic gates enforce under-claim, and a code-enforced allowlist enforces over-claim. This allows the LLM to be non-deterministic in its severity judgments, while the code ensures the verdict aligns with structural properties. The researcher calls this the \"frozenset\" mechanism - the LLM proposes a severity, but code checks whether the structural property supports it. This way, the LLM's label is decorative while the family it belongs to is load-bearing. The key takeaway is that when LLM judgment is placed on a critical path, it inherits the vulnerabilities of LLM judgment, but separating the critical path into deterministic code and advisory LLM judgment ensures consistency and safety.",
  "summary": "This is a companion to the PlannerCritic series . Article 2 was about a specific critic bug. This one is about the design principle I extracted from fixing it — and the measurement that proved it holds. I measured my LLM critic on identical input across five trials. It returned a different verdict every single time. label_flip_rate = 1.0. It also never let a defective plan through.…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}