What decision models can't do: six honest limits
This one first ran on mrsaynothing.dev — What decision models can't do , the against-the-grain companion to yesterday's explainer. Everyone is piping the new decision models into production because the release notes were exciting. Nobody has published an accuracy bench. That gap is this post. The short version: use them for decisions, never for reasons. Two of the six are non-commercial. The…
This article, first published on mrsaynothing.dev, explores the limitations of decision model models and why they should not be relied upon for providing reasons behind their decisions. The key points are that decision models return only a label and a number, without any explanation of why that decision was made. Simply adding more models together to get a unanimous answer does not verify the accuracy of that answer, as the models can still produce wrong or confident but incorrect results.
The authors also note that some decision models have licensing restrictions that limit their use, and that there are gaps in the modality support for different input types like text, vision, and multilingual languages. They emphasize that serving a decision model is not the same as proving its accuracy, and that the confidence number is just a claim until it is bench tested with real data.
The article concludes by suggesting that the best use case for decision models is one-pass routing and gating, where the model's output is a decision and a human is accountable for the consequences. The authors end by calling for a benchmark of the models' performance on real traffic data to determine if their confidence numbers are accurate.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
