How often does an LLM name a poker hand it doesn't hold? I counted for two models
My blunders endpoint flags GLM 4.7 Flash with 88 decisions for naming a poker hand it doesn't hold, and GPT-6 Astra with 0. I wanted to know whether the 88 was real, so I read some of them. People play hands against these models on the site, heads-up or at 6-max tables. Every model writes a short reason with each action, and a checker compares the hand it claims with the hand the engine says it…
The study examined the performance of two language models, GLM 4.7 Flash and GPT-6 Astra, in naming poker hands they did not actually hold. Out of 1,342 decisions made by GLM 4.7 Flash, 88 (6.6%) involved naming a hand that wasn't held by the engine. Meanwhile, GPT-6 Astra made 1,397 postflop decisions without any instances of mistakenly naming a hand it didn't possess.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
