Can AI tell whether an optimum is unique? A small Kaggle benchmark with exact answers
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked When a solver or an AI model returns "the optimal solution", it rarely says whether it is the only one. That matters: if many equally good solutions exist, a small change in the data can silently move you to a very different "optimal" answer. I wanted to know whether language models notice this, so the benchmark asks…
This benchmark explores whether language models can accurately determine if an optimum in a small optimization problem is unique, and if not, how many optimal solutions exist. 87 optimization problems of five types are used, ranging from subset selection to a toy portfolio selection with 16 to 20 binary variables. The exact optimal value and number of optimal solutions are known for each problem due to its small size.
Each model receives the problem and must output the solution, objective value, whether the solution is unique, and the exact number of optima. The answers are given in free text, with a JSON line at the end containing the solution, objective, unique flag, and objective count. The model is not given the count to determine.
Three metrics are scored: whether the solution is optimal, whether the "unique/not-unique" claim is correct, and whether the count of optima is exactly right. For the 29 instance board used for the Kaggle leaderboard, Gemini 3.7 Flash scored 90% (26 of 29) while gpt-oss-120b scored 55% (16 of 29). The gap between the two models was statistically significant (p = 0.007).
When counting the optima, Gemini 3.7 Flash achieved 70% accuracy with the plain prompt and 75% with an explicit prompt asking the model to find alternative solutions. Accuracy dropped dramatically as the number of optima increased, reaching only 42% when there were more than 50 optima. In the larger run of 87 instances, the plain prompt led to a 75% success rate with correct counts, while the explicit prompt improved to 85%. Most errors in counting were instances where the model underestimated the number of optima.
The stronger model, GPT-6.1-sol, performed better on the harder 10 instances where Gemini 3.7 Flash struggled. It got all 10 exact counts correct, including those with an extremely large number of optima like 2,411. However, it's unclear if GPT-6.1-sol reasoned internally or relied on external tools to arrive at the answers.
Interestingly, the model tended to claim uniqueness for non-unique optima more often than expected. In most cases, Gemini 3.7 Flash called a non-unique optimum unique. The failure was usually subtle, presenting a plausible lower bound as the exact count. Asking the model to explicitly check for other solutions did not clearly help, as the explicit prompt was only better for 8 out of 29 instances.
Additionally, the same instance gave different exact counts when tested multiple times, indicating substantial run-to-run variation.
This benchmark measures the answer produced by the model, not the actual methodology used to solve the problem. The results suggest that language models generally have difficulty precisely counting the number of optimal solutions, especially as the number of optima grows. Future work could involve more extensive testing, varying the models, incorporating tools, measuring counts in a less strict way, and applying the benchmark to larger real-world problem sizes.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
