{
  "id": 13379611,
  "title": "Can AI tell whether an optimum is unique? A small Kaggle benchmark with exact answers",
  "url": "https://urgent.news/2026/10/10/can-ai-tell-whether-an-optimum-is-unique-a-small-kaggle-benchmark",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-10T09:58:51.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/dimitar_kretski_329e6235e/can-ai-tell-whether-an-optimum-is-unique-a-small-kaggle-benchmark-with-exact-answers-5e27"
  },
  "original_language": "en",
  "account": "This benchmark explores whether language models can accurately determine if an optimum in a small optimization problem is unique, and if not, how many optimal solutions exist. 87 optimization problems of five types are used, ranging from subset selection to a toy portfolio selection with 16 to 20 binary variables. The exact optimal value and number of optimal solutions are known for each problem due to its small size.\n\nEach model receives the problem and must output the solution, objective value, whether the solution is unique, and the exact number of optima. The answers are given in free text, with a JSON line at the end containing the solution, objective, unique flag, and objective count. The model is not given the count to determine.\n\nThree metrics are scored: whether the solution is optimal, whether the \"unique/not-unique\" claim is correct, and whether the count of optima is exactly right. For the 29 instance board used for the Kaggle leaderboard, Gemini 3.7 Flash scored 90% (26 of 29) while gpt-oss-120b scored 55% (16 of 29). The gap between the two models was statistically significant (p = 0.007).\n\nWhen counting the optima, Gemini 3.7 Flash achieved 70% accuracy with the plain prompt and 75% with an explicit prompt asking the model to find alternative solutions. Accuracy dropped dramatically as the number of optima increased, reaching only 42% when there were more than 50 optima. In the larger run of 87 instances, the plain prompt led to a 75% success rate with correct counts, while the explicit prompt improved to 85%. Most errors in counting were instances where the model underestimated the number of optima.\n\nThe stronger model, GPT-6.1-sol, performed better on the harder 10 instances where Gemini 3.7 Flash struggled. It got all 10 exact counts correct, including those with an extremely large number of optima like 2,411. However, it's unclear if GPT-6.1-sol reasoned internally or relied on external tools to arrive at the answers.\n\nInterestingly, the model tended to claim uniqueness for non-unique optima more often than expected. In most cases, Gemini 3.7 Flash called a non-unique optimum unique. The failure was usually subtle, presenting a plausible lower bound as the exact count. Asking the model to explicitly check for other solutions did not clearly help, as the explicit prompt was only better for 8 out of 29 instances. Additionally, the same instance gave different exact counts when tested multiple times, indicating substantial run-to-run variation.\n\nThis benchmark measures the answer produced by the model, not the actual methodology used to solve the problem. The results suggest that language models generally have difficulty precisely counting the number of optimal solutions, especially as the number of optima grows. Future work could involve more extensive testing, varying the models, incorporating tools, measuring counts in a less strict way, and applying the benchmark to larger real-world problem sizes.",
  "summary": "This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked When a solver or an AI model returns \"the optimal solution\", it rarely says whether it is the only one. That matters: if many equally good solutions exist, a small change in the data can silently move you to a very different \"optimal\" answer. I wanted to know whether language models notice this, so the benchmark asks…",
  "key_points": [
    "Language models can determine if an optimum is unique in small optimization problems",
    "Gemini 3.7 Flash achieved 90% accuracy on 29 instance board for unique claim",
    "Counting optima accuracy drops as number of optima increases, reaching 42% for >50 optima"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}