JetBrains Ranked AI Agents on Real Kotlin Projects. The Token Column Is the Real Story.
JetBrains released the Kotlin Benchmark , its official benchmark for grading AI coding agents on real Kotlin engineering work. Claude Code with Opus 4.7 xhigh leads the first leaderboard at 85.7 percent , with JetBrains Junie and OpenAI's Codex right behind at 81.9 percent. But the resolution rate is the least interesting column on the page. The number that deserves your attention is tokens per…
The Kotlin Benchmark, released by JetBrains, evaluates AI coding agents on real-world Kotlin engineering tasks. While Claude Code with Opus 4.7 xhigh leads the leaderboard with 85.7% accuracy, the most telling metric is tokens per solved task. This metric ranges from 66,000 to 777,000 tokens across the top 20 setups, indicating a significant cost difference between setups with similar resolution rates.
The Kotlin Benchmark follows the SWE-bench methodology and uses a dataset of 105 engineering tasks from active open-source Kotlin repositories. Each agent must navigate the project and produce a working patch, which is only considered resolved if it passes the repository's tests. This benchmark differs from generic benchmarks like Kotlin_HumanEval, which only test a model's syntax knowledge.
Instead, the Kotlin Benchmark assesses the agent's ability to complete a validated engineering task within an existing project, a more realistic representation of how teams actually work. The leaderboard, sorted by token costs per resolved task, reveals that token efficiency is crucial. For instance, Claude Code + Opus 4.7 medium used only 66,000 tokens per task, compared to over 232,000 tokens per task for Junie + Opus 4.7 max, the runner-up.
Thus, while Claude Code + Opus 4.7 xhigh may have the highest accuracy, the expensive setups do not necessarily translate to better performance.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
