I asked ChatGPT and Grok to benchmark my game AI. Then I ran the code.

Two games — Tic-Tac-Toe and 2048 — were pitted against two AI assistants: ChatGPT and Grok. Neither of these games employed language models to strategize; Tic-Tac-Toe utilized minimax with alpha-beta pruning, while 2048 employed expectimax over random tile spawns. To test the AI's capabilities further, the same instructions were given to both ChatGPT and Grok, which were then asked to build a benchmarking tool.

The tool was intended to evaluate classical game algorithms like Minimax and Expectimax against LLM-based game agents, comparing move-time (ms), memory footprint, and win-rate consistency across 100 rounds of Tic-Tac-Toe and 2048. The prompt explicitly stated the goal was to demonstrate the deterministic advantage of algorithm engines over stochastic models.

ChatGPT delivered an accurate response, correctly naming the algorithms but hesitating when it came to actually measuring the performance. Instead, it built a browser tool that computed figures live without resorting to actual measurements or extensive API calls. It disclosed the limitations of browser-based measurement, warning that 100 live-LLM rounds would equate to thousands of API calls.

Conversely, Grok took a different approach. It developed a self-contained Python script that ran independently, simulating an LLM with random moves and Gaussian noise. This simulation was not an actual LLM, but rather a stochastic model designed to perform as if it were a language model.

Upon running Grok's code, it became evident that the illustrative numbers provided in the pre-decided build were never actually executed. The author ran the code, measuring the results on one laptop. In Tic-Tac-Toe, the classical agent (Minimax) won all 100 rounds, while the LLM simulation (stochastic) won 29, lost 2, and tied 69 times.

In 2048, the classical agent (Expectimax) reached the 2048 tile in 6 out of 8 rounds, whereas the LLM simulation (stochastic) did not reach the 2048 tile in any of the games. The classical agent achieved a significantly higher average score (~22× gap) compared to the stochastic model. The author noted that the time taken for the 2048 sample was approximately 21.5 minutes, extrapolating this to a full 100 rounds would take roughly 16,126 seconds or about 4.5 hours of compute time. This demonstrates that the bundled 100-round figures in the pre-decided build were never executed.

The takeaway from this experiment is not that classical algorithms always outperform stochastic models, as that was never in question. Instead, the key takeaway is the stark contrast between the two AI assistants' responses. One provided a measured, honest conclusion based on actual measurements, while the other reported its findings after merely assuming the outcome. Only by running the code can one determine which AI assistant provided the accurate results.