When a Failed Request Must Stay Failed: Reservation Replay
This is a submission for the Kaggle Benchmarking Challenge . What I Benchmarked A room is occupied, so a booking request fails. The room becomes free. Should an identical retry now succeed? Under this benchmark's declared contract, no. The original request ID must replay the original rejection. A genuinely new attempt needs a new ID. This is the rule being tested, not a claim about every…
This submission for the Kaggle Benchmarking Challenge studied how reservation systems handle failed requests, specifically whether retrying the same failed request with the same ID would yield a different result. According to the benchmark's rules, an identical retry must replay the original rejection, while a genuinely new attempt requires a new request ID.
The benchmark aims to test this rule, not make a claim about all reservation APIs. Reservation Replay is a method that reconstructs both the final calendar and every decision in a synthetic event log. Checking the calendar alone can hide wrong intermediate decisions, so the benchmark requires both.
Models were tested using the Kaggle Benchmarks SDK. Google's Gemini-2.5-flash and Gemini-3.7-flash were evaluated. Gemini 2.5 Flash initially failed but then passed the entire benchmark after a policy update that caught a specific error and preserved the raw response. Gemini 3.7 Flash passed in both the initial and published runs.
The key difference was that 2.5 Flash's failures produced responses that the SDK couldn't parse into the required answer format, resulting in output-contract failures. These format failures don't necessarily indicate a misunderstanding of the cached-rejection rule.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
