An AI Capture-the-Flag Tournament: What the Scoreboard Counted

In April, the author ran an AI capture-the-flag tournament with five games between small open-weight models ranging in size from 1.0B to 2.5B parameters. Qwen 3.5 at 2.3B captured 13 flags without losing any, while the other four models captured a total of two flags. The tournament was designed to test whether model size impacts security reasoning and if multi-step exploitation is feasible for smaller models.

After running 327 more games with larger models, the first conclusion drawn could not be confirmed. The tournament included five hosted models and one local 3B fine-tune, totaling 327 games with a scoreboard. However, two things in the final table contradicted the original report. The 189 bonus flags were distributed among the models, with the encrypted vault being the most challenging bonus flag, captured 23 times.

Despite having more parameters than all other models, RNJ-1 8B finished last, while a 3B fine-tune led on main flags, capturing two bonus flags.

The prompt used in the tournament did not specify attack methods, allowing models to use various techniques. RNJ-1, with 7,351 attacker commands, mostly focused on enumerating the machine it was on rather than exploiting targets. It did not address any opponents, resulting in no bonus flags being credited to it. The checker used for flag captures did not compare the output against the actual input, which allowed RNJ-1 to earn credits for turns where it wrote flags into its own command.