Mining 14,351 Chess Puzzles with Stockfish 18 (and the Illegal-FEN Trap in chess.js)

In building Brass Pawn, a team needed an offline dataset of chess puzzles that could run entirely offline, whether on a computer or mobile device. Transcribing puzzles from human annotators or scraping unreliable online collections proved problematic. Two major issues emerged: ambiguous solutions and horizon effects, where an engine's shallow evaluation failed to capture deeper strategic nuances.

To address these challenges, the team developed a verification pipeline using Stockfish 18. This pipeline aimed to ensure puzzle uniqueness and correct solution evaluation. The first stage focused on mining unique tactical puzzles, while subsequent stages involved depth-22 verification and independent checks for endgame stability.

A critical issue discovered was an "illegal-FEN trap" in chess.js that allowed positions where the non-moving side was already in check. Such illegal positions could be passed through chess.js without detection and would cause Stockfish to respond with bestmove (none). This issue posed a significant risk for puzzle validation and engine interactions. To mitigate this, the team created a helper function that checks for illegal FENs before processing them, ensuring only valid positions are used for evaluation.