Building a Multimodal Game-Box Scanner With VisionKit, IGDB, and GPT-5.6 Luna
Duo Queue combined VisionKit OCR, title dictionaries, image fingerprints, IGDB, and GPT-5.6 Luna to recognize Japanese game boxes.
On September 25, the author pointed their app at the Pokémon Red box on an Amazon page. It successfully found the game, but this was just the start. On September 17, a photo of the Japanese box for the game, ポケットモンスター赤 (Pocket Monsters Red), came in as "New Trestar," a non-existent game. The app's database, IGDB, had almost no Japanese names.
The author then created a dictionary on the server to convert Japanese and English nicknames into the official English titles. For example, スマブラ (Sumabura) became Super Smash Bros., and botw became The Legend of Zelda: Breath of the Wild.
The scanner also tried using the title, subtitle, and line of text in various combinations, up to four times. The title read from the photo could be edited. The scanner's performance improved over time. On September 22, the app went to App Review after the fourth version.
The author used the OpenAI API after trying Apple's VisionKit. The cheapest model, text-only, was used initially but faced challenges with OCR mistakes. A dictionary was created to address these issues. The model's performance improved significantly, with 43 out of 44 correct results when the dictionary was used alongside the model.
Finally, the app went live on September 22. A free first scan was offered, and subsequent scans would require payment. The app was tested on various cover images, with a 100% success rate in correctly identifying the games.
Written by urgent.news from HackerNoon's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
