Making a Small Decision Model: Part 3, Playing Doom

In Part 1 and Part 2 of this series, we created and trained Gev, a decision model with 270 million parameters, based on Google's T5Gemma 2 backbone. This model excels at answering closed text questions in just 45 milliseconds. From the outset, we selected T5Gemma 2 because it is a Vision-Language Model (VLM) with an integrated image reader (SigLIP vision tower) alongside its text encoder. This dual capability was crucial for handling multimodal data.

Imagine the input as live 35 frames per second gameplay from id Software's 1993 classic, Doom. The Doom engine updates the game state 35 times per second, known as game tics. Rather than asking Gev to react to every frame, we instruct it to make a move every four tics, approximately 9 times per second. At each step, Gev examines the screen and chooses one of 18 controller button combinations, such as turning left, moving forward, or firing a weapon.

Doom's pixel art and fast-paced motion are unlike any English text, necessitating fine-tuning. There are two primary methods to train a model to play a game: Reinforcement Learning (RL) and Behaviour Cloning (BC). RL involves the model learning through trial-and-error rewards over millions of simulated games, which is powerful but requires significant computational resources.

BC, on the other hand, treats gameplay as a supervised imitation learning task, where pre-recorded decisions from an expert game bot are used to train Gev to predict the expert's action from the visual frame.

To transform live video into a decision problem, a single still frame from Doom is insufficient as it lacks information about velocity or acceleration. The model requires motion history. Therefore, we feed Gev three frames: eight frames ago (t-8), four frames ago (t-4), and the current frame (t). The challenge lies in the way these frames are packed into the vision reader.

Initially, we stitched the three frames into a single image with the older frames at the top and the current frame at the bottom, reducing computational cost by passing through the vision reader once. However, this approach caused distant enemies to appear blurry. Later, we adopted a split layout, dedicating one 256-token image to the current screen and stacking the two history frames into a second 256-token image, resulting in a total of 512 visual tokens.

This arrangement provided a sharp view of the crosshairs at the moment while retaining the motion history.

Training Gev to perceive pixel art without losing its ability to understand English was a challenging process. Initially, we fine-tuned our trained text model on Doom gameplay using a single stitched image with the vision reader's weights frozen. This approach allowed Gev to play surprisingly well, averaging 13.7 frags per match.

However, detailed analysis revealed two issues: the frozen vision reader struggled with blurry distant sprites, and flipping images horizontally during training inadvertently swapped the older and newer history frames on every flipped example. To address these issues, we individually flipped each frame before stitching, switched to the two-image split layout, and incorporated several training habits: unlocking the upper visual layers of the vision reader for 1993 pixel art, mixing regular English questions into training batches to avoid catastrophic forgetting, and hiding previous button presses 50% of the time to force Gev to rely on visual cues rather than key-press inertia.