Sony patents AI that predicts a game's next frame before the GPU finishes rendering it

Sony has recently filed a patent for an innovative artificial intelligence (AI) system designed to predict a game's next frame before the graphics hardware completes its rendering process. This novel approach aims to minimize graphical stuttering without introducing additional latency that some frame-generation techniques may cause.

The AI system leverages the most recently completed frame and, optionally, recent controller inputs to generate a predicted frame, which can then be displayed as the hardware renders the subsequent frame.

What sets this method apart from other frame-generation techniques is its use of a frame already available, rather than waiting to interpolate between two completed frames. This approach could help maintain a more consistent frame delivery rate, particularly during moments when rendering slows down. The AI system utilizes a jointly trained encoder and decoder.

The encoder transforms a rendered frame into a compact numerical representation known as an embedding, while the decoder employs this representation to produce a predicted frame.

During gameplay, the AI system operates in tandem with the graphics pipeline. As the hardware renders the next frame, the AI system processes the previous frame and transmits its prediction to the display buffer, enabling the predicted image to appear even before the next rendered frame is ready. Moreover, the AI model can incorporate motion vectors, which depict the movement of image parts, and recent controller inputs, which reveal the player's latest actions.

These supplementary inputs offer more information than the previous frame alone, thus enhancing the AI's predictive capabilities.

Training for this AI model involves consecutive video frames. Initially, the network is equipped with randomly initialized weights and is tasked with predicting the subsequent frame based on the provided input frame. The system then compares its output with the actual subsequent frame, generating a loss value that quantifies the prediction error.

The patent's claims delve into checking this loss against a specified threshold during training. Since the video itself provides reference images, the training process does not require human labeling of the training material.

Additionally, the decoder outputs more color channels than the encoder receives, enabling it to generate multiple predicted frames simultaneously. Sony also discusses an alternative non-AI technique that shifts pixel blocks based on their motion, generating synthetic frames without relying on the trained encoder-decoder model. The filing further explores maintaining a fixed frame delivery rate without compromising visual quality, although the effectiveness of generated frames in preserving this quality remains unverified.

The practical test will determine whether the AI can swiftly generate accurate frames to fill gaps when rendering slows down. Although the patent outlines a method for achieving this, it does not demonstrate the system's performance in real-world scenarios.