What If a Transformer Never Had to Forget? Meet the Recurrent Looped Transformer (RLT)

A recent paper suggests a new type of transformer architecture called Recurrent Looped Transformer (RLT) that could potentially eliminate the need for forgetting as information flows through the model. The key innovation is a recurrent decoder that maintains a sliding window of key-value state across tokens, allowing the model to retain context over long sequences.

The RLT architecture consists of a causal encoder and a recurrent decoder. The encoder processes tokens in parallel and stores representations in key-value memory that the decoder can query. The decoder has two components: a final decoder output which serves as the recurrent hidden state, and a sliding window attention (SWA) cache that retains keys and values at each decoder layer. At each step, a gated merge combines the current encoder representation with the previous decoder output.

The design principles behind RLT are:

1. Latent reasoning with unbounded temporal depth - After processing t tokens, the state traverses t x 48 decoder blocks, allowing for unlimited depth compared to fixed transformer depth.

2. Model-hardware co-design - Encoder kernels process tokens in parallel while decoder transitions are sequential, but independent sequences can be batched together on the GPU.

3. Model-RL algorithm co-design - Pretraining, fine-tuning, sampling and RL all share a single state transition. The trainer rebuilds encoder memory, recurrent output and SWA cache during training based on current parameters.

The reference configuration uses 48 encoder and decoder layers with shared attention and feed forward network weights. Each token executes 96 logical blocks, where the decoder adds cross-attention that the encoder does not have. Moving the prompt-response split does not change the conditional distribution for a fixed token history, according to Proposition 3.1.

RLT is an architectural specification, not a trained system, so no measured results are provided. The report suggests potential benefits for long-range reasoning and model-hardware co-design, but emphasizes that the recurrent nature means the decoder is inherently sequential per sequence, requiring batching of independent sequences for parallel implementation.