Recurrent Looped Transformer

The Recurrent Looped Transformer (RLT) integrates a causal encoder with a recurrent decoder, allowing for continuous latent computation as sequences expand. The encoder builds a global key-value memory, while the decoder extends this computation over time. This design merges latent reasoning with limitless temporal depth, enabling model-hardware and model-RL algorithm co-design.

The system utilizes parallel encoder processing, sequence batching, memory reuse, and checkpointing surrounding a recurrent core. Pretraining, Score-Based Functions (SFT), sampling, and current-policy replay share the same state transition. The concept of infinite depth pertains to an expandable temporal path, not unlimited computational work per token.

As each token progresses, the recurrent path traverses a number of decoder blocks equal to the token count multiplied by the number of decoder layers. The per-token block count remains constant throughout this process.

Efficient reasoning, hardware utilization, and reinforcement learning (RL) scaling are yet to be validated. At any given time \(t\), the decoder has traversed \(tL_D\) decoder blocks, maintaining a consistent block count per token. Encoder work for known tokens is batched, while decoder updates are conducted independently. Shared weights and memory are employed, with activations being checkpointed while the original computation is preserved.

The full historical path can be reconstructed using current parameters, including prompt states and decoder stored key-value pairs. The sampled behavior probabilities remain linked to the actual sampler. This setup uses 48 encoder layers and 48 decoder layers, with shared attention and feed-forward network weights across stages.

The decoder traverses \(48t\) decoder blocks after \(t\) tokens, with each token executing 96 logical blocks. However, due to the cross-attention mechanism, these blocks do not all possess equal computational costs. Known tokens can be pre-processed in a causal batch, while the decoder's updates continue sequentially. Forward consistency and full gradient computation are distinct requirements.

Backpropagation Through Time (BPTT) encompasses paths through recurrent outputs, decoder key-value pairs, and encoder memory. Detaching any of these elements alters the gradient. Parameter adjustments invalidate old caches for precise current-policy replay.

Exact behavior probabilities must reflect the actual sampling distribution. Additionally, importance sampling necessitates support coverage. Shared transitions help eliminate structural prompt-bounds discrepancies. Despite this, numerical kernel parity and off-policy estimation remain separate considerations. For a detailed execution-level distinction, refer to the prefill-decode kernel mismatch note.