What Actually Happens When a Transformer Thinks Step by Step: The Mechanics of Chain-of-Thought

When a transformer model is prompted to "Let's think step by step," it utilizes a mechanism called chain-of-thought (CoT) that allows it to break down complex problems into sequential steps. The model does not actually think in the human sense, but rather follows a concrete computational process that involves intermediate token outputs.

The transformer decoder has a fixed serial depth, meaning it can perform a maximum of L rounds of computation in a single forward pass, where L is the number of layers. This limitation prevents the model from solving problems that require more sequential steps than L. However, when the model generates intermediate tokens during CoT, each token effectively provides an additional pass through the network, allowing it to tackle more complex problems.

The intermediate tokens serve as a form of working memory for the model, allowing information to travel from the output of one pass back to the input of the next. This mechanism enables the model to perform multi-step arithmetic, long division, or other tasks that require a sequence of logical steps.

Recent research has uncovered three key insights into what those intermediate tokens are doing:

1. Decoding Space Pruning (The Template Effect): As the model generates CoT, the probability distribution over next tokens becomes sharper and more concentrated at each step. These intermediate steps act as answer templates, constraining plausible next tokens and guiding the model towards the correct solution.

2. The Functional Rift: At the halfway point of the model (around layer 16 for a 32-layer model), there is a shift in function. The first half of the layers focuses on attention heads moving information between related entities, while the second half writes answer tokens to the residual stream. The model essentially switches algorithms halfway through, first aligning representations to the problem structure, and then harvesting answers from the context it has built.

3. Iteration Heads: Transformers develop a specialized two-layer attention circuit, known as iteration heads, which implements a state machine for serial computation tasks like parity, copying, and polynomial iteration. The first layer finds the end of input token and extracts the positional encoding, while the second layer retrieves the t-th input element and previous state, computing the next state using an MLP. This circuit is task-agnostic but can be fine-tuned for specific tasks.

The use of CoT doesn't just increase the number of neurons the model uses, but also modulates neuron activation differently depending on the task type. For open-domain tasks like GSM8K and SVAMP, CoT reduces average neuron activation, allowing the feed-forward neural networks to operate more selectively. In contrast, for closed-domain tasks like AQuA multiple choice, CoT increases neuron activation as the constrained answer space requires more active feature matching.

However, the reliability of CoT is limited. Every written token introduces a chance for error, and with a 0.1% error per token, a 2,500-token chain only survives about 8% of the time. To mitigate this, reasoning models are trained for self-repair (backtracking, checking) and can improve reliability by sampling multiple CoT chains and voting on the most probable answer. While longer chains provide more depth, they also introduce more compounding risk.

In summary, when a transformer model employs chain-of-thought, it leverages a concrete computational mechanism that allows it to perform multi-step reasoning by generating intermediate tokens that serve as working memory and answer templates. The actual computation happens in the forward passes between tokens, with the written steps providing a record of the model's reasoning process.