Are Latent Reasoning Models Easily Interpretable?

Despite the common belief that reasoning models are difficult to interpret, recent research has shown otherwise. The Coconut and CODI models were tested for their ability to use hidden reasoning steps during logical tasks such as PrOntoQA and ProsQA. Surprisingly, these models barely utilized their hidden reasoning steps for such tasks, often producing the same response even when instructed to stop thinking early.

This suggests that their high performance on logical tasks stems from their specific training data rather than the additional thinking during inference.

When faced with math problems, the researchers discovered that the models actually required those reasoning tokens. By projecting the hidden states back into regular vocabulary words, they found that correct intermediate math steps were present in the hidden states up to 93% of the time. This indicates that the models were performing standard math steps behind the scenes.

To verify this, the researchers adjusted numbers in the prompt and observed the reaction of the hidden states, allowing them to decode a reliable reasoning path for the majority of correct predictions.

However, the researchers found that they could rarely perform such analysis for incorrect predictions, suggesting that models are more interpretable than the AI community initially thought. Moreover, this interpretability can serve as a signal to predict whether the model is likely to provide a correct or wrong answer.