Can You Trust a Model’s Stated Reasoning?

The reliability of a model's stated reasoning in chain-of-thought prompting is a topic of considerable debate. While the visual appearance of explanations may suggest a clear causal connection to the model's output, published studies have revealed that this resemblance is not always warranted. One key finding is that the chain of thought itself can be entirely correct, yet still unfaithful to the actual reasoning that led to the output.

This unfaithfulness occurs when the explanation rationalizes the selected answer, rather than describing the true cause of it. Another study demonstrated that injecting biases into the model's training data can significantly alter its predictions, even without the bias being reflected in the generated explanation. This means that the chain of thought may not accurately represent the model's actual reasoning process.

Additionally, the faithfulness of these explanations varies depending on the specific task at hand, and cannot be assumed to improve simply due to model scaling. Furthermore, while training models to produce correct answers may not inherently result in more faithful traces, the absence of proper auditing mechanisms leaves room for potential misuse of these explanations as justifications for decisions.

It is therefore recommended not to present model traces as definitive reasons for outcomes, but rather use them as debugging tools to identify areas where the model may have gaps in its reasoning. The truncation test, which involves cutting the chain of thought at various points and forcing the model to provide an answer, can be a useful method to assess the actual relevance and faithfulness of the generated explanations.