When AI explains its decision, humans may stop thinking independently

AI-driven recommendation tools have been found to influence human decision-making, sometimes to a detrimental extent. A new study conducted by researchers from Harvard Business School, MIT, and the University of Washington reveals that AI recommender systems are persuasive enough to sway evaluators' judgments, leading them to adopt AI's recommendations over their own independent evaluations.

This is particularly concerning in the context of innovation screening, where human experts are expected to make impartial decisions based on merit.

In their experiment, the researchers evaluated how human evaluators were affected by AI recommendations, both with and without explanations from the model. The study included 228 experienced evaluators who assessed nearly 50 submissions to an MIT challenge. The evaluators were asked to review proposals under three scenarios: (1) human-only proposals with no AI assistance, (2) AI evaluations accompanied by a written rationale for the decision, and (3) black-box AI pass-fail recommendations without any accompanying explanation.

The evaluators' decisions were then compared to those made by four human experts, who served as the baseline for correctness. Across all scenarios, the evaluators accepted AI recommendations 67% of the time. When given both a black-box AI recommendation and a narrative explanation, they agreed with the AI 75% of the time, but only aligned with human experts' decisions 54% of the time.

Although black-box recommendations tended to improve decision quality by aligning more closely with human experts' judgments, the presence of narratives in AI explanations actually reduced the quality of decisions by discouraging independent verification of AI outputs.

The researchers suggest that this phenomenon can be attributed to negativity bias, where people tend to weigh negative information more heavily than positive information. In the context of rejection decisions, AI explanations provide ready-made justifications for going along with the AI's decision without independently verifying it.

This "illusion of explanatory depth" occurs because LLMs are linguistically fluent and expert-like, creating an impression of credibility that can be exploited by humans. Consequently, evaluators may overestimate their understanding of the decision, leading to a decrease in independent human judgment and an increase in false negatives.

The findings of this study have significant implications for enterprises designing AI-assisted evaluation systems. When implementing AI recommendation tools, especially in high-stakes decision-making contexts such as quality control, compliance screening, or fraud detection, enterprises should exercise caution and refrain from taking AI recommendations at face value.

It is crucial to verify AI outputs before deploying them to ensure accuracy and maintain human oversight. In tasks that involve early-stage screening, such as evaluating innovation proposals, simpler or more opaque recommendations may be preferable to preserve human discretion and verification, thereby preventing the suppression of independent judgment and productive human overrides.