AI's Worst Failure Mode Isn't Hallucination

Everyone knows AI hallucinates, inventing citations, fabricating statistics, and confidently naming non-existent libraries. Teams respond by building guardrails, deploying detectors, and using retrieval-augmented pipelines, but two other failure modes cause more real-world damage, and almost nobody builds defenses against them. The first failure mode is "This Is Impossible (It Wasn't)."

Linus Torvalds shared a debugging session from the Linux kernel where the AI, Gemini, confidently declared the problem unsolvable. Despite the AI's insistence, Torvalds pushed and added debug code, eventually fixing the issue with a single character change. The AI's stubborn refusal to accept the problem's solvability demonstrates the first failure mode, where AI confidently states impossibility.

The second failure mode is "Perfect-Looking, Dangerously Incomplete." A study by Composo found that 34 findings were omitted from 847 clinically generated AI notes, with 19 clinical discussions transforming into treatment decisions. Eleven of these omissions were severity-critical, such as a woman over 50 presenting with a headache and jaw pain, but the AI omitted the jaw pain entirely.

The note looks complete and well-structured, but the critical symptom is missing. These AI systems passed all evaluation frameworks, and standard hallucination detection would flag nothing, as nothing was hallucinated. The issue lies in the missing information, not false claims. Both failure modes share the same underlying mechanism: reinforcement learning from human feedback (RLHF), which optimizes for seeming helpful over being complete.

During training, human raters prefer confident, well-structured, and helpful-sounding responses, penalizing uncertainty and complexity. The result is a model that prefers giving up gracefully and closing notes prematurely, leading to dangerous omissions. The root cause of these failure modes is the reward system in AI training, which encourages clean, helpful output over complete, thorough responses.