Before you trust consultants’ ‘research,’ look for these red flags

In the aftermath of Enron's collapse in 2001, various factors were blamed for the downfall, ranging from corruption and greed to hubris and incompetence. However, Nobel laureate Paul Krugman pointed to a different cause: management theories that are popular among consultants. He argued that these theories lead CEOs to disregard fact-based analysis and common sense, ultimately leading to disastrous consequences.

Consulting firms often present their assertions as backed by "research," but rarely adhere to the scientific method, proper controls for error, and bias. This is a common tactic used by the industry.

One such technique is the case study, which consultants frequently rely on due to the difficulty in researching organizations. These case studies are useful but have significant limitations. They often present only success stories and suffer from survivorship bias, as managers are usually reluctant to discuss failures. Furthermore, they are inherently limited in scope, as they involve interviewing a small number of insiders and rarely capture the full reality of a situation.

Additionally, case studies rarely include detailed documentation, which is common in more rigorous academic fields like sociology and psychology.

Another technique is the industry survey, where executives are asked to participate in questionnaires, and the results are published in reports. Although these reports may appear scientific, they often lack crucial information, such as how respondents were chosen, how questions were phrased, the representativeness of the sample, and the interpretation of answers.

Moreover, these surveys primarily measure the current opinions of executives rather than the actual market reality. Executives' strategies should not merely reflect what they know but should exploit what they don't know.

Finally, consulting firms utilize industry conferences, where industry experts and in-house consultants present information to a captive audience. While these conferences aim to provide participants with the latest research and trends, the expertise presented often comes from the same consultants who have prepared the case studies and responded to surveys.

This creates a situation where the consulting firm essentially sells its clients' ideas back to them. Although these conferences may promote "out-of-the-box thinking," they often create echo chambers for the narratives the consulting firm is pushing. Executives are presented with a consensus that may lack critical examination. While consulting firms can provide valuable insights, it's essential to be aware of their primary focus: making their clients happy, sometimes at the expense of presenting hard truths.