Fool's Expertise

The recent Ezra Klein interview with Nvidia CEO Jensen Huang has sparked debate among AI doomerism enthusiasts. Depending on one's perspective, Huang may appear dismissive of potential dangers or reasonable in response. However, the interview left some opportunities on the table. While Huang acknowledged existing regulations like product liability laws could hold frontier labs accountable for harms caused by their products, co-host Klein continually appealed to the authority of AI pioneers like Geoffrey Hinton.

Despite Huang's correct point that Hinton has been incorrect in his predictions, such as his infamous 2016 claim that radiology would cease to exist by 2021, Huang missed a chance to explain why Hinton is wrong. Hinton's mistake stems not from a lack of foresight but from making claims far beyond his domain expertise. For instance, in the radiology example, asking Hinton about the role of a radiologist would have revealed that faster or better image interpretation does not replace the need for human practitioners.

Instead, it enhances their ability to serve patients. This issue of "Fool's Expertise" is prevalent among AI doomsayers, as their experience with one system (e.g., LLMs) grants them unwarranted authority over other areas, which is rarely checked by those like Klein who don't distinguish between different types of expertise. This becomes particularly dangerous when discussing catastrophe, as it involves long chains of causation that cross multiple domains.

Consulting experts from various fields is crucial, as a catastrophic accident may involve failures across several areas that must be understood in tandem. Additionally, if AI-inflicted doom relies on pathways into the real world, those pathways must be examined by experts. For example, discussing AI's cybersecurity threat requires talking to cybersecurity professionals, who will likely point out that the OpenAI escape depended on a simple containment failure.

Similarly, concerns about AI creating novel pathogens should involve experts in viral synthesis to determine the validity of those fears. The same applies to nuclear fears, where seven decades of mitigating global thermonuclear war have provided extensive safeguards and expertise to draw from. It's essential to remember that risks in the physical world are much more diffuse and attenuated when one follows proposed pathways.

For a serious, level-headed examination of these risks, one should consult RAND's "On the Extinction Risk from Artificial Intelligence" by Michael Vermeer, Emily Lathrop, and Alvin Moon. RAND's analysis draws from domain-specific experts, such as risk analysis, nuclear weapons, biotechnology, and climate change, and deliberately avoids engaging with AI experts to focus on what capabilities AI would require to achieve certain outcomes.

By relying on RAND's expert analysis, we can better inoculate ourselves from the fear contagion that often accompanies AI discussions.