The AI race has a democracy problem

The AI industry faces a significant democracy problem, as researchers like Anthropic's Monalisa Coxon warn that the race towards self-improving superintelligence is being conducted with little regard for the potential civilizational consequences. Coxon's resignation, echoed by other Anthropic researchers, highlights concerns about the industry's approach to AI development.

The debate surrounding AI safety is shifting from an internal technical issue to a question of democratic control, as companies develop infrastructures that may eventually be relied upon by governments, militaries, researchers, and ordinary institutions. Coxon argues that the decision over how fast to proceed should not lie solely with the companies seeking capital and strategic advantage, as their actions could move beyond democratic control.

The power of these companies lies not only in the intelligence of their models but also in the infrastructure they build, including compute, cloud systems, data, APIs, and software embedded in public and strategic functions. This concentration of power raises concerns about the fusion of public authority and private technological power, with companies increasingly involved in shaping the conditions through which governing takes place.

Additionally, the knowledge itself is becoming increasingly controlled by private institutions, as universities and researchers have contributed to the foundations of modern AI, but now rely on private gatekeepers for access to advanced compute and important datasets. This concentration of power and decision-making raises questions about accountability and oversight, as citizens have limited access to the inner workings of these companies and few opportunities to influence their decisions.

The race for AI dominance may now begin with the acquisition of infrastructural and political power by a small group of private institutions, potentially setting the stage for a future where the public has little say in the technological direction society takes.