Zuckerberg's Superintelligence Memo: The Whole Argument Rests on One Premise

Meta has published its policy document, <The Future is for Everyone> in which Mark Zuckerberg lays out his vision for superintelligence and its distribution. Central to Zuckerberg's argument is the notion that the safest outcome is a widely distributed one, where billions of people each have a capable AI agent working for them, rather than a handful of institutions controlling superintelligence.

This is based on the reasoning that if superintelligence is only in the hands of a few, whoever controls those institutions controls the outcome, leading to a potential loss of balance of power and lack of self-correction.

Zuckerberg's memo details several specific uses for personal superintelligence agents, including 24/7 assistance with relationships, creation tools for expressing ideas, business tools, personalized tutoring, and faster scientific discovery. He also commits to offering free or affordable versions globally and a fully private mode where Meta cannot access user information.

Regarding jobs, Zuckerberg dismisses the displacement premise, arguing that the ratio between automation and personal capability is a design choice, and building for invention pushes this ratio towards job growth.

However, the memo's most consequential move is Zuckerberg's redefinition of alignment as ensuring that agents share a person's goals and values, rather than just the company's. This personalization approach is meant to provide safety, as safety researchers point out that value pluralism across agents doesn't necessarily solve the problem of an agent pursuing its assigned goal through unauthorized methods.

Zuckerberg is confident that the two concepts are not separate, and recursive self-improvement is seen as a potential risk if one lab reaches it before others. Meta plans to resume releasing some open-source models, but not at the frontier, and advocates for the US to lead the open-source ecosystem globally while maintaining control over silicon exports and government infrastructure.