These AI Experts Want to Do High-Stakes Research Out in the Open

Nathan Lambert and Tom Zick, two industry scientists, have founded Trillium Labs, an open-source nonprofit dedicated to conducting AI research, including potentially risky areas like recursive self-improvement and agents, in a more transparent manner. They argue that the secrecy surrounding frontier AI labs hinders the ability of the scientific community to scrutinize ideas and contribute new approaches.

Lambert believes that allowing outside experts to see how models are built and fine-tuned could be vital for mitigating risks. He states, "Over the past few millennia, humanity has had the scientific method in our toolbox as a way to mitigate harms and build better futures." The most powerful AI models, such as those from OpenAI and Anthropic, are currently inaccessible and lack transparency regarding their development and behavior.

In contrast, some companies in China offer models that users can download and run on their own hardware. Other prominent AI labs like Ai2 and Hugging Face have taken more open approaches, including publishing details of model training. Trillium Labs aims to bridge this gap by focusing on post-training fine-tuning and recursive self-improvement, as well as studying how reinforcement learning impacts AI model behavior.

The founders hope that increased transparency will provide much-needed nuance to the ongoing debate about AI development.