OpenAI executive urges caution in US-China AI talks, warns of distillation threats

OpenAI's deputy lead for national security, Caroline Zier, cautioned on Monday about the potential risks of sharing information with Beijing during upcoming artificial intelligence (AI) talks between the US and China. Zier urged Washington to be "clear-eyed" and "aware" of what information they expect in return for sharing and start building a foundation. She expressed skepticism about both countries having the same understanding of the risks involved, particularly the dangers of loss of control.

Zier also highlighted the threat posed by alleged Chinese distillation attacks, which involve malicious actors using a "teacher" model's outputs to train a "student" model. She noted that such attacks often strip safety guardrails and boundaries from the models and that they primarily originate from Chinese labs. Beijing rejected US allegations of AI distillation last September, warning of retaliation.

Despite the rhetoric, technical teams from both countries have been holding regular talks to iron out details of a framework for AI cooperation. However, Sam Bresnick, a research fellow at Georgetown University’s Centre for Security and Emerging Technology, questioned the effectiveness of a proposed US-China "hotline" on AI risks, citing the Chinese side's reluctance to share information quickly. Both Zier and Bresnick expressed concerns about future dependence on Chinese models if access to them were restricted.