OpenAI executive urges caution in China-US AI talks, warns of distillation threats

A top OpenAI executive warned on Monday that Washington must approach sharing information with Beijing during upcoming AI talks with caution, highlighting the threat of alleged Chinese distillation attacks. Caroline Zier, OpenAI's deputy lead for national security, urged clear-eyed awareness of the information exchange, emphasizing the need to build a foundation for future cooperation.

While endorsing the bilateral dialogue, Zier expressed skepticism about shared risk perceptions between the US and China. She doubted that Washington and Beijing have the same understanding of the risks involved, particularly regarding safety and loss of control.

Zier expressed concerns about Chinese distillation attacks, which involve malicious actors using a "teacher" model's outputs to train a "student" model without authorization. She noted that these attacks often originate from Chinese labs and seek to steal protected intellectual property by stripping safety guardrails from models.

Chinese authorities rejected US allegations of AI distillation, accusing the US of using such claims to assert technological dominance. Despite the dispute, technical teams from both countries have been holding regular talks to finalize a framework for AI cooperation, with the goal of making AI a potential deliverable of upcoming talks between US President Donald Trump and Chinese leader Xi Jinping.

Experts cautioned that despite the talks, a US-China "hotline" on AI risks is unlikely to be an effective mechanism due to the Chinese side's reluctance to share information quickly. Sam Bresnick, a research fellow at Georgetown University's Centre for Security and Emerging Technology, highlighted that many US AI startups are increasingly preferring Chinese cost-effective open-weight models due to their lower prices and the ability to run them locally and fine-tune them with their own data.

Both Zier and Bresnick raised concerns about potential dependence on Chinese technology if access to these models were restricted, warning of significant rip and replace costs if the Chinese government decided to control model provisioning.