Nvidia is quietly staffing a new AI safety team as it doubles down on open models

Nvidia, the renowned chipmaker, is quietly assembling an AI safety and security engineering team to complement its push towards open-weight AI models, according to a series of recently posted job listings. The company is on the lookout for a distinguished engineer to spearhead the team, alongside roles for a security research engineer, an evaluation engineer, and a senior manager. The AI safety team will focus on evaluating AI agents prior to deployment and developing AI-powered tools to address software vulnerabilities.

This development comes as Nvidia intensifies its advocacy for open-weight models, which differ from closed systems by making their trained "weights" publicly accessible while keeping training data and source code private. Jensen Huang, Nvidia's CEO, recently voiced his support for open models, asserting that they enhance safety and cybersecurity. Further emphasizing this stance, Nvidia joined the Open Secure AI Alliance, a coalition of over 120 companies dedicated to creating open-source security tools for AI.

The AI safety team's establishment appears to be a strategic move, given the growing importance of AI safety as companies consider how best to deploy AI technologies. Open models, proponents argue, foster innovation through competition and bolster security through transparency and collective efforts. However, critics caution that open models could also make them more accessible to malicious actors.

Despite this, the push towards open models is seen as a crucial factor in driving broader adoption of AI, particularly as it evolves from chatbots to agents capable of accessing sensitive data and executing actions in the real world. Nvidia's AI safety team, therefore, aligns with its broader strategy to capitalize on the burgeoning demand for AI chips to power open-weight models.