ByteDance Is Training a 10-Trillion-Parameter Model To Chase the Frontier

ByteDance is reportedly in the process of training an AI model with an estimated 10 trillion parameters. This ambitious project is undertaken with the aim of narrowing the gap between ByteDance and leading AI systems such as Anthropic's Mythos. Although the model is currently in the early stages of pre-training, its future performance will be influenced by factors beyond just its sheer scale.

This endeavor highlights the intense competition among Chinese firms in the pursuit of frontier AI despite limitations on access to advanced chips. The size of the model itself is seen as a statement, as it is more than three times larger than Moonshot's Kimi K3, one of the biggest Chinese models currently available at approximately 2.8 trillion parameters.

While parameter count is not the sole determinant of a model's quality, the industry recognizes that aspects like data quality, training techniques, and efficiency can be equally important as raw scale. Nonetheless, investing the computational resources to train a model of this magnitude sends a clear message that ByteDance is determined to compete at the highest level, rather than merely offering a competent contender in the market.