Ai2 releases Olmo-core 3 to make developing large mixture-of-experts LLMs more efficient

Seattle-based AI research organization Allen Institute for AI introduced Olmo-core 3, a framework designed to simplify the development of large language models on Thursday. The framework improves the training process of mixture-of-experts large language models, enabling them to scale to a trillion parameters while maintaining computational efficiency.

Unlike dense models that utilize the entire model for computations, mixture-of-experts models allocate computation across specialized portions of the model for each token generated. Olmo-core 3 facilitates the expansion of expert pools from eight to 128 while still activating only four experts per token, allowing LLMs to scale to over one trillion parameters.

Benchmarks demonstrate that Olmo-core 3 processes 52,000 tokens per second on Nvidia B3000 GPUs for a 47-billion parameter model, a 2.7 times increase in throughput compared to Nvidia's established Megatron-core training architecture. The architecture employs expert parallelism, layer splitting, and distributed optimization to reduce memory overhead as models scale.

Additionally, Ai2 supports MXFP8, a number format that can decrease computation and data movement between GPUs. The new training architecture and improved efficiency are part of Allen Institute's goal to provide researchers with tools to create and train larger models. The framework, along with related systems, is currently available on GitHub for developers and the open-source community.