Gimlet Labs nabs $300M for its disaggregated inference platform

Gimlet Labs, a company that simplifies developers' inference workloads, has secured $300 million in funding at a $3 billion valuation. Andreessen Horowitz led the Series B round, joined by Arm Holdings Inc., Samsung Ventures, Microsoft Corp.’s M12 fund, and over a dozen other investors. The company's total outside funding now stands at $392 million.

Large language models (LLMs) encompass multiple software modules with varying hardware requirements, such as differing memory usage. Gimlet has created a platform that intelligently breaks down these LLMs into consistent modules and assigns each to the most suitable chip architecture based on its hardware needs. This process can involve running memory-intensive components on accelerators with large onboard RAM.

There are various methods of LLM disaggregation, with the most common being PD disaggregation, which separates the prefill and decode phases into different chips. Gimlet supports more granular disaggregation, allowing developers to divide the decode phase into smaller workflows that can be assigned to individual chips. Some developers even utilize a lightweight "drafter" model for initial prompt responses and a frontier LLM for refining those drafts, with each model running on a distinct chip architecture.

Gimlet's software streamlines the implementation of these disaggregation workflows and optimizes each LLM module for the specific chip architecture it's deployed on. It achieves this optimization through a combination of AI agents and a custom compiler. The AI agents explore multiple design approaches to find the best way to adapt LLM code to the target chip, running tests to ensure correctness. The compiler then applies both generic and chip-specific optimizations to the customer models.

The company offers its software in a serverless edition and as a managed service for enterprises to deploy on their own infrastructure. Gimlet has reportedly received billions of dollars in customer orders, including from one of the world's largest cloud providers and a top three AI lab. The funding will enable the company to expand the serverless edition's infrastructure capacity by adding several hundred megawatts of computing power.

Additionally, Gimlet plans to venture into the custom hardware market by developing an inference-optimized server without a motherboard, suitable for use outside data centers.