How LinkedIn Trains AI Job Search 8x Faster with Multi-Teacher Distillation

LinkedIn has unveiled the technical details behind its AI job search system, revealing a multi-teacher distillation pipeline that condenses knowledge from large teacher models into a compact 0.6B-parameter ranking model. The key to this achievement is not only the distillation technique but also the underlying systems work that makes it feasible at LinkedIn's scale.

The custom framework, built on SGLang, serves teacher models within the training loop, allowing the system to query one or more large teacher models for each training example. This serves the purpose of improving relevance and engagement metrics, such as click-through rates and applications. However, serving these teachers can slow down the process and become a bottleneck for a ranking system handling hundreds of thousands of queries per second.

To address this challenge, LinkedIn developed a multi-teacher distillation framework using SGLang. This system can load and serve teacher models of varying sizes, while also managing tensor-parallel and data-parallel setups. An asynchronous client in the system queries teachers during training, processes their outputs, and adds them to the distillation losses.

This method, called Online Multi-teacher Distillation, was shown to accelerate the distillation process by 3x when scaled across several nodes with local teacher replicas.

To further reduce serving overhead and avoid repeated calculations, LinkedIn introduced Offline Multi-teacher Distillation. In this approach, teacher outputs are precomputed and stored on HDFS or NFS, allowing them to be used directly in training instead of being queried in real time. This online/offline split works alongside a series of training-level optimizations, such as using LiGer to reduce memory usage and enable larger batch sizes, multi-node training with up to 3.5x additional speedup, FSDP2 for an extra 20% speedup, and H200 multi-node clusters for an additional 30% improvement.

The team found that the 0.6B student model, trained using an 8B relevance oracle and a 1.7B engagement teacher, improved job search results significantly. The model achieved a 24.48% increase in NDCG@10, raising it from 0.7583 to 0.9432. In addition to distillation, LinkedIn employed structured pruning and context compression techniques on the student model, which increased ranking throughput from approximately 290 to over 2,000 items per second per GPU.

LinkedIn's system is now live in production, powering the natural-language job search for their US users. They present this work as a guide for other teams to achieve cross-encoder-quality ranking while meeting real-time latency requirements, without resorting to high-cost inference of frontier LLMs for every request. The team suggests that teams creating similar LLM-supervised ranking systems can use the online/offline teacher-serving split, querying online in the early stages when teacher choices change, and then switching to offline caching once teachers stabilize and query volume increases.