Presentation: Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads
Sundara Raman Ramachandran shares how LinkedIn productionizes LLM ranking under strict P99 latency SLAs. He discusses distilling a 7B model down to 400M parameters, compressing job contexts into single-token embeddings for a 10x throughput leap, and eliminating hidden system bottlenecks - from custom flash attention masks to Python GIL workarounds, gc.freeze, and non-blocking CUDA memory copies.…
Sundara Raman Ramachandran, a lead engineer on LinkedIn's LLM Inference team, discussed scaling LLM-based ranking systems for latency-critical search and recommendation workloads at QCon AI. He shared how LinkedIn produces LLM ranking under strict P99 latency SLAs, focusing on distilling a 7B model down to 400M parameters, compressing job contexts into single-token embeddings for a 10x throughput increase, and eliminating hidden system bottlenecks.
LinkedIn processes up to 1.3 billion members across 200 countries, handling high QPS for ranking and recommendations. Search has evolved to natural language intent, such as "find me design roles in the entertainment industry" or "who in my network can help me file a patent for my business." These raw natural language texts are complex to understand, but LLMs can help interpret user intent and provide better results.
The talk covered the high-level architecture of query understanding and retrieval layers, with a focus on the ranking layer. The query understanding layer turns raw natural language text into a structured, normalized format for further systems to understand. This includes query planning, tagging, and rewriting to standardize the user intent and location.
In the retrieval layer, LinkedIn finds the top-K semantically relevant jobs for a user query. With millions of jobs, direct ranking isn't feasible, so the process begins with query understanding output, converting it into a query embedding. This embedding undergoes vector search using KNN on a pre-built index of job embeddings.
Written by urgent.news from InfoQ's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.