Google gooses Python AI performance by... removing Python
Newly open sourced XSE handles six million queries a second across services including Search, Ads, and YouTube Shorts
Google aims to boost Python AI performance by ditching the Python interpreter, according to industry insiders. Python's ease of use for data scientists is offset by the overhead and deployment challenges it poses in serving pipelines. To address this, the OpenXLA Project, an industry-backed initiative, offers tools for compiling machine learning computations into native code for various processors.
Google has now released XSE, short for the XLA Serving Engine, which specializes in serving smaller, task-oriented models with high queries-per-second and tight latency budgets. In tests, XSE delivered two to five times the throughput of optimized Python servers while using a quarter of the memory. The ML experts draft their model code in Python using Google's JAX, then export it to C++ serving on TPUs with integration planned for TorchTPU.
The compiled computations are executed by a C++ serving runtime using XLA to optimize the code for the target silicon, bypassing the Python interpreter entirely. Google anticipates a significant performance improvement, especially for ranking, recommendation, embedding, and vision models that are commonly used in services like YouTube Shorts, Google Ads, and Search.
Google has partnered with Arm and Intel to develop the XLA ecosystem, but prefers its own TPU accelerators for model execution.
Written by urgent.news from The Register Science's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
