{
  "id": 19196663,
  "title": "Google claims its new AI software cuts inference latency by over 90% — but there’s a catch",
  "url": "https://urgent.news/2026/11/09/google-claims-its-new-ai-software-cuts-inference-latency-by-over-90",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-09T18:40:01.000Z",
  "source": {
    "name": "The New Stack",
    "slug": "the-new-stack",
    "url": "https://thenewstack.io/google-tpu-sync-inference/"
  },
  "original_language": "en",
  "account": "Google’s latest AI software claims to reduce inference latency by more than 90%, but there are some caveats to consider. The company’s eighth-generation TPU announcement includes TPU Sync, an open-source system designed to tackle data movement issues between various types of memory that become more problematic as context windows grow. The software is targeted at developers running large models, especially those handling massive token contexts and reinforcement-learning workloads.\n\nTPU Sync moves data between accelerator memory, host DRAM, storage, and the network using zero-copy transfers and native PjRtBuffer hardware descriptors to avoid unnecessary copies. The company reports that KV-cache transfer latency and time to first token can be cut by over 90% in some tests. Inside the memory hierarchy, the framework achieves combined host-to-device and device-to-host throughput of 223 GB/s per Ironwood TPU host, nearing the theoretical 256 GB/s PCIe Gen5 limit.\n\nGoogle's benchmarks demonstrate significant reductions in time to first token (TTFT) when running the 235-billion-parameter Qwen3 model, decreasing by 60% for a relatively small number of requests. With a larger 480-billion-parameter model, the TTFT reduction reached 90%. When testing prefix caching with 64,000-token contexts, TTFT dropped by up to 90%, throughput increased by more than 50%, and P99 inter-token latency fell by 30%.\n\nHowever, developers may see varying results depending on their specific workload, the amount of reusable cached data, and the available network bandwidth. While SemiAnalysis suggests Google's gains result from infrastructure lacking effective prefill and decode separation, Nvidia still leads in some workloads. SemiAnalysis’s InferenceX data shows GB200 and GB300 NVL72 systems ahead in disaggregated-serving performance, though a direct comparison with a fully tuned TPU v7 setup remains unpublished.\n\nGoogle has made the new software compatible with existing tools by releasing it alongside TorchTPU, supporting vLLM and SGLang, two widely used open-source inference engines. Developers already using these frameworks can run disaggregated workloads on TPUs without rebuilding their serving stack. This aligns with Google's broader effort to make its custom chips more competitive by reducing inference costs through hardware changes.\n\nNotably, Google has emphasized designing chips around Gemini’s architecture, and now it addresses software problems that can leave chips waiting for data. However, the software isn’t production-ready for everyone yet. Google cautions that APIs are still changing and recommends using a verified commit or tagged release. Support is limited to Cloud TPU v7x and v6e, with specific JAX and PyTorch version requirements.",
  "summary": "Google’s eighth-generation TPU announcement on Monday included more than new chips, introducing TPU Sync, an open-source system designed to speed The post Google claims its new AI software cuts inference latency by over 90% — but there’s a catch appeared first on The New Stack .",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}