{
  "id": 5403981,
  "title": "Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens",
  "url": "https://urgent.news/2026/09/03/shopify-introduces-gisting-compressing-llm-system-prompts-into",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-03T20:00:00.000Z",
  "source": {
    "name": "InfoQ",
    "slug": "infoq",
    "url": "https://www.infoq.com/news/2026/09/spotify-gisting-llm-performance/"
  },
  "original_language": "en",
  "account": "Shopify has unveiled Gisting, a pioneering method to compress lengthy prompts for language models (LLMs) into a smaller set of learned tokens known as \"gist\" tokens. This technique enhances throughput and cuts down on inference expenses. By transforming expansive text into succinct gist tokens during the inference phase, Shopify aims to decrease end-to-end latency, lower infrastructure costs, and augment token throughput without altering the model's original weights.\n\nFor instance, Shopify's Sidekick GraphQL agent's system prompt was trimmed from approximately 6,000 tokens to merely 1,500 gist tokens, maintaining prediction quality while achieving a 4:1 reduction in context size. At 350 requests per minute (RPM), Spotify's median time to the first token (TTFT) decreased from 438 milliseconds to 354ms, median end-to-end request latency dropped from 6.8 seconds to 4.2 seconds, and throughput rose from 20.2 to 23.4 queries per second (QPS). Consequently, Shopify was able to reduce the number of allocated GPUs.\n\nGisting is rooted in a 2022 research paper on Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. The process involves two stages: a teacher pass where the model is executed with the actual prompt to obtain teacher logits, and a student pass where it is run with the gist tokens to derive student logits. The gist tokens are subsequently trained to minimize the Kullback-Leibler divergence between teacher and student logits, ensuring the student's predictions closely mirror the teacher's as closely as possible. Once trained, these gist embeddings are embedded into the model's embedding matrix and registered as special tokens within the model's tokenizer. During inference, the model operates like any other without the need for a custom attention mask, additional encoder, or distinct serving pathway.\n\nThe primary advantage of Gisting lies in its ability to replace conventional summaries of prompts with learned representations that enable the LLM to function as closely as if it had seen the original prompt. This approach facilitates reduced latency and increased throughput. For Shopify, TTFT dropped from 438ms to 354ms, and end-to-end latency fell from 6.8 seconds to 4.2 seconds, while QPS increased from 20.2 to 23.4. This enabled engineering teams to scale down GPU allocation.\n\nShopify emphasizes that Gisting can be combined with other optimization methods, such as prefix caching, which prevents the recomputation of KV tensors for cached prompt sequences. However, Gisting further minimizes overhead by substituting a lengthy prompt with a shorter sequence of learned gist tokens, thus compounding the benefits of both optimizations. For a comprehensive understanding of Gisting, including autosearch tuning and additional implementation details, readers are encouraged to review the original article.",
  "summary": "Shopify's engineering introduced gisting, a novel technique for compressing long LLM prompts into a smaller set of learned \"gist\" tokens, improving throughput and reducing inference cost. By Sergio De Simone",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}