Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens

Shopify has unveiled Gisting, a pioneering method to compress lengthy prompts for language models (LLMs) into a smaller set of learned tokens known as "gist" tokens. This technique enhances throughput and cuts down on inference expenses. By transforming expansive text into succinct gist tokens during the inference phase, Shopify aims to decrease end-to-end latency, lower infrastructure costs, and augment token throughput without altering the model's original weights.

For instance, Shopify's Sidekick GraphQL agent's system prompt was trimmed from approximately 6,000 tokens to merely 1,500 gist tokens, maintaining prediction quality while achieving a 4:1 reduction in context size. At 350 requests per minute (RPM), Spotify's median time to the first token (TTFT) decreased from 438 milliseconds to 354ms, median end-to-end request latency dropped from 6.8 seconds to 4.2 seconds, and throughput rose from 20.2 to 23.4 queries per second (QPS). Consequently, Shopify was able to reduce the number of allocated GPUs.

Gisting is rooted in a 2022 research paper on Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. The process involves two stages: a teacher pass where the model is executed with the actual prompt to obtain teacher logits, and a student pass where it is run with the gist tokens to derive student logits.

The gist tokens are subsequently trained to minimize the Kullback-Leibler divergence between teacher and student logits, ensuring the student's predictions closely mirror the teacher's as closely as possible. Once trained, these gist embeddings are embedded into the model's embedding matrix and registered as special tokens within the model's tokenizer.

During inference, the model operates like any other without the need for a custom attention mask, additional encoder, or distinct serving pathway.

The primary advantage of Gisting lies in its ability to replace conventional summaries of prompts with learned representations that enable the LLM to function as closely as if it had seen the original prompt. This approach facilitates reduced latency and increased throughput. For Shopify, TTFT dropped from 438ms to 354ms, and end-to-end latency fell from 6.8 seconds to 4.2 seconds, while QPS increased from 20.2 to 23.4. This enabled engineering teams to scale down GPU allocation.

Shopify emphasizes that Gisting can be combined with other optimization methods, such as prefix caching, which prevents the recomputation of KV tensors for cached prompt sequences. However, Gisting further minimizes overhead by substituting a lengthy prompt with a shorter sequence of learned gist tokens, thus compounding the benefits of both optimizations. For a comprehensive understanding of Gisting, including autosearch tuning and additional implementation details, readers are encouraged to review the original article.