A 10-email thread holds 487 tokens of actual text. With quoted replies the model reads 5,068.
You build a small tool that summarizes support tickets. Each ticket is an email thread, and your help desk stores every message the way it arrived. You join the messages, send them to a model, and get a good summary back. The token count in the response is about ten times what you expected from looking at the conversation. Nothing went wrong. The model read exactly what you sent, and most of it…
A support ticket summary tool built by the reporter reveals an unexpected reality about email threads. Each thread is stored as a separate unit in the help desk system, with every message retained in its entirety. When sent to a language model for summarization, the response is significantly larger than anticipated.
The model reads and processes the entire thread, copying the content without modification. This results in the stored thread being about ten times larger than the original, with most of the extra tokens coming from quoted replies. The quoted replies are the previous message pasted under the current answer, starting with indents and line breaks. Each level of quoting adds additional tokens, costing more to the model.
The reporter created a synthetic thread with ten messages to analyze the issue. The messages ranged from 26 to 65 tokens in length, with the last message alone containing over 1,000 tokens. Even the larger messages were multiple times larger than the original content. The quoted replies accounted for a significant portion of the token count, with the markers and signatures adding extra tokens to every line.
The reporter found that the primary cause of the inflated token count is the quoted replies. Each level of quoting adds an additional set of characters to the start of every line, which the tokenizer counts as separate tokens. Even if the quoting was minimal, the signatures and signature-like lines also contributed to the overall token count.
To optimize costs, the reporter suggested stripping out the quoted blocks before sending the thread to the model for summarization. This reduces the number of tokens the model processes, lowering the cost per thread. However, the reporter acknowledged that handling HTML mail formats and localized "wrote:" lines would require additional parsing and patterns for each specific use case. The savings from this approach varied depending on the specific implementation and the proportion of input versus output tokens.
The reporter concluded that the issue is more complex than simply copying and pasting content. The quoted replies, indents, and signature-like lines all contribute to the increased token count, making it essential to understand and account for these factors when working with email thread data and language models.
Rewritten by urgent.news; may contain errors — check the original.
Source: dev.to