Translating Full Books with LLMs: Our Chunking Strategy for Long-Form Context

How our team created a system to translate entire books, up to 100,000+ tokens in length, using Python, FastAPI, and Anthropic's Claude model, despite the model's 200,000-token context window limitation. Initially, we thought sending a full book directly to the language model would work, as Claude advertises a 200k token context. However, a typical novel contains around 100,000–150,000 tokens, which is technically within the limit but presents several challenges.

Firstly, the cost of translating a 100k token book in one go is high. Secondly, LLMs struggle to maintain attention to earlier chapters when processing long contexts. Additionally, rate limits and timeouts can disrupt the process, making it difficult to resume if an error occurs. To overcome these issues, we devised a chunking strategy that maintains context across chunks: terminology, character voice, and consistent style throughout the book.

Our approach involved splitting the book into overlapping chunks, translating each with a context buffer containing a running glossary of terms and character names, a summary of previous chapters, the current chunk's raw text, and the process was repeated. The pipeline began with parsing EPUB/PDF files into plain text with chapter metadata. The text was then split into token-aware chunks using an overlap of 500 tokens to ensure context preservation.

For each chunk, the system fetched the glossary and previous summary from PostgreSQL. The LLM was then called to translate the text using a system prompt that included the chapter summary and glossary. After translation, new terms were extracted from the response and updated in the glossary. A concise summary of the translated chunk was generated and stored for the next chunk. Finally, all translated chunks were assembled to form the complete translated book.

To handle token counting accurately, we switched from LangChain's RecursiveCharacterTextSplitter, which splits based on characters rather than tokens, resulting in some chunks exceeding the model's token limit. Instead, we used tiktoken, which precisely counts tokens. The chunking function takes the text, target token size, and overlap tokens as parameters. It encodes the text using tiktoken, splits it into chunks, and decodes them back to strings. If a chunk reaches the end of the text, the loop breaks.

However, since books have paragraphs and chapters, we didn't want chunks to split mid-sentence or mid-paragraph. We added a post-processing step to adjust chunk boundaries to the nearest paragraph break within the token window, which significantly improved translation quality. Lastly, we used Anthropic's Python SDK to integrate with Claude and a cheaper alternative, DeepSeek, for simpler passages.

The translation function formats a prompt with the system prompt, glossary, and chapter summary, sends it to Claude, and returns the translated text.