bge-reranker Truncates at 512 Tokens: 31% of My Chunks Got Cut
My RAG pipeline had a question it kept getting wrong. "How do I fix the replication lag alert on the orders DB?" The answer was in my runbooks. Retrieval found the right chunk at rank 3. Then the reranker moved it to rank 14, my top-5 cutoff threw it away, and the LLM confidently answered from a chunk about a different database. The reranker wasn't dumb. It was blind. My bge-reranker truncates at…
My RAG pipeline experienced a replication lag alert on the orders database. The issue stemmed from the reranker truncating chunks at 512 tokens, cutting off the correct answer for 31% of the chunks. The bge-reranker, a cross-encoder reranker, reads the query and passage together as a single input with a fixed number of position embeddings.
Anything beyond the 512-token limit gets cut off, resulting in the reranker losing answers that retrieval had already found. The problem lies in cross-encoders having a hard ceiling, while bi-encoders embed the query and passage separately. To detect this truncation, tokenize the query and passage using the reranker's tokenizer and log how often the length exceeds the max_length.
The fix involves chunking to the reranker's budget or scoring overlapping windows of the passage and taking the maximum.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
