Smarter RAG Chunking With Chunklet-py

Chunklet-py is a Python library designed to improve the process of splitting text for RAG pipelines. Traditional chunking methods often result in broken sentences or lack of context between chunks. Chunklet-py addresses these issues by understanding text structure and providing smarter chunking solutions.

Key features of Chunklet-py include support for over 60 languages, auto-detection of language, multiple constraint types (such as max_sentences, max_tokens, max_section_breaks, max_lines, and max_functions), and the ability to handle various file formats like PDF, DOCX, HTML, Markdown, LaTeX, and more. It also includes rich metadata with each chunk, like source references and character spans.

The library provides three main API components: DocumentChunker for general text and code chunking, CodeChunker specifically for source code, and SentenceSplitter for just sentences. It even offers a visualizer for interactive experimentation with chunking parameters. Chunklet-py can be installed via pip and supports structured documents, code, and visualization functionalities.

To use Chunklet-py, install it and choose the appropriate package (structured-document for documents, code for source code, or visualization for visualizing chunking). Afterward, import the necessary components and create an instance of DocumentChunker or CodeChunker, specifying the desired constraints such as max_sentences, max_tokens, max_section_breaks, max_lines, and max_functions. Then, call the appropriate chunking method (chunk_text, chunk_file, chunk_texts, or chunk_files) with your input text or file path.

When using the max_tokens constraint, a token_counter function is required to accurately count the tokens in the text. TokenCounterError will be raised if the token_counter is not provided. Chunkers return Chunk objects containing metadata, allowing you to access the content and metadata of each chunk.

Compared to other chunking libraries, Chunklet-py stands out for its lightweight design, multilingual support, and all-in-one approach. It is both versatile and easy to use, making it a valuable tool for improving RAG pipelines.