Chunking text for LLMs without losing the plot

What it does

semchunk is a Python library for carving text into chunks that preserve local semantic context rather than slicing at hard token boundaries. It accepts any tokenizer or token counter — from Tiktoken and Hugging Face Transformers to a plain lambda — and returns chunks sized to your limit. It also supports overlapping chunks, exact character offsets, and optional AI-powered enrichment via the Isaacus API.

The interesting bit

The core is a novel hierarchical chunking algorithm that the authors claim delivers 15% better RAG performance than rival splitters; the library is already baked into Docling, the Microsoft Intelligence Toolkit, and the Isaacus API, which suggests the theory has survived contact with production.

Key highlights

• Plugs into any tokenizer or token counter you already use, with no forced dependencies on heavy ML frameworks.
• Optional AI-powered chunking via Isaacus enrichment models like kanon-2-enricher for documents that need more than syntactic boundaries.
• Returns exact character offsets and supports configurable overlap, which is useful when you want sliding windows rather than clean breaks.
• Includes multiprocessing and progress-bar helpers for batch chunking jobs.
• Boasts “millions of downloads per month” and a Rust port (semchunk-rs) for teams that need the same logic outside Python.

Caveats

• The library does not account for special tokens your tokenizer might add to every input, so you may need to manually shrink your chunk_size to compensate.
• AI-powered chunking requires installing the separate Isaacus SDK and setting an API key; it is not a built-in, offline feature.

Verdict

Worth a look if you are building RAG pipelines and tired of chunks that bisect sentences or paragraphs at random. Skip it if you need a fully offline, zero-API-key solution with all bells and whistles; the core chunker works locally, but the advanced AI enrichment is cloud-gated.