Chunking strategies for RAG, and when each one bites you
Fixed-size, semantic, and structure-aware chunking all look fine in a demo. Here's how each one actually fails once real documents hit your production pipeline.
Chunking is the decision everyone makes in the first ten minutes of building a RAG pipeline and revisits six weeks later when retrieval quality mysteriously craters. It looks like a solved problem — pick a size, pick an overlap, move on to the interesting parts. It isn't solved. It's just deferred.
I've shipped all three of the common strategies below in production. Each one worked great on the eval set I built it against, and each one had a failure mode that only showed up once real documents — the messy, inconsistently formatted, occasionally malformed ones — hit the pipeline.
Fixed-size chunking
Split every document into N tokens with some overlap. It's the default in every RAG tutorial because it's trivial to implement and it's a reasonable baseline.
Where it bites you: fixed-size chunks don't respect document structure, so they cut sentences and sections in half at whatever byte offset the window lands on. A chunk boundary in the middle of a table row, a numbered list, or a code block turns that chunk into noise — the embedding captures neither the question nor the answer cleanly, and the chunk that does have the answer is missing the context that would make it useful in isolation.
The overlap parameter is supposed to paper over this, and it does, partially — at the cost of storing and searching over redundant text. I've seen teams push overlap to 50% to compensate for bad boundaries, which roughly doubles index size and search cost to avoid actually solving the boundary problem.
Where it's still fine: homogeneous prose with no meaningful internal structure — long-form articles, transcripts, narrative documentation. If the document doesn't have sections, tables, or lists that matter, fixed-size is close to optimal and not worth replacing.
Semantic chunking
Split at points where consecutive sentences or paragraphs diverge in embedding similarity, instead of at a fixed token count. The idea is sound: chunk boundaries should follow topic boundaries, not an arbitrary counter.
Where it bites you: semantic chunking is expensive — you're running an embedding model over sliding windows just to decide where to cut, before you've even embedded the chunks you'll actually store. On a large corpus that's a meaningful chunk of your indexing cost and time, and it's non-deterministic in ways fixed-size isn't — rerun it on a lightly edited document and you can get materially different boundaries, which makes debugging retrieval regressions harder because you can't be sure the chunk boundaries are even the same ones you tested against.
It also has a failure mode on documents with lots of short, topically-similar sections — FAQs, glossaries, changelogs. The similarity signal is weak between adjacent short sections, so the boundary detection either merges things that should stay separate or produces chunks so small they lose context on their own.
Where it's still fine: long, topically diverse documents where getting the boundary right matters more than the cost of finding it — research papers, long-form reports, multi-topic wikis. The win is real; it's just not free, and it's not universal.
Structure-aware chunking
Split along the document's own structure — markdown headers, HTML tags, code block boundaries, table rows kept intact. This is what I default to now for anything that isn't plain prose, because it sidesteps the two failures above: no mid-table cuts, no expensive similarity computation.
Where it bites you: it only works as well as the structure is consistent. The moment your corpus mixes well-formatted markdown with copy-pasted Word exports, scanned-PDF-to-text garbage, or Confluence pages with three different heading conventions, structure-aware chunking degrades to fixed-size with extra steps — except now you've also got a parser that occasionally throws on malformed input and needs a fallback path. I underestimated how much of "real" document ingestion is document-format normalization before chunking even starts, not chunking logic itself.
It also doesn't have an opinion about section length. A single H2 section that's 4,000 tokens long stays one chunk unless you add a secondary fixed-size split within oversized sections — which most structure-aware implementations don't do out of the box, and you'll find out you need it when retrieval starts returning huge, unfocused chunks for narrow questions.
Where it's still fine: technical documentation, API references, anything authored consistently in markdown or a known template. This is the common case for internal knowledge bases, and it's where structure-aware chunking earns its complexity.
What I actually run now
A hybrid: structure-aware as the primary split, with a fixed-size fallback for any resulting section over roughly 500 tokens, and no semantic chunking at all — the cost didn't pay for itself against the corpora I've worked with once structure-aware handled the common case well.
The chunking strategy that matters least is the one you pick on day one. The one that matters is whichever one you're willing to re-evaluate against your actual eval set once you have one — chunking decisions made before you can measure recall@k are guesses, and the guess that felt right in the demo is rarely the one that survives contact with your real documents.