Chunking Strategies for RAG, Measured

Introduction

Chunking is how a retrieval-augmented generation (RAG) system cuts its documents into the pieces it searches. When a question comes in, the system embeds it (turns it into a vector, a list of numbers that places similar meanings close together), finds the nearest chunks, and hands those to the model as context. The model can only answer from what it is handed, so the cut decides the ceiling on every answer before any model runs.

This post opens a series that builds one retrieval system over this blog, one layer per post. The corpus is the 23 posts published before this one, about 31,600 words, and the test is 80 questions written in a reader's words, each with the exact passage that answers it. Six chunking strategies were run against the same questions with the same embedding model. Cutting every 200 words put the answering passage in the top five results for 73.8% of questions. Searching 50-word pieces and returning the 200-word section around them reached 95.0%, the same as sending whole posts, while sending a ninth as many words.

Answer foundAnswer cut in halfAll routes

The path every RAG answer takes when the cut went well: the question is matched against chunks, and one of the five chunks returned holds the whole answering passage. Parent-child chunking got here for 95.0% of the 80 questions.

The same question when the chunker cut through the answer. Fixed 200-word chunks split 10% of the answering passages this way, and no search can return a passage that no chunk holds whole.

Every route at once: one dot leaves the start and splits wherever the routes go different ways.

1

A reader asks a question in their own words, such as how many virtual nodes even out a hash ring.

The question is embedded with the same model as the chunks, nomic-embed-text, into 768 numbers.

The index ranks every chunk by cosine similarity to the question and returns the 5 nearest.

One of the five chunks contains the whole answering passage, because no chunk boundary fell inside the passage.

The model reads the five chunks and answers from the passage, with every fact in one place.

A reader asks the same question in their own words.

The question is embedded with nomic-embed-text, exactly as before; retrieval is not the problem here.

The index returns the 5 nearest chunks, ranked by cosine similarity.

The answering passage straddles a chunk boundary, so no single chunk holds the answer. Fixed 200-word chunks did this to 10% of passages.

The model gets half the facts at best, and only if the neighbouring chunk also made the top 5.

All 5 steps

1. 1.A reader asks a question in their own words, such as how many virtual nodes even out a hash ring.
2. 2.The question is embedded with the same model as the chunks, nomic-embed-text, into 768 numbers.
3. 3.The index ranks every chunk by cosine similarity to the question and returns the 5 nearest.
4. 4.One of the five chunks contains the whole answering passage, because no chunk boundary fell inside the passage.
5. 5.The model reads the five chunks and answers from the passage, with every fact in one place.

1. 1.A reader asks the same question in their own words.
2. 2.The question is embedded with nomic-embed-text, exactly as before; retrieval is not the problem here.
3. 3.The index returns the 5 nearest chunks, ranked by cosine similarity.
4. 4.The answering passage straddles a chunk boundary, so no single chunk holds the answer. Fixed 200-word chunks did this to 10% of passages.
5. 5.The model gets half the facts at best, and only if the neighbouring chunk also made the top 5.

• Answer found · Answer found: one chunk holds the passageStep through
• Answer cut in half · Answer cut in half: a boundary splits the passageStep through

Use the zoom buttons, or focus the diagram and press plus or minus to zoom, arrow keys to pan, and 0 to reset.

How the Measurement Works

Every strategy went through the same pipeline: cut the 23 posts, embed every chunk with nomic-embed-text (768 dimensions, run on a homelab server through Ollama), embed each question, and rank chunks by cosine similarity. For each question, the five best chunks are what a model would receive.

Five numbers come out of each run:

The first two are strict, so a passage split across two chunks counts as not found. The coverage and precision columns are the token-level measures from Chroma's chunking evaluation, which give partial credit. A model can often answer from half of a passage, so the truth for any given system sits between the two views.

Why Not Send Whole Posts?

The simplest system skips chunking: embed each post whole and send the best five posts.

On 23 posts with distinct topics, picking the right post is easy, so whole posts find the answer 95.0% of the time. The cost is 7,114 words per question, of which a third of one percent is the answer. Every question pays for reading five full articles, and a larger or more overlapping collection would stop the right post from standing out at all. Long posts don't fit the embedding model whole either. nomic-embed-text reads 2,048 tokens, so 9 of the 23 posts were embedded from their opening 1,500 words or so, and whole-post search only worked because a post's opening says what the post is about.

Six Ways to Cut a Document

Each strategy below is a function from text to chunks. Every chunk is a slice of the original text with its character offsets, so any chunk can be traced back to the place it came from.

Use the zoom buttons, or focus the diagram and press plus or minus to zoom, arrow keys to pan, and 0 to reset.

Fixed size cuts every size words. With overlap, each chunk repeats the last words of the one before, a sliding window that steps forward by size - overlap, the same window the two-pointer post walks across an array.

1/** Fixed size: every `size` words, optionally overlapping the previous2 *  chunk by `overlap` words (a sliding window that steps size - overlap). */3export function fixedSize(text: string, size: number, overlap = 0): Chunk[] {4  if (overlap >= size) throw new RangeError("overlap must be smaller than size");5  const w = words(text);6  const out: Chunk[] = [];7  for (let i = 0; i < w.length; i += size - overlap) {8    const last = Math.min(i + size, w.length) - 1;9    out.push(slice(text, w[i][0], w[last][1]));10    if (last === w.length - 1) break;11  }12  return out;13}

Sentence-aware chunking never cuts inside a sentence: it packs whole sentences until the next one would push the chunk past the limit.

Recursive chunking tries the largest boundary first. It splits on headings, and only a piece still over the limit gets split again on paragraphs, then sentences, then words. Neighbouring pieces are then packed back together up to the limit. This is the splitter LangChain recommends for generic text.

1/** Recursive: split on the largest boundary first (headings, then2 *  paragraphs, then sentences, then words) and only go finer for a piece3 *  that is still over `max` words; then pack neighbours back up to `max`. */4export function recursive(text: string, max: number): Chunk[] {5  const levels: ((s: number, e: number) => [number, number][])[] = [6    (s, e) => splitAt(text, s, e, /\n(?=#{1,6} )/g),7    (s, e) => splitAt(text, s, e, /\n\s*\n/g),8    (s, e) => sentenceSpans(text.slice(s, e)).map(([a, b]) => [a + s, b + s]),9    (s, e) => words(text.slice(s, e)).map(([a, b]) => [a + s, b + s]),10  ];11  const pieces: [number, number][] = [];12  const split = (span: [number, number], level: number) => {13    if (countWords(text, span) <= max || level === levels.length) return void pieces.push(span);14    for (const part of levels[level](span[0], span[1])) split(part, level + 1);15  };16  split([0, text.length], 0);17  return pack(text, pieces, max);18}

Semantic chunking embeds every sentence and starts a new chunk wherever two neighbouring sentences stop pointing the same way, measured by cosine similarity against a threshold.

Parent-child chunking indexes small chunks for precise matching, then hands the model the larger section each match came from. LangChain ships this as the ParentDocumentRetriever; LlamaIndex calls it small-to-big retrieval.

1/** Parent-child: index small chunks for precise matching, but hand the2 *  model the whole section (up to `parentMax` words) a match came from. */3export function parentChild(text: string, childMax: number, parentMax: number): { parents: Chunk[]; children: Chunk[] } {4  const parents = recursive(text, parentMax);5  const children: Chunk[] = [];6  parents.forEach((p, i) => {7    for (const c of recursive(p.text, childMax))8      children.push({ ...slice(text, p.start + c.start, p.start + c.end), parent: i });9  });10  return { parents, children };11}

The Results at 200 Words

Two strategies stand out for opposite reasons. Recursive chunking sends the fewest words of any strategy that never cuts an answer, 803 per question, because it ends chunks at headings and paragraphs instead of mid-thought. Parent-child chunking finds the most, 95.0%, by separating the two jobs a chunk does: the small child is what gets compared with the question, and the 400-word parent is what the model reads.

With parent lookupWithout parent lookupAll routes

Parent-child retrieval splits the two jobs a chunk does: a small child is compared with the question, and the larger section around the child is what the model reads. This found the answer for 95.0% of questions, against 85.0% for recursive 200-word chunks alone.

The same match without the lookup step. The small chunk matched precisely, but 100 words is often too little to hold the whole answer and the context that explains the answer.

Every route at once: one dot leaves the start and splits wherever the routes go different ways.

1

A question arrives and is embedded like any other query.

The question is compared with 100-word children. A small chunk is about one thing, so a good match is not diluted by the text around the answer.

One child wins: the closest small chunk by cosine similarity.

Every child stores a parent index, so the lookup returns the 400-word section the winning child was cut from.

The model reads the whole section, so the answer arrives with the sentences around the answer. Found: 95.0%, at 1,586 words sent per question.

A question arrives and is embedded like any other query.

The question is compared with the same 100-word children.

The same child wins the match.

The model reads only the 100-word child, so any part of the answer just outside the child never reaches the model.

All 5 steps

1. 1.A question arrives and is embedded like any other query.
2. 2.The question is compared with 100-word children. A small chunk is about one thing, so a good match is not diluted by the text around the answer.
3. 3.One child wins: the closest small chunk by cosine similarity.
4. 4.Every child stores a parent index, so the lookup returns the 400-word section the winning child was cut from.
5. 5.The model reads the whole section, so the answer arrives with the sentences around the answer. Found: 95.0%, at 1,586 words sent per question.

1. 1.A question arrives and is embedded like any other query.
2. 2.The question is compared with the same 100-word children.
3. 3.The same child wins the match.
4. 4.The model reads only the 100-word child, so any part of the answer just outside the child never reaches the model.

• With parent lookup · With parent lookup: match small, read the sectionStep through
• Without parent lookup · Without parent lookup: the model reads only the childStep through

Use the zoom buttons, or focus the diagram and press plus or minus to zoom, arrow keys to pan, and 0 to reset.

Semantic chunking is missing from that table because its result depends on one number, the similarity below which two neighbouring sentences count as a change of subject.

At 0.5 no neighbouring sentences were ever that far apart, so every cut came from the 200-word cap and the result is identical to sentence chunking, chunk for chunk. At 0.7 nearly every sentence boundary qualifies, and the posts break into 996 pieces of 32 words. Only 0.6 behaves like the idea on paper, and it lands exactly where sentence chunking already was. Qu, Bao and Tu found that semantic chunking's extra cost is not repaid by consistent gains over simpler splitters. The threshold result adds a practical cost on top, since the value that makes the method work has to be found again for each corpus and embedding model.

What Overlap Buys

Plain 200-word chunks left 10% of the answering passages split across a boundary. A 50-word overlap fixed all of them, and the strict measure rose by 7.5 points. The partial-credit measure did not move, because when an answer straddles a boundary, both halves sit in adjacent chunks that look alike to the embedding, so both tend to come back.

Overlap's benefit is coherence, since the model gets the answer as one passage instead of two halves to reassemble. How much overlap that takes is a question of cost.

Twenty words of overlap bought the whole strict gain of fifty for 10% more chunks instead of 26%. A 100-word overlap went further, but half of every chunk is then a repeat, and the index grew by 80%. Chroma reports the same trade: overlap helps recall for small chunks and hurts precision, because the model is sent the same sentences twice.

Bigger Chunks Are Not Safer

A larger chunk cuts fewer answers in two, so it seems like the safe choice. The sweep says otherwise.

The two lines part ways at 200 words. Past that point almost every answer sits whole inside some chunk, but search stops finding the chunk: at 400 words the answer reached the top five for 36.3% of questions, half the rate at 200. Recursive chunking showed the same cliff, 85.0% at 200 words and 62.5% at 400.

Two effects stack. The first is dilution. A 400-word chunk covers several ideas, so its single embedding sits between them and matches none of them closely, the effect Chroma describes as relevant information being diluted within a chunk. The second is specific to how these posts are written. Each one ends with a conclusion and key takeaways that restate its measured numbers, and once a large chunk swallows that ending, the chunk looks relevant to every question about the post. Removing those sections and re-running raised fixed 400 from 36.3% to 43.8% and fixed 800 from 43.8% to 55.0%. The summaries matter, but they explain less than half the drop, and dilution accounts for the rest. Any document that summarises itself, from a report's executive summary to a README's overview, sets the same trap.

The Parent-Child Dial

Parent-child retrieval has two sizes to choose. The child decides how precisely a question is matched, and the parent decides how much the model reads.

Small children with 200-word parents found as many answers as the 400-word version while sending about half the words, 823 per question, nearly the same as recursive chunking's 803. For the same reading budget as recursive chunking, the hit rate went from 85.0% to 95.0%. Larger parents buy a little more, 97.5% at 800 words, but the model reads four times as much for it.

When You Do Not Need It

• The whole collection fits in the prompt. A few short documents can be sent whole every time. Chunking pays off when the collection outgrows the context window, or when each question should not pay to read everything.
• Documents are already small and self-contained. FAQ entries, support tickets and product records are natural chunks. Cutting them further only separates questions from answers.
• Search is by exact identifiers. Error codes, part numbers and names match better with keyword search, a later post in this series.

Conclusion

Chunking sets the ceiling for a retrieval system before any model runs. On this blog's 23 posts, cutting every 200 words found the answering passage for 73.8% of 80 questions. Respecting the document's own structure raised that to 85.0%, and searching 50-word children while returning their 200-word section reached 95.0% for the same number of words sent. Bigger chunks kept more answers whole but hid them from search, overlap mostly reassembled answers that were already arriving in pieces, and semantic chunking depended on a threshold that either did nothing or shattered the text.

Key Takeaways

• Fixed 200-word chunks split 10% of the answering passages across a boundary, and found the full passage in the top five for 73.8% of questions.
• Recursive chunking (headings, then paragraphs, then sentences) found 85.0% while sending 803 words per question.
• Parent-child retrieval with 50-word children and 200-word sections found 95.0%, as many as whole posts, while sending 823 words instead of 7,114.
• Doubling fixed chunks from 200 to 400 words cut the hit rate from 73.8% to 36.3%, even though more answers stayed whole.
• A 20-word overlap gave the same strict gain as 50 words (73.8% to 81.3%) for 10% more chunks; answer coverage barely moved.
• Semantic chunking ranged from identical to sentence chunking (threshold 0.5) to 996 32-word fragments (threshold 0.7).

Sources

• Chroma, Evaluating Chunking Strategies for Retrieval
• Qu, Bao and Tu, Is Semantic Chunking Worth the Computational Cost? (Findings of NAACL 2025)
• LangChain, recursive text splitter and parent document retriever