RAG Chunking Strategies: How to Compare Fixed, Structural, Semantic, and Parent-Child Retrieval

There is no universally optimal RAG chunk size. Chunking is an information-retrieval decision: a chunk must be narrow enough to rank for a specific need, broad enough to contain usable evidence, and traceable enough to cite. Choose it by evaluating complete retrieval-and-answer behavior on representative documents and questions.

This guide compares four strategy families with one controlled protocol. It does not claim a winner without data.

Define the unit the reader actually needs

The original RAG formulation combines a generator with retrieved non-parametric memory. Once a production system makes documents into passages, passage construction affects what the retriever can return and what provenance the generator receives. The motivations and architecture are described in Lewis et al..

A useful chunk has:

• semantic sufficiency: enough surrounding information to support an answer;
• retrieval specificity: little irrelevant text that dilutes matching;
• structural integrity: lists, tables, code, and headings remain intelligible;
• stable provenance: the chunk maps to an exact source version and region; and
• permission integrity: it never combines material with different access policies.

Token count alone cannot establish these properties.

Compare four strategy families

Fixed windows

Split after a fixed token or character count with optional overlap. This is deterministic and easy to reason about, but it may cut a sentence, table, or code block. Overlap can preserve boundary context while increasing index size and duplicate evidence.

Use fixed windows as a baseline. A more complex approach should beat it on a metric readers care about.

Structure-aware chunks

Parse headings, paragraphs, lists, tables, and code, then group compatible blocks up to a size budget. Store the heading path with each chunk. This often improves citations because the output mirrors author-defined boundaries, but quality depends on parsers and malformed source documents.

Never concatenate across a security boundary merely to reach a target size.

Semantic boundaries

Estimate topical change between adjacent units and split where it is large. This can keep discussions together when formatting is weak, but introduces model dependence, thresholds, cost, and less obvious reproducibility. Fingerprint both the boundary model and configuration.

Parent-child retrieval

Index small child units for precise matching, then return or expand into a larger parent section. This decouples retrieval granularity from context granularity. It also creates new decisions: how many parents to expand, how to deduplicate siblings, and how to keep the total prompt inside budget.

Use a factorial experiment, not four unrelated demos

Freeze everything except the chunking strategy:

• same source versions and parser output;
• same embedding model and distance;
• same index configuration;
• same retriever and candidate count;
• same reranker, if any;
• same prompt, model, and generation settings; and
• same held-out question set.

Record each configuration as data:

{
  "strategy": "structure-aware",
  "max_tokens": 480,
  "overlap_tokens": 0,
  "preserve_tables": true,
  "parser": "internal-html@3",
  "tokenizer": "model-family@revision"
}

Do not use an approximate word-to-token conversion for the configured limit. Use the tokenizer associated with the model or embedding contract and record its revision.

Build questions that expose boundary failures

Include:

1. facts contained in one sentence;
2. answers that require a heading and its first paragraph;
3. procedures spanning a numbered list;
4. table questions requiring headers and one or more rows;
5. code questions requiring a declaration and its explanation;
6. evidence on opposite sides of a plausible split;
7. multi-document and multi-hop questions;
8. exact identifiers and paraphrases; and
9. questions the corpus cannot answer.

The long-context study “Lost in the Middle” found that model performance can vary with the position of relevant information rather than remaining uniform across a long input. That result is a reason to test evidence order and context packing, not proof that one fixed chunk size solves the problem. See Liu et al., TACL 2024.

Measure retrieval and generation separately

Retrieval metrics diagnose whether evidence was found:

• recall at the number of chunks or tokens the application can afford;
• reciprocal rank of the first sufficient evidence;
• nDCG for graded relevance;
• percentage of retrieved tokens marked relevant; and
• duplicate or overlapping token rate.

Generation metrics diagnose whether the model used evidence:

• answer correctness against a rubric;
• claim-level support from retrieved spans;
• citation precision and completeness;
• abstention on unanswerable questions; and
• latency and cost for retrieval, reranking, and generation.

A strategy can improve chunk recall while producing worse prompts through duplication or fragmented evidence. Report both layers.

Treat overlap as a costed parameter

Overlap is not free insurance. It increases embedded text, index storage, ingestion work, and the chance that near-duplicate chunks crowd the top results. Sweep overlap alongside chunk size and measure duplicate evidence in the final context.

For parent-child retrieval, measure expansion amplification:

tokens returned to model / tokens in matched child chunks

High amplification may be justified when parents supply essential definitions. It should still be visible in the results.

Preserve tables and code deliberately

Flattening a table into disconnected rows can lose header meaning. Repeating all headers in every row can create substantial duplication. Test at least two representations and keep coordinates that resolve a citation to the original table.

For code, avoid splitting between a declaration and its body when the language parser can identify the construct. However, syntax-aware chunks can become very large; support children for individual methods plus a parent for the containing type or module.

Choose the simplest strategy that clears the bar

Select a strategy based on a declared priority order, for example:

1. citation support must exceed a fixed threshold;
2. retrieval recall must clear its threshold for every critical cohort;
3. p95 latency and prompt tokens must remain inside budgets; and
4. among passing configurations, choose the least operationally complex.

Do not select the configuration with the best aggregate score if it breaks a critical document type or tenant. Publish cohort results for HTML, PDFs, tables, code, short pages, long manuals, and permission classes.

Re-run the benchmark when inputs change

Chunking results are tied to the parser, corpus, embedding model, retriever, model context behavior, and evaluation set. Re-evaluate after meaningful changes to any of them. Store old chunk sets side by side until the new configuration passes and can be rolled back.